Are you all defragging regularly?

[jiclark]

I've noticed that my OS X 10.1.2 disks (I have two, my internal and a firewire external) get fragmented really quickly running OS X. I reboot with a Mac OS 9 CD, then run Norton SpeedDisk (from the v.6 package). I've been running it more than once a month for several months now, and it seems to ALWAYS say that I have "severe" fragmentation on my two OS X disks...

Ideas? Just keep dealing with it? Doesn't this somehow seem weird, or is this just the "brave new world" of OS X????

TIA,
John

[mervTormel]

probably a couple of factors:

there's probably a lot of writes and deletes on everybody's new OSX installation. gotta dl those apps, try them out, keep them or delete them. and fink is a monster on root.

as soon as you delete a file, you create a hole that has the potential to fragment a write of a file.

if you just wrote new files to an unfragmented disk, it would remain largely unfragmented. as files are changed (preferences, data), they might grow a little bit and can't fit back in their happy space, or shrink, so they fragment, or create a hole.

here's where partitions are very good.

i have an OSX install that i have kept mostly vanilla+DevTools+fink. most directories in my $HOME are populated with aliases/symlinks to off-world ( / ) partition's dirs. the only dir in my $HOME that gets written to is my ~/Library, and i don't want to mess with that. note: even fink's /sw is now on an off-world partition with a symlink to it at root ( / ).

the results have been pretty: when i run my weekly disk maintenance from OS9, i have minimal fragmentation on my OSX boot drive, and it's all largely in the ~/Library directory.

of course, my data and download partitions are completely polluted because i write/change a lot to data, and delete/write a lot from downloads.

tip:

minimize deletes until just before disk maintenance then do a big cleanup. you can do this by never emptying the trash until disk maint, or moving unwanted files to a pre-trash staging area if you're wiggly about things in the trash.

[jiclark]

Thanks merv,

Great info! But, not being a "major-leaguer", I'm a little lost on some of your details. And I'm actually planning on moving to a new 80GB drive this weekend. It'd be a good time to do a clean install, and I'd love more specifics on what you're talking about when you say "here's where partitions help". How do you recommend creating the ideal setup on the new drive? Could you give a little more info on how "aliases/symlinks" help? What's the difference, anyway? Any and all elaboration you can offer would be sooooo appreciated...

Thanks a lot for answering my questions!
John:)

[mervTormel]

john asks if i would share a little about partitions. i said i would write something up.

as for the aliases vs. symlinks, here goes:

finder aliases are pointers to other filesystem objects, with some intelligence. the finder is often forgiving when you move the target, and it can sometimes re-establish itself to it's intended target without a hitch. at least, this has been my perceived experience with finder aliases. and perhaps explains why their size is so variable. there must be some meta-data built into them to help the finder's 'alias resolution engine' follow/find targets when they have been moved or renamed. anyone know?

as for symlinks, they are a UNIX manifestation of pointers in the filesystem. but, briefly, a symlink merely points to another object named in the filesystem, with no intelligence. if anything in the path to the object changes, the symlink breaks.

From % man 7 symlink :

... A symbolic link contains the name of the file to which it is linked, i.e. it
is a pointer to another name, and not to an underlying object ...

Symlinks are good for static stuff that will probably never move.

A good candidate is the fink install in /sw

I have moved the entire /sw dir to another partition, and defined a symlink in root ( / ) to point to it.

% ll /sw
lrwxr-xr-t 1 root admin 31 Feb 6 12:00 /sw -> /Volumes/whiskey/sw//

# note the l (ell) in the file type position, indicating a link

% ll /sw/
total 0
drwxr-xr-x 17 root admin 534 Feb 3 13:09 LessTif/
drwxr-xr-x 795 root admin 26k Feb 3 13:09 bin/
drwxr-xr-x 38 root admin 1.2k Jan 21 03:41 etc/
drwxr-xr-x 13 merv wheel 398 Jan 20 20:09 fink/
drwxr-xr-x 7 root admin 264 Jan 16 15:24 fink.old/
drwxr-xr-x 12 root admin 364 Feb 3 13:09 grace/
drwxr-xr-x 146 root admin 4.8k Feb 3 13:09 include/
drwxr-xr-x 3 root admin 264 Jan 13 14:15 info/
drwxr-xr-x 688 root admin 23k Feb 3 13:09 lib/
drwxr-xr-x 4 root admin 264 Jan 19 16:43 libexec/
drwxr-xr-x 3 root admin 264 Jan 19 16:42 man/
drwxr-xr-x 17 root admin 534 Jan 19 16:42 sbin/
drwxr-xr-x 87 root admin 2.8k Jan 19 16:43 share/
drwxr-xr-x 5 root admin 264 Jan 16 15:28 src/
drwxr-xr-x 9 root admin 264 Feb 6 08:49 var/
drwxr-xr-x 3 root admin 264 Jan 19 16:43 vnc/

It works well and is completely transparent. References to /sw work for all intents and purposes.

Here's how I did it:

% sudo ditto -rsrc /sw /Volumes/whiskey/sw # make no assumptions about contents

% sudo rm -rf /sw # remove without impunity

% sudo ln -s /Volumes/whiskey/sw /sw # create symbolic link

[hembeck]

Hey all,

Does OS X come loaded with a defragging utility, maybe accessed via the Terminal, or do we have to fork over 100 bucks and get something like Nortorn Utilities?

Thanks for the feedback,
Fernando

[mervTormel]

nope. no native OSX disk defrag utility. but, you prolly shoudn't worry about it.

i've worked for years in big shops. some sysadmins are fastidious about fragmented files. they go out of their way to find a utility to do it, or schedule downtime for exclusive access so they can do a full backup and restore, which will linearize the data.

other sysadmins (me included) have never even considered it. and with the huge RAID arrays that are created, who'd want to?! it's an enormous task with a lot of risks and little benefit.

so, these fragmented disk systems are all up and running and shoving electrons around. and the sysadmins not fretting about fragmentation have one or more less issues to fret about.

let the bits fall where they may.

[jiclark]

Thanks merv,

But you're definitely speaking a language I don't understand! I'm not sure what you're telling us to do. I understand the difference between an alias and a symbolic link, but how does creating symlinks via the Terminal help keep fragmentation to a minimum? And, also, how does partitioning help? You spoke in your original thread about the directories in your Home directory being "populated by aliases/symlinks to off-world (/) partition's dirs..." [BTW, is every folder a "directory", or is there a difference?] Does that mean that your Documents, Movies, Music, Pictures, etc. folders are all actually just full of "pointers" to other partitions? So how does that work; you have a "data" partition with all your documents, as well as a "downloads" partition, and you can defrag these individually more quickly than the entire drive?? Actually, you posted above that you don't defrag at all, but you also talked originally about your "weekly disk maintenance from OS 9"... What exactly do you DO weekly from OS 9???

Forgive my ignorance, and my pestering questions; I just want to learn how best to manage and optimize a large hard drive in this new environment!

Thanks again for your patience with me...

John

[robh]

I don't defrag my OSX disk. I never defragged a unix filesystem before, never will.

I see lots of people reporting disk problems with OSX and more of often than not they also run third party disk repairers or optimizers. Coincidence ?, I doubt it.

Leave the dirty work of managing the filesystem to the unix layer of OSX.

[stetner]

I must admit, I have never defragged a unix file system (and never will damn it, they just don't need it! 8-), but, we must remember that most people are not using UFS, but are using HFS+.

So the question is really, does *HFS+* suffer from slowdown as it gets fragmented? That I don't know.

The other question is how effective is darwin's file buffer cache? If it is really good, then it would make a slow f/s less apparent.

Personally, I find the time spent in defragmenting more annoying than any slowdown I have ever noticed.

YMMV,

Doug

[jiclark]

Originally posted by stetner
...but, we must remember that most people are not using UFS, but are using HFS+.

So the question is really, does *HFS+* suffer from slowdown as it gets fragmented? That I don't know.

The other question is how effective is darwin's file buffer cache? If it is really good, then it would make a slow f/s less apparent.

I think stetner has brought up some great points! How do we find out the answers to his questions about the differences between UFS and HFS+? In response to robh; I know I've had disk related problems, but I've ONLY run disk utilities AFTER they crop up. After the amount of time I spent running utilities for OS 9, I was really looking forward to NOT having to do that for OS X! I don't think they're causing my problems, but they might not be fully dealing with them, especially since there really aren't any mature disk utilities for OS X yet. (Could there be some inherent problem with HFS+?) Do you UNIX geeks use fsck (I think that's what it's called)? What kinds of things prompt your use of that utility, and is there a good description of HOW to use it somewhere?

Thanks, all of you, for your responses; this is good stuff for all the newcomers to OS X to have as a resource. And hopefully we can get to the bottom of some of this stuff...

[robh]

I did a search a few weeks back for official info on defragging HFS+ under OSX. At the time I found an Apple document that said defragging wasn't important. Before you ask, no I didn't save the URL :(

For those who have disk problems with OSX, do you still boot into OS9 ?. I don't, not because I don't like OS9, but more importantly, I don't trust OS9 with my disks. From what I've read, it seems that OS9 offers no protection to the filesystem from rogue/broken apps... if something goes wrong, OS9 can crap all over files it should have no right to touch. OSX is far more protective of the filesystem.

[jiclark]

Originally posted by robh
For those who have disk problems with OSX, do you still boot into OS9 ?...

Myself, I don't boot into OS 9 hardly at all... So I don't think that's my problem. I'll do some searching over in the KB at Apple myself...

Thanks, John

[mervTormel]

Originally posted by jiclark

But you're definitely speaking a language I don't understand...

hmmm, time for schoolin' then, eh? a lot of this will assume a passing familiarity with disks and such.

> ... but how does creating symlinks via the Terminal help keep fragmentation to
> a minimum?

They don't, but you asked about them. Understanding them as a tool will give you a foundation for managing partitions and their contents.

> And, also, how does partitioning help?

By segmenting files by type/importance/activity, you can backup less and fret less about partitions that have less meaning (e.g., app downloads) if compromised, right? You don't backup downloaded apps, right?

And with a few different boot partitions you can avoid booting from CD, and quickly boot and repair when needed.

And partitions are protected from one another. If everything is sitting on an 80GB single-partition disk, and the catalog structures get frazzled, the whole drive could be compromised.

This is all part of pro-active disaster recovery planning.

> You spoke in your original thread about the directories in your Home directory
> being "populated by aliases/symlinks to off-world (/) partition's dirs..."
> [BTW, is every folder a "directory", or is there a difference?]

A UNIX filesystem directory is the equivalent of an OSX Finder folder

> Does that mean that your Documents, Movies, Music, Pictures, etc. folders are
> all actually just full of "pointers" to other partitions?

Exactly.

> So how does that work; you have a "data" partition with all your documents, as
> well as a "downloads" partition, and you can defrag these individually more
> quickly than the entire drive??

Exactly. Or diagnose structures and files only, forgetting about frags.

> Actually, you posted above that you don't defrag at all, but you also talked
> originally about your "weekly disk maintenance from OS 9"... What exactly do
> you DO weekly from OS 9???

I currently have a dependency on a grandfather-father-son-offsite backup set to DVD-RAM in OS9 retrospect.

-Mostly- weekly disk maintenance regimen:

- full # ditto backup of OSX root ( / ) to off-world partition
------ complemented by daily ditto bkup of /Users, /var, /etc
- OSX boot to single user mode
- # /sbin/fsck -n to see about it
- if issues that I trust can be repaired # /sbin/fsck until clean
- if filesystem was repaired, # reboot -n (don't flush buffers)
- boot to OS9
- full retrospect backup of 'data' partition to DVD-RAM

Then, if there's a problematic disk (rarely) you could repair thusly:

- disk first aid
- disk warrior
- norton disk doctor
- perhaps a casual look at fragmentation

- boot to OSX

If I know a partition has been beaten bloody, and will lose another round in the short term, I may chose to defrag the broken files just to give it a little break when the bell rings.

--
"A computer lets you make more mistakes faster than any invention in human history -- with the possible exceptions of handguns and tequila." -Mitch Ratliffe

[osxpounder]

I did a search a few weeks back for official info on defragging HFS+ under OSX. At the time I found an Apple document that said defragging wasn't important. Before you ask, no I didn't save the URL :(

Perhaps you refer to this topic at Apple Support; it concerns "disk optimization" as well as fragmenting, explaining why, in Apple's opinion, "there is little benefit to defragmenting."

Mac OS X: About Disk Optimization (http://docs.info.apple.com/article.html?artnum=25668&sessionID=anonymous|55986749&kbhost=kbase.info.apple.com%3a80%2f)

In short: "If you think you might need to defragment, try restarting first.

If your disks are almost full, and you often modify or create large files (such as editing video, ... there's a chance [your disks] could be fragmented. In this case, you might benefit from defragmentation, which might be performed with third-party disk utilities."

It eases my mind on the subject. I come from a Windows background, where my use of massive files caused me to pay attention to the fragmentation issues that could really slow down my Windows machines.

[edalzell]

....that in Panther, the OS automatically defrags files as it goes!

Check it (http://episteme.arstechnica.com/eve/ubb.x?a=tpc&f=8300945231&m=9900929295) out!

[jiclark]

Can someone explain just exactly how "...the OS automatically defrags files as it goes!"??

I mean, I know that it's somehow built into the journaling process or something, and I think it only happens on restart or cold boot, but I'd like to know more detail on just how this process is working in the background...

Thanks!

[ulrichm]

Can someone explain just exactly how "...the OS automatically defrags files as it goes!"?? As far as I know, when OS X reads a file which is fragmented, it writes the file way at a new location (if possible) unfragmented. That's how it goes.

[Old Toad]

FWIW, this was just posted by Apple: About Disk Optimization (http://docs.info.apple.com/article.html?artnum=25668)

[yellow]

That's the same kbase doc as the one osxpounder posted above..

[Old Toad]

Opps. I missed that one entirely. These eyes aren't what they used to be. Sorry about that.

[osxpounder]

Well, they do claim, on the newly linked page, that it was updated ... but I don't see what's different.

[vancenase]

i suppose it's good that apple doesn't have a defrag utility if most people do not need to use it (so they will not use it 'just because'). but, for those select few that do, we are left at wondering what software we should use ...


from apple (http://docs.info.apple.com/article.html?artnum=25668): If your disks are almost full, and you often modify or create large files (such as editing video, but see the Tip below if you use iMovie and Mac OS X Panther), there's a chance they could be fragmented. In this case, you might benefit from defragmentation, which might be performed with third-party disk utilities. Another option is to back up your important files, erase the hard disk, then reinstall Mac OS X and your backed up files.
but ... the only specific resolution they recommend is reformatting and reinstalling. what about a) offering defrag software or b) suggesting software with the apple stamp of approval?

sheesh.

[hayne]

Perhaps you missed the part about this only being a problem "if your disks are almost full". I think what Apple really recommends is not having your disk almost full.

Buy more disk. Or clean up your existing disk.
Both are better solutions than any disk utility. Especially since if you are working with large files, the speed of your computer will be largely determined by the speed of your disk. And all disks effectively slow down as they fill up - quite dramatically too: about one half the speed when 3/4 full if I recall correctly.

[vancenase]

i didn't miss that part -- my eyes work quite fine (most of the time). they can recommend that all they want, but i still don't have the money to invest into infinite hard drive space (or even an additional hard drive ... nor do they support me replacing the hard drive in my albook).

what i normally do is hover near full, and when it's time i'll write my critical data to DVD (that my boss will beat me alive if its lost).

so, yes, i hover around full capacity. perhaps i am one of 'those' they say should wipe my drive or defrag. do they suggest i wipe my drive every few months, or simply run a defrag utility? wiping every few months just because i keep my drive at full capacity seems a bit overboard.

while on the subject, is the easiest way of cloning/format/reinstall to use carbon copy cloner to make an image and write during reinstall or copy over afterwards (i've never done it)?

[hayne]

I recommend pretending that your disk drive is about 20% smaller than it is in reality. E.g. if your disk is a nominal 40 GB drive, pretend that it is a 32 GB drive. Then you will need to clean up when you get to the 30 GB level or so (leaving 2GB room for operations). And then you won't need to worry about fragmentation.

In other words, full capacity in a disk drive is like the redline RPM in a car. It's not a good idea to be anywhere near that for very long.

Or just run your drive as you do now, and don't worry about fragmentation. As I pointed out, if your drive is almost full, it is necessarily running much slower than it would be if there was more available space and if you don't mind that slowness, a bit more slowness from fragmentation won't be a problem either.

Re: Carbon Copy Cloner:
You wouldn't reinstall when using this. You would just make a copy of your disk on an external drive, boot off that external drive, erase your internal drive, make a copy of the external drive onto your internal drive, and you are done.

[vancenase]

I recommend pretending that your disk drive is about 20% smaller than it is in reality. E.g. if your disk is a nominal 40 GB drive, pretend that it is a 32 GB drive. Then you will need to clean up when you get to the 30 GB level or so (leaving 2GB room for operations). And then you won't need to worry about fragmentation.

that's good advice. maybe someone can come up with a script that will lie to me when it tells me how much disk space i have left (i'm one of 'those' who sets the clock backwards a bit, sets the alarm early, does the calculation, and hits the sleep button).


Re: Carbon Copy Cloner:
You wouldn't reinstall when using this. You would just make a copy of your disk on an external drive, boot off that external drive, erase your internal drive, make a copy of the external drive onto your internal drive, and you are done.
that is exactly what i meant by cloning/format/reinstall (as a general process; not specific actions within a program) ... i apologize for not being clear the first time. i just wondered which of these options were better (i think i've seen others talk about both):

1. mirror the data to an external drive, format/reinstall, then copy back over after booting into the fresh install

2. during the OS X install, i thought you can use a disk image (which one could create using CCC) to 'restore' your old drive ... i could be wrong on that though.

[hayne]

that's good advice. maybe someone can come up with a script that will lie to me when it tells me how much disk space i have left
There is a kernel parameter that tells it how much disk space to preserve. But as far as I recall, it's not changeable without recompiling your own kernel.

i just wondered which of these options were better (i think i've seen others talk about both):

1. mirror the data to an external drive, format/reinstall, then copy back over after booting into the fresh install

2. during the OS X install, i thought you can use a disk image (which one could create using CCC) to 'restore' your old drive ... i could be wrong on that though.

I don't recall anything like what you describe in option #2.
And in option #1, there is no point to doing a "reinstall" of the OS after the format if you are going to follow it by a carbon-copy-cloner operation that will completely replace the contents of the drive. You do it as I described in my previous post - no reinstall involved anywhere.

[vancenase]

I don't recall anything like what you describe in option #2.
And in option #1, there is no point to doing a "reinstall" of the OS after the format if you are going to follow it by a carbon-copy-cloner operation that will completely replace the contents of the drive. You do it as I described in my previous post - no reinstall involved anywhere.

gotcha. in reference to this and this below (which was from a few posts above)


You wouldn't reinstall when using this. You would just make a copy of your disk on an external drive, boot off that external drive, erase your internal drive, make a copy of the external drive onto your internal drive, and you are done.

so this involves reformatting the drive, but does not involve reinstallation using the OS X install cds? just copy everything (include system files) from drive A (main drive) to B (backup drive), reboot into B, wipe A, then copy all (including system files) back from B to A, and reboot?

sorry for asking so many questions about this. i'm used to doing this on windoze (well, reformatting/reinstalling every 6-12 months) --- just not much experience on the mac side of things. it seems if you tried to do something similar to the above in windoze, the system would not be happy on boot :)

[hayne]

so this involves reformatting the drive, but does not involve reinstallation using the OS X install cds? just copy everything (include system files) from drive A (main drive) to B (backup drive), reboot into B, wipe A, then copy all (including system files) back from B to A, and reboot?

Correct, but you can't just copy the files using Finder or the 'cp' command, there is more that you need to do - and that is why you need Carbon Copy Cloner to do the copying. Mike Bombich (author of CCC) has an article on his site explaining all the extra stuff you need to do to make the drive bootable.

And of course, I would check that the drive B copy was good (either by inspection of the folders or by using some utility like '/usr/bin/diff') before erasing my original drive.