Yikes--hard drive flooding with MBs!
 

I've got a 1999 Lombard Powerbook 333 Mhz with extra RAM and an Orinoco wireless card. Okay, I don't have much hard drive space to begin with--4GB!--so I usually keep 1 GB free. Now I've got 600 MB....but recently it keeps filling up to 500, then 300, then 150 and the system craaaawls and breaks. You can watch it filling; it's doing it right now. We're at 348 MB free and stuff is slow.

It happened after I decided to try a wake-from-sleep fix of using Open Firmware to reset the nvram. As expected, some of my preferences are gone--but preference like the Safari home page don't seem to stick anymore. Word won't save programs. Photoshop often. says the "scratch disks"are full. Felt like a virus, but ran Virex and nope. Reset permissions. fsck. Nothin'.

What did I do to my computer? Many thanks.
Andy

Re: Yikes--hard drive flooding with MBs!
 

I don't know what you've done to your machine but I do have a fair idea as to what might be happening. It sounds as though your system is paging out excessively; i.e. creating multiple swap files that are taking up your hard drive space. To confirm this open Terminal and type:

cd /private/var/vm <return>
ls -al <return>

You should see a file called swapfile0 and, if your system has paged out, swapfile1 and so on. The more swapfiles there are the more your system has paged out. Given that you don't have a lot of hard disk space whatever is causing this is causing you two problems. One, you're running out of hard drive space, and two, something is eating its way through your ram. If your system creates enough swapfiles to fill your hard drive this will, in all probability, crash your system.

It sounds as though you have a memory leak of some description. In terminal try typing:

top -w

the last two columns will show you the amount of memory that each app is currently using. If any of these figures increase significantly over time this would indicate that that app has a memory leak; i.e. it uses more and more memory over time.

What I'd suggest is that you reboot and follow the instructions above and report back what happens.

Thanks, that makes somes sense to me, but I'm not sure I came up with anything. After I read your message, there seemed to be a few swap files; I rebooted and there was just the one:

total 156256
drwxr-xr-x 4 root wheel 136 Mar 10 11:00 .
drwxr-xr-x 19 root wheel 646 Mar 10 11:00 ..
drwx--x--x 8 root wheel 272 Mar 6 16:45 app_profile
-rw------T 1 root wheel 80000000 Mar 10 11:00 swapfile0

running top gave me some files with curious hunger (Pith) and also Microsoft Word, which I had expected. Word has been acting wonky recently, but I'm not sure it's a culprit. Here's the result:

Processes: 41 total, 2 running, 39 sleeping... 102 threads
Load Avg: 1.23, 1.58, 1.52 CPU usage: 48.4% user, 20.6% sys, 31.0% idl
SharedLibs: num = 86, resident = 20.0M code, 2.32M data, 6.68M LinkEdit
MemRegions: num = 2683, resident = 46.8M + 6.89M private, 83.0M shared
PhysMem: 38.0M wired, 58.3M active, 174M inactive, 271M used, 113M free
VM: 1.18G + 48.8M 5899(0) pageins, 0(0) pageouts

PID COMMAND %CPU TIME #TH #PRTS(delta) #MREGS VPRVT RPRVT(delta) R
398 Pith 0.0% 0:02.77 1 58( 42) 118 3.39M 1.95M 5
397 top 11.2% 0:13.71 1 14 17 1.25M 192K
396 Microsoft 8.8% 0:31.26 4 125( 44) 204 15.0M 11.0M 3
394 tcsh 0.0% 0:00.12 1 10 15 1.12M 340K
393 login 0.0% 0:01.42 1 12 33 1.31M 244K
392 Terminal 1.6% 0:07.78 3 58 109 3.34M 1.31M 5
391 Microsoft 0.8% 0:09.52 10 137 198 16.0M 8.98M 2
390 Safari 36.0% 2:25.10 4 111( -1) 274 14.8M 9.69M 1
382 Microsoft 0.0% 0:02.67 2 69 101 4.45M 2.25M 7
380 Finder 0.0% 0:05.72 1 71 126 4.18M 2.54M 1
379 SystemUISe 0.0% 0:01.69 1 140 103 2.96M 1.20M 4
378 Dock 0.0% 0:03.99 2 89( 4) 112 2.56M 892K 5
372 pbs 0.0% 0:02.20 2 27 25 1.64M 484K
357 DirectoryS 0.0% 0:01.14 3 56 137 3.27M 752K 3
333 loginwindo 0.0% 0:02.25 4 149 107 3.68M 1.41M 6


And I ran it later, to see if anything had changed (of course *now* my computer won't reproduce it's problems; I've still got my hard drive space). here's what it looks like now:

Processes: 49 total, 4 running, 45 sleeping... 115 threads
Load Avg: 2.24, 2.13, 2.03 CPU usage: 71.2% user, 28.8% sys, 0.0% idle
SharedLibs: num = 92, resident = 22.4M code, 2.39M data, 7.63M LinkEdit
MemRegions: num = 3819, resident = 80.4M + 8.86M private, 97.1M shared
PhysMem: 43.4M wired, 68.9M active, 230M inactive, 342M used, 41.9M free
VM: 1.40G + 58.8M 7197(0) pageins, 1(0) pageouts

PID COMMAND %CPU TIME #TH #PRTS(delta) #MREGS VPRVT RPRVT(delta) R
476 top 8.2% 0:00.82 1 14 17 1.25M 200K
473 tcsh 0.0% 0:00.07 1 10 16 1.13M 344K
472 login 0.0% 0:01.65 1 12 33 1.32M 248K
468 lookupd 0.0% 0:01.14 2 33 46 2.25M 348K
421 Microsoft 0.0% 0:15.51 10 139 235 16.6M 9.72M 2
420 Safari 0.0% 1:50.43 5 170 431 35.9M 18.7M 1
419 usb 0.0% 0:01.54 1 11 15 1.18M 76K
418 rastertohp 0.0% 0:00.05 1 9 14 1.18M 80K
417 cgpdftoras 0.0% 0:00.27 1 19 47 1.75M 616K 1
405 Print Cent 27.3% 20:50.91 2 79 105 3.30M 1.96M 8
401 SpamSieve 0.0% 0:06.03 1 53 131 5.33M 4.11M 5
400 Microsoft 3.1% 18:56.85 4 132 317 22.1M 19.1M 3
398 Pith 0.0% 3:12.17 2 85 218 8.74M 6.73M 1
397 top 8.2% 26:50.91 1 14 18 1.29M 248K
394 tcsh 0.0% 0:00.12 1 10 15 1.12M 340K

Do you see anything there that's curious? Lots of pageins? Grateful--ASG

Pageins are fine and are nothing to worry about. But no, I don't see anything curious. Given that you're not experiencing any problems at the moment though I wouldn't necessarily expect to see anything obvious. Keep monitoring what's going on and if it does happen again check on the number of swapfiles prior to rebooting as a reboot clears all but one of your swapfiles.

One thing you might want to consider is getting some more RAM. While 384mb wont cause any major problems more memory would make things run a little more smoothly.

Just happened again and memory swooped down to 384. Here's the results:

total 468768
drwxr-xr-x 6 root wheel 204 Mar 11 12:01 .
drwxr-xr-x 19 root wheel 646 Mar 10 11:00 ..
drwx--x--x 8 root wheel 272 Mar 6 16:45 app_profile
-rw------T 1 root wheel 80000000 Mar 10 11:00 swapfile0
-rw------T 1 root wheel 80000000 Mar 11 10:25 swapfile1
-rw------T 1 root wheel 80000000 Mar 11 12:01 swapfile2
[Andrew-Greers-Computer:/private/var/vm] agreer%


Processes: 46 total, 3 running, 43 sleeping... 120 threads
Load Avg: 1.58, 1.32, 1.11 CPU usage: 79.7% user, 18.9% sys, 1.4% idle
SharedLibs: num = 100, resident = 19.9M code, 1.92M data, 6.47M LinkEdit
MemRegions: num = 4331, resident = 83.2M + 5.55M private, 76.5M shared
PhysMem: 62.3M wired, 158M active, 114M inactive, 335M used, 49.2M free
VM: 1.42G + 60.8M 14471(2) pageins, 18138(0) pageouts

PID COMMAND %CPU TIME #TH #PRTS(delta) #MREGS VPRVT RPRVT(delta) R
738 Preview 0.0% 0:31.67 2 85( 74) 173 4.93M 3.80M 7
723 tcsh 0.0% 0:00.08 1 10 15 1.12M 340K
722 login 0.0% 0:01.51 1 12( 3) 33 1.31M 232K
711 lookupd 0.0% 0:01.99 2 33 46 2.25M 364K
637 SecurityAg 0.0% 0:01.80 2 74( 61) 92 2.79M 1.29M 4
478 AppleSpell 0.0% 0:00.14 1 16 22 1.35M 444K
421 Microsoft 0.6% 8:54.33 12 151( 139) 345 23.6M 13.9M 2
401 SpamSieve 0.0% 0:21.85 1 60( 44) 153 6.61M 4.83M 6
400 Microsoft 9.7% 63:07.78 4 131( 114) 487 34.6M 24.6M 3
398 Pith 0.0% 12:34.91 2 86( 70) 398 15.0M 12.5M 1
397 top 11.1% 74:12.05 1 14 18 1.29M 260K
394 tcsh 0.0% 0:00.12 1 10 16 1.11M 0K
393 login 0.0% 0:01.42 1 12 33 1.28M 0K
392 Terminal 55.2% 8:30.89 5 83( 25) 204 4.32M 1.98M 8
382 Microsoft 0.0% 0:53.93 2 69 109 5.00M 1.69M 6

Have since restarted (another aspect is my computer is having even more wake-from-sleep problems than before). Any clues here?
Thanks-ASG

missing columns
 

One thing I forgot to mention is that you need to ensure that your terminal window is wide enough to display all the data - you're missing a couple of columns (RSIZE(delta) and VSIZE(delta)) that would be the most useful ones under these circumstances. You're also truncating the list somewhat; i.e. there are other processes running that aren't included.

What I would suggest is that after a reboot you open two large terminal windows. Start top -w in one of them then use ctrl-C to stop it. This will leave the display on the screen and will give you a good baseline to work from. When the problem reoccurs type top -w in the second window and look for any differences in the amount of ram used in the final two columns between the current state of affairs and your baseline:

Apologies for not being specific enough the first time round, I wasn't paying sufficient attention to your output.

Hopefully this will give you some idea as to where your hard drive space is going. Incidentally, swapfiles probably don't account for all the hard drive space you're losing as (unless you've changed anything) they're normally 76.2Mb each. So, two extra would only account for 152.4Mb extra and wouldn't take your hard drive space from 1Gb down to 384Mb. Do any of the programmes you use write relatively large caches to the hard drive?

It's still worth checking this out though as it might yet give us some clues.

3 further things
 

1) How soon after restarting did this happen?

2) I've read (here and elsewhere) that it's a good idea to have around 25% of your startup drive free. This means that as soon as you dip below 1Gb free you're potentially going to have problems.

3) Has this problem occured when you're not running PhotoShop? It doesn't show up in the data you've posted but, as I mentioned, you didn't include the entire list so if you run PhotoShop fairly soon after booting it might not appear in your top output unless you display more items in the list. PhotoShop's scratch files can be quite large and could account for your problems.

This is easy for me to say but you would be much better off with a bigger HD and some more ram.

top is not really a good tool to diagnose this issue in the forum.

when you see large amounts of disk space disappear, 1GB you mention, and swap accounts for some of it, then i suspect photoshop scratch space gets the rest.

either get omnidisksweeper and examine the drive, or...
that should find the large files that are modified and newer than the file we touched

No, you're probably right. Had the issue just been one of proliferating swapfiles it might have been ok but if other things are involved too I'd tend to agree.

ASG:

djn1 is correct about the free space on your system disk. 25% is arbitrary, since 25% of a 160GB drive is a hell of a lot of free space, but your 1GB free is hitting the wall, prolly with use of photoshop scratch space.

when the system disk gets full, unix folds faster than superman on laundry day.

Good point. My startup disk has around 50Mb free (of 60Mb) - what would be a reasonable limit for this size of drive? Clearly I'm not likely to run up against it any time soon but it would be useful to know.

And what to do about it short from restart ?
 

Hi Andy (original poster), I have exactly the same machine as you do.
However, I have crammed it with 512 MB of RAM and a 20 MB internal disk.

Upgrading from 384 to 512 MB has helped a lot.

However, I experience the very same problem: Obviously one or more apps I use have memory leaks:

VM: 3.02G + 65.9M 217480(0) pageins, 651231(0) pageouts

is the result after a couple of days.

While I could try to hunt down the stupid app that has eaten a gig or two of core for breakfast, I'd rather not, because one of the better things about unix as I understand it, is that you can let stupid apps do their thing, kill -9 em if they don't die horribly on their own and be a happy camper.

In my feeling, there is something terribly wrong in the VM-design of Mac OS X. While it is fine by me that mem-leaking apps will run slow as soon as they start to page, (and that their authors should be required by law to buy users more RAM until they wisen up :) ) the horror should instantly stop once the process has ceased to exist. This is sadly not true.

As performance degrades slowly, I actually started to notice this, when Toast burnt a coaster: it turned out the SCSI-performance, that is normally around 6,6 MB/s was down to around 2 MB/s and so a tad slow to burn at x12 speed which needs at least 2,3 MB/s. Nothing but a restart will get it back to the previous performance, not even quitting all apps, turning of journalling on all disks etc.

To be completely fair, I must admit that I have a couple of zombies, most of them Safaris, in my ps-listing. Do zombies still own their arena ?

What I am looking for are any good ideas how this can be fixed ?

Would turning off swap (and turning it on again) help ?
Is there anything else to be done to get the same benefit ?

While I have read about (and am convinced of the effectiveness) of different hacks to get swap on a partition with unfragmented swap-files, I believe that the +real+ fix would be to convince the OS that there is no longer a need to keep track of 3 gigs of absolutely useles memory.

My understanding of this is that while OS X's VM system will cope reasonably well under normal circumstances it will not manage "stupid apps" at all well. In other words, you can't, by definition, manage memory that is being eaten away by an errant application because the OS can't determine that the app in question has a memory leak - it will do its best to accomodate that application's unwarranted memory demands and other apps will suffer as a consequence.

i'm sure you mean 50GB / 60GB, no?

needed free space depends on RAM size and disk geometry and daily consumption.

if you have a large RAM footprint, and your apps don't leak, and you don't use all available RAM free pages, then, perhaps, there will never be anything but swapfile0. that is the case here, anyway, with 1.5GB of RAM.

there are rules for free disk space that take into account physical RAM, disk geometry and daily usage to calculate a free space threshold. i'll try and hunt them down.

i learned long ago that a bounty of RAM and disk space solved many administrative and technical hassles.

--

TT,

zombie processes do not take any appreciable resources.

here are some commands that examine real and virtual memory usage:

ztpm is aliased to `ps axm -o pid,stat,vsize,rss,time,command | head -31'
ztvm is aliased to `echo " PID STAT VSZ RSS TIME COMMAND" ; ps ax -o pid,stat,vsize,rss,time,command | sort -nr +2 | head -31'

This has all gone far over my head, but it's been mighty helpful. I think I misled you by stating I'd lost 1 GB of disk space--looking back, I'm afraid my disk *was* as overfull as you all suspected (about 550 MB free) and the two swapfiles that appeared would have brought it to its knees (which happened). Photoshop is not usually on--only Word and Safari are the biggies. But when PS is on, those scratch disks fill up fast.

More RAM and a bigger hard drive are the answer--but not for me at the moment. I'm going to free up another 500 MB of disk space and see if this disappears.

And yet--why would I have also experienced preferences not sticking and a computer that won't wake from sleep? Could these really be related? And my sense still is that all this happened after resetting nvram--yet how could that affect all this?

Thanks for all your time and thought
Andy

From what I've read, a lack of hard disk space can cause all sorts of problems which might include the ones you originally described. Free up some space and let us know how you get on.

Thanks Merv, I'd be interested in seeing these. And yes, I did mean 50GB/60GB :D

Lack of hard disk space can cause some really strange things to happen. I use Carbon Copy Cloner to do backups to an external HD. I set it up to do it automatically at night. My external HD got disconnected and CCC ran the script and started backing my hard drive, which was about 3/4 full, to itself. Needless to say, I get up the next morning and my computer is incredibly slow. All of my prefs are screwed up and couldn't be reset. Restarting had no effect. FSCKing had no effect. :confused: I finally realized that I only had about 10 meg left on the HD as I recall, maybe less. I eventually tracked it down to file that CCC had made in /Volumes. I had to use the CL to rm it. What a hassle. CCC no longer is automated.

Agreed, those are mighty commands.
Thank you about the statement concerning the living dead, nice to know they don't take memory out there in the twilight zone...

Agreed again, those who retrocompute or can't throw lots'a RAM at their problems should compensate by being clued about what they do, how to tune and what not to do or they'll be out in the cold...
Economically and in a corporate context, you're probably right from a pragmatic point of view.

But still, I'm not happy with the idea to consider the problem with ever-growing and never again shrinking memory-usage is just something we'll have to live with.

While the original poster is certainly well advised to clean up his disk, I still consider that my machine with several gigs free for swapfiles and maxed out with RAM should not have its I/O performance slashed by a factor of three just because the memory management is flawed.

Any ideas
(short from suggesting I should +fix+ it in Darwin and shut up (else I might consider to start bickering about how lousy both HFS and the entire concept of swap-files rather than swap partitions are anyway and nobody really wants to get that far, I hope))
are most welcome.

Hint to Andy: depending on your plans for the future buying a SCSI-IDE-adapter, a cheap IDE-disk and putting it in an old external SCSI-drive case and end up with lots of disk-space or do the same with a Firewire-PC-card and external case might be a Good Thing (tm). I picked opton 1 and it's way cool to have 120 GB extra space to play with...

Do these swap files get deleted when you restart? Cuz uh... I've got 3 of 'em.

17" iMac w/SuperDrive. 256 megs of RAM.