Any harm, any good?
 

I use an eMac in my office at work and it has 384 MB of RAM. This is sufficient most of the time, but occasionally I get a lot of pageouts. I will not be able to get more RAM for it, so I took to typing into the terminal the command
sudo du -sx / every now and then because it seems to free up some memory (and therefore, I hope, decrease the amount of pageouts). Eventually I scheduled cron to do this every hour, and it is completely in the background now so that I would not even know it is happening except that I have it send me an email via sendmail when it does this.

My questions are:
1) Does this cause any harm...might something else I am doing simultaneously conflict with this and cause problems (corrupt memory, or something).

2) Is it really doing any good or is it akin to wearing a magnetic bracelet? (Apologies to people who believe in this).

I know people have said 'Who cares about pageouts and swapfiles?', but once the second swapfile appears, the third and fourth never seem to be far behind (and so on, and so on).

I was under the impression that du didn't do anything except display the hard disk usage that each filesystem was using. It's just a statistical display and has no impact what-so-ever on your swapping.

You're right...at least that's what it says in the man pages. But run top, look at your free memory, and then run sudo du -sx / and look at your free memory again.

du -sx /
 

OK I did gain a little. Hmm.. well, I'm not sure what or what that happens there, but I only gained 4mb of free space back. And surely that is outweighed by the collosal amount of cpu cycles wasted when du is running. How much memory are you regaining on average?

I'll have to watch for awhile and do some arithmetic to be able to tell you that...but if my memory serves me correctly, sometimes there is a jump of over 50 mb. I imagine the gain might be more if you do a few intensive tasks with some (gui) applications, then close their windows, and then make the command.

I don't do anything *so* intensive that I can't spare the cpu cycles (if it really works).

You must be doing something pretty memory intensive if you're swapping out that much. What exactly are you running that sucking up so much of your memory? You might want to try and find of one of your apps have a memory leak and is sucking down unecessary amounts of memory.

After a big download (iMovie 3) I had just 11 MB of free memory left. After running
sudo du -sx /
I had 174 MB of free memory.

Apparently we both answered about the same time. To answer your question with respect to the data above, while this data was obtained I had
Mail, Safari, Calendar, AppleWorks, TextEdit, Terminal, X11, System Preferences, and PrintCenter open. Terminal was running top with the option that it updated only every 10 seconds. Safari had downloaded iPhoto and iMovie (responsible for a large loss in free memory), although du -sx / had run between the downloads. With X11, I had run nano, tex, xdvi, and xgalaga (only one game, so I wasn't goofing off too much). I also installed iMovie and iPhoto after their dmg files were mounted. I then stuffed their installation folders (because the dmg files were gone).

Now I'm stumped. Totally stumped. There's got to be something else going on here, like top only reports largest unused blocks in memory and du simply gathers all the blocks "together" and top then reports more "free" memory.
With top running, while I downloaded iPhoto3, and installed it, ran du, and watched my "free" memory shoot back up.
ACK! du doesn't do anything like this! *confused*
Still, I'm not convinced. Forgive me for being a skeptic. :)

Alternately, you don't run top all the time do you?

No, I don't have top on all the time, but being new at all this and curious, I check it from time to time.

For whatever it is worth, I'm not contending that it du *really* helps. I don't know enough to contend anything. I just noticed the change and thought it helped.

Did I read your post correctly...after downloading and installing iPhoto and running du -sx / did you too see a large jump in free space?

Anyway, perhaps top is not reporting that number accurately and du -sx changes its reporting of that number but not the actual number?

*boggle*
 

Yes, I did see a jump. I tried to see if it was cleaning up cached pages or what, but I have no idea. And I am only *guessing* that top is incorrectly reporting the "free" physical memory and somehow du is "fixing" that. *!boggle!* Thanks for giving me a brainbender to work on this weekend dude! This is a mystery I'd like to solve! :D

I did a search of these forums. Check out this
thread

I do it also
 

I have Memory Monitor in the Dock, and when the graph is full, I also run du -sx /. I don't even use sudo. Of course, I get a whole column of permission denied messages, but it still frees up the memory. Running the weekly maintenance script will give the same effect.

sustained disk i/o seems to release inactive pages to the free pool.

a find / will accomplish the same thing, which is akin to what the weekly maint task runs a la building the locate database.

Unless I've missed something, there's a big confusion about what the different functions do and how the system deals with virtual pages.

du, top, vm_stat display only statistics on the memory usage, each at its own way, but do not free anything.

For example:

du -sx display statistics without taking in account the subdirectories in filesystem mount points.

du -k reports the statistics in blocs of 1024 bytes, the default (with du -sx for example) is in blocs of 512 bytes.

vm_stat without interval display accumulative statistics, with an interval reports only the change between intervals.

top -a displays statistics relative to the last time the command was launched

top -d with a delta relative to the last time the command was launched

find -P reports the same as vm_stat but it returns the info for symbolics as those of the symbolic links (not the referenced files)

find -H or -L reports for symbolics the stat of the referenced files.

etc..., etc...

Briefly, those commands have nothing to do with the instant image of the memory usage and virtual pages. And never free anything.

As for virtual pages, (very, very) roughly speaking, there is a certain amount of memory uses to load process, resources, etc.... When a process wants a resource, the corresponding page is loaded in memory (that's pageins). There are algorithms which swap memory when a page is considered as no longer needed (that's pageouts) - the concept of no longer needed may be for example it has not been used since a certain delay or there are another processes with higher priority which request memory usage, or the process wait for an I/O device, or the process is blocked...

Then, the more you launch processes concurrently and the more thoses processes request memory (for resources for example: it's the case with graphics or video files), the highest is the need of swaping memory. Hence you can have a tremendous number of pageins and pageouts.

Miga, to reduce confusion...

du reports disk space usage, not memory

find -P has nothing to do with vm_stat


du a large disk space and watch some inactive pages release to the free pool. it does effect memory management. it is demonstrable.

As a matter of fact, to reduce disk access, blocs are loaded in memory, it's related to memory usage.
Not with vm_stat, but with statistics about file yes.
Quoted from find:
-P The -P option causes the file information and file type (see
stat(2)) returned for each symbolic link to be those of the link
itself. This is the default.

No that's not due that does it, that the scheduling policies, the swap algorithms and so on. Pages are swapped, then at a regular interval, there are algorithms which free them (eventually after storing modified pages on the disk). That's why you see pages released to the free pool.

A good place to learn about this is:

<http://www.williamstallings.com/OS4e.html>

I don't mean to be annoying with this question, just interested in the coincidence of this phenomenon...

Does du invoke these algorithms? Even if du does not directly result in the increase in what top reports as free memory, the increase in what top reports as free memory seems to always coincide with the running of du -sx /

Why is it that running du coincides with the increase in what top reports as free memory?

Again, no disrespect intended.

Sorry Miga, but I don't think your explanation holds water. I just tried this and my memory usage went from 40mb free to 256mb free (taken from inactive memory) over the space of 60 seconds and watched it happening with iPulse and AquaMon. It wasn't a scheduling policy that did this, it was running 'sudo du -sx /'.

After writing the above I did it again and freed up another 30mb.

I have no idea why this would be the case but I think you do have to accept the empirical evidence irrespective of whether it makes sense or not.

this:
is what i still don't understand. downloading files reduces your system memory to 11 MB? can that even be possible? just to keep myself in check, i'd like to ask if we're talking about FREE MEMORY or FREE DRIVE SPACE. clarity, please?

Free Memory (I had 25 GB of drive space). I think Safari holds the download in RAM, at least while it is mounting the dmg file.

No. Those algorithms are managed as the kernel level. You don't have access to them. In fact it's a mix of hardware mechanisms and very low level software.

I try to explain it a bit.

1 - The processor works roughly at ns speed, the disk at ms speed (taking into account positionning heads (very slow), seek time (very fast), track confirmation (intermediate)). You have more than one level of memory: the mass storage memory (your disk: the slowest), the main memory (intermediate speed), the cache memory (the fastest: maybe more than one level of cache depending of your computer). To speed up things, data loaded into memory are loaded into the cache, so that when requested the total seek time is dramatically reduced.

2 - There are a bunch of algorithms which are always running on your computer for priority access, refreshing display, scheduling, etc... There as also a bunch of tasks always running. Some of them you can see (all daemons - inetd, syslogd, configd, kextd, ...), another ones you cannot see (hardware mechanisms, ...).Generally speaking, there is a queue (in fact there are many queues of such tasks) of tasks with priority level. Each task is given a very short delay to run (2 ns or so) taking into account its priority level, then it is suspended and the next task in the queue is given the same delay, etc.. That's what is called round robin policy.

3 - One of this algorithm tries to guess the portion of data your computer will use in the near future (the concept of future for your processor is very far from your concept of future - nano-seconde vs seconde). It's based on the statistics you can see with top and dup or whatever primitive you call (note I write see, I don't write build). Note also that I write data, that means anything your computer can deal with: code, resources (files, IO devices,...). When a process is new (I mean really new, never run on the computer) to the computer, it is given a default behaviour regarding to near future, then as soon as it runs, the algorithms begin to store statistics: how long it runs, how much data access, from where, which etc., etc.. The algorithms adjust the statistics based on round robin policy (in fact, the policies used are improved round robin policy). That process is applied to ALL processes.

4 - The purpose of all above is to always keeping your processor running as much as possible, but not too much to impeed loading new processes when requested.

5 - Then there are tools which allow you to get a snapshot of the processes running on your computer at a given time. For example, dup, top and so on. Those processes refresh the display every second by default. Hardly can you imagine how much the processor has cycle during this time (remember nanoseconde - 10 ower - 9). Hence you see with those tools memory changes, but in no case that implies that those tools perform the changes.

Hope this help understanding the matter (which in fact is far much more complicated than what I explained here).

Here is what I think is going on ....

The OS has a system that handles allocation of memory. That memory is used for processes, but also for 'file-system buffer cache', and it is dynamic.The effect of running du or find is that there is a lot of filesystem activity. The OS (probably, I have not perused the code) starts putting a lot of 'new' file-system info into cache, gets rid of lots of 'old' stuff, and at the end of the day it has decided to keep a lot less file-system buffer cache than when you started.

A lot of speculation, but it is my gut feel about what is going on.....

Cheers,

[QUOTE]Originally posted by stetner

The effect of running du or find is that there is a lot of filesystem activity.
[QUOTE]

No, that's not the effect of du, this activity is always there, but du enables you to see it. That makes a big difference.

Not at the end of the day, circa every 2 nanosecondes. That makes also a big difference.

I've have some web pages which explain the process (very roughly). Some are already translated into English, others are on the way on being translated from French (sorry for the delay exams are in a few days).

Here they are in case you're interesting:
<http://micmacfr.homeunix.org/progsysdet/progsys01.shtml>
<http://micmacfr.homeunix.org/progsysdet/progsys07.shtml>
<http://micmacfr.homeunix.org/progsysdet/progsys08.shtml>
<http://micmacfr.homeunix.org/progsysdet/progsys11.shtml>

and a very old page describing the Power PC Architecture (you can have the same form Mac OS Inside pages in English). This one won't be translated into English (too old).
<http://micmacfr.homeunix.org/classid...1/CPPEdit.html>

But evidently the true reference is as already mentioned here:
<http://www.williamstallings.com/OS4e.html>
it has a bunch of links to lectures notes, graphics, courses, other web sites related to almost every computer operating sytem running on earth and maybe on Jupiter (who knows :-).

Cheers :-)

basic understanding
 

Here's my very basic understanding of what is happening.

Memory (RAM) is used for two different purposes:
1) temporary storage space for processes
2) disk cache (allows fast access to disk files that have recently been read from the hard disk)

The 'top' utility reports how much memory (RAM) is in use and how much is free.
It breaks the "in use" memory down into three categories:
wired - memory that is reserved (by privileged processes)
active - the remaining memory that is currently in use by processes & open files
inactive - memory previously used by processes & previously open files

A lot of the "inactive" memory is typically used for files that have been read or written to recently. The kernel prefers to keep the contents of files in memory (disk cache) so that they can be accessed faster. Even if you write to a file it doesn't actually write it to the physical disk until it is has to. Having a lot of memory in the "inactive" state is not a bad thing - it means that the kernel is making good use of the available memory to avoid having to use the disk (which is far slower).

The memory that is available for a new process is the sum of the free and inactive memory. But as explained above, some of the inactive memory may hold disk files which were modified but not yet written to the physical disk, so if a new process needs this memory, it has to wait for the files to be physically written to disk first. And the kernel always likes to keep a certain minimum amount of free memory (a few percent) for elbow room.

So, why does the use of 'du' result in memory being transferred from inactive to free? I'm not sure of the details but it would seem that as part of its job in figuring out how much disk space is used, 'du' somehow prompts the kernel to physically write out any pending file changes and this frees up some of the inactive memory.

Here's an interesting test that you can do to see some of this in action.
First run 'du' to get your free memory relatively high.
Now run 'top' in one Terminal window and run the following command in a second window:
cat /dev/random > foo
This command writes random bytes into a file "foo" and it keeps doing so until you stop it (with control-C). If you watch the inactive memory in 'top', you will see it going up and up as this file gets written. Do the control-C to stop the file being written and look at the size of the file with:
ls -l foo
You should find that your inactive memory has gone up by about the same amount as the size of the file. Now remove that file with:
rm foo
and look at the inactive memory in 'top' - you should see that it has gone back down to what it was before.

Bottom line:
An OS X system is designed to make maximum use of the RAM for disk caching so having a low "free" memory is not bad - it is a sign that the system is making the best use of the hardware resources.
Something that reduces your "inactive" memory and increases your "free" memory is not necessarily a good thing. It might have run the disk unnecessarily and thus slowed you down.
I would be very interested in any experiments that show that the system has better performance in some way (e.g. faster time to completion of some task) after the use of 'du' or other methods of forcing the system to move memory from "inactive" to "free".

Something must be getting lost in the translation here Miga, from the man page (although it should be obvious ...):Which means it traverses the given file system reading sizes and calculating how much disk space is used. Therefore lots of file-system activity. The same effect is seen when 'find' is used, lots of disk activity.

You seem to be talking about process scheduling, instruction caching etc. which I do not see as directly connecting with the topic at hand.

Cheers,

This is the same as disk caching is analog to memory caching (analog mechanisms, analog algorithms, basically a computer understands only changes in electronics, sorry I miss the proper terms in English) , so that means every time your request data: code, resources, ...
the requested data is loaded in memory via paging, segmentation, buffering, etc, etc... and a part of it is copied into cache.

On second request, either the data is in cache and is loaded from it, or is not in cache and is loaded from main memory into cache, or is not into main memory and is loaded from secondary storage.

Regularly the kernel realesed pages in the cache based on not recently used algorithm and not modified algorithm. Basically it has two lists, the free list and the modified list. When unloading page from memory, the page are not unloaded, simply the entries in list are updated.

Then on a regular basis, the kernel loads new pages in cache based on the statically next pages in the near future algorithm using clustering. And write out pages to secondary storage using the not recently modified page algorithm using clustering. The near future and nt recently used or modified concept are based on gathering statistics each time a page is loaded, unloaded, modified, etc.... This gathering is made by the different algorithms

So, "display" is not "build", top, dup and the like "display" statistical memory usage. That's the daemons which build statistics, and another ones which change the real memory usage.

If you say dup frees memory, it is as if you were saying, I'm in a train, another train is running side by side at the same speed in the same direction, we are not moving as I can see no move. Or another analogy, with kaleidoscope: you say the kaleidoscope builds images, no the kaleidoscope builds absolutely no image, there is a mechanism which make the film moving regularly (the builder) and continuously and then there is a window which allows you to see the moving film at regular time intervals, so it appears to you as distinct images, but there is no distinct images, just a film, just a film mover, just a window and just you.

Briefly you're not alone on the computer, even if you are a single user. They are a bunch of processes which are running continuously without you to have requested them and without you having the slightest right to stop them (and this even at boot time, and even at shut down time).

Miga, Stetner, Hayne, thank you all very much for the time, thought, and energy evident in your replies. I have a much better understanding of things now: I'm going to delete that line from my crontab and no longer assume that I know better than my computer.

I'm not saying that I have a much better truly technical understanding, but a much better basic view of things. I'm going to basically assume that because of all the activity it causes, du gives these statistical algorithms an unbalanced view of what is no longer needed and they release a lot of pages they originally had thought were going to be needed soon. These very same pages would have been released for something else later, if/when needed anyway. I watched du in operation in top just now and saw it consume up to 60 percent of the cpu for awhile.

I apologize, Miga, for taking you away from studying for your exams. Thank you for the links.

Ed

[Edited to add more clarity to my fuzzyness (although I probably increased the fuzzyness of my former clarity).

Yes, Miga, I understand about how a UNIX system works. The but the thread started with the question of why 'du -sx /' (which could have been any command that thrashes its way through the file-system, ie find / ....) seems to free memory that was on the inactive list. But I did not say that. I surmised that the underlying file-system buffer cache algorithm in the kernel was causing the freeing of memory.

If it is not tied into the file-system buffer cache, then running any command (ie top or vm_stat) should do the same thing. But from the comments in the thread, it is file-system intensive commands that free the memory.

I think we both know that it is *not* the command itself, 'du' in this case, in and of itself freeing memory, it is causing the kernel to free memory.

Anyway, my point is that I feel it has more to do with the file-system buffer cache algorithms than the process scheduling etc that you eluded to.

I think I will leave this one as it sits now.....

Top reports it as free physical memory. Who knows if this is accurate or not.

Please see my post (http://forums.macosxhints.com/showpo...65&postcount=9) in a different, related thread - it might be useful to anyone still curious about this topic.

-- Aiello