Free Memory (I had 25 GB of drive space). I think Safari holds the download in RAM, at least while it is mounting the dmg file.

No. Those algorithms are managed as the kernel level. You don't have access to them. In fact it's a mix of hardware mechanisms and very low level software.

I try to explain it a bit.

1 - The processor works roughly at ns speed, the disk at ms speed (taking into account positionning heads (very slow), seek time (very fast), track confirmation (intermediate)). You have more than one level of memory: the mass storage memory (your disk: the slowest), the main memory (intermediate speed), the cache memory (the fastest: maybe more than one level of cache depending of your computer). To speed up things, data loaded into memory are loaded into the cache, so that when requested the total seek time is dramatically reduced.

2 - There are a bunch of algorithms which are always running on your computer for priority access, refreshing display, scheduling, etc... There as also a bunch of tasks always running. Some of them you can see (all daemons - inetd, syslogd, configd, kextd, ...), another ones you cannot see (hardware mechanisms, ...).Generally speaking, there is a queue (in fact there are many queues of such tasks) of tasks with priority level. Each task is given a very short delay to run (2 ns or so) taking into account its priority level, then it is suspended and the next task in the queue is given the same delay, etc.. That's what is called round robin policy.

3 - One of this algorithm tries to guess the portion of data your computer will use in the near future (the concept of future for your processor is very far from your concept of future - nano-seconde vs seconde). It's based on the statistics you can see with top and dup or whatever primitive you call (note I write see, I don't write build). Note also that I write data, that means anything your computer can deal with: code, resources (files, IO devices,...). When a process is new (I mean really new, never run on the computer) to the computer, it is given a default behaviour regarding to near future, then as soon as it runs, the algorithms begin to store statistics: how long it runs, how much data access, from where, which etc., etc.. The algorithms adjust the statistics based on round robin policy (in fact, the policies used are improved round robin policy). That process is applied to ALL processes.

4 - The purpose of all above is to always keeping your processor running as much as possible, but not too much to impeed loading new processes when requested.

5 - Then there are tools which allow you to get a snapshot of the processes running on your computer at a given time. For example, dup, top and so on. Those processes refresh the display every second by default. Hardly can you imagine how much the processor has cycle during this time (remember nanoseconde - 10 ower - 9). Hence you see with those tools memory changes, but in no case that implies that those tools perform the changes.

Hope this help understanding the matter (which in fact is far much more complicated than what I explained here).

Here is what I think is going on ....

The OS has a system that handles allocation of memory. That memory is used for processes, but also for 'file-system buffer cache', and it is dynamic.The effect of running du or find is that there is a lot of filesystem activity. The OS (probably, I have not perused the code) starts putting a lot of 'new' file-system info into cache, gets rid of lots of 'old' stuff, and at the end of the day it has decided to keep a lot less file-system buffer cache than when you started.

A lot of speculation, but it is my gut feel about what is going on.....

Cheers,

[QUOTE]Originally posted by stetner

The effect of running du or find is that there is a lot of filesystem activity.
[QUOTE]

No, that's not the effect of du, this activity is always there, but du enables you to see it. That makes a big difference.

Not at the end of the day, circa every 2 nanosecondes. That makes also a big difference.

I've have some web pages which explain the process (very roughly). Some are already translated into English, others are on the way on being translated from French (sorry for the delay exams are in a few days).

Here they are in case you're interesting:
<http://micmacfr.homeunix.org/progsysdet/progsys01.shtml>
<http://micmacfr.homeunix.org/progsysdet/progsys07.shtml>
<http://micmacfr.homeunix.org/progsysdet/progsys08.shtml>
<http://micmacfr.homeunix.org/progsysdet/progsys11.shtml>

and a very old page describing the Power PC Architecture (you can have the same form Mac OS Inside pages in English). This one won't be translated into English (too old).
<http://micmacfr.homeunix.org/classid...1/CPPEdit.html>

But evidently the true reference is as already mentioned here:
<http://www.williamstallings.com/OS4e.html>
it has a bunch of links to lectures notes, graphics, courses, other web sites related to almost every computer operating sytem running on earth and maybe on Jupiter (who knows :-).

Cheers :-)

basic understanding
 

Here's my very basic understanding of what is happening.

Memory (RAM) is used for two different purposes:
1) temporary storage space for processes
2) disk cache (allows fast access to disk files that have recently been read from the hard disk)

The 'top' utility reports how much memory (RAM) is in use and how much is free.
It breaks the "in use" memory down into three categories:
wired - memory that is reserved (by privileged processes)
active - the remaining memory that is currently in use by processes & open files
inactive - memory previously used by processes & previously open files

A lot of the "inactive" memory is typically used for files that have been read or written to recently. The kernel prefers to keep the contents of files in memory (disk cache) so that they can be accessed faster. Even if you write to a file it doesn't actually write it to the physical disk until it is has to. Having a lot of memory in the "inactive" state is not a bad thing - it means that the kernel is making good use of the available memory to avoid having to use the disk (which is far slower).

The memory that is available for a new process is the sum of the free and inactive memory. But as explained above, some of the inactive memory may hold disk files which were modified but not yet written to the physical disk, so if a new process needs this memory, it has to wait for the files to be physically written to disk first. And the kernel always likes to keep a certain minimum amount of free memory (a few percent) for elbow room.

So, why does the use of 'du' result in memory being transferred from inactive to free? I'm not sure of the details but it would seem that as part of its job in figuring out how much disk space is used, 'du' somehow prompts the kernel to physically write out any pending file changes and this frees up some of the inactive memory.

Here's an interesting test that you can do to see some of this in action.
First run 'du' to get your free memory relatively high.
Now run 'top' in one Terminal window and run the following command in a second window:
cat /dev/random > foo
This command writes random bytes into a file "foo" and it keeps doing so until you stop it (with control-C). If you watch the inactive memory in 'top', you will see it going up and up as this file gets written. Do the control-C to stop the file being written and look at the size of the file with:
ls -l foo
You should find that your inactive memory has gone up by about the same amount as the size of the file. Now remove that file with:
rm foo
and look at the inactive memory in 'top' - you should see that it has gone back down to what it was before.

Bottom line:
An OS X system is designed to make maximum use of the RAM for disk caching so having a low "free" memory is not bad - it is a sign that the system is making the best use of the hardware resources.
Something that reduces your "inactive" memory and increases your "free" memory is not necessarily a good thing. It might have run the disk unnecessarily and thus slowed you down.
I would be very interested in any experiments that show that the system has better performance in some way (e.g. faster time to completion of some task) after the use of 'du' or other methods of forcing the system to move memory from "inactive" to "free".

Something must be getting lost in the translation here Miga, from the man page (although it should be obvious ...):Which means it traverses the given file system reading sizes and calculating how much disk space is used. Therefore lots of file-system activity. The same effect is seen when 'find' is used, lots of disk activity.

You seem to be talking about process scheduling, instruction caching etc. which I do not see as directly connecting with the topic at hand.

Cheers,

This is the same as disk caching is analog to memory caching (analog mechanisms, analog algorithms, basically a computer understands only changes in electronics, sorry I miss the proper terms in English) , so that means every time your request data: code, resources, ...
the requested data is loaded in memory via paging, segmentation, buffering, etc, etc... and a part of it is copied into cache.

On second request, either the data is in cache and is loaded from it, or is not in cache and is loaded from main memory into cache, or is not into main memory and is loaded from secondary storage.

Regularly the kernel realesed pages in the cache based on not recently used algorithm and not modified algorithm. Basically it has two lists, the free list and the modified list. When unloading page from memory, the page are not unloaded, simply the entries in list are updated.

Then on a regular basis, the kernel loads new pages in cache based on the statically next pages in the near future algorithm using clustering. And write out pages to secondary storage using the not recently modified page algorithm using clustering. The near future and nt recently used or modified concept are based on gathering statistics each time a page is loaded, unloaded, modified, etc.... This gathering is made by the different algorithms

So, "display" is not "build", top, dup and the like "display" statistical memory usage. That's the daemons which build statistics, and another ones which change the real memory usage.

If you say dup frees memory, it is as if you were saying, I'm in a train, another train is running side by side at the same speed in the same direction, we are not moving as I can see no move. Or another analogy, with kaleidoscope: you say the kaleidoscope builds images, no the kaleidoscope builds absolutely no image, there is a mechanism which make the film moving regularly (the builder) and continuously and then there is a window which allows you to see the moving film at regular time intervals, so it appears to you as distinct images, but there is no distinct images, just a film, just a film mover, just a window and just you.

Briefly you're not alone on the computer, even if you are a single user. They are a bunch of processes which are running continuously without you to have requested them and without you having the slightest right to stop them (and this even at boot time, and even at shut down time).

Miga, Stetner, Hayne, thank you all very much for the time, thought, and energy evident in your replies. I have a much better understanding of things now: I'm going to delete that line from my crontab and no longer assume that I know better than my computer.

I'm not saying that I have a much better truly technical understanding, but a much better basic view of things. I'm going to basically assume that because of all the activity it causes, du gives these statistical algorithms an unbalanced view of what is no longer needed and they release a lot of pages they originally had thought were going to be needed soon. These very same pages would have been released for something else later, if/when needed anyway. I watched du in operation in top just now and saw it consume up to 60 percent of the cpu for awhile.

I apologize, Miga, for taking you away from studying for your exams. Thank you for the links.

Ed

[Edited to add more clarity to my fuzzyness (although I probably increased the fuzzyness of my former clarity).

Yes, Miga, I understand about how a UNIX system works. The but the thread started with the question of why 'du -sx /' (which could have been any command that thrashes its way through the file-system, ie find / ....) seems to free memory that was on the inactive list. But I did not say that. I surmised that the underlying file-system buffer cache algorithm in the kernel was causing the freeing of memory.

If it is not tied into the file-system buffer cache, then running any command (ie top or vm_stat) should do the same thing. But from the comments in the thread, it is file-system intensive commands that free the memory.

I think we both know that it is *not* the command itself, 'du' in this case, in and of itself freeing memory, it is causing the kernel to free memory.

Anyway, my point is that I feel it has more to do with the file-system buffer cache algorithms than the process scheduling etc that you eluded to.

I think I will leave this one as it sits now.....

Top reports it as free physical memory. Who knows if this is accurate or not.

Please see my post (http://forums.macosxhints.com/showpo...65&postcount=9) in a different, related thread - it might be useful to anyone still curious about this topic.

-- Aiello