Terminal Memory Management?

[kerim]

I wonder if we aren't making too much of the memory gains. I tried running the vm_stat a few times in a row and found that the pages free can fluctuate quite a bit without doing anything (except running programs in the background)...

[kerim]

I think I found one place where memory is getting hogged and I have an idea about how to fix it, but I need UNIX help!

I tried to keep everything as simple as possible. First I changed my ImageMagick command to output mulitple PBM (portable bitmap) files instead of one single file.

Then, I restarted my machine and ran the command:

 nice +10 convert -compress zip *.pbm tiff:output%02d.tiff

(This converts them to individual Tiff files with numbers output01.tif, output02.tif, etc.)

These PBM files are about 500K each, and after it went through 29 of the 59 files, I got this error:

Quote: convert: Unable to open file (output29.tiff) [No space left on device].

For all the remaining files.

Now, what I realize in using +adjoin in ImageMagick is that it seems to first do all the conversions in memory (for all the files) and then to write them to disk afterwards!!! What I would like to know how to do is to tell ImageMagick to do this for one file at a time!

Something like:

For each file *.pbm
Convert to compressed Tiff
Renaming with the same name but changing the extension.

I imagine that there is a UNIX command for this but I don't know where to look - especially for the naming part!

TIA!

[pmccann]

What about a simple foreach loop: that might start/stop imagemagick once per file, thus leaving lighter footprints.

% foreach file (*.pbm)
foreach? nice +10 convert -compress zip $file (etc etc)
foreach? end

Worth a go!

Cheers,
Paul

(You can of course throw this into a file --say "scriptname" with

#!/bin/sh

on the first line (and no "foreach?" on lines 2 and 3; the shell adds these in when you do this interactively), then make it executable and just use

./scriptname

to run it)

[mervTormel]

better use #!/bin/csh with foreach

sh no got foreach

sh has:

for NAME [in WORDS ... ;] do COMMANDS; done

for file in /path/to/*.pbm;
do
nice...;
done

[kerim]

Amazing. This is definately the way to go! It was much faster, and it did all the files without error.

I want to sort out one more thing before I go trying to write a script for this (I'm still doing it manually), and that is get the process of file naming using the "foreach" command down. I tried reading the man (I really did), but what I came up with is this and it doesn't work:

% foreach file ( *.pbm )
foreach? nice +10 convert -compress zip $file $file:h.tiff
foreach? end

If the original file was file01.pbm, the output file should be file01.tif, but I get: "file01.pbm.tif" instead. I though the :h would strip the tail leaving only the header, but I guess I misunderstood the manual.

Thanks.

[oldfogey]

Quote: Originally posted by kerim % foreach file ( *.pbm ) foreach? nice +10 convert -compress zip $file $file:h.tiff foreach? end If the original file was file01.pbm, the output file should be file01.tif, but I get: "file01.pbm.tif" instead. I though the :h would strip the tail leaving only the header, but I guess I misunderstood the manual. Thanks.

Try replacing $file:h.tiff by "`basename $file .pbm`.tif"

And yes, that is a space after $file. (Is it '.tif' you want or '.tiff'? You say both :-) )

hth

[kerim]

Great. In another thread someone suggested "basename" but I didn't notice that it required a space. Now it works! Such small things. Thanks about the tif/tiff thing too. Doesn't really matter as far as I can tell, but it is good to be consistent!

[kerim]

Now the question is how to get this script to work not on the top level folder, but for all the sub-folders in it. That is, I have a folder called "scans" and it contains multiple folders "scanfolder 1" "scanfolder 2" etc. How do I get the script to operate on the TIF files inside each of these folders, rather than just the top level?

#! /bin/csh

if ( $#argv == 0 ) then
  echo "Usage: $0 <path1> [<path2>...]"
  exit 1
endif

foreach dir ($*)
cd $dir
echo "converting all files to Tiff! in" $PWD
foreach file (*.pbm)

nice +10 convert -compress zip $file tif:`basename $file .pbm`.tif
end
end

I'm also having a problem with this code not accepting directories with spaces in the name, even if they are input in quotes or with backslashes or even both ...

[kerim]

I haven't solved the issues from my last post, but here is the full script as it now stands:

#! /bin/csh

if ( $#argv == 0 ) then
  echo "Usage: $0 <path1> [<path2>...]"
  exit 1
endif

foreach dir ($*)

echo "Changing directory to: " $dir
cd $dir

echo "Cropping, Roating, and Filtering TIF to PBM!"
foreach file (*.tif)
nice +10 convert -gravity South -crop 1700x2200+0+0 -rotate "+90" \
  -level 10000,1,50000 -unsharp 6x1+100+0.05 \
  $file pbm:`basename $file .tif`.pbm
echo $file "done"
end

echo "Converting all files to compressed TIFF!"
foreach file (*.pbm)
nice +10 convert -compress zip $file tif:`basename $file .pbm`.tiff
echo $file "done"
end

echo "Converting all TIFF Files into a single landscape PDF!"
nice +10 convert -compress zip -page 792x612 -adjoin *.tiff pdf:document.pdf

echo "All Done!"

end

[edit: readability -mt]

[kerim]

I finally got the recursive directory thing! Now I just need to figure out why it isn't accepting spaces in directory paths...

Here is the final script. All I intend to do now is fix some of the feedback, and set it to work with fixed top-level directories. I want to have one directory called "portrait" and one called "landscape" - that way I can simply save scans to the appropriate directory and the script will treate them each correctly. When done I will post back here. But for now, this is what I have:

#! /bin/csh

if ( $#argv == 0 ) then
  echo "Usage: $0 <path1> [<path2>...]"
  exit 1
endif

foreach dir ($*)

echo "Changing directory to: " $dir
cd $dir

foreach dir ($dir/*)
cd $dir
echo "Cropping, Roating, and Filtering TIF to PBM!"
foreach file (*.tif)
nice +10 convert -gravity South -crop 1700x2200+0+0 -rotate "+90" \
  -level 10000,1,50000 -unsharp 6x1+100+0.05 \
  $file pbm:`basename $file .tif`.pbm
echo $file "done"
end

echo "Converting all files to compressed TIFF!"
foreach file (*.pbm)
nice +10 convert -compress zip $file tif:`basename $file .pbm`.tiff
echo $file "done"
end

echo "Converting all TIFF Files into a single landscape PDF!"
nice +10 convert -compress zip -page 792x612 -adjoin *.tiff pdf:document.pdf

echo "All Done with directory: " $dir
cd ../
end

end

[edit: readability -mt]

[Eckbert]

would this make sense if you run du on a system that seems to become sluggish to avoid a crash? I mean if you catch it early enough?

[tomholland]

although not a unix solution, I just deleted my loginwindow plist files and the system seems to be handling memory much better.