Finding and moving files to a common directory
 

I currently store all my image files (*.dmg) in the same directories as the associated applications. These are all contained in a Volume called "Main" in a directory called "AppsX".

I planned to move all these image files to a single directory within AppsX called "Installers". I based my code on a good article in O'Reilly's FreeBSD Basics column called "Finding Things in Unix". Based on the column writer's code I tried the following code structure:

[localhost:/Volumes/Main/AppsX] jfwoods% find . -name "*.dmg" -print | xargs -J X mv X ./Installers/

My system apparently does not accept the '-J' switch since it replied as follows:

xargs: illegal option -- J
usage: xargs [-0t] [-n number [-x]] [-s size] [utility [argument ...]]
[localhost:/Volumes/Main/AppsX] jfwoods%

The article states that, "The J switch tells xargs to take all of the files it receives and assume that the file you specify with the command is to be the destination....I also defined a character (X) and put that character on either side of the mv command."

Could someone please tell me what I did wrong here, and also what code I should be using to achieve the desired result.

Many thanks

Julian Woods

julian, let's build the commando you need to execute this manuveur...

first, see if this gives you expected results...
you'll need to test for switches (you don't have -J) and test to see that the pipe gets filenames to xargs as expected without errors (spaces in filenames will cause errors without switch -print0 for find and -0 for xargs; see the man pages for explanation)

Yikes! I'm a newbie so don't really understand your proposed code. Neither does my Terminal, which replied as follows:

[localhost:/Volumes/Main/AppsX] jfwoods% find /pathTo/imgBucket -type f -name "*.?mg" -print0 | xargs -0 ls -l
find: /pathTo/imgBucket: No such file or directory
[localhost:/Volumes/Main/AppsX] jfwoods%

Cannot understand what you mean by /pathTo/imgBucket. Could not find these terms either in man or in my huge "Inside UNIX" reference book. Wouldn't the find . -name "*.dmg" -print be sufficient to find all the .dmg files in AppX? I tried the following:

[localhost:/Volumes/Main/AppsX] jfwoods% find . -name "*.dmg" -print0 | xargs -0 ls -l

and this duly listed all the .dmg files in AppsX - a total of 62 files. But now I have to move them all to the directory called 'Installers' that also resides within AppsX.

Could you elaborate? This is a good learning experience for me.

Thanks

Julian Woods

PS. How come Mac OS X does not have the '-J' switch?

julian, regrets for the obfuscation.

/pathTo/imgBucket was meant to signify for you to provide the full pathname to the directory holding the files you want to find.

the command
searches the current directory and below, the dot ( . ) dictates this. if you hadn't cd'd to the dir of interest, then the first arg to find is the "where" arg; you specify the dir to start the finding. this could be a full pathname or a relative pathname. the point being that if you use dot as the where arg, it presumes you have cd'd to the place of interest first. i was trying to generalize your command and teach something.

now then, so, you got the command to list the files without error, good. we're part way there. here's the tricky part...

where are you moving them relative to where you are finding them?

show us the fulll pathname to your dot ( . ) [ cd there and type pwd ] and show us the full pathname to your target dir of the move ( mv )

something like this...
your results will be different.

what we want to do is find the target so we can be explicit in the commands to move x to y

i could find no references to command xargs' -J switch

Well Merv, if I follow your procedure I get this:

Welcome to Darwin!
[localhost:~] jfwoods% cd /Volumes/Main/AppsX
[localhost:/Volumes/Main/AppsX] jfwoods% pwd
/Volumes/Main/AppsX
[localhost:/Volumes/Main/AppsX] jfwoods% cd Installers
[localhost:Main/AppsX/Installers] jfwoods% pwd
/Volumes/Main/AppsX/Installers
[localhost:Main/AppsX/Installers] jfwoods%

In other words my Installers folder (where I want my files moved) is a sub-folder of AppsX (where I want to start my find).

So now we have to do a move — perhaps by piping to the mv command or using -exec or something. The thing is simplified somewhat by the fact that the found files are simply moved to a sub-folder within the same directory. I could for example try:

[localhost:/Volumes/Main/AppsX] jfwoods% find . -name "*.dmg" -print | mv "*.dmg" /Volumes/Main/AppsX/Installers

or perhaps:

[localhost:/Volumes/Main/AppsX] jfwoods% find . -name "*.dmg" -print -exec mv /Volumes/Main/AppsX/Installers {} \;

but I think I'd rather wait upon your superior wisdom before I screw things up for myself.

Thanks

Julian

Addendum

Thought I'd just quote from the text I mentioned above (written by Dru Lavigne at http://www.onlamp.com/pub/a/bsd/2002...SD_Basics.html):

"Brave enough to try something even more useful but complicated-looking? Let's get find to not only find this file, but move it to the correct directory using the following command:

find . -name "*.pdf" -print | grep -v "^\./pdfs/" | xargs -J X mv X ./pdfs/

To see if it worked, let's repeat the original find command:

find . -name "*.pdf" -print
./pdfs/50130201a.pdf
./pdfs/50130201b.pdf
./pdfs/50130201c.pdf
./pdfs/IWARLab.pdf
./pdfs/DoS_trends.pdf
./pdfs/Firewall-Guide.pdf
./pdfs/2000_ports.pdf

So it worked. Let's see why. Once grep finished filtering the find output, we piped that result to the xargs command to finish the job for us. The J switch tells xargs to take all of the files it receives and assume that the file you specify with the command is to be the destination. For example, before I ran the find command, I had no idea how many files needed to be moved. There may have been one, or there may have been several. I needed to let xargs know that regardless of how many files were found, I want them all moved and I want them all moved to the pdfs directory. That bit of magic is the job of the J switch. To get the J switch to work properly, I also defined a character (X) and put that character on either side of the mv command."

Hers is a bit more complicated in that she first looks for her stray .pdf files (i.e. those that are NOT in the proper directory) and then moves them to the correct directory. But, then, she apparently has that magic 'J' switch that I don't seem to have.

Oh well!!

you have the means of the -J switch, it's just not called the -J switch. it's called the -i switch and it works differently...

% blah | xargs -n1 -i mv {} ./installers

but, answer me what i asked for above because this has an element of danger and i want to craft a test for you first, then you can run it and investigate it's parts.

Oh! I thought I'd answered your question above when I put:

"Well Merv, if I follow your procedure I get this:

Welcome to Darwin!
[localhost:~] jfwoods% cd /Volumes/Main/AppsX
[localhost:/Volumes/Main/AppsX] jfwoods% pwd
/Volumes/Main/AppsX
[localhost:/Volumes/Main/AppsX] jfwoods% cd Installers
[localhost:Main/AppsX/Installers] jfwoods% pwd
/Volumes/Main/AppsX/Installers
[localhost:Main/AppsX/Installers] jfwoods%

In other words my Installers folder (where I want my files moved) is a sub-folder of AppsX (where I want to start my find). "

Isn't this what you wanted? Or are you after something else?

whoops! your back to back posts caused me to miss the one with your dir specs. regrets. okay then, we shall start with this caveat...

could it be that moving a file you found searching subdirs into a directory that is in the subdirs you are searching be problematic? could be. to be really safe, it's best to use logical and reasonable sources and targets and specify full paths.

let's see if we can get the expected command to run. try:
note the backslash at the end of command line #1 - it just tells the shell, more to come, keep prompting for more. copy/paste the blue to the shell and give 'er a go.

that looks like it would work, but there are some things to look out for, which we'll cover when you've exercised that command and pasted some brief results back here.

let's talk about the commandage so far. we've told find to only consider regular files (no dirs, links, pipes, fifos) and only go two dirs deep (at this point we don't want to go deeper because we risk junking packages) and then print0 the results (print0 stages us for better handling of spaces and other weird chars in the filenames)

next, we tell xargs -0 (expect -print0 args from a pipe) -n1 (only deal with one line [filename] at a time) -i (i'm going to place the piped input in the {} position (this is the -J of Dru) then we just echo a string of what command we want to perform.

let's see your run...

one thing you'll have to look out for is "collisions". you may not know what's down the tree of your find and there may be duplicate file names. and mv can be pretty harsh on the fact that a file exists at the target.
in my case i had dupes of the above images in my hierarchy, so investigate...
the second find/mv combo would have stomped on the first. in my case, the second find result is the same or newer file, date-wise, than the first, but that is a serendipitous result of my filing madness, and may not be your case.

the GNU version of mv allows you to specify a condition to make backup versions in case of collisions, but we'll proceed like you don't have that.

so, run the above find pipe to basename/sort/uniq and see if there are any "collisions"

another thing that will bite you here is that mv does not honor resource forks for mv's to other partitions. mv's on the same partition merely adjusts the directory entries (inode) and doesn't really touch the file. but a mv target of another partition will result in a copy (cp, not honoring rez fork) then a remove (rm, not honoring that you didn't mean it :( )

Thanks Merv, here is what happened when I first copied your code, and then re-typed it calling "installers" "Installers" (with a capital I as I have it).

First try

[localhost:Main/AppsX/Installers] jfwoods% find . -type f -name "*.?mg" -maxdepth 2 -print0 | xargs \
-0 -n1 -i echo "mv {} ./installers/" -> -0 -n1 -i echo "mv {} ./installers/"
xargs: illegal option -- i
usage: xargs [-0t] [-n number [-x]] [-s size] [utility [argument ...]]
find: -maxdepth: unknown option

Second try

[localhost:Main/AppsX/Installers] jfwoods% find . -type f -name "*.?mg" -maxdepth 2 -print0 | xargs \ -0 -ni -i echo "mv {} ./Installers/" -> -0 -ni -i echo "mv {} ./Installers/"
find: -maxdepth: unknown option
[localhost:Main/AppsX/Installers] jfwoods%

I think the problem with the first try was that my copy of your code left big space gaps (which you don't see too well from my paste above. My second try (which I typed manually) was more successful as you can see. Should I now leave out "-maxdepth"? and see what happens?

Eagerly looking forward to the next exciting episode.

Julian

lo, regrets. one of my problems is having many packages/flavors installed and i forget to vanilla-ize to stock OSX for crafting this kind of stuff. but, i get to play with a lot of new(er) features 8-]
"space gaps" are one of the shell's word delimiters (usually) and shouldn't matter much in this case. by that, i mean, any space in the above command line could be one or more of tab, space or \<return> . all unquoted space is flattened to one space by the shell interpreter.

let's simplify so we can make some progress. try:
that should just show us what xargs gets piped.

O.K. That code gives me a long list of .img and .dmg files by folder and sub-folder within the folder AppsX.

Looks fine to me. What next?

okay, good. how deep do the subdirectories run? we don't want to break packages, so you might want to hand move the entire app with packages out of the way. e.g.,
here, it is considered bad form to move the above images out of their packages (which is why i used the switch -maxdepth 2 of my version of the find command).

also, let's run the query for collisions:
the \<return> in the above blue command is the shell's "command continuation" designator. if you type the command all on one line, you'll want to remove the \

so, let's see your results.

also, just to be safe, i want you to consider moving the target dir of the mv out of the source dir of the find. right?

Thanks. The first code you now offer is the same as what I have already run. I did run it again though with the same result.

The query for collisions produced the following:

[localhost:/Volumes/Main/AppsX] jfwoods% find . -type f -name "*.?mg" -print0 | \ xargs -0 -n1 basename "$@" | sort | uniq -d
Illegal variable name.
[localhost:/Volumes/Main/AppsX] jfwoods%

Can't understand which variable this is. Can you?

i asked you how deep the subdirs run. are there any packaged image files that are going to be broken?

i instructed you to remove the backslash ( \ ) if you entered this as one line.

also, we're going to have to run the collision command in sh shell, so try this:

Sorry, didn't get it about how deep etc. I checked back and the depth goes up to five (5), not including the AppsX folder. I cannot find any package files though. How come they get broken?

Now running the collisions query resulted in the following:

[localhost:/Volumes/Main/AppsX] jfwoods% sh
localhost% find . -type f -name "*.?mg" -print0 | xargs -0 -n1 basename "$@" | sort | uniq -d
Xtools.dmg
localhost% exit
[localhost:/Volumes/Main/AppsX] jfwoods%

I checked and I do have two copies of Xtools.dmg in different sub-directories. Should I get rid of one of them?

On re-reading your posts I realize I didn't respond to your question as to whether "there are any packaged image files that are going to be broken." I did a find:

[localhost:/Volumes/Main/AppsX] jfwoods% find . -name "*.pkg" -print
./divosx119/qtffmpeg.pkg
./Fink 0.3.2a Installer/Fink 0.3.2a Installer.pkg
./Interarchy 5.0.1/Interarchy 5.0.1/Extras/tcpflow-0.20.pkg
./libhistory.pkg
./MacGimp Manuals etc./MacGIMP_Download.pkg
./QuickTime/QuickTime.pkg
./Streamripper/streamripper-1.0.5.pkg
./USB Overdrive X Beta/USB Overdrive X.app/Contents/Resources/USBOverdrive.pkg
./Xfree86Complete-4.2.0.pkg
[localhost:/Volumes/Main/AppsX] jfwoods%

so I do have a few package files in AppsX. Does this mean I should move them out. Don't quite understand that bit about being 'broken'. If we are only searching for "*.?mg" 's the .pkg's should be left alone, no?

more crossed wires
 

what i mean by package is that an app you've downloaded can have many things in it's "package". for instance, the BBEdit demo has

BBEdit 6.5 Demo/BBEdit Extras/BBEdit Unix Tools Installer.dmg illustrated here

if you break the "package", no harm may come, but it isn't logical. i think you're just wanting to organize standalone image files. we are not looking for .pkg files

as for your two Xtools.dmg's, the mv command is destructive. either you select which one to throw away, or the mv is going to stomp on the first one it moved with the second one find feeds to it.

now is the time to move the "installers" target directory out of the AppsX dir up to Main, so it sits along side AppsX right?

okay, so we're ready to unleash hell? got a backup?

we're going to have the use find's -exec option; i couldn't get mv to work with OSX's xargs.

we'll run two phases. 1) just list the commands that will run 2) actually execute the move...

here's the test:
eyeball that output real good. that is what is going to happen on execution below. the commands should look like:

mv ./pathto/file /fullpathto/dir/

copacetic?

then here's the execution:
be sure to validate your results

Oh! THAT kind of package! I may have a couple such packages but most of my .dmg etc. files are with installed applications so they are really only backups (which is why I want to isolate them). And if I do separate a package I can always reunite the files again if necessary, so I can safely ignore the possibility.

I have back-ups of most of these apps on CD's, though I may have left out a couple — difficult to say at this point. I guess I'll have to chance it. My AppsX file is 18.2 GB in total. But surely this code is not likely to actually delete, trash or damage the files themselves, is it????

I have duly placed my Installers folder in Main. However, since it is already 10:45 p.m. here in the East I think I'll wait and see whether you respond with a 'WAAAIIIIIIT!!!!' before I try my luck in the morning.

Be sure I'll let you know one way or another.

Julian