The macosxhints Forums - Ditto Within a For Loop

The macosxhints Forums (http://hintsforums.macworld.com/index.php)

- UNIX - General (http://hintsforums.macworld.com/forumdisplay.php?f=16)

- - Ditto Within a For Loop (http://hintsforums.macworld.com/showthread.php?t=53579)

Ditto Within a For Loop

Hi all,

I'm not a Unix scripter and I am wondering why this part of my simple shell script does not work. My objective is to put the 2 most recently modified files of a directory into a disk image. I believe I first have to ditto them to a temp directory.

So, this doesn't work...

for i in "ls -1t | head -2"
do
ditto -rsrcFork "$i" /Temp/"$i"
done

The file names do have spaces in them. I tried putting the quotes in various places. The loop seems to pass ditto both file names at once.

Any help greatly appreciated.

Thanks.

for i in `ls -1t | head -2`;

?

Quote:

Originally Posted by DoubleEdd

for i in `ls -1t | head -2`;

?

Thanks, but I tried that and no go. Ditto thinks the file names with spaces are seperate files and returns "no such file or directory".

Tried escaping the quotes with \? edit - hmm, actually I don't think that'll work. You could use sed to mangle the variable contents though. Something along the lines of passing it through a
sed -e 's/ /\\ /g'

Quote:

Originally Posted by DoubleEdd

Thanks for your help. Yes, I've used sed to append quotes to the end of the filename. However, it looks like the problem is with ditto.

Say there are two files within "Folder One"; "File One" and "File Two"...

cd ~/Desktop/"Folder One"
x="$(ls -t | head -2)"
ditto "$x" ~/Desktop

Returns...

ditto: /Users/me/Desktop/Folder One/File One.rtf
File Two.rtf: No such file or directory

It loses the path to the second file.

?

This works...

cd ~/Desktop/"Folder One";
for i in "`pwd ls -t | head -2`";
do
ditto "$i" ~/Desktop;
done

Thanks.

Quote:

Originally Posted by MacGuyver

I think you are misinterpreting the error message from 'ditto'
The problem is that the variable $x contains a newline:

Code:

% echo $x

File One

File Two

Since you quoted the variable $x when invoking 'ditto', its contents are passed as is to 'ditto' (i.e. the shell does not break it up into separate arguments) and thus 'ditto' is receiving two arguments:
Argument #1:
File One
File Two
Argument #2:
/Users/me/Desktop
Thus 'ditto' will be looking for one file whose name is:
"File One
File Two"
(with an embedded newline)
and the fact that there is no file with that name is what the error message is saying.

The xargs command can be a useful addition to your Unix bag of tricks. This lets you do a one-liner:

Code:

ls -t "Folder 1" | head -2 | xargs -i ditto "Folder 1/{}" "Folder 2/"

I note that another way of doing this would sidestep the loop entirely and instead pass all of the files to be copied in one call to 'ditto' via the use the 'xargs' command.
E.g.:

ls -1t | head -2 | enquote | xargs -J % ditto % ~/Desktop

In the above I have used my Bash function 'enquote' (in my ~/.profile):

Code:

# enquote: surround lines with quotes (useful in pipes) - from mervTormel

enquote () { /usr/bin/sed 's/^/"/;s/$/"/' ; }

Quote:

Originally Posted by UncleJohn

Code:

ls -t "Folder 1" | head -2 | xargs -i ditto "Folder 1/{}" "Folder 2/"

Are you using an enhanced version of 'xargs'?
I don't see any "-i" option in the 'xargs' on my 10.4.5 system.

Quote:

Originally Posted by hayne

Are you using an enhanced version of 'xargs'?
I don't see any "-i" option in the 'xargs' on my 10.4.5 system.

Oops, you're right. :o

I'm used to the Solaris version of xargs, and sadly have no Macs at work. Your version of the command is the right one for OS X. Sorry for the misinformation.

Quote:

Originally Posted by hayne

I note that another way of doing this would sidestep the loop entirely and instead
pass all of the files to be copied in one call to 'ditto' via the use the 'xargs' command.
E.g.:

ls -1t | head -2 | enquote | xargs -J % ditto % ~/Desktop

In the above I have used my Bash function 'enquote' (in my ~/.profile):

Code:

# enquote: surround lines with quotes (useful in pipes) - from mervTormel

enquote () { /usr/bin/sed 's/^/"/;s/$/"/' ; }

This is cool stuff. I've been trying to grok xargs for a few months now.

I just did a few quick tests, and it seems the -0 (zero) option to xargs
will discern between spaces within filenames and spaces between files.
(I think the trailing newline after each filename is the key).

So, I believe this line will work the same:

ls -1t | head -2 | xargs -0 -J % ditto % ~/Desktop

But I still like that "enquote" function. ;)

-HI-

Quote:

Originally Posted by Hal Itosis

I just did a few quick tests, and it seems the -0 (zero) option to xargs
will discern between spaces within filenames and spaces between files.
(I think the trailing newline after each filename is the key).

So, I believe this line will work the same:

ls -1t | head -2 | xargs -0 -J % ditto % ~/Desktop

I don't think that will work. Look at this:

Code:

% mkdir Foo

% cd Foo

% touch "File One" "File Two"

% ls

File One        File Two



% ls -1t | head -2 | enquote | xargs ls

File One        File Two



% ls -1t | head -2 | xargs -0 ls

ls: File One

File Two

: No such file or directory

This works (yes, on my G5 this time) :)

Code:

G5:~$ ls -1t | head -2 | xargs -I % ls "%"

File One

File Two

The difference is that -I operates on an entire line of input while -J will split on white space. In my previous (incorrect for OS X) example, the -i option of Solaris (and Gnu) xargs is just shorthand for "-I {}".

The -0 option tells xargs to split on a "null" character, which you will have a hard time generating using ls. It's meant to be used in conjunction with the -print0 option of the find command.

Quote:

Originally Posted by UncleJohn

Code:

G5:~$ ls -1t | head -2 | xargs -I % ls "%"

File One

File Two

The difference is that -I operates on an entire line of input while -J will split on white space.

Thanks - that works well.
I was worried about the 255 character limit mentioned in the man page but it doesn't seem to affect things here. I guess I don't understand it - the 'xargs' man page isn't a model of clarity. :)

I.e. ls -1t | head -100 | xargs -I % ls %
works fine after doing: for ((i=0; i<100; i++)); do touch "file "$i; done

Thanks to all for your replies. This has been very interesting. I was wondering if someone could explain the meaning of the percent signs in the following code...

ls -1t | head -2 | xargs -I % ls "%"

And why is the last one quoted?

Thanks.

Quote:

Originally Posted by MacGuyver

I was wondering if someone could explain the meaning of the percent signs in the following code...

ls -1t | head -2 | xargs -I % ls "%"

And why is the last one quoted?

The xargs man page says this about the "-I" option:

Code:

     -I replstr

             Execute utility for each input line, replacing one or more

             occurences of replstr in up to replacements (or 5 if no -R flag

             is specified) arguments to utility with the entire line of input.

             The resulting arguments, after replacement is done, will not be

             allowed to grow beyond 255 bytes; this is implemented by concate-

             nating as much of the argument containing replstr as possible, to

             the constructed arguments to utility, up to 255 bytes.  The 255

             byte limit does not apply to arguments to utility which do not

             contain replstr, and furthermore, no replacement will be done on

             utility itself.  Implies -x.

The % is an arbitrary choice of placeholder - what is called "replstr" in the man page. You could use anything else as long as it doesn't occur in the command elsewhere.

It turns out that there is no need to quote the % in the 'ls' example.
But quoting it means that the replacement strings (from the results of 'head -2') will be quoted.

Quote:

It turns out that there is no need to quote the % in the 'ls' example.

Darn it, hayne, you got me again. :)

I just assumed the quotes were needed when you had file names containing spaces. But you're right, it works without them. This must mean that xargs tokenizes the command line before doing placeholder replacement, then calls some form of exec with an argument list, rather than passing the whole command line to a sub-shell. Which makes perfect sense if you think about it.

And you're right, that man page is pretty murky. This excerpt from the Cygwin man pages makes it a little clearer. I think it still applies to the OS X version.

Quote:

Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separator is the newline character.

Quote:

Originally Posted by UncleJohn

This must mean that xargs tokenizes the command line before doing placeholder replacement, then calls some form of exec with an argument list

Code:

% curl -s http://www.opensource.apple.com/darwinsource/10.4.5.ppc/shell_cmds-74.1/xargs/xargs.c | grep execv

                execvp(argv[0], argv);

Yep - hypothesis confirmed. :)

Quote:

Originally Posted by UncleJohn

Code:

G5:~$ ls -1t | head -2 | xargs -I % ls "%"

File One

File Two

The difference is that -I operates on an entire line of input
while -J will split on white space.

Brilliant. Thanks for that description.

Quote:

Originally Posted by UncleJohn

The -0 option tells xargs to split on a "null" character, which you will have a hard time generating using ls. It's meant to be used in conjunction with the -print0 option of the find command.

While that's true, it doesn't mean a null terminator is required for xargs to function
at all... plus, there is still a discernible difference when used in this example.
Back to the Foo directory:

Code:

$ ls -1t

File One

File Two

$ ls -1t | head -2 | xargs

File One File Two

$ ls -1t | head -2 | xargs -0

File One

File Two



$ ls -1t | head -2 | xargs | od -cb

0000000    F   i   l   e       O   n   e       F   i   l   e       T   w

          106 151 154 145 040 117 156 145 040 106 151 154 145 040 124 167

0000020    o  \n                                                        

          157 012                                                        

0000022

$ ls -1t | head -2 | xargs -0 | od -cb

0000000    F   i   l   e       O   n   e  \n   F   i   l   e       T   w

          106 151 154 145 040 117 156 145 012 106 151 154 145 040 124 167

0000020    o  \n  \n

          157 012 012

0000023

Anyway, I'm glad to have learned something new. Thanks all.

Sorry to beat this horse again... but when I actually type it out,
I sometimes start to understand it better.

Guess this won't be any big news, but -- while the magical xargs -I%
works wonders on paths with spaces -- it still stumbles on single quotes:

Code:

$ cd Foo; ls -1 

File One

File Two

$ touch "Pike's Peak"; ls -1 

File One

File Two

Pike's Peak

$ ls -1 | xargs -I% file % 

File One: empty

File Two: empty 

xargs: unterminated quote

Minor nuisance.
Employing hayne's handy enquoter makes all go well:

Code:

$ ls -1 | sed 's/^.*$/"&"/' | xargs -I% file % 

File One: empty

File Two: empty

Pike's Peak: empty

Okay... but what about double quotes then?

Code:

$ ls -1

File One

File Two

Pike's Peak

$ touch "\"Bartlett's Quotations\""; ls -1

"Bartlett's Quotations"

File One

File Two

Pike's Peak

$ ls -1 | xargs -I% file %

Bartlett's Quotations: cannot open (Bartlett's Quotations)

File One: empty

File Two: empty

xargs: unterminated quote

$ ls -1 | sed 's/^.*$/"&"/' | xargs -I% file %

xargs: unterminated quote

Oh oh, enquoter quoted a quote... so that turned
"Bartlett's Quotations" into ""Bartlett's Quotations""
which -in effect- becomes Bartlett's Quotations

Let's tweak enquoter so it won't quote any double quotes:

Code:

$ ls -1 | sed '/"/!s/^.*$/"&"/'

"Bartlett's Quotations"

"File One"

"File Two"

"Pike's Peak"

That much looks right. Now throw xargs back in:

Code:

$ ls -1 | sed '/"/!s/^.*$/"&"/' | xargs -I% file % 

Bartlett's Quotations: cannot open (Bartlett's Quotations)

File One: empty

File Two: empty

Pike's Peak: empty

Shoot. How about... "%"

Code:

$ ls -1 | sed '/"/!s/^.*$/"&"/' | xargs -I% file "%"

Bartlett's Quotations: cannot open (Bartlett's Quotations)

File One: empty

File Two: empty

Pike's Peak: empty

Hmmmm... make that \"%\"

Code:

$ ls -1 | sed '/"/!s/^.*$/"&"/' | xargs -I% file \"%\" 

"Bartlett's Quotations": empty

"File One": cannot open ("File One")

"File Two": cannot open ("File Two")

"Pike's Peak": cannot open ("Pike's Peak")

Hey... it worked?
(Umm, not really).
FORGET IT.

It's hopeless that way. Must resort to null chars
to manage *all* possible cominations of quotes...

Code:

$ find . -name "*i*" -print0 | xargs -0 -I% file %

./"Bartlett's Quotations" ./File One ./File Two ./Pike's Peak: cannot open (./"Bartlett's Quotations" ./File One ./File Two ./Pike's Peak)

Whoops... xargs is feeding file all 4 found items at once.
Add the -n1 option to pass them one at a time.

Code:

$ find . -name "*i*" -print0 | xargs -0 -n1 -I% file %

./"Bartlett's Quotations": empty

./File One: empty

./File Two: empty

./Pike's Peak: empty

Finally. (whew)

--

But: must we always use find's print0 to talk to xargs -0 ?
The original ls -1t | head -2 had a particular purpose.
Isn't there some way we can insert nulls after any command's list output?

Let's try tr

Code:

$ ls -1 | tr "\n" "\000\n" | xargs -0 -n1 file

"Bartlett's Quotations": empty

File One: empty

File Two: empty

Pike's Peak: empty

HOT DAMN!!!!!!!
Going for broke...

Code:

$ ls -1t | head -2 | tr "\n" "\000\n" | xargs -0 -n1 -I% cp -v % %.bak

"Bartlett's Quotations" -> "Bartlett's Quotations".bak

Pike's Peak -> Pike's Peak.bak

$ ls -1

"Bartlett's Quotations"

"Bartlett's Quotations".bak

File One

File Two

Pike's Peak

Pike's Peak.bak

Now I'm really happy! :D

-HI-

Using 'tr' to change newlines into null characters is a good idea.
But do you still need the newlines if you do that?
I.e. wouldn't
tr "\n" "\000"
work just as well?

Thanks hayne... guess I got so "caught-up-in-the-moment" that
I wasn't thinking about further code reduction. I wonder though
if that newline might be missed... further down the pipeline. (?)

[No, I guess you're right.]

Well, we're even then:
You trimmed nuller() { /usr/bin/tr "\n" "\000"; } ...and
I tweaked enquoter() { /usr/bin/sed '/"/!s/^.*$/"&"/'; }

;)

(Not of much use I guess, unless the entire "name" was quoted.)