anomalous double-dot (..*) files produced en masse during rsync to AFP shares

[Hal Itosis]

Most often reported in context with ..DS_Store.random filenames...

• Samba.org: "osx 10.6 strange rsync errors" (03 Apr 2010)
  
  
• MacRumors Forums: "rsync issues" (12 Apr 2010)
  
  
• Apple Discussions:
  1. "rsync issues with 10.6.3" (13 Apr 2010)
  2. "rsync broken with 10.6.3?" (14 Apr 2010)
  3. "rsync broken in Snow Leopard 10.6.4 when copying to network volumes" (04 Jul 2010)

...this phenomenon is not limited to .DS_Store files alone (those are simply the most ubiquitous). The same strange results will occur with any "dot" file in the source (e.g., .bashrc, .localized or .whatever).

The aberrant files do indeed have names that start with two leading dots (e.g., ..DS_Store) and —contrary to what one might expect —such "double-dot" filenames are visible in Finder (not hidden):

$ touch ~/Desktop/'..i see you'

--

They seem to be the byproduct of rsync when it copies to an AFP share, while running Snow/Leopard... but i'm not certain if a precise culprit (or specific minimal combination of culprits) has been pinpointed. Anyone know???

STEPS TO REPRODUCE (i'm running 10.6.6 btw)

1. on our HD, create a small "test-source" folder which includes 2 or 3 token files, plus a dot file:
   

   $ cd ~/Desktop && mkdir test-source
   $ touch test-source/{.whatever,file{1,2}}
   $ ls -logA test-source
   total 0
   -rw-r--r--  1   0 Jan 22 00:46 .whatever
   -rw-r--r--  1   0 Jan 22 00:46 file1
   -rw-r--r--  1   0 Jan 22 00:46 file2

2. connect to another machine via AFP, mount a volume, and create an empty "test-target" folder:
   

   $ mkdir /Volumes/your/path/to/test-target
   $ ls -la /Volumes/your/path/to/test-target
   total 0
   drwxrwxrwx  2 halito  halito   264 Jan 21 21:28 ./
   drwxrwxrwx@ 9 halito  halito   264 Jan 21 21:28 ../

3. rsync the source to the target:
   First, using the all-important -n option, preview a dry run...
   

   $ rsync -axhuvin --no-g --stats ~/Desktop/test-source/ \
   /Volumes/your/path/to/test-target/
   building file list ... done
   .d..tp... ./
   >f+++++++ .whatever
   >f+++++++ file1
   >f+++++++ file2
   
   Number of files: 4
   Number of files transferred: 3
   Total file size: 0 bytes
   Total transferred file size: 0 bytes
   Literal data: 0 bytes
   Matched data: 0 bytes
   File list size: 77
   File list generation time: 0.001 seconds
   File list transfer time: 0.000 seconds
   Total bytes sent: 117
   Total bytes received: 44
   
   sent 117 bytes  received 44 bytes  322.00 bytes/sec
   total size is 0  speedup is 0.00

   That all looks fine thus far.
   
   
4. so, finally take the plunge and remove the -n option:
   With Leopard (10.5.8), the run looks like this...
   

   $ rsync -axhuvi --no-g --stats ~/Desktop/test-source/ \
   /Volumes/your/path/to/test-target/
   building file list ... done
   .d..tp... ./
   >f+++++++ .whatever
   >f+++++++ file1
   >f+++++++ file2
   rsync: mkstemp "/Volumes/your/path/to/test-target/..whatever.P5geMP" failed: 
     No such file or directory (2)
   
   Number of files: 4
   Number of files transferred: 3
   Total file size: 0 bytes
   Total transferred file size: 0 bytes
   Literal data: 0 bytes
   Matched data: 0 bytes
   File list size: 77
   File list generation time: 0.001 seconds
   File list transfer time: 0.000 seconds
   Total bytes sent: 225
   Total bytes received: 92
   
   sent 225 bytes  received 92 bytes  634.00 bytes/sec
   total size is 0  speedup is 0.00
   rsync error: some files could not be transferred (code 23) 
     at /SourceCache/rsync/rsync-35.2/rsync/main.c(992) [sender=2.6.9]

   But at least Leo caught itself there, and only excreted one oddity...
   

   $ ls -logA /Volumes/your/path/to/test-target
   ls: ..whatever.P5geMP: No such file or directory
   total 32
   -rwxrwxrwx  1   0 Jan 22 00:46 file1
   -rwxrwxrwx  1   0 Jan 22 00:46 file2

   Snow Leopard however takes that singular weirdness just shown in Leopard and "divides it by zero". I.e., in Snow Leopard the situation doesn't politely error out automatically after producing one oddball file, but rather it will continue to crank them out by the thousands until something else (like ^C, or too many files in a directory?) stops the process externally.
   Snow Leopard's run goes like this...
   

   building file list ... done
   .d..tp... ./
   >f+++++++ .whatever
   >f+++++++ file1
   >f+++++++ file2

   ^and it just sits there, seemingly still copying some file. Actually however, something more bizarre is taking place. NOTE: After 2 or 3 seconds, go ahead and control-C out of that task...
   

   ^C
   rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at 
   /SourceCache/rsync/rsync-40/rsync/rsync.c(244) [sender=2.6.9]

5. and now, survey the damage:
   

   $ ls -la /Volumes/your/path/to/test-target
   ls: ..whatever.0F92Hm: No such file or directory
   ls: ..whatever.0G92Hm: No such file or directory
   ls: ..whatever.0H92Hm: No such file or directory
   ls: ..whatever.1F92Hm: No such file or directory
   ls: ..whatever.1G92Hm: No such file or directory
   ls: ..whatever.1H92Hm: No such file or directory
   :
   : # hundreds (if not thousands) of nearly identical error messages
   :
   ls: ..whatever.aG92Hm: No such file or directory
   ls: ..whatever.aH92Hm: No such file or directory
   ls: ..whatever.bF92Hm: No such file or directory
   ls: ..whatever.bG92Hm: No such file or directory
   ls: ..whatever.bH92Hm: No such file or directory
   total 0
   drwxrwxrwx  128 halito  halito  4308 Jan 22 01:19 .
   drwxrwxrwx@   6 halito  halito   264 Jan 22 01:17 ..

   The error messages there claim "no such file" exists... yet, open the folder and view its content with Finder to see all the droppings:
   

   $ open /Volumes/your/path/to/test-target

   
   
   More fun: those double-dot files can not be deleted over our LAN. We have to login to the target's hardware device itself and do the deletions directly. [see my Time Capsule question below.]

--

It would seem then that —to totally *prevent* any possible freak out like this with vanilla (rsync 2.6.9) machines —we would probably need to exclude all dot files in the source from rsync via:

--exclude='.*'

Seems drastic, but is that true? Anyone have more insights on this phenomenon? Did rsync 3.0.7 fix this? If so, did it clean up an rsync bug, or workaround a Snow Leopard/AFP bug? Why does Finder (over a network) display the curious critters, while simultaneously Terminal (over a network) claims they don't even exist??? [note: i did search here at macosxhints, but found no mention of any of this stuff anywhere.]


And BTW, how can we remove these pesky double-dot files from a Time Capsule disk after-the-fact? [i.e., not talking about a sparsebundle image like what Time Machine backs up to... but from within folders on the actual Time Capsule disk itself?]

TIA and have a nice day.

-HI-

[agentx]

Wow i think you win the competition for longest post ;-)

I have long stopped using rsync 2.6.9(2008) and now use 3.0.7 compiled for snow which works fine for all my needs without any issues that i have found. I would certainly install and try a different version.

[fracai]

With both the SL supplied rsync (2.6.9) and MacPorts built (3.0.7) I don't see this error.

Using this script, I see the attached output.

This could be a sign of other filesystem damage.

#!/bin/sh

SOURCE=/Users/me/test-source/
TARGET=/Volumes/AFP/Users/me/test-target/

mkdir "${SOURCE}"
touch "${SOURCE}"/{.whatever,file{1,2}}
ls -la "${SOURCE}"
rm -rf "${TARGET}"
/opt/local/bin/rsync --version
/opt/local/bin/rsync -axhuvin --no-g --stats "${SOURCE}" "${TARGET}"
/opt/local/bin/rsync -axhuvi --no-g --stats "${SOURCE}" "${TARGET}"
ls -la "${TARGET}"
rm -rf "${TARGET}"
/usr/bin/rsync --version
/usr/bin/rsync -axhuvin --no-g --stats "${SOURCE}" "${TARGET}"
/usr/bin/rsync -axhuvi --no-g --stats "${SOURCE}" "${TARGET}"
ls -la "${TARGET}"

[Hal Itosis]

Quote: Originally Posted by agentx I have long stopped using rsync 2.6.9(2008) and now use 3.0.7 compiled for snow which works fine for all my needs without any issues that i have found.

Yeah, good idea... but i'm also thinking in terms of "users in general" and the posting of scripts in public forums which might bork up other users' disks with double-dotted debris.

Quote: Originally Posted by agentx I would certainly install and try a different version.

Totally agree in spirit. I would extend that sentiment however, and say that: Apple should install a bug-free (or less bug-prone) version of rsync onto everyone's machines.

[Hal Itosis]

Quote: Originally Posted by fracai With both the SL supplied rsync (2.6.9) and MacPorts built (3.0.7) I don't see this error. Using this script, I see the attached output. This could be a sign of other filesystem damage.

Hmm, okay... thanks. But —as evidenced by the links at the start of my post (and google does detect even more than those five) —this is not something that i alone am seeing. So perhaps there are some other factors needed to bring about this behavior. (idunno: WiFi versus direct Ethernet connection?, being logged-in with the same username/uid on both ends?, running in 64-bit mode versus non-64-bit mode?, who knows?).

It is interesting though that you're not getting bit by this. That does tell us something... i'm just not sure what (exactly), yet.

I ran across this thread and found your description of a problem which exactly matched mine. I wanted to chime in for a moment and say that I found a solution for deleting files on a Time Capsule.

First, mount the disk via SMB instead of AFP. To do this, choose, Go->Connect to Server from the finder. Then, login using smb://yourservername.

Once mounted, I executed the following command in the directory. In my case, the files were ..DS_Store ... files.

% find . -name '..DS_Store*' -print0 | xargs -0 rm

It took a while, but this did the trick and I was able to delete the files successfully. Hope this helps with your problem as well.

[Hal Itosis]

Quote: Originally Posted by crscheid First, mount the disk via SMB instead of AFP. To do this, choose, Go->Connect to Server from the finder. Then, login using smb://yourservername. Once mounted, I executed the following command in the directory. In my case, the files were ..DS_Store ... files. % find . -name '..DS_Store*' -print0 | xargs -0 rm It took a while, but this did the trick and I was able to delete the files successfully. Hope this helps with your problem as well.

Oh sweet... blessed relief.
That indeed was the ticket.
THANKS!

--

Still open to other ideas/solutions from anyone which might serve to prevent this situation from occurring in general cases (vis-à-vis posting rsync scripts online for the masses to run). Solutions besides using the heavy-handed --exclude=.* i mean.

Now the explanation.

To avoid other problems Rsync creates a temporary file to transfer to, then moves the file over the original copy being overwritten.

The temporary naming scheme is .{filename}.random
But as you can see in your case it results in files being ..filename.random

That may be an illegal filename to start with a double dot, preventing the move command from executing. When that fails Rsync starts a new temporary file and tries it again.

Two more options:
If the file(s) is not in use at the time, use rsync's in-place option to over-write the file without using a temp file.
or, configure it to use a temp directory and set it to not use temporary file names so it will never create an illegal filename with ..x.y

I've created what I think is a pretty good workaround for this issue. The workaround is to modify the way rsync creates temporary files so that is only prepends a dot if there isn't already one. This should appear in the official rsync when it reaches version 3.0.9, but until then you can apply my patch as follows;

Type the following commands in terminal;

curl http://rsync.samba.org/ftp/rsync/src/rsync-3.0.8.tar.gz > rsync-3.0.8.tar.gz
tar zxvf rsync-3.0.8.tar.gz
curl http://www.mudflatsoftware.com/rsync/nodotdot.diff > nodotdot.diff
cd rsync-3.0.8
patch -p1 < ../nodotdot.diff
./configure
make

If you don't feel like compiling your own rsync you could try my rsync GUI application, DropSync. I've built this workaround into that app, but it isn't the default (you'll need to select "latest rsync" instead of "system rsync" in the advanced tab of the configuration window). -- sorry for the shameless plug

[mikewong]

i'm not sure what happened, but maybe my rsync 2.6.9 crashed last time i ran it. anyways i found out i had a lot of files like this, all with the modification date of my last backup:

.F204N2_000774.dat.Vh0Mzm
.F204N2_000774.out.kX4Eze
.F204N2_000775.dat.b27Als
.F204N2_000775.out.I9Mkc8
.F212N2_000226.dat.pJe8Gs
.F212N2_000226.out.ezc1SC

this thread, and especially ppostma1's comment, helped me figure out what happened. i've updated my rsync to 3.0.9.

this was for syncing a macbook pro (10.7) to a desktop mac pro (10.6) over an ethernet cable.

it was hard to figure this out since you can't just google .*.??????
=)