rsync revisted...

a11en · 05-27-2006, 04:07 AM

Ok, guys, this rsync thing is getting a bit crazy... what are the version of rsync out there that are up to date, and what are the current problems with them? I'm having trouble getting this worked out.

I believe there's a version of rsync with osX, that handles the -E option, for extended attributes/resource forks, one version which is out there from samba.org, which doesn't handle the extended attributes (I think the current version is 2.6.6?), and RsyncX which handles resource forks, but only on hfs+ drives (are there other problems with RsyncX?).

What I'm looking for is essentially an hfs+ only option right now. I need backups which can resurrect my machine if it goes down, and keep my thesis work safe. I'd prefer if it doesn't copy every darned file with an extended attribute (which apple's version seems to like to do) even when there is no change. So, for now, I'm only interested in a mac-only solution. However, is there a solution for a unix drive backup option?

I've currently been running the following command line through a bash script with a patched version of 2.6.3 it appears. [I think there were 2 patches out there... one for extended attributes, and one for a date issue? I need to track that down again.]

sudo $RSYNC -r -E -a -S -vv -x -p -e --delete-after \
--exclude-from=backup_excludes.txt $* /Volumes/Macintosh\ HD/ /Volumes/BackupDrive/PowerbookBackups

The problem with the above command, is that every extended attribute file is transferred it appears. (like all my photos in iPhoto, even though they haven't changed in years...)

Any and all help greatly appreciated. I'm >< close to a great solution here to my backup woes... I'm confident I have a good backup at the moment, but not confident I'm doing it the best/quickest way at the moment.

My hope in this thread is to get a rundown of the versions out there, and the possible problems/solutions that they pose in one place/thread, so that others like me can get to the bottom of this quickly.

Thanks, guys!!
-Allen

Gnarlodious · 05-27-2006, 05:02 AM

I also would like to know if there is a newer version of rsync than the one that came with 10.4, which is rsync 2.6.3. This version works great, unless the "extended attributes" switch is enabled. I suspect the -E feature was a kludge and there is a newer one out there for OSX that works right.

a11en · 05-27-2006, 05:09 AM

BTW, I just found out about rdiff-backup (check out fink packages- unstable tree).. wondering if anyone could add to this comments about rdiff-backup.

Thanks, guys! I know that we can't be the only ones who are confused about rsync vers.

-Allen

tjj · 05-27-2006, 11:37 AM

See this blog

Gnarlodious · 05-27-2006, 12:48 PM

Where I see rsync failing is under these circumstances:

1) When the -E and --delete option are used together, rsync attempts to delete the file twice, throwing an error for each nonexistent file.

1) When the -E and --delete option are used together, rsync may arbitrarily delete the resource file on the target machine while copying the data file. This has resulted in missing image icons, no big deal.

3) When the -E option is is used, rsync may arbitrarily fail to copy the newer resource file, causing the new data file to be merged with an older resource file. I see this problem rarely.

4) The worst bug is, when the -E option is used, rsync sets the modtime on the target machine to the time rsync was run regardless of whether the file was copied. This means if you are simply backing up it is irrelevant, but if you are synchronizing two machines, every file will be copied from the target machine because they are all marked as newer than the source machine.

5) When the -E option is is used, rsync reports every file checked regardless of copy status, resulting in huge amounts of Terminal output and the inability to see what has actually been copied.

Notice that all these failings are associated with the -E option. Using rsync without the -E option works as well as on any Linux machine.

Despite these 5 bugs, I use rsync daily and it works well enough for me.

Hope this information helps someone.

tjj · 05-27-2006, 05:02 PM

Thanks Gnarlodius,
That's good to know.
However, it's annoying that the -E flag is broken, and when used for sync'ing -even if it's 'just' back-ups - the modification date issue is, well, inconvenient.

I just tried to rdiff-backup my / to an external Firewire drive - apparently it succeded, though permissions for some of the toplevel folders on the backup appear to differ.
I have had asr and Superduper choke on different files within the NeoOffice application, rdiff-backup didn't. Haven't thoroughly checked the backup yet - e.g. if it's bootable...

Edit: Hm, lots of files and dir not backed up - according to the output this was due to running out of space. Apparently, it takes more overhead thant the the 3+GB I had on the external drive.
And btw this file caused an error (also in asr and superduper):

Code:

Applications/NeoOffice.app/Contents/share/dict/ooo/it_IT.dic [Errno 5] Input/output error

a11en · 05-30-2006, 02:41 AM

Hi again everyone,

I wanted to let you know I installed the version of rsync that comes with RsyncX. It appears to use the same commands as the updated rsync with patches to enable HFS/extended attributes.

Unfortunately, from what I can see, it still insists on upping files with extended attributes even though they have not been updated since the last rsync command. So, it appears this version of rsync also suffers from the faulty -E flag option. If I'm wrong, please post any comments here, I greatly appreciate it. BTW, I should mention that I used the version as rsync through the CLI, so if there's some sort of special scripting to take care of the extended attributes, it's possible I'm missing them.

Next up on the chopping block: rdiff-backup...

Will let you know how it goes!
-Allen

olealf · 05-30-2006, 05:01 AM

The -E bug is the reason I'm still using version 2.6.0 installed by RsyncX.

Beside it filling up my logfiles with '/rsrc paths are deprecated (..namedfork/rsrc)' it works like a charm.

a11en · 05-31-2006, 12:49 AM

So, 2.6.0's -E flag works without constantly updating the extended attribute files?

Just working my way through rdiff-backup. The idea that the database can go south doesn't seem like too much fun... but it appears the filestructre that would have been made last would be useable. So, it's not a complete failure of the system, I feel. It just means a rebuilt backup most likely.

olealf · 05-31-2006, 05:26 AM

Quote:

Originally Posted by a11en


	So, 2.6.0's -E flag works without constantly updating the extended attribute files?

The -E flag wasn't present in that version. Instead you had to use -eahfs flag.

I got a lot of ideas for writing my own backup scripts from those two sites:
http://www.mikerubel.org/computers/rsync_snapshots/
http://gja.space4me.com/things/Backups_With_Rsync.html

They are not Mac related but the general ideas can be used on a Mac.

05-27-2006, 04:07 AM	#1
a11en Triple-A Player Join Date: Nov 2005 Posts: 71	rsync revisted... Ok, guys, this rsync thing is getting a bit crazy... what are the version of rsync out there that are up to date, and what are the current problems with them? I'm having trouble getting this worked out. I believe there's a version of rsync with osX, that handles the -E option, for extended attributes/resource forks, one version which is out there from samba.org, which doesn't handle the extended attributes (I think the current version is 2.6.6?), and RsyncX which handles resource forks, but only on hfs+ drives (are there other problems with RsyncX?). What I'm looking for is essentially an hfs+ only option right now. I need backups which can resurrect my machine if it goes down, and keep my thesis work safe. I'd prefer if it doesn't copy every darned file with an extended attribute (which apple's version seems to like to do) even when there is no change. So, for now, I'm only interested in a mac-only solution. However, is there a solution for a unix drive backup option? I've currently been running the following command line through a bash script with a patched version of 2.6.3 it appears. [I think there were 2 patches out there... one for extended attributes, and one for a date issue? I need to track that down again.] sudo $RSYNC -r -E -a -S -vv -x -p -e --delete-after \ --exclude-from=backup_excludes.txt $* /Volumes/Macintosh\ HD/ /Volumes/BackupDrive/PowerbookBackups The problem with the above command, is that every extended attribute file is transferred it appears. (like all my photos in iPhoto, even though they haven't changed in years...) Any and all help greatly appreciated. I'm >< close to a great solution here to my backup woes... I'm confident I have a good backup at the moment, but not confident I'm doing it the best/quickest way at the moment. My hope in this thread is to get a rundown of the versions out there, and the possible problems/solutions that they pose in one place/thread, so that others like me can get to the bottom of this quickly. Thanks, guys!! -Allen

05-27-2006, 11:37 AM	#4
tjj Major Leaguer Join Date: Dec 2002 Location: Copenhagen, Denmark Posts: 271	I'm also confused... See this blog __________________ -- Thomas 10.4.11 on Pismo 500 MHz/1GB 10.5.6 on iMac intel 2.16 GHz core duo/2GB

05-27-2006, 12:48 PM	#5
Gnarlodious Major Leaguer Join Date: Oct 2003 Location: Santa Fe Posts: 372	rsync bug list Where I see rsync failing is under these circumstances: 1) When the -E and --delete option are used together, rsync attempts to delete the file twice, throwing an error for each nonexistent file. 1) When the -E and --delete option are used together, rsync may arbitrarily delete the resource file on the target machine while copying the data file. This has resulted in missing image icons, no big deal. 3) When the -E option is is used, rsync may arbitrarily fail to copy the newer resource file, causing the new data file to be merged with an older resource file. I see this problem rarely. 4) The worst bug is, when the -E option is used, rsync sets the modtime on the target machine to the time rsync was run regardless of whether the file was copied. This means if you are simply backing up it is irrelevant, but if you are synchronizing two machines, every file will be copied from the target machine because they are all marked as newer than the source machine. 5) When the -E option is is used, rsync reports every file checked regardless of copy status, resulting in huge amounts of Terminal output and the inability to see what has actually been copied. Notice that all these failings are associated with the -E option. Using rsync without the -E option works as well as on any Linux machine. Despite these 5 bugs, I use rsync daily and it works well enough for me. Hope this information helps someone.

05-27-2006, 05:02 PM	#6
tjj Major Leaguer Join Date: Dec 2002 Location: Copenhagen, Denmark Posts: 271	-E flag Thanks Gnarlodius, That's good to know. However, it's annoying that the -E flag is broken, and when used for sync'ing -even if it's 'just' back-ups - the modification date issue is, well, inconvenient. I just tried to rdiff-backup my / to an external Firewire drive - apparently it succeded, though permissions for some of the toplevel folders on the backup appear to differ. I have had asr and Superduper choke on different files within the NeoOffice application, rdiff-backup didn't. Haven't thoroughly checked the backup yet - e.g. if it's bootable... Edit: Hm, lots of files and dir not backed up - according to the output this was due to running out of space. Apparently, it takes more overhead thant the the 3+GB I had on the external drive. And btw this file caused an error (also in asr and superduper): Code: Applications/NeoOffice.app/Contents/share/dict/ooo/it_IT.dic [Errno 5] Input/output error __________________ -- Thomas 10.4.11 on Pismo 500 MHz/1GB 10.5.6 on iMac intel 2.16 GHz core duo/2GB Last edited by tjj; 05-27-2006 at 06:48 PM.

05-27-2006, 05:02 AM	#2
Gnarlodious Major Leaguer Join Date: Oct 2003 Location: Santa Fe Posts: 372	I also would like to know if there is a newer version of rsync than the one that came with 10.4, which is rsync 2.6.3. This version works great, unless the "extended attributes" switch is enabled. I suspect the -E feature was a kludge and there is a newer one out there for OSX that works right.

05-27-2006, 05:09 AM	#3
a11en Triple-A Player Join Date: Nov 2005 Posts: 71	BTW, I just found out about rdiff-backup (check out fink packages- unstable tree).. wondering if anyone could add to this comments about rdiff-backup. Thanks, guys! I know that we can't be the only ones who are confused about rsync vers. -Allen

05-30-2006, 02:41 AM	#7
a11en Triple-A Player Join Date: Nov 2005 Posts: 71	Hi again everyone, I wanted to let you know I installed the version of rsync that comes with RsyncX. It appears to use the same commands as the updated rsync with patches to enable HFS/extended attributes. Unfortunately, from what I can see, it still insists on upping files with extended attributes even though they have not been updated since the last rsync command. So, it appears this version of rsync also suffers from the faulty -E flag option. If I'm wrong, please post any comments here, I greatly appreciate it. BTW, I should mention that I used the version as rsync through the CLI, so if there's some sort of special scripting to take care of the extended attributes, it's possible I'm missing them. Next up on the chopping block: rdiff-backup... Will let you know how it goes! -Allen

05-30-2006, 05:01 AM	#8
olealf Triple-A Player Join Date: Jan 2002 Location: Hamburg; Germany Posts: 241	The -E bug is the reason I'm still using version 2.6.0 installed by RsyncX. Beside it filling up my logfiles with '/rsrc paths are deprecated (..namedfork/rsrc)' it works like a charm.

05-31-2006, 12:49 AM	#9
a11en Triple-A Player Join Date: Nov 2005 Posts: 71	So, 2.6.0's -E flag works without constantly updating the extended attribute files? Just working my way through rdiff-backup. The idea that the database can go south doesn't seem like too much fun... but it appears the filestructre that would have been made last would be useable. So, it's not a complete failure of the system, I feel. It just means a rebuilt backup most likely.