rsync by mtime

[saltydog4791]

Hello all,

Stumped a little with what I am trying to do. I backup all my files once a month to some drives that I keep offsite. I would like to do daily backups to another drive at home so I have some kind of disaster recovery within the month. I have tried the following which seems to work on my os x box but I also need this to run on an ubuntu fileserver. For some reason the following will not work on the fileserver. Does anyone know of an alternate approach that might work? This is what I am trying to work with:

rsync --files-from=<(find src -mtime -1 -print) src dest

I can't think of any other way to do this. I would like to mirror the directory structure if at all possible. Thanks to anyone who has some clues.

mike

[lar3ry]

Look at the rsync manual page for --backup and --backup-dir -- these allow you to create daily incremental backups.

It's not automatic--you'll need to add some logic to create the incremental directories. (SMP: It's a "S"imple "M"atter of "P"rogramming...)

Here's some pseudo-code I use for my daily hotbackup and network backup scripts:

- Create BACKUP_DIR/inc/2007/03/03
- rsync -avz --delete --backup --backup-dir=BACKUP_DIR/inc/2007/03/03 from_dir/ to_dir/

This will do the following:

- Make "to_dir/" a mirror copy of "from_dir/" every day

- If a file is changed or deleted from the old "from_dir/", it will be copied to the same file structure within "BACKUP_DIR/2007/03/03" (today's date). Example: Assume "from_dir/dir1/dir2/dir3/file3" changes between yesterday and today. The new value will be found in "to_dir/dir1/dir2/dir3/file3" as expected, but the old contents will be found at BACKUP_DIR/2007/03/03/dir1/dir2/dir3/file3"

You will need either a large backup drive or put into place some method of pruning the daily backups you don't need (do you need daily incrementals from six months ago?).

Also note that if you copy disconnected sets of folders, you'll need some naming convention. For example, let's say that I backup /Applications, /Library, and my home directory /Users/lar3ry every day. I'd need to run rsync three times as follows:

$ rsync -avz --backup --backup-dir=/Volumes/Back/2007/03/03 /Applications/ /Volumes/Back/Applications/

$ rsync -avz --backup --backup-dir=/Volumes/Back/2007/03/03 /Library/ /Volumes/Back/Library/

$ rsync -avz --backup --backup-dir=/Volumes/Back/2007/03/03 /Users/lar3ry/ /Volumes/Back/Users/lar3ry/

You'll want to make sure you change the name of the incremental directory every day... this can be done using the "date +%Y/%m/%d" command.

My own personal script is written in perl and works the way I describe. It's saved my butt a number of times. Also note that rsync allows you to backup over the network, which can give you more flexibility:

- If you back up, make sure you back up to a different system (preferable), drive, or (at least) partition. Backing up onto the same drive/partition won't save you when the drive goes bad.

- USB and Firewire drives are great for backup/archival purposes. Use them!

- DoSomethingWhen (DSW) can be used to have a backup happen automatically once a USB drive becomes available to OS X. In other words, simply turn on your drive and DSW will start your backup for you.

- There are some good "network drives" available nowadays that might be perfect for off-system backup. Most use Windows file systems, so you might lose resource forks, etc, but it's better than losing data.

lar3ry

[Squirtypoo]

I'm confused. What's the reason you can't you use 'rsync -a --delete-after src dest'? This will mirror the directory structure and preserve the files' metadata.

[saltydog4791]

Thanks for your replies guys. Is there a way I can get this to work for just backing up what has changed since the last full backup? I only want to backup the changes since the last full backup while retaining the right directory structure.

[cpragman]

I did this a while ago using a version of tar (I named it hfstar) that respected hfs (that might have been rolled into the base tar since then). Instead of trying to figure out how to name a backup based on a specific date/time, I used the system's aproach of just rotating the files.

[Squirtypoo]

Quote: Originally Posted by saltydog4791 Thanks for your replies guys. Is there a way I can get this to work for just backing up what has changed since the last full backup? I only want to backup the changes since the last full backup while retaining the right directory structure.

I thought the default behaviour of rsync only copied incremental changes?

[lar3ry]

Quote: Originally Posted by saltydog4791 Thanks for your replies guys. Is there a way I can get this to work for just backing up what has changed since the last full backup? I only want to backup the changes since the last full backup while retaining the right directory structure.

What this will do is keep the "BACKUP_DIR" set to an up-to-date mirror of your files, only copying the files that changed since the last time you ran your backup. The files that were changed between the two dates will be placed into the "incremental" folder named with the date/time in my example. Thus, the changes between two backups can be found in the dated directories (in the proper directory structure), and BACKUP_DIR will always contain the latest versions of all the files.

rsync will only copy the files that changed between runs.

• The first time you run the script (when nothing is in the destination), the rsync command will copy every file to the destination folder and the dated directories will be empty.

• The second time you run it, it will only copy those files that changed since the first run, making the destination folder completely up to date, and all changes between the first run and the second run in the dated folder.

• The third time you run it, it will again only copy those files that changed since the second run, once again making the destionation folder up to date and putting the changes between the second run and the third run into the dated folder.

Here is a script to get you started. You will almost definitely need to tweak this:

#! /bin/sh

BACK_DRIVE="/Volumes/BackupDrive"
DIRS="/Applications /Users/userName"

FULL_DIR="$BACK_DRIVE/Full"

INC_DIR="$BACK_DRIVE/`date +%Y-%m-%d`"

for i in $DIRS
do
    rsync -az --backup --backup-dir="$INC_DIR" "$i/" "$FULL_DIR/$i/
done

There are many options that can be added, but this should be a start.

This script assumes the following:

• Your backup drive is /Volumes/BackupDrive

• You wish to back up the /Applications folder and your HOME folder (/Users/userName).

• Your full backup would be found at
  • /Volumes/BackupDrive/Full/Applications (your system applications folder)
  • /Volumes/BackupDrive/Full/userName (your HOME folder)

• Your incremental backup would be found at
  • /Volumes/BackupDrive/2007-03-07/Applications (changes to your applications folder)
  • /Volumes/BackupDrive/2007-03-07/userName (changes your HOME folder)

I hope this explains things better than I did the first time.