How to Backup
 

I've been researching backing up my home folder. My problem is that I want to exclude certain folders from my reoccurring backups. for example my music folder is 10GB and is already backed up (on my iPod :) ) I don't want to waste that space on my backup drive. My question is this. Does anybody know of a way of excluding a file from a command? Is there a way to use wildcards to do this?

I have looked into using ditto, Synk, psync and hfspax and only hfspax seems to offer this ability built in. Unfourtunely, the complexity of hfspax has me a little worried. Does anybody have any friendly advice?

Thanks!

well, you could move your music dir out of your home, and symlink to it. that would be one method.

Sounds like a good idea, I figured there was probably something easy I was overlooking.

I haven't created my own symlink before, and I'm sure I can find some resources on the web on how to do it but can you give many any pointers (I did look at the man page). My two biggest questions would be (A) whats the difference between a symlink and a link? and (B) should I be fairly confident that iTunes would behave itself with it?

Thanks for the suggestion,

Richard

There is no need to symlink. Look in Itunes:Preferrences:Advanced. You can move your music folder and then just tell iTunes where you put it.

Again, another thanks, I don't know how many times I've seen that preference, but I just didn't make the necesary connection for this little problem. Always good to ask others with a fresh perspective.

I always think twice before going to the forums and bothering other people with questions. I tell myself that I should just do more research on my own. But each time I do write in, I'm always glad I did.

Thanks again,

Richard

Umm, why not making a list of the files and folders to backup:

cd ~
ls -1 > backuplist_1

then excluding your music folder:

cat backuplist_1 | grep -v Music > backuplist_2

and taring and gzipping those files so that you can burn them:

/sw/bin/tar -cvzT backuplist_2 -f burn_me.tar.gz

Note that you can't use the "-T" option in the standard tar which is used by OS X 10.2.4, well not on my system, at least. You have to use the GNU tar utility that can be easily installed with fink (fink install tar). I made a symlink called "tor" in my /usr/bin directory.

cd /usr/bin/
sudo ln -s /sw/bin/tar tor

You can put the first instructions in a shell file, if you tend to making backups quite often. And don't forget to delete the filelists in those directories. And in the end you could put all those commands in one command:

ls -1 | grep -v Music > backuplist; tor -cvzT backuplist -f burn_me.tar.gz; rm backuplist

tar is not an acceptable format for backing up Mac OS files (which in many cases contain resource forks).

A better solution would be something like psync or rsync. These utilities allow you to include/exclude files.

There is a version of tar called hfstar that is acceptable for backing up Mac OS files - but in my many hours of testing - it is unreliable and I would not recommend it.

Also - Carbon Copy Cloner does a great job of backing up Mac OS files using the ditto command. I would highly recommend that. One nice feature of CCC is it generates shell scripts for backing up and is a great way to learn some shell scripting.

You are welcome, glad I could help out. I have used that preference on a family's computer. Each person had their own user id, but they all shared the same iTunes library.

bluehz,

I have fixed my original problem, but your response brings up another question for me. You mention that psync allows you to include and exclude files, but I've looked at the various man pages and didn't see anything like that. I'm courious how it would be done.

Thanks

psync - selective backups
 

I use psync for a variety of purposes including backing up to an Apacer HandyDrive. The procedure I use is to run the following AppleScript:

tell application "Finder"
set theDisks to name of every disk
if {"NO_NAME"} is in theDisks then
do shell script "/Volumes/NO_NAME/backup"
end if
end tell

where the shell script referenced above is:

sudo /usr/local/bin/runpsync -d 0 0 '/Users/djn1/Documents/Bolton' '/Volumes/NO_NAME/Bolton'
sudo /usr/local/bin/runpsync -d 0 0 '/Users/djn1/Documents/Publications/Final versions' '/Volumes/NO_NAME/Final versions'
sudo /usr/local/bin/runpsync -d 0 0 '/Users/djn1/Documents/Publications/current publications etc' '/Volumes/NO_NAME/current publications etc'
sudo /usr/local/bin/runpsync -d 0 0 '/Users/djn1/Documents/Other professional/Job apps + cv/CV' '/Volumes/NO_NAME/CV'
sudo /usr/local/bin/runpsync -d 0 0 '/Users/djn1/Documents/Publications/Articles' '/Volumes/NO_NAME/Articles'

In other words, I specify in advance which folders I want to back up.

I am not familiar with that. Would you please explain that to me a litte?

Also I don't have psync on my system - fink doesn't know it either. Would you tell me where to obtain it?

Some Mac files (mostly older apps, etc) are composed of two components - a resource component and a data component. This is completely different from the rest of the world and other platforms - e.g. Windows, Linux, Unix, etc. In order to properly backup Mac files - you need to use a utility that will properly recognize and store both portions of the Mac files - the resource and the data. Since tar comes from a unix background it knows nothing of the resource portion of Mac files and will simply ignore it when backing up the file. The result being that when you restore the file it is incomplete.

Mac OS X has moved away from using resource portions of files and is more in line with the rest of the world these days by only using teh data portion of the file. In those cases - it is perfectly fine to backup with teh tar utility. The problem is - the transition from resource/data Mac format to strictly new style data only formats is still in transition. Meaning many of the files on your machine - even though they may be completely new apps or whatever - are still using old style formats. You would never be able to ascertain which file you could safely backup with tar.

psync and rsync are specifically created to be aware of Mac formatted files.

...and yes - I was mistaken - psync will not let you use a exclude/include setup - like reading it from a text file - but you can stil specifically command which dirs you wanted synced (ala backed up).

psync is available here
http://sourceforge.net/projects/psyncx

Lots of good info here:
http://www.dan.co.jp/cases/macosx/backup-volume.html

If just starting out using OS X and backing up, I would highly recommend CarbonCopyCloner - which uses Ditto. Very reliable, great GUI - no scripting required (unless you want to). Get it here:
http://www.bombich.com/software/ccc.html

it's also available here:

http://www.versiontracker.com/morein...d=16089&db=mac

Thanks. But since I'm using almost only Linux apps, I think I'll stick with tar.

Problem with using tar is you just won't know what's broken until its too late.... suit yourself though. Good luck!

My only problem with ditto is that it always backs up the whole folder. Psync (and I think hsfpax too) will just backup the changes that was made between the two folders. Seems like its a lot more..... elegant, for a lack of a better word. My problem now is that I put together a little script (my first!) that runs fine in the terminal, but wont run correctly from the crontab. I think it is because psynce is perl (correct?). Here is my little script (backup.sh):

#!/bin/sh
mkdir /Volumes/disk
mount -t afp afp://user: password@10.0.1.2/user /Volumes/disk
psync -r -d ~/Desktop/Test/ /Volumes/disk/Next/
umount /Volumes/disk
disktool -r

(FYI-there isn't a space between user: and password, but if I didn't add it, the result would be a smile face added by the forums server)

This mounts my backup volume (through airport using AFP), compares the two folders, makes changes, unmounts volume, and then refreshes my desktop. When it runs from contrab (sh ~/bin/backup.sh) It mounts the volume, it unmounts it, and then refreshes the desktop, but doesn't run psync. From TOP I can see the tcsh shell open, then I see the sh shell open, but no perl.

Is there a way to fix this? If not I'll try other ways, but I wanted to understand why this wasn't working first.

Thanks for any insight, I might repost this in the Forums under a different title, since it the question has evolved a little bit. Would this be proper etiquette?

Thanks again for any feed back/help!

system crontab or user?
 

Are you running the cron job through your user crontab or the system crontab? I ask as PsyncX (a gui front end for psync) writes scheduled copies to the system crontab. Having said all that I don't know if this is at all relevant but it might be worth trying to run your script from the system crontab if you haven't already tried it.

I suspect you are goingto have to run the script from the system crontab. Also- make sure your path variables are at the top of your crontab.
Can you open the terminal an watch as cron attempts to run the backup? Post results here.

Also - I believe you are correct about ditto copying all files (as opposed to syncing). I kinda of wonder though as the man pages say:
I am not sure about the term "merge". It almost sounds like it might sync - but I have never done any testing.

Here is a script I use - that was actually saved out of Carbon Copy Cloner and modified. It may give you some ideas.
This script creates a disk image and then backs up my home dir to the disk image.I have had trouble with mounted stuff before - but this scrpt works so maybe you can find something in there that might be the key.

Also - that damn smile thing got me for months - b efore I noticed the little "Disable Smilies in this post" at the bottom of the reply form.

[edit: fold -mt]

You'll definitely want to run your backup script using the system crontab (unless you're completely sure that everything in the directories is owned by you; indeed, I'm not even sure that that's good enough).

One big thing: don't use "~" in your crontabs: always thrown in the full path to the directory to which you're referring. In this case --with cron running as root-- you're probably backing up a directory that doesn't exist: hence the difficulty. (root's home dir is, ummm, /var/root).

By the way, CCC uses psync for its synchy bits and pieces now.

Cheers,
Paul (avoiding writing lectures: ugh, here comes first semester, like a rush...)

Psync is available with Fink. Install the 'macosx-file-pm' package (Collection of modules to manipulate files on MacOS X)

There is no function in Fink commands to allow you beforehand to check what's inside the 'macosx-file-pm' package. So, if you search for psync with 'fink' it will not find it. The only way I know to do this, is to make a 'google' search for 'psync fink' . That will bring some results.

Thanks for all the info! As you might have guessed there was a couple of minor problems. I removed all the "~" (-thanks to pmccann), but the biggest problem was that I needed to add the entire path for psync (/usr/local/bin/psync) when writing the command. I'm not sure which paths contrab uses but obviously this wasn't one of them. I am also planning to add some ideas from the script that bluehz posted (from Mike Bombich's script-thanks to both).

One note I would make is that I didn't use Carbon Copy Cloner because it states that it only works localy. Psync has the ability to back up over AFP (it saves an additional file that contains all the original permissions, some of which I assume must change when saving to another machine. When you retrieve the backup psync reads the additional saved file and returns the original permisions) I don't think ditto has this ability (maybe CCC does now but the current documention makes me think not). I hope this helps others-I definately learned a lot.

Here is some general info I picked up over ther course of my investigation. Hope its useful.

utility - preserves permissions, preserves resource fork & creator type, corrects permissions with AFP

cp - yes, no, no
CpMac - no, yes, no
ditto - yes, yes, no
rsync - yes, no, ?
RsyncX - yes, yes, ? - slower than ditto
psync - yes, yes, yes
pax - yes, no, ?
hsfpax - yes, yes, ? - many options, complicated (to me)
tar - yes, no, ? - I believe there is a MacOSX version of tar but I don't remmember the name


GUI interfaces

Carbon Copy Cloner - uses ditto
PsyncX - uses psync
Sync - not sure what it uses - yes, yes, ? - nice archiving ability

(please let me know if you see something wrong)


Thanks again to everybody!

Richard

Thanks!

Here are some other options summarized:

http://chemistry.ucsc.edu/~wgscott/x..._and_fink.html

I have been experimenting with psync lately and overall I am very happy with it. I do have one question. Some of the stuff I was synching up were VISE installers from HP. After the synch the destination system doesn't recognize them at all. It seems like the forks are stripped, or something like that. Does anyone know why this happens with these types of files?

Thanks,

saltydog4791

I tried using psync for long time and ran into various problem. Mainly issues with errors during the backup. The output was never enough to detail what was going wrong and the way I look at it - any errors druing a backup means you have an incomplete, unreliable backup. I have also seen the issue with the applications getting borked - even though psync states it is completely OS X safe. I therefore gave up on psync. Of all the stuff I have tried - psync, rsync, hfstar, hfspax, ditto - only ditto suits my needs for a errorless, reliable backup. This is unfortunate too since ditto really has no fine grain control over it - its an all or nothing prospect.

I don't know for sure how stable psync is now verses earlier versions, but in my limtied experience I've had no problems. I recently move my iTunes folders (over 9 GBs) using psync without error. Of course this was mostly mp3s but still I thought I would mention my limited experience.

Richard

Hello,

I keep my backups with psync, I recently tested it and it all looked fine. Don't get any errors either. You might want to give it a second chance.


v

I second vonleigh's suggestion because I've been using psync for nearly two years now and have had no problems with it that weren't user created. It's very solid and my only backup method on OS X. And I've tested all the other popular utilities for backup, finding the closest second to be CCC.

I can't use ditto because of its problems with locked files. hfspax is good but too easy to make a mistake with. hfstar isn't solid enough to be used reliably.

The fact that you can make incremental backups with psync to a mirror of your drive is why I think it's the easiest and most reliable method.

Till now, my experience with "psync" has been quite good. No complaints here.

RsyncX is freeware and it's definitely faster than ditto when it comes to large volumes of data where only a small part is changed between each back-up. I use it in a cronjob (root's crontab) for frequent back-ups of data that I would really, really hate to loose. Has remote and exclude features, and copies resource forks. There's also a GUI if you are so inclined. I use carbon copy cloner for system cloning prior to updates (well, that was the idea...) and has used psync to update the clone from time to time without problems.
Appears that the tool to be chosen depends on the job to be done, if you'll forgive the platitude.
I think there's room for diversity here.

My psync always completes - but it always has errors... usually cryptic...

If you're getting the same errors on the same backup scheme using psync then you need to wipe the target clean and start that backup again from scratch. In my experience, most of those errors are from orphaned sym links because of files that have been deleted by the user. Use the -v switch to have verbose output that you can look through to make certain the backup is clean. If not, then it will give you the line number of the error and you can check that out. It's not really so cryptic.