back up/ file compression

[pyrogen]

Is there a way I can compress my home directory information in the terminal? I want to back up my home folders and preferences on CD-rw. I had read something recently about tar. compression or something like that. I would be very grateful for a tutorial link or info.

Thanks,
Destin

[cadriopo]

Hello, take a look at "hfspax". You can find it here: http://homepage.mac.com/howardoakley/
It contains a very good tutorial and many infos about unix compression & archive tools.

[osxpez]

Depending on how much data you have in your home folder you might get away with tar:ing it directly on the CD-R. From your home directory:

$ tar cvzf /Volumes/NameOfYourCD-R-Disc/MyHomeFolder.tgz .

tar takes several flags and arguments. The flags above are c for create archive, v for be verbose and print the names of the files you are archiving, z for gzip-compress the archive, f for file archive (as opposed to spitting out the archive on standard out I think). The last argument here is . (dot) for the current directory. But you can give a list of specific directories as well.

[pyrogen]

Thank you both!
Destin

[pyrogen]

Osxpez,
Well, I just attempted to tar my home directory to a cdr, but the pipe failed because there was not enough space on the disk.

Is their a way that i can tar the individual folders in my home directory?

Would this work?:

% tar cvzf /Volumes/NameOfCD-R_Disc/Documents.tgz .

Doing a tar for each folder? Pictures, movies etc....

I believe my psd. docs take up alot of space.

Thanks
Destin

[JayBee]

Quote: Originally posted by pyrogen % tar cvzf /Volumes/NameOfCD-R_Disc/Documents.tgz .

You realise what's going on here? The "." at the end is telling tar which directory to tar up. This will try to tar up whatever directory you're currently sitting in. If you try to run this in the root directory of your system, you could be in for a long wait...

For the sake of completeness, you really want to use the following:

% tar cvzf /Volumes/NameOfCD-R_Disc/home.tgz /Users/<your username>/

I think. It should be obvious from here how to tailor this to specific directories.

My own backup solution involves backing up my home directory every morning to a disk image, and then compressing THAT. At the end of each week, I burn the latest image on to a multi-session CD. I think that's a little safer, as I'm not sure how nicely tar treats resource forks. I've had a few headaches from backing up files to linux servers, then restoring files that Mac OS can't recognise. Nasty. Disk Images may even compress better, but I'm not sure...

[pyrogen]

Thanks jaybee. Now that I reread osxpez's post, this seems obvious. Thanks for taking the time and patience to post. I appreciate the help.
Thanks,
Destin

[pyrogen]

Every thing worked perfectly, my user folder is completely backed up. Thanks for all the help.
Cheers,
Destin

[mervTormel]

er, uh, doesn't tar miss the resource forks?!

[pyrogen]

er, uh, doesn't tar miss the resource forks?!


Merv.......

Do I need to be concerned about resource forks when I am backing up the contents of my Pictures, Video, and Documents folders?

I am sorry if this sounds stupid.

Destin

[mervTormel]

Quote: Originally posted by pyrogen Do I need to be concerned about resource forks when I am backing up the contents of my Pictures, Video, and Documents folders? I am sorry if this sounds stupid.

not at all; it's a very good question.

the point is, you have to know. are there rez forks that i am missing? does my backup have any integrity?

if tar misses the rez forks, then i can't rely on it to backup my $HOME at all, because i don't want to have to know whether a file has a rez fork or not. i just want the backup to have integrity. e.g.,

$ pwd
/Users/merv/Library/Preferences

$ ls -Ghln "DragThing Preferences" "DragThing Preferences/rsrc"   
-rwxr--r-- 1 501      0 May 28 23:11 DragThing Preferences
-rwxr--r-- 1 501   4.6M May 28 23:11 DragThing Preferences/rsrc

that one file, alone, disqualifies using a backup facility that will miss an important file.

it boils down to situational awareness and reasonable assurity. if you're aware that files in your backup set don't have rez forks and you're assured that new files or changes won't create them, then tar backup should have integrity.

[JayBee]

Quote: Thanks for taking the time and patience to post.

Not a problem pyrogen. Glad to be able to be of help to somebody for a change, instead of being the one asking the questions!

Anyway, as I pointed out, I'm not sure if tar munges resource forks. It may not be important for data files such as your images etc - these are platform independent mainly, and you can just re-associate them by dragging on to app icons etc.

However, if you want a universal backup solution, you really have to take resource forks into account. As I say, the safest way to do ANY CD based backup (or more generally, any backup which involves a non-HFS+ filesystem), you really want to use disk images.

A disk image will preserve EVERYTHING about your directories by creating an exact sector-for-sector replica. You can then burn the image AS A FILE on to the CD (or upload it onto a server, or whatever) safe in the knowledge that all the underlying structure of the filesystem you've backed up is being preserved.

if you "man hdiutil" and have a scan through there, you'll find out how to create images in the terminal. This is easily scripted up, and can be used to automate your backups for those times you feel ultra-paranoid

In fact, right at the end of the man page, there's a worked example of how to backup a complete folder to a disk image:

Image from folder:
$ du -s myFolder   # du(1) seems to find resource forks, even on HFS+
1234

$ hdiutil create -sectors 1334 folderImage  # add 100 sectors for file system

$ hdid -nomount folderImage.dmg
       /dev/disk1s2

$ newfs_hfs -v myFolderImage /dev/rdisk1s2

$ hdiutil eject disk1

$ hdid folderImage.dmg
       /dev/disk1s2            Apple_HFS         /Volumes/myFolderImage

$ ditto -rsrcFork myFolder /Volumes/myFolderImage

$ is the shell prompt, # are comments that you don't have to type.

The "du" line measures the total size of your backup candidate. You then use hdiutil to create an image slightly larger than this, to make room for file-system stuff. The rest of the code just takes this image, attaches it as a device, formats it, mounts it in finder, then uses ditto to copy the files across ("ditto -rsrc <source> <target>" or "ditto -rsrcFork <source> <target>" are identical commands to copy files while preserving resource forks).

You can then eject the image in finder, and you end up with a file (called "folderImage.dmg" in this case) that you can compress into a tarball to your heart's content. This is a much safer way to do things IMHO, although I've never compared the compression ratios of diskImage.tar vs plainFiles.tar. Frankly however, unless you're REALLY pushing it to compress under 650MB, I don't think you'll be wasting many CDs by using the disk image method - CDs are cheap, and resource forks are expensive

Hope this helps!

[mervTormel]

Quote: Originally posted by pyrogen Do I need to be concerned about resource forks when I am backing up the contents of my Pictures, Video, and Documents folders?

with pictures and video, you'll waste CPU cycles running them thru gzip. they're already compressed. so, you might want to gzip your documents to a temp space then backup, without compression, your docs, pix, and viddies. seems that would be more efficient.

[pyrogen]

Thanks guys, I have alot to learn ...............

jaybee ...... If I follow the prompts you outlined, I will effectively create a disk image( on the desktop or in the finder) which has a copy of my home folder:

ditto -rsrcFork myFolder /Volumes/myFolderImage

is the 'myFolder' my home folder name, or is it just part of the script?

I am guessing that /Volumes/"my home folder name" is correct.

Sorry if I am unclear.

If I make my home folder a disk image, I maintain the resource forks?
If I tar the disk image, does the resource forks remain intact?
Thanks,
destin

[pyrogen]

never mind, figured it out...... sorry

[mervTormel]

Quote: Originally posted by pyrogen is the 'myFolder' my home folder name, or is it just part of the script? I am guessing that /Volumes/"my home folder name" is correct.... If I make my home folder a disk image, I maintain the resource forks? If I tar the disk image, does the resource forks remain intact?

i think whenever you see a reference to a filesystem object, like 'myFolder', you'll need to interpolate that to mean 'full path to the dir of interest'

so, your home is /Users/username and can be short-handed by ~ or $HOME in scripts.

re: disk image, that's a really interesting thing you said. since you ditto -rsrc to the image, rez forks are captured, and since they are 'wrapped' in another 'container', tar will get the goods; tar will be backing up one file, the image that contains what it contains. you see? slick. but a little onerous, stepwise.

[JayBee]

<edit>
Speed was never my strong point, guys!

Ach, I'll leave this up anyway, for the sake of completeness...
</edit>

Quote: Originally posted by pyrogen If I make my home folder a disk image, I maintain the resource forks? If I tar the disk image, does the resource forks remain intact?

Yup - that's basically the idea here. It's like, if you compress a PSD, you retain all the structural information in there (like layers etc) because the structure is stored in a FILE. Here we're doing the same thing. Making a FILE out of the structure, and then compressing that file.

Anyway, on to your specific. After pointing out that using a "." location was unclear, I went and did the same thing! Let's go through this

Let's say your home folder lives at /Users/pyrogen/, and that you want to back it all up (including your Library, which is where all your settings and your mail is stored). If you want to back up something else, just replace this path with another.

Let's also say that we want to put the resultant image into the Shared folder (/Users/Shared). Putting it on the Desktop is a Bad Idea, as we are going to be backing up the desktop, and it's going to be really tricky to try to copy a file into itself...

% du -s /Users/pyrogen

The first step is to find out the SIZE of disk image we need to create. This command returns the size of your home folder in sectors. You can have this returned in various flavours if you like - as always, "man du" will sort you out if you need more help.

Anyway, this returns a number. In my case I got 227780. This is the number we use to work out the size of disk image we need to create in order to back up the folder.

% hdiutil create -sectors 244864 /Users/Shared/pyrogen_backup

Here we're using "hdiutil" to create a disk image which is 7.5% larger than our backup candidate. This is to allow for the extra file-system gubbins that will have to be copied over in addition to the actual files. All file-systems have a system of recording their system of filing (this is what is meant by a disk's "format") and this takes up a little space.

I got 7.5% by trial and error - I'm sure there's a more precise way to work this stuff out, but basically if you are told that you've run out of disk space later on, try altering the factor here and see if that sorts it.

Anyway, this line creates a file at /Users/Shared/pyrogen_backup.dmg. This is our disk image file. However, right now, it's just a file of a certain size.

% hdid -nomount /Users/Shared/pyrogen_backup.dmg

Now, what this does is "attach" our image file to our computer as though it's an external disk. HOWEVER, the "-nomount" option prevents Finder from seeing it. If Finder spots our image, it'll offer to format it. We're going to do that anyway, but we want to control the process in the terminal for the sake of consistency.

In particular, if you've scripted this process up, you don't want the script to hang while it waits for user input.

The output from this command will be along the lines of:

/dev/diskX              Apple_partition_scheme         
/dev/diskXs1            Apple_partition_map            
/dev/diskXs2            Apple_HFS

/dev/diskXsY is a generic way for Unix to know what DEVices are attached to your system. The format is Disk Number X, Slice (partition) Number Y. Here, we can see the way the disk is laid out - slice 1 holds our partition map (the "file system extras" we've made space for), and slice 2 will hold our backup.

First, however, we have to format the disk. To do this, you need to find out what disk number we're dealing with - ie what the X value is for your system. In my case it was 3, but you'll probably find it's 2 (if you have no CDs/disk images already mounted, and you only have one hard disk).

% newfs_hfs -v pyrogensBackup /dev/diskXs2

Here we're using newfs_hfs to create a NEW FileSystem of the HFS type on /dev/diskXs2. Looking at the previous output, we see that /dev/diskXs2 is the partition already labelled Apple_HFS. This is the choice you should make on your own system.

The pyrogensBackup bit is the NAME of your disk. This is the label you'll see when you mount the disk in Finder.

% hdiutil eject diskX

Now that we've formatted the disk, we "eject" it from the DEVice directory, and then...

% hdid /Users/Shared/pyrogen_backup.dmg

... remount it, this time allowing Finder to see it. We now have:

/dev/diskXs2            Apple_HFS         /Volumes/pyrogensBackup

So, we've got a new Volume mounted called pyrogensBackup. If you open a new Finder window and go to the top level (where you can see all the disks on your system) you'll see a white drive icon called pyrogensBackup. Congrats! All that now remains is to:

% sudo ditto -rsrcFork /Users/pyrogen /Volumes/pyrogensBackup

Which will copy your home directory on to the disk image. This is basically making an EXACT copy of your home directory. We use sudo here to get around any possible file ownership problems - you'll have to enter your admin password before the copying starts.

Once that's done, you can just eject the disk from Finder as you normally would, or, to keep it nice and command-line, use

% hdiutil eject diskX

once again. Now, your home directory is stored as a file in /Users/Shared, and is a perfect copy. You can compress this as much as you like, but the integrity of the copy will hold. Also, even if you store the image somewhere that resource forks are lost, the resource forks of all the files inside the image are safe - worst case scenario is that pyrogen_backup.dmg loses its association with Disk Copy. Big deal, eh?

Anyway, hope this rather long winded explanation clears up my mistakes

[mervTormel]

Quote: The output from this command will be along the lines of: ---------------------------------------------------------------- % hdid -nomount /Users/Shared/pyrogen_backup.dmg /dev/disk3 Apple_partition_scheme /dev/disk3s1 Apple_partition_map /dev/disk3s2 Apple_HFS ---------------------------------------------------------------- First, however, we have to format the disk. To do this, you need to find out what disk number we're dealing with. In my case it was 3, but you'll probably find it's 2 (if you have no CDs/disk images already mounted, and you only have one hard disk). ---------------------------------------------------------------- % newfs_hfs -v pyrogensBackup /dev/rdisk3s1 ---------------------------------------------------------------- Here we're using newfs_hfs to create a NEW FileSystem of the HFS type on /dev/disk3s1. Looking at the previous output, we see that /dev/disk3s1 is the partition already labelled Apple_HFS. This is the choice you should make on your own system.

JayBee, et. al.,

the target of that newfs is wrong. i think it should be /dev/disk3s2 . JayBee, you might want to edit that.

you definitely want to hit the right target here, both with the slice label and its /dev/disk representation. anything else could lead to unexpected results and even disaster.

you're script should be oh, so careful here. you might want to use the -N switch in newfs in testing this sucker...

/dev/disk3s2 Apple_HFS

you absolutely have to get the /dev of the "Apple_HFS" slice.

[pyrogen]

Thanks guys!!!!!
Jaybee, thanks for breaking it down for me, thats exactly what I needed.
Merv. nice catch!
I am off to try this out, I will post back.
Thanks agian,
Destin

[pyrogen]

well when I attempt:

% hdiutil create -sectors 2676896 /Users/Shared/pyrogen_backup

i get the message:

hdiutil: create failed - No such file or directory

i believe this is because i have no 'Shared' folder in '/Users'

is it acceptable to:

% hdiutil create -sectors 2676896 /Users/pyrogen_backup

forgoing the 'Shared' folder and creating the disk image in '/Users' ?

OR

Should I create a shared folder in '/Users'?
If so would I use 'mkdir' ?

Thanks,
Destin