let me illustrate better:

We have a big directory(240GB) called proj. Within /proj there are many directories around 20.

/proj
/dir1
/dir2
/dir3
and so on.....

When I use tar it gives me a single proj.tar.gz(180GB) file at the destination external HD.

When I use iBackup I get:

/dir1.zip
/dir2.zip
/dir3.zip
and so on......

This makes it easier to retrieve the information.
Thanks fracai!!

One thing I don't think you have mentioned. Is this a dual core/processor machine ?
If so it could (no guarantee ) be faster running it as several jobs. If so how many cores ?

That will compress each directory found in "proj/" and put the compressed files at "backup/proj/"

The way the script is written now you'd have to execute the script from within the directory above "proj". You should probably modify the script to use absolute paths for both the source and destination values.

i wasn't able to find how many cores. This is the info. I have:

Mac OS X Server 10.4.6
2Ghz Power PC XServe G5
2 GB DDR SDRAM
machine model:RackMac3.1
machine name: XserveG5
CPU type: PowerPC G5 (3.0)
Number of CPUs: 1
CPU Speed: 2Ghz
L2 Cache (per CPU): 512KB
Memory: 2GB
Bus Speed: 1GHz
Boot ROM Version: 5.17f2

I have another concern. I know I'm able to view the contents of the tar.gz file, but this takes forever. Is there an application like Winrar that would enable me to view the contents of this tar.gz file? I just want to verify the folders inside the proj/.
Thanks..

Hmm, okay thanks for clearing that up. I was under the impression that -E did all that by preserving all the extended attributes... and you are right.

There is also an option for -z which does in fact say compress data. I am not sure how it compresses data though. You may want to look into it

To find core:s info try (in terminal)

And you'll get back sothing like:

Hardware:

Hardware Overview:

Model Name: MacBook
Model Identifier: MacBook2,1
Processor Name: Intel Core 2 Duo
Processor Speed: 2.16 GHz
Number Of Processors: 1
Total Number Of Cores: 2
L2 Cache (per processor): 4 MB
Memory: 2 GB
Bus Speed: 667 MHz
Boot ROM Version: MB21.00A5.B07
SMC Version: 1.17f0
Serial Number: W87222FBYA8
Sudden Motion Sensor:
State: Enabled


And if it doesnt say anything about cores then you have a single core system.

tar -czf tarball.tgz files is basically the same as tar -cf - files | gzip > tarball.tar.gz

This is the monster 200Gb tarfile? tar stores it's index throughout the file, it's going to take a while to scan through that much data, uncompressing internally as it goes. Be happy you're not dumping directly to tape, the original use for tar (Tape ARchiver)

I ran the command:
system_profiler SPHardwareDataType
and got:

Hardware Overview:

Machine Name: Xserve G5
Machine Model: RackMac3,1
CPU Type: PowerPC G5 (3.0)
Number Of CPUs: 1
CPU Speed: 2 GHz
L2 Cache (per CPU): 512 KB
Memory: 2 GB
Bus Speed: 1 GHz
Boot ROM Version: 5.1.7f2
Serial Number: QP53901KSLX

So it must be a single core system. acme.mail.order I think tlarkin is refering to -z with rsync NOT tar. thanks for all the help!

Note the item that specifies "during transfer". rsync will compress the data while transferring the data and uncompress it at the destination. This is useful if your computer and the destination computer are fast, your data is easily compressible, and the network is slow. You'll get an overall speed boost by pre-compressing the data, transferring the smaller amount of data, and decompressing on the other side.

rsync will not create a compressed archive for you, however nice that feature would be.

can we pipe it to tar?

rsync <whateverfile> | tar <whateverfile>

I don't think it would work because the destination HD is smaller than the source. It would be great if rsync could do such thing! Can cpio do this? iBackup uses cpio and it zips all the directories as I metioned before. thanks!

rsync doesn't pipe its output so this wouldn't work.

cpio, zip, or even tar can pipe their output, but the main benefit of rsync in making backups smaller is the ability to reference a backup that is already in place and only backup changed data (ie. the backup is incremental).

the bottom line is:
compressing large media files is slow and not typically not very beneficial.
the only way to fit a large backup in a small space is compression.
incremental backups are faster than full backups.
large drives are cheap.

The comment about the number of cores you have is correct, running parallel jobs could be faster, but with the media files you're compressing I'm not sure how much benefit you'd see. You might even just be limited by disk speed at that point. You can check your processor strain by opening "Activity Monitor" while running your backup script. If you have more than 1 core or CPU you'll see more than 1 graph charting your CPU usage. If your usage is maxed out already, parallel jobs won't provide a benefit. If each chart is around 50% you could look into running parallel jobs in your script.