Easy Backup Script?

[brownpanda]

I am having some trouble "backing up" some of my files. Let me explain. I use a Mac running OSX 10.6.8. I use TimeMachine to backup the entire system, so I'm covered in terms of overall backup. What I want to do is to save (not really backup, since I will be doing this irregularly, as and when I need to) my User folder into an external hard-drive. This User folder is the one you find when you click on your Mac's internal harddrive and then click on Users. I want to save the one with my name on it.

My main objective is to save all my own files that I have accumulated over the years. There's about 350GB in my User folder. Some of them are applications, system folders and stuff I don't really need. They are often useless once they are taken off the system, anyway. I don't really care if these "useless" files are also saved since the memory used in total is manageable.

Problem 1:
When I try to simply pull the folder into the external drive, it hits a snag and some of the files won't copy over. The error message is often something like "The folder xxxx already exists.". When I then open up the folders and copy stuff over manually, I find that I can do that if I go in deep enough. Then I have to go back folder layer by layer and copy the files in each folder. I figured that the problem is path length. As someone explained it to me: the recipient file system (i.e. the formatting of the external hard disk) can't support the depth of the folder structure. As a rule of thumb most file systems can't handle the full path to a file being more than 255 characters.

Problem 2:
I really want the external harddrive to be readable by most computers (Macs, Windows and their various versions). I travel quite a bit and some of the computers I have access to are quite old. My home computer, for example, runs on Windows XP. So, I would prefer to keep my harddrive formatted to FAT32, as this is the most widely readable format. I don't have large files, everything is less than 2 GB, so I don't really need to use the new formats.

Problem 3:
Most of the backup programs I've looked at are not easily accessed directly. Programs like CrashPlan compact the files and I need to have web access to restore and read the files. I want a simple "backup" which just copies the files into the harddrive. Preferrably, byte-to-byte but if additional info is added that bloats up the files a little, I don't mind that either.

So, from my point of view (being a computer illiterate, just a heavy user), what I need to do seems so simple - copy one BIG folder into the harddisk. I don't need to schedule this or anything, just do a copy when I need to. If possible, I'd like it to update old files but not delete what was there - in other words, I am not looking to sync the two folders, just backup. I do save my own old versions so I don't need all the fancy stuff that TimeMachine can do (plus I am using TimeMachine anyway to do that).

Can anyone help me do this?

I'm asking this question here because I'm thinking that a simple Applescript might do this for me. The steps would be as follows:

Open User Folder in Mac
Open destination User Folder in external drive
Count the number of Subfolders
Create similar subfolders in destination
Copy remaining loose files over

For each of the Subfolders
Open Subfolder
Count the number of subfolders inside
Create similar subfolders in destination
Copy loose files over
[For each of the subfolders inside,
open the subfolder
count
create
copy
[For the next layer of subfolders
.....]
]

Next Subfolder

The idea is to actually open the folders in the destination drive so that the Copy command does not have a long pathname. Hopefully, this will bypass the 255 pathlength problem. I don't know how to write programs so I don't know if this is doable or not. If so, can someone write me a script for this? If not, can you explain what's the problem?

Many thanks.
PS: Just to be greedy - is there a nice copy command that won't add to the copied file but just copy it over byte for byte?

[hayne]

Easiest thing to do would be to just delete the existing folder on the destination drive (external drive) at the start of the script. Then a simple copy will work fine.

If you want to be safer (and you have enough space for two copies of the folder during the time the copying is being done), you could rename the existing folder (e.g. to xxx_old) and then delete it at the end (when the copy has been done successfully).

[cpragman]

rsync ought to handle this.

Another option (not exactly what you asked for) is to write a backup script using tar. You can "rotate" backup files by renaming them every day, the same way the system does with log files. My sample is based on the open source "hfstar" that handled extended attributes. It is a few years old. I think apple has added extended attributes to tar since, sou you can probably use the regular "tar" nowadays. Since you would be compressing the files into an archive, long pathnames wouldn't matter.

Attached is a sample I used to use.

Quote: PATH=/bin:/sbin:/usr/sbin:/usr/bin:/usr/libexec export PATH host=`hostname -s` echo "Subject: $host Daily Backup of User Documents" # Find recent files, use xargs to back them up. This finds files modified recently. # Other find approaches are equally possible. find -x /Users/*/Documents -mtime 1 -and ! -type d -print0 > DAILYBACKUPFILES #append file list from second source location to the first file list find -x /Users/shared/staff/Documents -mtime 1 -and ! -type d -print0 >> DAILYBACKUPFILES /usr/local/bin/hfstar -cvf /users/shared/staff/backups/DailyBackup DAILYBACKUPFILES cat DAILYBACKUPFILES | xargs -0 /usr/local/bin/hfstar -rvf /users/shared/staff/backups/DailyBackup #rm DAILYBACKUPFILES echo -n "Rotating Backup files:" cd /users/shared/staff/backups for i in DailyBackup; do if [ -f "${i}" ]; then echo -n " $i" if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi if [ -f "${i}.9${gzext}" ]; then mv -f "${i}.9${gzext}" "${i}.10${gzext}"; fi if [ -f "${i}.8${gzext}" ]; then mv -f "${i}.8${gzext}" "${i}.9${gzext}"; fi if [ -f "${i}.7${gzext}" ]; then mv -f "${i}.7${gzext}" "${i}.8${gzext}"; fi if [ -f "${i}.6${gzext}" ]; then mv -f "${i}.6${gzext}" "${i}.7${gzext}"; fi if [ -f "${i}.5${gzext}" ]; then mv -f "${i}.5${gzext}" "${i}.6${gzext}"; fi if [ -f "${i}.4${gzext}" ]; then mv -f "${i}.4${gzext}" "${i}.5${gzext}"; fi if [ -f "${i}.3${gzext}" ]; then mv -f "${i}.3${gzext}" "${i}.4${gzext}"; fi if [ -f "${i}.2${gzext}" ]; then mv -f "${i}.2${gzext}" "${i}.3${gzext}"; fi if [ -f "${i}.1${gzext}" ]; then mv -f "${i}.1${gzext}" "${i}.2${gzext}"; fi if [ -f "${i}.0${gzext}" ]; then mv -f "${i}.0${gzext}" "${i}.1${gzext}"; fi if [ -f "${i}" ]; then mv -f "${i}" "${i}.0" && if [ -x /usr/bin/gzip ]; then gzip -9 "${i}.0"; fi; fi touch "${i}" && chmod 640 "${i}" && chown root:admin "${i}" fi done

[ganbustein]

Your problem is that some of the items in your home folder have an ACL (Access Control List) that specifies:

group:everyone deny delete

There's one of these on your home folder itself, but Finder kindly deletes it from the copy. Unfortunately, it does not delete it from the subfolders. Since it can't delete the subfolders, it can't replace them either.

The easiest fix was suggested by Hayne. If you have enough space for two copies, rename the previous copy before making a new one. Delete the previous copy afterwards if you don't still need it. Less safe but using less disk space would be to delete the previous copy before making the new one. You may have to hold down the option key while emptying the Trash.

Or, enter the Terminal command:

chmod -RN /path/to/previous/copy

(which deletes all ACLs at all levels within the copy) and then just drag the folder over. Finder will ask if you want to replace the existing folder, and will no longer have any trouble doing so if you consent.

I doubt that it's a path length problem. The default maximum path length in OS X is 1024 Unicode characters, as you can learn from the Terminal command:

getconf PATH_MAX /

The 255-character limit is the maximum length of a single component of a path. (That is, each file name, and each directory name along the path, is limited to 255 Unicode characters.) You can see that with:

getconf NAME_MAX /

On MtLion (and probably Lion as well), instead of refusing to copy Finder asks for an admin authentication, which allows it to just blast through permissions problems.

[benwiggy]

Quote: Originally Posted by brownpanda There's about 350GB in my User folder. Some of them are applications, system folders and stuff I don't really need. They are often useless once they are taken off the system, anyway. I don't really care if these "useless" files are also saved since the memory used in total is manageable.

Unless you have specific multi-user requirements, applications should be in the top-level /Applications folder, and not inside the User folder. Similarly, by default, the only "system folders" in your user folder should be the Library folder, which stores preferences, settings and application data specific to your user account.
It should be easy to exclude the ~/Library folder from the copy, if you just want copies of your files.

Quote: Originally Posted by brownpanda I really want the external harddrive to be readable by most computers (Macs, Windows and their various versions). I travel quite a bit and some of the computers I have access to are quite old. My home computer, for example, runs on Windows XP. So, I would prefer to keep my harddrive formatted to FAT32, as this is the most widely readable format. I don't have large files, everything is less than 2 GB, so I don't really need to use the new formats.

Bear in mind that some OS X file data does not travel well across other filing systems. It's mostly metadata that may get lost, but it's something to be aware of.

[anthlover]

Crash plan + does indeed offer web based restore and access from mobile too. It is a good part of a multi pronged approach.

You have not really told us enough about your macs and back up drives and how your using them with time machine, etc. (hope your not using time machine wirelessly or for that matter over network with their sparse bundles).

Both Carbon Copy Cloner and Super Duper can do what you want. Offer Scheduling and let you choose folders. They also of course do bootable back ups which is a very good idea.

I should point at also that crash plan itself is able to do some of the things you have in mind also it can back up to other computers and NAS as well as to crash plan central (off site encrypted back ups).

Note that EXFAT can store files larger then 4GB and with the appropriate update XP can read/write EXFAT. Windows 7 has native support.

I would not necessarily take "your back up mobile" risking its loss. If you need access between systems that can easily be done cross platform via your local network. PCs can map your Mac drive and vs. versa. If there is a subset of your data you want to travel with a USB stick or Dropbox might be a better choice.