Time Machine Backup failure?
 

Did my weekly Time Machine back up last night and got this warning:

This backup is too large for the backup disk. The backup requires 605.35 GB but only 550.00 GB are available.

FYI, the disc in question is a 1TB disk, so the disk itself is large enough; perhaps not for the grand total, but for some percentage OF the total...I'm under the impression that TM backs up what it can, sacrificing older backups to make room for the new backed-up files...I did get such warning last week, but now this one, so I guess I don't know what I'm meant to do...

Do I need to manually chuck stuff? Am I right to expect that TM will continue to sacrifice the old to make room for the new files?

thanks for any clarification on this,

a

Hmm. Are you expecting an Incremental backup to be that big? i.e., have you actually written or modified 600 Gigs-worth of data in the last hour?
How much free space is reported in the Finder -- 550 Gb?

Look at the TM log messages in Console, which are left every hour and are quite chatty.

I would be wary about manually removing stuff. Don't do it in the Finder -- there is a "delete archive" contextual menu in the Time Machine app when you click on a file. That deletes all versions of it, IIRC.

Assuming TM has got its knickers in a twist, and assuming that there's nothing on the disk that you've deleted from your internal and may want to restore later, then it might be worth wiping the disk and starting your backup again.

I spoze my expectations are: TM doing its business, as it's been doing for me for several months, and last week, reporting that it tossed some older stuff to make room for the newer; to me, that makes sense. Finder reports the volume in question has 551.8GB free

The only TM-related entries I see in console are: "12/15/12 8:49:30.271 AM iPhoto[249]: This process is attempting to exclude an item from Time Machine by path without administrator privileges. This is not supported." Nothing TM-related from last evening when I did this backup (or tried to).

I guess I am assuming some twisted knickers...and I guess in the back up my mind, wiping the volume and doing a fresh TM backup had crossed my mind.

Last week and this week are for me the first experiences watching what TM does as it contends with a "full" volume, and decisions it's been programmed to make.

I did do a lot of housekeeping, this past week which included taking on new files, and trashing files I no longer needed, or duplicates thereof, from my Boot Volume/Documents, which TM has been instructed to back up.

So, lots of moving around. Some new files in terms of GB size. perhaps too many changes for TM to resolve?

I guess I'm saying that I accept that, when backing to a disc of finite size (they all are!) I expect TM to get to the point of being told to back up a volume, coming up short, then deciding what needs to go, and that those decisions are based upon what's older, and what's newer.

Sort of like a hot air balloon losing altitude..something has to go over the side, and I'm good with that. I don't see how a human would do any better, confronted with the same choices.

TM appears to me, NOT to have done that last night. Is this a reasonable theory for me to come up with? Do I need to accept my file management scrapes and bruises, wipe the volume, and begin anew with a fresh TM backup?

THank you,

a

One good tip would be to allow TM to work as it is designed. That is, to leave the backup drive connected, incremental backups are done each hour, and, after the initial full backup, would be quite quick (you would likely not notice it working), the space needed would change only slightly, and would be a good choice for what you expect - clearing out the oldest incremental backups when it gets low on space.
If you only connect the backup each week, and then do a large update/change to much of the drive, then you can expect it to need additional space on the backup drive.
So, consider that if you want to use TM as the occasional full backup, then you are going to need a larger backup volume.
TM is supposed to work to delete _increments_ when space is needed. It can't delete an increment if that last "increment" is essentially a full backup, and, in your own example, needing more the half the maximum space on the hard drive.
So, the take-away for you is to allow TM to backup normally. It's not a weekly experience that you have to remember. Just leave your TM volume connected, and this should not normally happen.

DeltaMac;

I can fully see the merits of your excellent point. My concern in allowing once-only back ups is to reduce the wear and tear I feel could be caused by the constant backing up. I have googled this and some report that they have experienced drive failure owing to "excessive" backing up.

Perhaps not scientific data, but it is some peoples' either experience or viewpoint.


But your point about too-large last increments being undeletable if those last increments are full backups is crystal clear.

would daily TM backups be better?

than you for reading and weighing in.

a

I think that's the nature of incremental backups.
The backup volume would certainly be accessed more often than a weekly, or even once a day backup.
But, each write-time would be short, and only once per hour. Most times, that activity won't even be noticeable.
And, hard drives, surprisingly, are designed for use as active storage, which would naturally include reading and writing. Even without TM, OS X does a _lot_ of disk activity, and certainly much more than just once an hour.
I think your theory about "excessive" backups is just a red herring. People have drive failures, which can ultimately be from use - each read/write cycle would put one a step closer to hard drive death, but TM use is not the excessive use that some think it is, and certainly not compared to the use that the _rest_ of the OS does naturally to any hard drive.

OK...makes sense...the boot volume certainly IS constantly busy while powered up and not sleeping..

so, with respect to the problem I had last night, is my only action now to wipe and re-TimeMachine that volume?

Or, try, try again with having incremental back ups turned on and seeing how that goes?

thank you!

a

You might like a different product to do occasional back ups use Carbon Copy Cloner or Super Duper. They do fully bootable image based back ups. There both truly excellent and come in free and paid versions.

I currently favor CCC.

You should have more then 1 back up separate physical drives. And it is good idea to also have an online back up crash plan is what of the best. It also keeps track of versioning of your documents and even deleted files. They offer unlimited storage for around $60 a year.

TimeMachineEditor is installed on every machine that i have TM users on !
http://timesoftware.free.fr/timemachineeditor/

I've never quite seen the point of that. From the webpage:
"This is useful if you don’t need to backup every hour and don’t want the performance penalty."

If you haven't saved any files, then TM isn't going to backup anything, so there's no performance penalty. (And what kind of performance penalty is there on a modern machine when saving to a different drive from the system?)

If you have saved some files, then presumably you want them backed up?

My experience with TM is that if you use it exactly as it was intended and don't tinker with it, it will work well. Otherwise, not so much. (Same goes for SpotLight.)
If TM doesn't work well for your setup, then you need to use something else, rather than modify it.

@ben There has always been a performance penalty and quirks with TM.
It has got a lot better with ML. But I tend to set schedules for TM as it does effect some users workflow. Indeed modern hardware helps now. Many people do not tweak TM selection options maybe they still use entourage or have Outlook 2011.....or other program's that are not always graceful with TM.

I can vouch for TM editor as being completely rock solid way of doing just that. It uses it own scheduling method using launchd and does not mess with TM plists.

Was just throwing it in as an option as we use scheduling when we use TM.

Corporate?
 

Are we veering into a corporate or educational direction with TM?

If so what is the efficacy compared to other products?
e.g. Letting users control there own restore destiny

Also do you network shares? Are we able now to avoid sparse bundle hell and use the attached version of TM backups with a network share.

Did you perchance rename a folder that had large descendants, like media? TM would consider this to be "new" and cheerfully make another copy of it.

And do you use an email program that isn't time machine friendly, like Entourage?

;-) you have got to be off your rocker to use TM with anything more than a few machines and in home situations only. It is also very unstable using in network backup situations and obviously does not scale.

So not veering away from Home users too much. My key issue with TM is when it goes bad, it goes very bad and you pretty much have to wipe the drive holding backup.....and as such getting rid of your backup of files that potentially will not be on your main hard disk.

My other main issue is the backup trimming.....what gets trimmed and when ? I am sure it has a clever algorithm but it is a bit too set, forget and hope that it does not trim something you may need one day.

Crashplan - Is my chosen tool for incremental data backups across the board (Mac, Windows, Linux). The data de-duplication is one of its strengths.
I also use full disk clone weekly for quick restores of machine, or in business environments a mixture of imaging and data restores.

Dont get me wrong i think TM is useful and is certainly better than no backups, i just do not use it as i have been burnt many times with it.

I can't speak to non-home use or scaling, but I've had a very good experience backing up over the network with multiple machines to one NAS.

Not recently, but in the past I did have a backup go bad a few times, but at the time my backups were on a Drobo and I put more blame there than I do on TimeMachine itself. I also fairly quickly decided I didn't care if I had to restart the backups. For one, I've always backed up to a sparse bundle DMG, so it's just a matter of replacing the bad backup image with the empty image, though I'd think it'd be the same procedure of just deleting the Backups.backupd folder for a local disk. Indeed, it was a pain to redo the initial backup.

And why would you be "getting rid of files that potentially will not be on your main hard disk"? Are you say that you would run TimeMachine and then delete the files from your drive? That's not a backup. Or do you just mean that you're losing the backup when you'd need to restore data from that backup? That's a risk and is why you should use as many backup strategies as is appropriate for your data.

The algorithm is fairly simple. For the past 24 hours, keep every backup. For the past week, keep one per day, anything older than that keeps only one per week. When there isn't enough room to make a new backup, TimeMachine starts deleting the oldest backups until there's space for the new backup.

Is de-dupe really a strength though? How much duplication do you really have on your disks? Also, keep in mind that de-dupe only occurs per machine, so if you have the same files on multiple machines (say, a music library) Crashplan is going to backup those files on each machine. Unless you're talking about the incremental backup behavior where changing the header of a movie is only going to backup what changed in the header. That's a nice feature.

There's also the problem of restoring from Crashplan. With TimeMachine, your data is already in house. With Crashplan, you're limited by your bandwidth or receiving up to 1TB of data in the mail. It's a trade off, and I'm happy to have in-house and remote backups.

I am really speaking from extended experience in a multitude of environments, equipment and software (backup solutions...i have used far to many).
I am certainly not saying do not use TM ! just my reservations.

Overall I have bullet proof backup systems in place at my home/office and all my client sites. I will not be reaching for TM that much regardless....that is not to say it is not in use ;-) It actually does a pretty good job of backing up OS X Server machines since ML which in the past has been a PITA to keep everything backed up correctly.

The NAS backup route works OK until the .sparsebundle goes bad ;-) not just limited to Drobos seen it on NAS too, but Diskwarrior fixed a few of the backups but a few were dead the water.

I have had countless run ins with TM over the years and LOTS of data loss and quirks.
Hard to go back to something that has burnt me & users so many times.
ie. Backed up using TM and removing content from main hard disk, TM goes bad....files lost or even worse trimmed without them knowing. This is users of course left to their own devices ;-) not me.

Also had quite a few failed and/or slow restores and sometime Apple shoddy system updates have led to many issues too ! How good is a backup if it does not work for 4 weeks due to Apple breaking it ?

Yes its "killer" feature, once people wrap their head round it, is the GUI restore interface.

I personally do not have many duplicated files but deal with people that do.
ie. Graphics people who have Job folders which have files that are used in other job folders etc.....maybe only 50GB of duplicated data but all in all it all adds up as they are often changing large INDD files, EPS, PDFs etc.

Crashplan can do Local backups/simultaneous multiple disk backups (thats what we mainly do), machine to machine backups ie. network (use this lots), cloud backups etc. Has many clever options and uses high level encryption for any data that goes offsite. Our key reason for using this is the fact it is fully cross platform.

Anyway different strokes for different folks.

And of course Crashplan is a paid service, TM is Freeeeeee !
although Crashplan's free version does an OK job as well just missing some features.

And its only drawback is that it uses Java ;-) which is a dirty word to some.

O I do not trust TM at all wirelessly. I went the whole wireless sparse bundle to hell routine.
Local TM is probably fine since no sparse bundle.
Unless something has changed i do not TM should use sparse bundles ever.

I use CCC or Superduper for local and Crashplan for off site. It is amazing. The free version of crash does work locally, and I am sure to a NAS. Java or not crash Rocks.

This describes my recent TM woes to a T. Apple Sr. Advisors were clue-free as to what the heck happened. A Mystery of the Universe, seems to be.

when it comes to data backin' up, I don't care much for mystery and suspense. I like boring, every day, meat n potatoes predictability. No hairpin turns/bridge out. No mummies in the closets. No seltzer water in the face.

Thinking about my and others' experience, I'm getting a picture that TM is great until it isn't and then it stinks.

I will be re-considering my back-up strategy, quite possibly going back to CCC.

Thanks to everyone who weighed in on this.

a

@acme Your comments brought a smile to my face ;-)

There is no mystery Apple have butchered a perfectly working mechanism like rsnapshot to shoehorn a backup system into OS ;-)

My basic strategy for my devices is full disk Clones weekly with daily incremental of my data. Also offsite and cloud strategy for good measure.

Agreed
 

I think if Apple is not going to drop sparse bundles then they should drop wireless. Have the TMs and Time Capsule local. Then it is at least good, except for the rolling over the data thing but at least the data integrity is there.

I'm at a loss as to what "rsnapshot" is..

a

rsnapshot is a rock solid UNIX utility for filesystem snapshots & for making backups of local and remote systems.
http://www.rsnapshot.org

I have used this on Linux/UNIX based system for years, it is excellent.

I am pretty sure this is what Apple has used for the backend of TM and no doubt extra tinkering.

I'm fairly certain that TM is not using rsync or rsnapshot as a backend.

For one, they use the Spotlight index to keep track of which files need to be added to the backup (really, which directories need to be scanned for changes), while rsync always scans everything.

Two, rsync will copy only changed blocks, while TimeMachine always sends a full copy of the file.

Three, rsnapshot has an entirely different method for maintaining backup intervals. Rsnapshot explicitly names things as hourly.0, hourly.1, daily.3, weekly.4, and then renames the backups as they need to be rotated. TimeMachine only uses the date of the backup for the name, deletes expired backups, and never does any renaming.

Finally, rsync has always been GPL licensed and newer versions are GPL v3. At the least, the source would have to be available if backupd (the TimeMachine process) was based on rsync or rsnapshot and for later versions it would explicitly be disallowed from being part of the OS. As far as I'm aware, the TimeMachine source code has not been released.

TimeMachine is 100% Apple.

That said, rsync and rsnapshot are solid tools that I use every day. I even exclude my email backup folder from TimeMachine and use rsync (and zfs snapshots, previously rsnapshot) to maintain a backup history.

thx my learned Friend ;-)......Was a stab in the dark

I love ZFS too ;-) Most of my storage is going that way next year (RAID-Z2)

fracai;

I can't tell whether a guy like me would benefit from having rsync/rsnapshot on my macs...you know far more about this subject than I do and I get the feeling that rsync and rsnapshot are solid tools for those who really know their beans about backing up, the underlying system ramifications and needs, etc.

Or, is that an unfair interpretation; can they work for a guy like me...basically an artist who wants to back up his data, uses TM and wants a little extra insurance?

Thank you!

a

I think for most carbon copy cloner and super coupled with an offsite program like crashplan is the best way to go. TM is there to be a built in solution. Back ups and strategies are not a simple tooic and TM makes it simple. I would only say that TM should never be done with anything but local attached storage be uase ofnsoarse bundles. Non local shoould have used file level. And of course apple should finally update their file system.

For extra insurance I'd look at backing up to a different location, preferably offsite.

Right now, I back up the majority of my machines with TimeMachine (to a FreeNAS hosted share, maybe I've been lucky, but I've not had many problems at all with TimeMachine and networkshares [1]). In the past I then backed up a few directories using rsnapshot / rsync when TimeMachine got bogged down by large numbers of files. Now those directories are managed directly on the NAS so I'm not using rsync for anything but on demand file transfer. I also backup my machines to Crashplan for offsite protection.

So, rsync and rsnapshot are very appropriate tools if you have a need for them. In my opinion though, TimeMachine is way easier to setup and for most users rsync won't be necessary. Also, as anthlover suggested, CCC and SuperDuper are likely to be way easier to configure than dealing with rsync or rsnapshot.

[1] Backing up with TimeMachine to FreeNAS has actually been a really good match. First, I'm seeing fewer problems than I did when my TimeMachine backups were on a Drobo. Second, when I've had a backup go bad (once in the past six months), I rolled back the TimeMachine image to a good version (thank you ZFS snapshots) and started backing up again. In the past I'd need to wipe away the backup history and start from scratch. Now I lose just the bad backup and the downtime until it's fixed. I'm also fairly certain that the corruption was due to shutting my laptop while the backup was running. That doesn't always cause problems, but it's not rare.

Remind me not to type responses from my Cell Phone. :)

Should I not have sent that ambulance?

acme - I had very similar issues with TM when I first bought my Mac. My 1TB TM drive would have 600GB free, and the TM run would fail saying that it didn't have enough space. At the outset, TM thought that there was plenty of space (it thought it only had to back up 30GB). But as the backup progressed, the estimate for how much data had to be written kept growing. I spoke to Apple support several times about this, and what we finally discovered was that this problem was being caused by the fact that I was running a virtual machine actively (it was running, not shot down) at the time of the backup. For some reason, TM couldn't initially determine the size of the virtual disk, but later on it did make that determination. But it was too late, because it had already passed the delete step of the process.

My solution was to exclude all virtual machine files from the TM backup. I also run CCC monthly, and they get backed up then. I also backup key data from the virtual machines as needed.

Once I excluded the vm's, TM has been rock solid ever since.

I don't know if you might be running vm's, but perhaps this info may spur your thought process as to what might be happening.

Time machine absolutely should not be used for VM backup or for that matter things like Entourage database. This is where TM falls over.....it does my not handle big data files very well and 1 little change will lead to a completely new copy of file. Totally inefficient. This is why I use CrashPlan.

Nor do I...I do not believe I am, but how would I determine whether or not?

thank you!

a

Acme if you do not know you are running VMs - Virtual Machines then you definitely are not !
ie. using with VMware, Virtialbox, parallels etc. to run other OS such as Windows and Linux.

well good to know, thank you/

In other news, I gave rsync a twirl...it seemed to work fine, except for a pile of
"skipping" various files. I guess I'll need to learn all the commands to include when running it. seems like a worthy tool.

a

Generally the most basic command and command i use most is

Note using sudo as system level files will be skipped if backing up as a User.
Note training slash means copy content into target

ah...that explains that..thank you for sharing commands you use with it.

a

Are there any security risks doing rsynch while sudo is active?

a

No using sudo gives you full access rights to all files/folders. I tend to use rsync for User folder backups or clones of certain folders to network disks etc.

Overall for full disk clones and automation Carbon copy cloner is an amazing reliable tool.