What's Wrong With This Shell Script?
 

I'm trying to write a Postflight shell script for Carbon Copy Cloner that will:

1 - Unmount the backup drive

2 - Write the result to the CCC log file

3 - Send e-mail consisting of the last 50 lines of the CCC log file

Here's the script. It's actually one line in the script, but I broke it up here for easier reading:

1 & 2 work fine, but the e-mail never gets sent.

If I run the shell script from the command line, it works fine: It unmounts the drive, writes to the CCC log file and sends e-mail.

But, when run as a CCC Postflight shell script, the last part always fails.

Any ideas?

Try /usr/bin/mail at the end.

Thanks, but that didn't work. Here's a fragment of the CCC log. You can see that the Posflight script has been run and that the unmount command was successful. But, the mail is never sent:

Try this:

Thanks for the suggestion. I tried this:

(Again, the third line of the script is one long list run asynchronously.)

If I run this as a CCC Postflight shell script the log /var/log/myscript.log is completely empty, nothing in it at all.

If I run this script from the command line (as root) here's what's in the log:

Just what you'd expect if the script ran through to the end.

So, there's something about how this script runs as a CCC Postflight script that simply prevents some stuff from happening.

I don't get it.

Perhaps that's your problem. What happens if you run it synchronously? I don't see any need for an asynchronous routine here, and 2 minutes is way longer than needed to let a log process finish.

If I run it synchronously then it unmounts the drive before CCC is finished. This causes CCC to end with an error and a dialog box left on the screen. The author of CCC suggests the following:

&

and

I just want to combine these into one script. If I put them in separate lists, then the second list runs before the first one has had a chance to write the results of the unmount command to the CCC.log. The "sleep" time doesn't seem to matter. Even if I put 'sleep 120' the second list executes before the first one is finished.

The parentheses () cause the commands to be run in a sub-shell.
I think what you want is to put all of the commands inside one set of parentheses.

Thanks hayne. But, I've tried all in one set of parenthesis. If you look at the CODE in the OP, you'll see that it's all in one line. But, if I do that, the 'mail' command never gets executed if the script is run as a CCC postflight script.

But, if I run the exact same script from the command line, then the mail command does get executed.

Start by using full paths for all executables and files referred to in the script.
E.g. "/usr/bin/tail" instead of "tail"

We've been there. mail is the issue, not tail. mail will still deliver an empty message. Still, full paths remove the question.

What happens when you replace the mail command with a file write? This will say once and for all if it's mail or something else that's misbehaving.

..... | /usr/bin/tail -n 9 > /var/log/myoutput.txt

I haven't been following this thread, but upon reading it quickly I don't see that it has been established that 'mail' is running at all. In fact an earlier post (with the 'echo'ing to the log) seemed to indicate that the script wasn't running at all.

Exactly. At the first sign of trouble with a script, you should do two things:
- use full paths for everything
- insert logging statements (writing to a file) after each command

OK. I just combined the script into one long list to be run asynchronously. I ran it from the command line. I got e-mail with the last 50 lines of CCC.log. Everything works exactly right if I run the script from the command line like this:

But, when I run it as a CCC Posflight script the only part that doesn't work is the mail command:

The 'unmount' command works!

The write to the CCC.log file works!

He says, up in #1, that the script runs fine from the command line. So either something is killing the background job (difficult) or it's failing before the final step.

mnewman: something else to throw in. Replace the `tail` part with a simple echo command to test the mail function.

After re-reading #5 and #7 I'm thinking it's still a timing error that will be solved with synchronous coding. Waiting x seconds for an unrelated process to stop is just a lot of guesswork, we should have a way to test, definitively, when it's ready to go. Mike Bombich knows his stuff - there must be a better solution than 'wait a while and hope'.

The script might run fine from the command line but fail miserably in some other environment (like that provided by CCC) if the script makes any assumptions at all about its environment. A common difference is that the execution PATH is different from that used on the command line. And aliases etc are not operative - since the shell set up files (dot files) are (often) not used in other environments.

OK. I've put full paths for all of the commands. I have replaced 'mail' with a simple 'tail' that writes the last 50 lines of the CCC.log file to an now-empty log file. It works from the command line. I'll try it now with CCC.

Here's the one line list of commands:

My thoughts exactly- that's the usual cause but doesn't seem to be the main culprit here. Hence full paths and some script step logging.

I tried this:

The first two 'echo' statements ($USER and date) never made it into /var/log/myscript.log

The order of the other echo statements was:

1 - beforemail + date

2 - aftermail + date

3 - duringmail + date

But, if I ran the above from the command line, all of the echos made it into the log file.

OK, ran the above as a CCC Postflight script. Everything worked except the write to /var/log/myscript.log.

This is my old CCC Postflight script:

It always worked just fine. So, we know that the 'mail' command can work when a script is run as a CCC Postflight script.

I understand that you've been running this script from the command-line as 'root'.
Are you sure that the CCC environment runs the script as 'root' ? If not, permission problems are an obvious thing to look at.

I've tried running it both as "me" and as root. Should I try running it as some other user?

I don't really think it'll help, but I would try the code without the trailing & as that seems to be the only difference between the older working and the current non-working code.

The problem isn't in mail directly as you've demonstrated merely tailing the contents into a file doesn't work.

The other difference is when you run the code manually, there is a controlling terminal. When run via Postflight, there isn't. Maybe a nohup is needed.

This has to be run with the trailing & because it has to run asynchronously. Otherwise the disk gets unmounted before CCC is finished causing CCC to exit with an error.

Where would the 'nohup' go?

It Finally Works
 

Here's the version that works:

The only thing I can see that is different from the versions that didn't work is the length of the second 'sleep' command; ten seconds instead of 60 or 120.

Oddly, although this script works, it fails to unmount the disk:

The only thing the & is doing in this case is making the shell script return immediately while everything else happens in the background synchronously. I've broken the code down to be more readable:

(
sleep 60;
diskutil unmount 02352EBA-9C0A-3992-AA28-BBCBC7A29367 >> /library/logs/CCC.log 2>&1;
sleep 10;
tail -n 25 /library/logs/CCC.log | mail -s "CCC" root
) &

The & is clearly outside the () and is just putting everything inside the () in the background. The first thing happening is the program sleeps for 60 seconds for no apparent reason. AFTER sleeping, it unmounts the drive putting both stdout and stderr into your log file. I would really expect diskutil to NOT return until it has finished, but a short sleep can't hurt.

You said this worked, but failed to unmount the drive. Did you have anything else open on the drive when you ran the script?.

When you were testing it before, were you waiting 3 minutes for all the sleeps to expire before deciding it hadn't worked?

The nohup would go at the top of script right below the opening line, but I don't think now that is your problem.

Have a look here for the reason: Mounting a volume before a scheduled task and unmounting a volume after the scheduled task

Nothing.

Yes.

CCC ran as a scheduled task last night. From the log file you can see that the postflight script ran. But, it neither unmounted the drive nor sent e-mail. This is driving me nuts:

Why not simply eliminate the problem entirely? Skip the CCC postflight step and do the entire job as a regular shell script. CCC is little more than a wrapper around a couple of commands like hdiutil and rsync - duplicating it's functionality in a shell script is very easy.

I wouldn't have the slightest idea where to begin.

If I don't even have the brains to get a one line shell script to run I can't imagine doing something as complex as cloning a drive. Remember, this is the "Unix Newcomers" subforum…..

Cloning a drive is easy! And you've already worked out some essential concepts.

But lets back up a bit - what is your primary goal? A bootable backup or a restorable image?

Is this on a personal machine or on a server?

This is on our main personal machine. My goal is a bootable backup. I run CCC automatically once a week. I want to be notified via e-mail that CCC ran. I want to have the backup drive mount only on demand and unmount when CCC is done.

I want the backup drives (CCC and Time Machine) to be unmounted when not in use because it makes a significant improvement in the speed of loading file dialog boxes.

The bootable backup portion of CCC will do something like this:
(note carefully the spacing around the single / )

rzync -aqAXx --del --log-file=/var/log/myrsync.log / /Volumes/backupVolume

As for the mount/unmount part, Bombich does some fancy things to hide the process from the Finder - no need to duplicate.

To mount an unmounted volume we need to find it's device name, diskutil plus some string massaging:

Everything else can be run syncrounously entirely in the background. Get Lingon to create an auto-launch background job.

AMO - Thanks for this. I will have a look and see what I can do. Right now the monitor for that machine (a MacMini) is in for repairs and I'm doing everything via VNC from an ancient iMac. When I get the monitor back I'll give it a shot.

I'm still curious as to why the script in the OP doesn't work as expected.

Why don't you enable remote login on the Mini and use Terminal and ssh? Much faster than VNC to the other desktop then using Terminal there.

Good idea. I've got SSH enabled with public key authentication.

The long standing way to make sure disk io was completed in the past was to issue "sync; sync; sync", not sleeping and hoping it magically got done. I'd be hard pressed to believe that no longer works on a system that is "certified unix".

I tried this:

Both e-mail messages got sent, but the drive still failed to unmount:

I thought the unmount failure might be because mds was trying to index the clone, so I turned indexing off. Obviously, that didn't work. Neither did the sync; sync; sync…

There can be nothing using this volume as it is not mounted until just before CCC runs. No other process uses the volume.

I remain baffled.

You might try using 'lsof' to check for open files on that volume.

Here it is:

This is too weird. If I put 'sleep 60' as follows:

unmount FAILS
mail SUCCEEDS

If I put 'sleep 120' as follows:

unmount SUCCEEDS
mail FAILS

Here's what was happening with the volume (lsof) just before the 'sleep' command:

Anything there that would keep the volume from unmounting?

I think any open file (or folder) will keep the volume from unmounting. Each of the lines you showed from 'lsof' indicates an open file ("REG") or folder ("DIR").
The first entry in each line is the program that has opened the file/folder.
I assume that "Path\x20F" is Path Finder.

You're right. It is PathFinder. But, I never have trouble manually unmounting the volume, even if PathFinder has it displayed in the sidebar under 'Devices'.

I may have overcome, if not solved the problem. I tried setting the 'sleep' time to 90 as follows:

And, the volume was unmounted and the e-mail went out. I guess 60 was too short and 120 too long. I have no idea why. How could the extra 30 seconds cause the final command in the list to fail?

From the CCC.log fragment that was the body of the e-mail:

I guess it will forever remain a mystery. Unless, of course, it fails again tomorrow…

What is update_dy?
 

Which it did. This morning the Volume failed to unmount. I did an lsof of the volume immediately after the sleep command:

and it is absolutely full of stuff like this:

What is update_dy? I can't find a reference to it.

Thanks hayne. I tried both 'man' (without the switch) and 'which' on 'update_dy' and, of course, came up with nothing.

I'm assuming that this is running on the newly blessed clone and that the volume cannot unmount until this is done?