mv command and busy files....?

[chriscaldes]

I'm not sure how to describe this challenge I'm having easily...but here goes. This is in sh

We run a variety of shell scripts here at work that deal with the movement (mv) of files from one folder to another.

With a wait loop - we are 'watching' certain folders, and moving/filing the contenets of them at timed intervals.

Now - for the most part - the scripts work fine, however, if a client is in the process of copying a large file to that 'watched' folder when the script executes, the script will move the file before the copy from the client has been completed- which results in a corrupt file.

So - what we've done to get around this - is to use fstat to determine if the file is busy, or not, and deal with it accordingly.

That works, and works great, but we have like 30 mv statements in the script - and I don't want to have to pass thru an if/then function every time I want the shell to move something.

In any case - like I said - what we have works, but I was wondering if any of you experts had any better, more efficient ways to deal with this issue.

Here's the snippet of code we're using that works..
find /Volumes -type f \! -name ".*" | grep -i .psd | while read FN
do
# Check to see if file has finished copying...
X=`fstat "$FN" | wc -l`
if [ $X -le 1 ]
then
/Developer/Tools/MvMac "$FN" /Volumes/PI_Raidset/

if [ $? == 0 ]
then
printf ""${FN##*/}"\tmoved to zIn_Progress...\t`date +%b\ %d\ %y\ %H\:%M\:%S`\n" >>$LOGPATH
fi
else
printf ""${FN##*/}"\tis still copying....not put moved to zIn_Progress yet...\t`date +%b\ %d\ %y\ %H\:%M\:%S`\n" >>$LOGPATH
fi
done

[breen]

Are the source and target directories on the same partition?

You could hard link new files to the new location without needing to check whether they're busy.

Then come along behind with a cleanup script and delete any files in the original directory that are not busy.

Just a thought.....I haven't actually tried writing a script to do that.

Breen

[chriscaldes]

Yes the source and the target directories for the mv are on the same partition. I'm not sure what you mean by hard-linking them though.

[breen]

Okay, look at this output:

[lab-g4s-Computer:~] labg4% ls -l man.ps
-rw-r--r-- 1 labg4 staff 15494 Sep 17 14:57 man.ps
[lab-g4s-Computer:~] labg4% ln man.ps Desktop/man.ps
[lab-g4s-Computer:~] labg4% ls -l man.ps
-rw-r--r-- 2 labg4 staff 15494 Sep 17 14:57 man.ps

Notice the second field of the long listing -- it tells you how many links the file has. (man ls will give you more details.)

The ln command without the -s switch creates another name for the file. It's not an alias to the file but another directory entry pointing to the same inode, which points to exactly the same disk blocks.

[lab-g4s-Computer:~] labg4% rm man.ps
[lab-g4s-Computer:~] labg4% ls -l Desktop/man.ps
-rw-r--r-- 1 labg4 staff 15494 Sep 17 14:57 Desktop/man.ps

And now I can remove the first link -- and the file is still there on disk with its directory entry in a different location.

Making the hard link and then deleting the first name accomplishes exactly the same thing as moving the file -- but you can do the first step while the file is busy.


Really understanding this requires knowing about how Unix-like systems organize their filesystems. The O'Reilly book on the Linux kernel has a good description of one such filesystem.

[hayne]

1) I presume that you are looking for files ending in ".psd" or ".PSD" in which case you could use:

find /Volumes -type f -name '*.[pP][sS][dD]' | while read FN

(This avoids the need for the grep command)


2) To find out if a file is open, you could use:

fstat FN | grep -q FN

since the status from that command will be 0 if the file is open.


3) You can use "-exec" in a find to execute a command and get its status. The symbols {} in the command are substituted with the name of the file. The subsequent operands in the find only execute if the status was 0. You can reverse this as usual with a \! in front of the -exec


So, putting together the above points 1, 2, & 3, you could change your first line to:

find /Volumes -type f -name '*.[pP][sS][dD]' \! -exec sh -c "fstat {}|grep -q {}" \; -print | while read FN

and then your loop would only be dealing with files that are not open. (At least they were not open at the time the find command ran - it is possible that someone else opened the file in the short interval after the find command ran.)

And you really should use full pathnames for all the executables used in your script. E.g. /usr/bin/fstat instead of just fstat

[hayne]

Quote: Originally posted by breen The ln command without the -s switch creates another name for the file. It's not an alias to the file but another directory entry pointing to the same inode, which points to exactly the same disk blocks. [...] Really understanding this requires knowing about how Unix-like systems organize their filesystems. The O'Reilly book on the Linux kernel has a good description of one such filesystem.

But the HFS+ filesystem (which is used by default on OS X) does not actually have inodes in the traditional UNIX sense. The metadata is stored together with the file contents. Thus hard links in OS X are sort of an emulation.

There is a very interesting article by Wilfredo Sánchez
on the design issues faced when integrating Mac OS and BSD UNIX:
http://www.mit.edu/people/wsanchez/papers/USENIX_2000/

Here's an excerpt about hard links:

Quote: we now emulate hard links by creating a "kernel-level" symbolic link which is visible only to and interpreted by the HFS+ file system. This was necessary due to the lack of support in the volume format. The resulting behavior is very similar to that of hard links when viewed from above the kernel, though they are relatively inefficient in comparison.

[chriscaldes]

You guys are incredible - you just gave me about 500 more ideas for me to make my script better.. THANK YOU!

The whole ln thing has got me thinking... I bet I can use this linking thing for a lot of other stuff.

Hayne - thanks for explaining your thinking on the snippet - I bet if you looked at my whole 300 line script you could turn it into about 5 lines......

For those of you that are interested - we put a large volume of images through our production department that need to be converted to 7 different variations each. So, with applescript controlling Photoshop, the shell controlling file movement and Filemaker keeping track of it all - we've developed a pretty cool little system. Prior to the shell - I was using Applsecript to do it all (yes - that was a nightmare).

So thanks again!

[hayne]

In case you aren't using this yet, it is easy to call AppleScript from a shell script and vice versa.

Apple has a page about using 'do shell script' to invoke UNIX scripts from AppleScript:
http://developer.apple.com/technotes/tn2002/tn2065.html

Calling an AppleScript from a shell script is easily done via /usr/bin/osascript and a "HERE document" as illustrated in the script I contributed to this thread:
http://forums.macosxhints.com/showth...threadid=10149

[breen]

Quote: Originally posted by hayne But the HFS+ filesystem (which is used by default on OS X) does not actually have inodes in the traditional UNIX sense. The metadata is stored together with the file contents. Thus hard links in OS X are sort of an emulation. There is a very interesting article by Wilfredo Sánchez on the design issues faced when integrating Mac OS and BSD UNIX: http://www.mit.edu/people/wsanchez/papers/USENIX_2000/

Thanks for the clarification, Hayne. I've added Wilfredo's aritcle to my reading list.

[robJ]

Quote: Originally posted by hayne In case you aren't using this yet, it is easy to call AppleScript from a shell script and vice versa.

I can do a fair job when it comes to AppleScript but I'm still noobin' my way around the shell. I have an AppleScript script that needs to detect when a file has finished downloading. Can I use fstat to do this? If so, would you please provide a simple shell example that targets one file (I can handle the integration with AppleScript)? I've tried but I don't seem to have the correct syntax because I get nothing but the column headers back.

Quote: USER CMD PID FD INUM MODE SZ|DV R/W MOUNT NAME

Thanks for any light that you might shine on the darkness known as the shell.

-- Rob

[hayne]

Quote: Originally posted by robJ I've tried but I don't seem to have the correct syntax because I get nothing but the column headers back.

Well if you are getting only the headers, that is because the file is not open.
Maybe you want to show us some of your script.

Here's an example:
filename=~/Library/Preferences/com.apple.finder.plist;
fstat $filename

[robJ]

Quote: Originally posted by hayne Well if you are getting only the headers, that is because the file is not open. Maybe you want to show us some of your script. Here's an example: filename=~/Library/Preferences/com.apple.finder.plist; fstat $filename

The reference to headers was from my attempt in the Terminal. For example, I've tried all of the following with the same 'headers only' result:

fstat Desktop/tester.txt
fstat ~/Desktop/tester.txt
fstat /Users/me/Desktop/tester.txt

When checked, the text file was in a open, modified, unsaved state (tested while open in TextEdit and BBEdit).

Thanks for your time,
Rob

[hayne]

Quote: Originally posted by robJ I've tried all of the following with the same 'headers only' result: fstat Desktop/tester.txt fstat ~/Desktop/tester.txt fstat /Users/me/Desktop/tester.txt When checked, the text file was in a open, modified, unsaved state (tested while open in TextEdit and BBEdit).

Well I don't understand what is happening then. What you are doing should work and show you info on the file that is open. I guess you need to do more experiments and see if you can figure out what might be the problem. Read the man page for fstat in case that gives you any clues. And (just to be sure) try using the full path to fstat: /usr/bin/fstat

[robJ]

Quote: Originally posted by hayne Well I don't understand what is happening then. What you are doing should work and show you info on the file that is open. I guess you need to do more experiments and see if you can figure out what might be the problem. Read the man page for fstat in case that gives you any clues. And (just to be sure) try using the full path to fstat: /usr/bin/fstat

Ok, thanks. I was hoping that the shell would come through where AppleScript, or maybe OS X, has fallen short. In past attempts to determine the open/busy status of files in OS X, I have always failed, even though the AppleScript dictionary of the Standard Additions osax indicates that it should be possible by using its 'busy status' property. I guess that an application or process must explicitly flag a file as busy/open. Evidently this isn't a common practice since it appears that a file can be open in more than one application at the same time.

I just did a test with TextEdit and BBEdit by opening the same file in both. I edited and saved in TextEdit and the change was updated live in BBEdit. This (live update) must be a feature of BBEdit because it didn't happen in TextEdit when the file was edited and saved in BBEdit.

I don't think that this was possible prior to OS X and it seems like it isn't a good idea to allow a file to be open in more than one app at a time.

I might be just barking up a dead tree.

Thanks again for your input,
Rob (who didn't intend to hijack the thread)

[chriscaldes]

Robj - I agree, I think different programs treat files different when they are "open".

I am using fstat to determine if a file is being written to, ie: copied, saved, etc... - so they are not 'open'

One thing you could try if fstat isn't working is to do an ls -la on the file every x seconds - then when the size in bytes = the size - the file is finished downloading, (or is stalled, incomplete) Either way it's done I guess.

Seems like a hack - but I suppose you could do something like

while true
do
X=`ls -l ~/Desktop/Untitled.rtf | awk ' { print $5 }'`
sleep 5
Y=`ls -l ~/Desktop/Untitled.rtf | awk ' { print $5 }'`
if { $X == $Y ]
then
echo the file is done
fi
sleep 10
done

I was thinking of doing this - but then I found fstat to work.

Let us know if you find something better.

[robJ]

Quote: Originally posted by chriscaldes One thing you could try if fstat isn't working is to do an ls -la on the file every x seconds - then when the size in bytes = the size - the file is finished downloading, (or is stalled, incomplete) Either way it's done I guess.

Yep, I'm using a similar size check now, but with AppleScript instead of the shell. It works satisfactorily so I guess I should quit chasing the elusive open/busy flag and move on.

Thanks for your input!

-- Rob

[hayne]

I think I now understand what is happening. Some editors (e.g. TextEdit & BBEdit) open the file and copy the file contents into memory and then immediately close the file. I.e. it only looks like the file is open but actually it is not. If you make any changes to the file and then save the file, they will re-open the file and save the changes. Thus it is quite understandable that if you do 'fstat' on the file after opening it in these editors, it will not show as being open.

Not all editors do this - for example, the UNIX 'vi' editor keeps the file open.

I guess you need to rethink why it is that you are asking the question about which files are open.

[robJ]

Quote: Originally posted by hayne I guess you need to rethink why it is that you are asking the question about which files are open.

I have a folder action script that triggers the shell version of Virex whenever new files arrive in my download folder. The script uses a loop to detect that the file size has stopped changing, indicating that the download is complete. This works fine unless there are network problems so I was looking for a different way to determine that it was safe to proceed with the Virex scan. It's really no big deal in this particular script but it seems that there should be an easy way to detect that a file is in use and that it shouldn't be mucked with until closed by whatever opened it. <shrug>

-- Rob

[chriscaldes]

in an update to this thread - I've fund that dring large file transfers, my previous post using FSTAT does not appear to be working properly.

(to recap- the issue I'm having is that a shell is trying to move files before they've finished copying)

fstat "$FN" | wc -l

For some reason - the above code isn't doing what I had hoped it would do - that is - determine if the file is still be copied....It seemed to work initially - but now it is not...

So - I am back to square 1 I guess. I read that article by Wilfredo - but there's not enough detail there for me to try and figure something else out.

At this point - I can live with a loop:

 while true
 do
 X=`ls -l ~/Desktop/Untitled.rtf  | awk ' { print $5 }'`
 sleep 5
 Y=`ls -l ~/Desktop/Untitled.rtf  | awk ' { print $5 }'`
 if { $X == $Y ]
 then
 echo the file is done
 fi
 sleep 10
 done

But I'm not real happy with that - it's too duct-taped if you ask me.

Is there something else I could try that anyone can think of?

[hayne]

What you really want is for the process that is doing the copying to indicate when the copying is finished.
If the copying is always going to be done via the Finder, you could check if the Finder is making the files have a type "bzy " (note the space at the end) as in this MacOSXHints article:
http://www.macosxhints.com/article.p...20920055805577
If so, that should provide an easy solution to your problem.

Otherwise your "duct tape" solution of checking to see if the file size has changed in N seconds seems as good as anything. You could refine this a bit by looking at the modification times etc. See the Perl script in this MacOSXHints article:
http://www.macosxhints.com/article.p...31001073403810