How to copy files and verify? (firewire copying problems)

A Little Peaved! · 06-04-2003, 03:39 PM

Trying to copy large amounts of data (approx. 60GB from 60GB drive) from external firewire drive (source) using Finder to an internal drive (destination).

Copying sometimes fails, other times it appears to complete successfully. Finally, after a few attempts, and copying in smaller portions, it appeared everything was copied sucessfully.

However, I have discovered that many, but not all, of the files are corrupt. I am not sure what the pattern is.

Examples- text files have beginnning or ending replaced with garbage, or, in some cases, what appears to be HTML text from another file. PDF's cannot be opened because they are too corrupted. Meanwhile, many files are jsut fine.

I checked the original files on the source disk, they are fine.

Disk Utilities and other utilites say both disks are "OK", fwiw.

I believe the copying problem is due to the source disk is a flaky, slow, temperamental Maxtor 60GB. Other drives used in the same enclosure work just fine. I think this drive is jinxed. Someone gave it to me for free, I think because of problems. But, it seems to work most of the time. But, the problem is that it is unreliable, as described above.

So, what is a good way to copy files, and ensure that they are verified so that the copies are identical to the originals?

Preferably, this is via a GUI technique rather than unix command.

Also, I heard Retrospect may do this. If so, how does that work?

Of course, a free solution or built-in Mac OS X would be ideal.

Suggestions?

thanx!

yellow · 06-04-2003, 04:22 PM

I agree with your notion that the source disk is flaky.
Retrospect is not cheap. It does do copy and comparision against the original file. However, it's backup software, so using it to 'copy' the data from the source disk will output it all into a single backup file on your internal drive. Then you'd have to restore the file to your local drive. Basically using almost 2x the space on the internal drive.. Also, there's no guarrantee that the files won't be corrupt when restored. Plus Retrospect is NOT cheap.

When this has happened to me in the past, I always go the safe and slow route. Copying a directory at a time to try and get as much good data as possible. Sometimes a little as a few files at a time.. boring yes, tedious yes, but effective.

mervTormel · 06-04-2003, 04:39 PM

i thought the finder did some sort of rudimentary parity checking, and would fail with an error at the first smell of trouble!

what are the results of using something robust like CCC? which uses ditto and rsync (?) commands.

A Little Peaved! · 06-05-2003, 11:45 AM

Retrospect- if it can only "copy" from Retrospect backup, then that is no good here. (Plus, I hear Retrospect is NOT cheap.

)

CCC- what is it?

ditto- ditto, er, what is it?

rsync- don't know that either... in some previous threads, it was mentioned that rsync did not understand macintosh file's resouce fork, which is a huge flaw.

Finder error checking- my intuition is that the Finder just checks if the proper number of bits were copied, the devices reported no errors, and the File Manager is happy.

As I recall, hard drive firmware and driver software contains error detection and verification capabilities. These options were accessible in nitty-gritty detail with software such as FWB Hard Disk Toolkit.

Unfortunately, now with OS X and the state of the Mac software market, this product and others have become part of history and there are no current OS X replacements.

So, the level of error checking in OS X, I believe, is determined by the default settings of hard disk firmware, driver and perhaps Apple Disk Utility (if it touches those settings).

Isn't there something easy, free/cheap and effective?

yellow · 06-05-2003, 12:41 PM

CCC (Carbon Copy Cloner)

Ditto is a CLI command. It's quite nice because it'll copy resource forks. Check out the man page.

A Little Peaved! · 06-05-2003, 03:05 PM

Carbon Copy Cloner does not look like it has very robust copying capabilities (for this purpose). The documentation recommends manually verifying that copying was successful- that's not very encouraging, and it's exactly what we are trying to avoid.

Carbon Copy Cloner uses ditto as its copying engine.

The ditto man is cryptic to me. Is there a friendly guide for ditto? I tried google, but too many wrong hits for "ditto".

This situation has become a bigger problem than I thought, so I am in dire need for a good solution...

Also need a good comparison tool to run, prior to re-copying files. This is necessary both to check for corrupted files, and to idnetify changed/new files to sync everything.

Still looking... (sigh)

tia

A Little Peaved! · 06-16-2003, 08:50 PM

???

grrl_geek · 06-16-2003, 09:03 PM

Well, there's the command "diff". That will compare two directories or files and tell you if they differ (by size or content or whatever)

At the command line, do this:

Code:

diff -qr /path/to/volume1  /path/to/volume2

The "q" means "quick". Otherwise, it will print out the actual textual differences in the files.

The "r" is for recursive.

It will print out something like this:

Code:

[Supernova:~/Documents] grrl% diff -qr test1 test2
Files test1/Test_one.doc and test2/Test_one.doc differ
Files test1/half_puzzle and test2/half_puzzle differ
Only in test2: half_puzzle copy

The only thing I am not sure about is whether this will report information in the resource forks.

A Little Peaved! · 06-17-2003, 11:07 AM

Thanks, grrl_geek!

Quote:


	Originally posted by grrl_geek The only thing I am not sure about is whether this will report information in the resource forks.

Can someone confirm if diff is resource fork-savvy?

There is also "psync" which I found and am investigating further. Does anyone know how well it works?

CCC may be a solution. I am not sure how robust its verification is, though.

Verification and accurate comparison are still big concerns. Also, robustness and understanding Mac file structures (such as resource forks, packages, etc.)

It's frustrating that it is so difficult to find good solutions!

hschickel · 06-17-2003, 11:39 AM

Disk Copy - one of those unsung heroes of the utilities folder. It does just about everthing.

sudo ditto -rsrc /Volumes/source /Volumes/destination - make sure you've unchecked the "Ignore ownership on this volume" box (get info from the finder) for your source and destination if you're transferring OSX system stuff where this is important

EDIT - md5 - run a checksum on the original and the copy. It'll be much faster than running diff or somesuch.

Hugh

djn1 · 06-17-2003, 11:51 AM

Quote:


	Originally posted by A Little Peaved! There is also "psync" which I found and am investigating further. Does anyone know how well it works?

I've been using PsyncX for around six months and have had no problems with it. As best I can tell it doesn't include file verification, but it is straightforward and seems to be a relatively efficient solution.

hschickel · 06-17-2003, 11:55 AM

Quote:


	I believe the copying problem is due to the source disk is a flaky, slow, temperamental Maxtor 60GB. Other drives used in the same enclosure work just fine. I think this drive is jinxed. Someone gave it to me for free, I think because of problems. But, it seems to work most of the time. But, the problem is that it is unreliable, as described above.

Quote:


	It's frustrating that it is so difficult to find good solutions!

I can only assume / hope that this drive will go in the garbage as soon as you get your data off it...

Hugh

A Little Peaved! · 06-17-2003, 07:22 PM

Quote:


	Originally posted by hschickel I can only assume / hope that this drive will go in the garbage as soon as you get your data off it...

Yes, a quick toss in the garbage is too good for this drive- it needs to be thrashed viciously to a slow death, then thrown in the garbage!

A Little Peaved! · 06-17-2003, 07:24 PM

I found diff (I believe), but only as source.

see http://ftp.gnu.org/pub/gnu/diffutils/

Is there a Mac OS X installer for diff?

Mac OS X gui for diff?

tia

hschickel · 06-17-2003, 07:52 PM

Code:

[hschickel:~] hugh% which diff
/usr/bin/diff
[hschickel:~] hugh%
[hschickel:~] hugh% fink list diff
Information about 2627 packages read in 19 seconds.

     algorithm-diff-pm  1.15-1        Perl module, compute differences between two fil...
     diffutils          2.8.1-1       Tools to compare files
     wdiff              0.5-11        Word-based frontend to GNU diff
[hschickel:~] hugh%

My system has diff (maybe from the DevTools?) and you can get diffutils via fink.

Hugh

ps - you really ought to look at md5.

A Little Peaved! · 06-17-2003, 08:54 PM

Sorry, I'm not familiar with fink.
Found http://fink.sourceforge.net/ but I don't understand how to use what's there.

md5, either.

Found rsync_hfs, but it is also source.

http://www.opendarwin.org/cgi-bin/cv...cts/rsync_hfs/

is there a Mac OS X installer or GUI for rsync_hfs, or any other of these?

thx

jbc · 06-17-2003, 09:12 PM

You might look at rsyncx. I think it must be related to rsync hfs somehow; it's a hfs/resource savvy version of rsync. Both seem to reference opendarwin.org. The latest build should be available here:

http://www.macosxlabs.org/rsyncx/rsyncx.html

Haven't come across anyone with any hands-on experience with this app yet. I downloaded it, but have yet to try it. May be worth a look for what you're trying to do.

grrl_geek · 06-18-2003, 02:23 AM

Fink is a (really nifty) project to port/recompile Unix software for the Mac OS X platform. If you want to take a look at it, install FinkCommander, which makes the whole thing way easier.

On the matter at hand, I'll bet diff gets installed with the Developer Tools. I seem to have a copy from Fink, too.

locate diff nabs me two copies:

/usr/bin/diff
/sw/bin/diff

I also noticed some documentation on the various diff commands:

/Library/Documentation/Commands/diffutils/diff_toc.html

Peaved (fyi: it's really hard to not call you Peeves, a la Harry Potter), there should be a CD that came with your Jaguar install called "Developer Tools". The install is about 400 MB, but about 350 of that is documentation, which you can choose not to install.

Hope that helps.

A Little Peaved! · 06-24-2003, 12:54 PM

Thanks for all the info.

Qdea Synchronize! looks like it might be a good solution...

I am surprised no one here has mentioned it. Actually, surprised that it is not more popular in general. People who have used it say very good things about it...

A Little Peaved! · 07-01-2003, 12:33 AM

Also considering Tri-Backup

See also this related thread:
Which BackUp Program is Best or your Favorite?
http://forums.macosxhints.com/showth...?threadid=8348

06-04-2003, 03:39 PM	#1
A Little Peaved! Major Leaguer Join Date: Apr 2002 Posts: 463	How to copy files and verify? (firewire copying problems) Trying to copy large amounts of data (approx. 60GB from 60GB drive) from external firewire drive (source) using Finder to an internal drive (destination). Copying sometimes fails, other times it appears to complete successfully. Finally, after a few attempts, and copying in smaller portions, it appeared everything was copied sucessfully. However, I have discovered that many, but not all, of the files are corrupt. I am not sure what the pattern is. Examples- text files have beginnning or ending replaced with garbage, or, in some cases, what appears to be HTML text from another file. PDF's cannot be opened because they are too corrupted. Meanwhile, many files are jsut fine. I checked the original files on the source disk, they are fine. Disk Utilities and other utilites say both disks are "OK", fwiw. I believe the copying problem is due to the source disk is a flaky, slow, temperamental Maxtor 60GB. Other drives used in the same enclosure work just fine. I think this drive is jinxed. Someone gave it to me for free, I think because of problems. But, it seems to work most of the time. But, the problem is that it is unreliable, as described above. So, what is a good way to copy files, and ensure that they are verified so that the copies are identical to the originals? Preferably, this is via a GUI technique rather than unix command. Also, I heard Retrospect may do this. If so, how does that work? Of course, a free solution or built-in Mac OS X would be ideal. Suggestions? thanx!

06-04-2003, 04:39 PM	#3
mervTormel League Commissioner Join Date: Jan 2002 Posts: 5,536	i thought the finder did some sort of rudimentary parity checking, and would fail with an error at the first smell of trouble! what are the results of using something robust like CCC? which uses ditto and rsync (?) commands. __________________ On a clear disk, you can seek forever.

06-16-2003, 09:03 PM	#8
grrl_geek Major Leaguer Join Date: Feb 2003 Location: Berkeley, CA Posts: 270	Well, there's the command "diff". That will compare two directories or files and tell you if they differ (by size or content or whatever) At the command line, do this: Code: diff -qr /path/to/volume1 /path/to/volume2 The "q" means "quick". Otherwise, it will print out the actual textual differences in the files. The "r" is for recursive. It will print out something like this: Code: [Supernova:~/Documents] grrl% diff -qr test1 test2 Files test1/Test_one.doc and test2/Test_one.doc differ Files test1/half_puzzle and test2/half_puzzle differ Only in test2: half_puzzle copy The only thing I am not sure about is whether this will report information in the resource forks.

06-17-2003, 11:39 AM	#10
hschickel All Star Join Date: Jan 2002 Location: NY, NY Posts: 776	Disk Copy - one of those unsung heroes of the utilities folder. It does just about everthing. sudo ditto -rsrc /Volumes/source /Volumes/destination - make sure you've unchecked the "Ignore ownership on this volume" box (get info from the finder) for your source and destination if you're transferring OSX system stuff where this is important EDIT - md5 - run a checksum on the original and the copy. It'll be much faster than running diff or somesuch. Hugh __________________ First they laugh at you, then they fight you, then you win. Last edited by hschickel; 06-17-2003 at 11:48 AM.

06-17-2003, 08:54 PM	#16
A Little Peaved! Major Leaguer Join Date: Apr 2002 Posts: 463	Sorry, I'm not familiar with fink. Found http://fink.sourceforge.net/ but I don't understand how to use what's there. md5, either. Found rsync_hfs, but it is also source. http://www.opendarwin.org/cgi-bin/cv...cts/rsync_hfs/ is there a Mac OS X installer or GUI for rsync_hfs, or any other of these? thx Last edited by A Little Peaved!; 06-17-2003 at 08:59 PM.

06-04-2003, 04:22 PM	#2
yellow Moderator Join Date: Jan 2002 Posts: 10,677	I agree with your notion that the source disk is flaky. Retrospect is not cheap. It does do copy and comparision against the original file. However, it's backup software, so using it to 'copy' the data from the source disk will output it all into a single backup file on your internal drive. Then you'd have to restore the file to your local drive. Basically using almost 2x the space on the internal drive.. Also, there's no guarrantee that the files won't be corrupt when restored. Plus Retrospect is NOT cheap. When this has happened to me in the past, I always go the safe and slow route. Copying a directory at a time to try and get as much good data as possible. Sometimes a little as a few files at a time.. boring yes, tedious yes, but effective.

06-05-2003, 11:45 AM	#4
A Little Peaved! Major Leaguer Join Date: Apr 2002 Posts: 463	Retrospect- if it can only "copy" from Retrospect backup, then that is no good here. (Plus, I hear Retrospect is NOT cheap. ) CCC- what is it? ditto- ditto, er, what is it? rsync- don't know that either... in some previous threads, it was mentioned that rsync did not understand macintosh file's resouce fork, which is a huge flaw. Finder error checking- my intuition is that the Finder just checks if the proper number of bits were copied, the devices reported no errors, and the File Manager is happy. As I recall, hard drive firmware and driver software contains error detection and verification capabilities. These options were accessible in nitty-gritty detail with software such as FWB Hard Disk Toolkit. Unfortunately, now with OS X and the state of the Mac software market, this product and others have become part of history and there are no current OS X replacements. So, the level of error checking in OS X, I believe, is determined by the default settings of hard disk firmware, driver and perhaps Apple Disk Utility (if it touches those settings). Isn't there something easy, free/cheap and effective?

06-05-2003, 12:41 PM	#5
yellow Moderator Join Date: Jan 2002 Posts: 10,677	CCC (Carbon Copy Cloner) Ditto is a CLI command. It's quite nice because it'll copy resource forks. Check out the man page.

06-05-2003, 03:05 PM	#6
A Little Peaved! Major Leaguer Join Date: Apr 2002 Posts: 463	Carbon Copy Cloner does not look like it has very robust copying capabilities (for this purpose). The documentation recommends manually verifying that copying was successful- that's not very encouraging, and it's exactly what we are trying to avoid. Carbon Copy Cloner uses ditto as its copying engine. The ditto man is cryptic to me. Is there a friendly guide for ditto? I tried google, but too many wrong hits for "ditto". This situation has become a bigger problem than I thought, so I am in dire need for a good solution... Also need a good comparison tool to run, prior to re-copying files. This is necessary both to check for corrupted files, and to idnetify changed/new files to sync everything. Still looking... (sigh) tia

06-16-2003, 08:50 PM	#7
A Little Peaved! Major Leaguer Join Date: Apr 2002 Posts: 463	???

06-17-2003, 07:24 PM	#14
A Little Peaved! Major Leaguer Join Date: Apr 2002 Posts: 463	I found diff (I believe), but only as source. see http://ftp.gnu.org/pub/gnu/diffutils/ Is there a Mac OS X installer for diff? Mac OS X gui for diff? tia

06-17-2003, 09:12 PM	#17
jbc All Star Join Date: Feb 2003 Location: Chico, CA Posts: 686	You might look at rsyncx. I think it must be related to rsync hfs somehow; it's a hfs/resource savvy version of rsync. Both seem to reference opendarwin.org. The latest build should be available here: http://www.macosxlabs.org/rsyncx/rsyncx.html Haven't come across anyone with any hands-on experience with this app yet. I downloaded it, but have yet to try it. May be worth a look for what you're trying to do.

06-18-2003, 02:23 AM	#18
grrl_geek Major Leaguer Join Date: Feb 2003 Location: Berkeley, CA Posts: 270	Fink is a (really nifty) project to port/recompile Unix software for the Mac OS X platform. If you want to take a look at it, install FinkCommander, which makes the whole thing way easier. On the matter at hand, I'll bet diff gets installed with the Developer Tools. I seem to have a copy from Fink, too. locate diff nabs me two copies: /usr/bin/diff /sw/bin/diff I also noticed some documentation on the various diff commands: /Library/Documentation/Commands/diffutils/diff_toc.html Peaved (fyi: it's really hard to not call you Peeves, a la Harry Potter), there should be a CD that came with your Jaguar install called "Developer Tools". The install is about 400 MB, but about 350 of that is documentation, which you can choose not to install. Hope that helps.

06-24-2003, 12:54 PM	#19
A Little Peaved! Major Leaguer Join Date: Apr 2002 Posts: 463	Thanks for all the info. Qdea Synchronize! looks like it might be a good solution... I am surprised no one here has mentioned it. Actually, surprised that it is not more popular in general. People who have used it say very good things about it...

07-01-2003, 12:33 AM	#20
A Little Peaved! Major Leaguer Join Date: Apr 2002 Posts: 463	Also considering Tri-Backup See also this related thread: Which BackUp Program is Best or your Favorite? http://forums.macosxhints.com/showth...?threadid=8348