PDA

View Full Version : How do I NOT preserve resource forks?


Rich B
08-11-2005, 09:16 AM
Hello people,

I'm rather new to my Mac (and seriously considering blatting OS-X with Linux instead). I seem to be having the opposite problem to everyone else in that I DON'T want to preserve resource forks when I tar up my files.

This is on a 10.4.2 system on the system disk (which is formatted as HFS+ I believe - I've not changed the format it since I bought it... but I'm considering it!)

Scenario is...
1/ I have some stuff that I edit
2/ I tar this up in the usual way
3/ Copy the tar to another (non Mac) machine and untar it
4/ A bunch of "._***" resource fork files get unpacked along with the files that I was actually interested in.

I have tried this several times to make sure I wasn't doing something stupid, but no, this seems to be what is happening.

I can't see any resource files at all on the Mac. I read that "ls -l <file>/rsrc" should show up the resource forks but there's nothing there when I try this. So either I've got this wrong or the ._*** files are being generated on the fly when tar is run.

I would rather the resource fork files were not being created at all, but it seems that this is a pipedream (unless someone knows otherwise - oh, PLEASE say you do). But if I can't see them from the command line, I can't delete them!

I'm assuming Apple have hacked the 'ls' command to hide this stuff? Does anyone know if a standard (separate) 'ls' command is available that will actually do what I ask and not lie to me, or will I have to install a complete new shell? Oh, for a Mac that worked like a PROPER unix box!!!

Any clues would be much appreciated.

regards

Rich.

JDV
08-11-2005, 10:11 AM
This isn't an elegant suggestion, but it plays on what is often regarded as a defect in ftp programs and make it work to your advantage. Most FTP programs that aren't designed for Mac-to-Mac transfer don't preserve the resource fork. Perhaps if you transfer the files from your Mac to the other machine via FTP, you'll achieve your goal without purposely deleting anything.

Joe VanZandt

Rich B
08-11-2005, 10:35 AM
Thank for the suffestion, but...

a/ FTP is not an option to tranfer my files to the other machine
b/ The ._*** files are in the tarball so FTP wouldn't see them anyway

Also, I may just want to tar stuff up and leave it on the Mac (and STILL not want the .+*** files in there)

Ta ayway,

regards

Rich.

vancenase
08-11-2005, 11:21 AM
i think you could use rsync to copy it to a new directory. and, i know there is a version of rsync that supports resource forks, but there may be a flag to turn them off during the copy.

hayne
08-11-2005, 11:55 AM
I can't see any resource files at all on the Mac. I read that "ls -l <file>/rsrc" should show up the resource forks but there's nothing there when I try this. So either I've got this wrong or the ._*** files are being generated on the fly when tar is run.

That is correct - the ._*** files are being created by 'tar'.
In Tiger (10.4) there is also arbitrary metadata that is stored in these ._*** files - not just resource forks. See this ArsTechnica review page for details:
http://arstechnica.com/reviews/os/macosx-10.4.ars/7

Unfortunately, there doesn't seem to be an option to 'tar' to suppress the saving of this metadata via ._**** files. One thing you could do, however, is use a different version of 'tar' - e.g. the GNU version of tar that is available via Fink.

I'm assuming Apple have hacked the 'ls' command to hide this stuff? Does anyone know if a standard (separate) 'ls' command is available that will actually do what I ask and not lie to me, or will I have to install a complete new shell? Oh, for a Mac that worked like a PROPER unix box!!!

As noted above, the ._*** files are created on the fly. The 'ls' command does show you the files that exist on disk just like any other Unix.

Rich B
08-11-2005, 12:03 PM
Ah! Thank you for the insight.

I'll grab gtar and take it from there. Thank you.

I have read comments about ._*** files popping up on flash cards and external drives etc. This really is an awful mess that Apple have created!!!

regards

Rich.

hayne
08-11-2005, 12:30 PM
Unfortunately, there doesn't seem to be an option to 'tar' to suppress the saving of this metadata via ._*** files.

I just looked at the source code for the version of 'tar' that is suppled with 10.4.2 (it is a modified version of gnutar):
http://www.opensource.apple.com/darwinsource/10.4.2/
and if I understand it properly, it seems that you can disable the ._*** files if you define the environment variable "COPY_EXTENDED_ATTRIBUTES_DISABLE".
It doesn't seem to matter what this is defined as - it just has to be defined.
E.g. in bash:
export COPY_EXTENDED_ATTRIBUTES_DISABLE =true

Rich B:
Please give this a try and let us know if that does indeed work to suppress the ._*** files. If so, I'll submit it as a hint to the main macosxhints site.

I edited this to correct the name of the environment variable:
COPY_EXTENDED_ATTRIBUTES_DISABLE instead of COPYFILE_DISABLE_VAR

Rich B
08-11-2005, 01:24 PM
A brave effort, but I'm afraid setting COPYFILE_DISABLE_VAR didn't make any difference.

regards,

Rich.

hayne
08-11-2005, 01:57 PM
A brave effort, but I'm afraid setting COPYFILE_DISABLE_VAR didn't make any difference.

Oops - my mistake. I didn't look at the source code carefully enough.
COPYFILE_DISABLE_VAR is a defined constant in the source code - it is defined as "COPY_EXTENDED_ATTRIBUTES_DISABLE".
So the environment variable that needs to be set is COPY_EXTENDED_ATTRIBUTES_DISABLE
E.g. in bash:
export COPY_EXTENDED_ATTRIBUTES_DISABLE=true
I tested this (creating a 'tar' archive after setting this variable) and it does work - there are no ._*** files in the archive.

(I edited my posting above to supply the correct name for the environment variable.)

By the way, the handling of "extended attributes" in Tiger's improved versions of Unix commands like 'tar' is implemented via the code in the files "copyfile.h" and "copyfile.c" that are part of "libc" (available via the Apple opensource link above). I note that there is a useful diagram in copyfile.c that shows the format of the ._*** files.

Rich B
08-11-2005, 02:09 PM
...but setting COPY_EXTENDED_ATTRIBUTES_DISABLE does work!

Excellent. Thanks very much indeed.

So, what's the next 'standard' unix command that's broken then :-) and is there a more general way of dealing with this nonsense?

regards

Rich.

hayne
08-11-2005, 02:22 PM
So, what's the next 'standard' unix command that's broken then :-) and is there a more general way of dealing with this nonsense?

I think the correct interpretation is that the standard Unix commands have been fixed - not broken. The information (resource forks, metadata) exists as part of the files in OS X, so the 'tar' command should archive this info somehow. I think we can expect that Apple's changes will get integrated into the 'standard' GNU tools at some point.

Rich B
08-11-2005, 02:37 PM
Oh dear - I hope not :-(

It's a nice idea on the surface but unfortunately, as the original reason I started this thread amply demonstrates, it's simply not portable. Even if the system I unpacked this tarball on (OpenBSD) understood the concept of extended file attributes etc, the format would almost certainly be different and mean something quite different.

R.

JDV
08-11-2005, 03:35 PM
I was under the impression that resource forks were NOT a part of OS X, but left over from previous Mac OSes. Have they been retained as the carrier of meta-data now? I agree that this would be a step backward if they are retained. Rich is correct that they don't travel well.

Just BTW, Rich, in my original (impractical and inelegant) suggestion, I had it in mind FTPing before archiving, for I know that archiving it first would include the resource fork. It was a bad idea, but not as careless as it may have seemed.


Joe VanZandt

hayne
08-11-2005, 03:52 PM
I was under the impression that resource forks were NOT a part of OS X, but left over from previous Mac OSes. Have they been retained as the carrier of meta-data now? I agree that this would be a step backward if they are retained. Rich is correct that they don't travel well.

1) Some programs (those that started life under OS 9) still use resource forks for storing important information.

2) The new (since Tiger) metadata is not stored in resource forks - it is stored as a part of the file in HFS+. See the ArsTechnica article referred to above for details.

3) The ._*** files that are included by 'tar' store both resource fork info and metadata. While it is true that these files may not be useful on non-OS X systems, they do preserve the info that was in the original OS X files. You can just ignore (or delete) them on other systems.

The situation is partially analogous to that at the introduction of colour television in the United States. The colour info was added on as a separate component to the monochrome picture in broadcast signals. Black & white televisions simply ignored the colour component.
http://en.wikipedia.org/wiki/NTSC

biovizier
08-11-2005, 04:22 PM
I was under the impression that resource forks were NOT a part of OS X, but left over from previous Mac OSes
I think the status of resource forks is still something more than mere "carry overs" at this time... that is to say, old ones aren't merely tolerated - new ones are being created by OS X. They are still used for a lot of "convenience" features in OS X that are still without non-resource fork based counterparts.

For example, "clippings", those ".xxxloc" internet location files (webloc, afploc, etc), and "alias" files are all completely resource fork based.

Other examples include, custom icons (the cut and paste ones added via "Get Info"), and the information when an individual file is set to "Open with" a specific application (though if "Change all" is used, then "Launch Services" handles it).

An Applescript saved as an "Application" (as opposed to "Application Bundle") will have (and require) a resource fork, and many old fonts have their data in resource forks.

No doubt there are others...

dmacks
08-11-2005, 08:34 PM
So the environment variable that needs to be set is COPY_EXTENDED_ATTRIBUTES_DISABLE
This does not appear to be in the manpage for tar, and it's a useful thing to know. File a radar on it?

hayne
08-11-2005, 09:07 PM
This does not appear to be in the manpage for tar, and it's a useful thing to know. File a radar on it?

The source code file "copyfile.c" makes it clear that this is an experimental implementation and that things are likely to change. As such I don't think Apple intends the environment variable COPY_EXTENDED_ATTRIBUTES_DISABLE to be part of the "public" interface for 'tar'.

If you were to file a radar (bug report) on anything, it should be on the lack of a command-line option for suppressing the ._*** files.

hayne
08-11-2005, 10:05 PM
By the way, I just noticed that the Fink people seem to have known about this work-around (COPY_EXTENDED_ATTRIBUTES_DISABLE) for a while:
http://www.mail-archive.com/fink-commits@lists.sourceforge.net/msg35785.html

NovaScotian
08-12-2005, 10:42 AM
.... While it is true that these files may not be useful on non-OS X systems, they do preserve the info that was in the original OS X files. You can just ignore (or delete) them on other systems. ....
I don't think this is universally true. You can filter those files out so they don't make it to the new system, but if you don't and they land on a WinXP disk, some of them (and I don't know how to tell which) will give an error if you try to move them, delete them or otherwise interact with them. Until I recently did a clean install of XP, I had dozens of such files - completely untouchable - littered all over my document folder. To avod this, I used Crystalfire Wormhole which will prefilter the .*** files out of a transfer from OS X to WinXP.

brucoder
12-01-2005, 08:08 PM
First, the "._" convention is the AppleDouble convention that allows multi-fork files to be stored onto flat filesystems such as UFS, Linux, Windows, et al that don't support the Finder attributes or multi-fork files. Since the tar format also doesn't have a way to handle these multi-fork files, this method was adopted for tar, pax, rsync, cp, mv, and a few other Unix tools in 10.4.

I believe that the original issue that affected Rich's tarballs is that the 'copyfile' changes to GNUTAR result in ALL files processed getting the AppleDouble "._" file - even if there were no Extended Attributes, ACLs, or Resource forks that would require its creation. Of course, these are all components that can't easily be seen in an 'ls' call, so they are a mystery to most non-Mac folks.

Unfortunately, the thing that was missing from that list above was "Finder info". If you've ever examined a file in the Finder, it will have Finder information assigned; drag it to a new folder - Finder Info, double-clicked to open it - Finder info... Because this attribute info is not part of a normal POSIX flat file's attributes, voila! tar generates the "._" file to hold the non-POSIX Finder attribute info. This is probably what has happened in Rich's case; the files were either handled within Finder, or the tool used to create/edit them was Finder aware and set the properties. If this is the case, when these files are restored to a flat filesystem, it is safe to delete them. Also, if this is the reason for the creation, the "._" file will be 82 bytes.

The other issue with this is that once the forks and metadata have been separated by the tar creation, even if you are restoring to an HFS filesystem, the AppleDouble format is retained rather than having the multi-fork, HFS file restored as it was originally. So, a folder that used to have 40 files inside can now potentially have 80. :eek:

HTH,
Tim

hayne
12-02-2005, 12:03 AM
I believe that the original issue that affected Rich's tarballs is that the 'copyfile' changes to GNUTAR result in ALL files processed getting the AppleDouble "._" file - even if there were no Extended Attributes, ACLs, or Resource forks that would require its creation.
I don't think that is true - I think the ._ files are only created when there is some metadata to be saved.

The other issue with this is that once the forks and metadata have been separated by the tar creation, even if you are restoring to an HFS filesystem, the AppleDouble format is retained rather than having the multi-fork, HFS file restored as it was originally

I don't think this is true - I think the software that does the restoring (e.g. the new version of 'tar') knows about the AppleDouble files and recombines them to reproduce the original files (on HFS+ systems).

tlarkin
12-02-2005, 09:32 AM
this is a most interesting thread. I for one, have an external 80 gig fw/usb 2.0 2.5" HD I got for my basic usage at work. I keep data, and utilities on it, and keep it in FAT32 format so it can be mounted in winxp, linux, and mac os x. When I hook it up to my linux machine or my win xp machine I see all the resource files, and though they are all small in size it can be somewhat annoying. For the most part I just have had other OSes hide them.

It is a part of the HFS+ file system so I don't think its left over from os 9 and I see apple probably sticking with it. Especially since Tiger had a bunch of meta data features added, and things like spot light which are suppose to scan meta data from files very fast and allow the user to do more robust searching.

So, my question is, what if I just copy the native files over via finder, and I am not archiving them in a .tar file, can I then still get rid of the resource fork? Is there perhaps a flag in the cp command to just copy it with out the resource fork? That would be a really cool feature, for people like me who use multiple platforms. It is not something I majorly want and could easily live with out, but after reading this thread I am now curious.

Anyways, thanks for that info hayne, I am sure I will run into someone who will want me to do this exact thing for them in the future.

hayne
12-02-2005, 09:45 AM
Is there perhaps a flag in the cp command to just copy it with out the resource fork?
No - but see this other thread where I supply an AppleScript droplet for copying without the ._ files:
http://forums.macosxhints.com/showthread.php?t=45098
(see final version of the AppleScript in post #17 and note that it uses the Perl script from earlier in the thread.)

tlarkin
12-02-2005, 10:05 AM
No - but see this other thread where I supply an AppleScript droplet for copying without the ._ files:
http://forums.macosxhints.com/showthread.php?t=45098
(see final version of the AppleScript in post #17 and note that it uses the Perl script from earlier in the thread.)


cool thanks for the info I book marked it and will try it out next time I do a major utility/data transfer to my external drive from the mac side.

brucoder
12-14-2005, 01:56 AM
I don't think that is true - I think the ._ files are only created when there is some metadata to be saved.

My point was that any file that is touched by Finder will have Finder Info assigned - poof, an 82 byte ._ file.

I don't think this is true - I think the software that does the restoring (e.g. the new version of 'tar') knows about the AppleDouble files and recombines them to reproduce the original files (on HFS+ systems).

On a test that I just ran with tar under Tiger, the ._ files are in the tarball, and when I restore them to a fresh HFS+ directory, they are restored in the Appledouble format and not recombined as hoped.

I agree that this is probably not what Apple is intending, but it is what I'm seeing here and what I believe Rich was seeing originally.

hayne
12-14-2005, 09:56 AM
My point was that any file that is touched by Finder will have Finder Info assigned - poof, an 82 byte ._ file.
Maybe you can be more precise about what you mean by "touched by Finder". I interpreted it to mean that merely viewing a folder with Finder would be "touching". My tests (see below) don't show any Finder Info being assigned by merely viewing a folder, nor by viewing Get Info on a file.

On a test that I just ran with tar under Tiger, the ._ files are in the tarball, and when I restore them to a fresh HFS+ directory, they are restored in the Appledouble format and not recombined as hoped.

Are you sure that you used the same (Apple-supplied) version of 'tar' in both cases? I'm not seeing that in my tests on 10.4.3 - see below.

% mkdir AAA
% cd AAA
% echo "hello" > foo.txt
% echo "goodbye" > foo2.txt

% # go into this folder in Finder
% # do File/Get Info on both foo.txt and foo2.txt
% # change foo2.txt to open with TextEdit (default was TextWrangler)

% xattr --list foo.txt
foo.txt

% xattr --list foo2.txt | cut -c 1-80
foo2.txt
com.apple.FinderInfo
com.apple.ResourceFork <0000, 0000, 0x01, 0000, 0000, 0000, 0xaa, 0x1e, 0000, 0

% /usr/bin/tar cvf foo.tar foo.txt foo2.txt
foo.txt
./._foo2.txt
foo2.txt

% /usr/bin/tar tvf foo.tar
-rw-r--r-- fred/fred 6 2005-12-14 10:11:14 foo.txt
-r-------- fred/wheel 43719 2005-12-14 10:16:07 ./._foo2.txt
-rw-r--r-- fred/fred 8 2005-12-14 10:16:07 foo2.txt

% mkdir BBB
% cd BBB

% /usr/bin/tar xvf ../foo.tar
foo.txt
./._foo2.txt
foo2.txt

cd ..
% ls -la BBB
total 104
drwxr-xr-x 4 fred fred 136 Dec 14 10:32 .
drwxr-xr-x 7 fred fred 238 Dec 14 10:32 ..
-rw-r--r-- 1 fred fred 6 Dec 14 10:11 foo.txt
-rw-r--r-- 1 fred fred 8 Dec 14 10:32 foo2.txt

% diff -r . BBB
Only in .: .DS_Store
Only in .: BBB
Only in .: foo.tar

% xattr --list BBB/foo.txt
BBB/foo.txt

% xattr --list BBB/foo2.txt | cut -c 1-80
BBB/foo2.txt
com.apple.FinderInfo
com.apple.ResourceFork <0000, 0000, 0x01, 0000, 0000, 0000, 0xaa, 0x1e, 0000, 0


The 'xattr' command used above is that from the Ars Technica article on Tiger by John Siracusa.