The macosxhints Forums - Undeletable files

The macosxhints Forums (http://hintsforums.macworld.com/index.php)

- UNIX - General (http://hintsforums.macworld.com/forumdisplay.php?f=16)

- - Undeletable files (http://hintsforums.macworld.com/showthread.php?t=50018)

xApple

01-14-2006 02:20 PM

Hello everyone,

A strange thing happened lately on one of my external drives:
4 files seam to have been corrupted and cannot be deleted. (which is quite bothersome since they take up lots of space.)

When trying to erase them thru the Finder I get the following error message:
"The operation cannot be completed because the item HardBox is in use"
where "HardBox" is the name of my external USB drive.

So I decided to remove them myself using the Terminal.
after entering the usual "cd" command to go to the right directory containing the four files, I try to execute a simple "ls"; the names appear normally as you can see:

Code:

xApple:/Volumes/HardBox/toKill/bene xapple$ ls

Greystoke.la.l??gende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi

Le Seigneur Des Anneaux - La Communaut?? De L'Anneau CD1 .avi

Le Seigneur Des Anneaux - La Communaut?? De L'Anneau CD2.avi

Sin.City.FRENCH.SVCD.TS.DiVX.REPACK-BlueeeMasKKK.v??rifi??.par.divxorama.net.avi

xApple:/Volumes/HardBox/toKill/bene xapple$

but if I try a "ls -al", things start getting strange:

Code:

xApple:/Volumes/HardBox/toKill/bene xapple$ ls -al

ls: Greystoke.la.légende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi: No such file or directory

ls: Le Seigneur Des Anneaux - La Communauté De L'Anneau CD1 .avi: No such file or directory

ls: Le Seigneur Des Anneaux - La Communauté De L'Anneau CD2.avi: No such file or directory

ls: Sin.City.FRENCH.SVCD.TS.DiVX.REPACK-BlueeeMasKKK.vérifié.par.divxorama.net.avi: No such file or directory

total 0

drwxr-xr-x   6 xapple  xapple  204 Jan  9 23:46 .

drwxr-xr-x   4 xapple  xapple  136 Jan 14 19:10 ..

xApple:/Volumes/HardBox/toKill/bene xapple$

Needless to say, if I execute a "rm", I get the same error message:

Code:

xApple:/Volumes/HardBox/toKill/Bene xapple$ rm Greystoke.la.l\303\251gende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi 

rm: Greystoke.la.légende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi: No such file or directory

xApple:/Volumes/HardBox/toKill/Bene xapple$

What could I do to get rid of the files and free the space ?

I thought of a operating a "Repair Disk" with the "Disk Utility" but it says the "HardBox" disk needs no repairs.

Thanks in advance for any suggestions you could bring !

hayne

01-14-2006 03:16 PM

I would guess that the problems you are seeing is due to strange characters in the filenames. The mere presence of accented characters shouldn't be a problem - so I suspect some other weirdness.

1) Could you please email me the smallest one of the files (if you can get it into a mail message as an attachment). I'm interested in this problem - it has come up a few times before and I've never had a sample to investigate myself.
You can sent it to: macdev@hayne.net
Please use "MACOSXHINTS strange file" as the subject of the message so I will be sure to notice it among the junk mail messages in case it gets classified as junk.

2) Can you rename these files in Finder? If you can, do so and then try deleting them again.

3) You could try removing the whole containing folder:

cd /Volumes/HardBox/toKill
rm -rf bene

If neither of these methods work, let us know and we can suggest other ways.

xApple

01-14-2006 07:08 PM

1) No, I cannot rename the files in the Finder. When I try to do so, the Finder retorks
"The operation could not be completed. An unexpected error occurred (error code -43)"

2) I would be glad to eMail you one of the files, but I'm scared it would be a bit cumbersom on your mailbox... the smallest of the files weighs 698 [MB]... but you're weird characters theory is intresting...

3) If I try to operate a "rm" on the directory containing the four files I get the following:

Code:

xApples-PowerBook:/Volumes/HardBox/toKill xapple$ rm -rf bene

rm: bene: Directory not empty

xApples-PowerBook:/Volumes/HardBox/toKill xapple$

There must be a way to modify the catalog in order to be gone with these bothersome files...

xApple

01-14-2006 07:34 PM

Oh yeah and I tried doing everything I mentioned as "sudo rm" or "sudo ls -al" and it gives the exact same results...

hayne

01-14-2006 07:49 PM

Quote:

Originally Posted by xApple

3) If I try to operate a "rm" on the directory containing the four files I get the following:

Code:

xApples-PowerBook:/Volumes/HardBox/toKill xapple$ rm -rf bene

rm: bene: Directory not empty

Try the same command but preface it with 'sudo' - this will make it ask you for your password and then execute the command with 'root' privileges:

sudo rm -rf bene

hayne

01-14-2006 07:54 PM

Here's two Perl scripts that might help in giving us more diagnostic info.
The first one ("listdir") lists the files like 'ls -l' does. I suspect that it will fail on the files due to strange characters - but it's worth trying.
The second one ("listdir_diag") tries to avoid any operation that might fail due to strange characters - and it prints out the numerical values (in hexadecimal and octal) of the characters in the filenames.
For info on running scripts, see this Unix FAQ.

Code:

#!/usr/bin/perl

use strict;

use warnings;



# listdir:

# This script lists the files in the specified directory

# It gives the info from 'stat' (i.e. similar to 'ls -l')

# Cameron Hayne (macdev@hayne.net)  May 2004



# If no command-line argument, use the current directory

my $dirname = scalar(@ARGV) >= 1 ? $ARGV[0]: ".";



# function declarations

sub list_directory($);



MAIN:

{

    list_directory($dirname);

}

exit;





#----- FUNCTIONS -----



sub printable($)

{

    my ($str) = @_;



    # replace anything other than normal characters with a question mark

    $str =~ s/[^\w\d.\- ]/?/g;

    return $str;

}



sub time_as_date($)

{

    my ($time) = @_;



    my $date = localtime($time);

    return $date;

}



sub print_info($$)

{

    my ($dirname, $filename) = @_;



    use File::stat;

    my $info = stat("$dirname/$filename")

        or warn "Can't stat file $filename: $!\n" and return;



    # see 'man stat' for the meaning of the following fields

    my $name = printable($filename);

    my $ino   = $info->ino;     # inode

    my $mode  = $info->mode;    # permissions etc

    my $nlink = $info->nlink;   # number of links to this file/directory

    my $uid   = $info->uid;     # user ID of owner

    my $gid   = $info->gid;     # group ID

    my $size  = $info->size;    # size in bytes

    my $atime = $info->atime;   # time of last access (seconds since epoch)

    my $mtime = $info->mtime;   # time of last data modification

    my $ctime = $info->ctime;   # time of last file status change



    # discard file type info from 'mode' and put in usual numeric format

    my $perms = sprintf("%04o", $mode & 07777);



    # convert times (in seconds since the epoch) to date strings

    my $adate = time_as_date($atime);

    my $mdate = time_as_date($mtime);

    my $cdate = time_as_date($ctime);



    # print out the desired fields in tab-separated format

    print "$name\t$uid\t$gid\t$perms\t$mdate\n";

}



sub list_directory($)

{

    my ($dirname) = @_;



    # open the directory and call 'print_info()' for each file/sub-directory

    opendir(DIR, $dirname) or die "can't opendir $dirname: $!";

    while (defined(my $filename = readdir(DIR)))

    {

        next if $filename =~ /^\.\.?$/;     # skip . and ..

        print_info($dirname, $filename);

    }

    closedir(DIR);

}

Code:

#!/usr/bin/perl

use strict;

use warnings;



# listdir_diag:

# This script lists the files in a directory,

# supplying both hexadecimal and octal translations as well.

# It is intended for use in diagnosing problems with strange characters.

# Cameron Hayne (macdev@hayne.net)  October 2003



# If no command-line argument, use the current directory

my $dirname = scalar(@ARGV) >= 1 ? $ARGV[0]: ".";



opendir(DIR, $dirname) or die "can't opendir $dirname: $!";

my $file_count = 0;

while (defined(my $filename = readdir(DIR)))

{

    next if $filename =~ /^\.\.?$/; # skip . and ..



    ++$file_count;



    # print out $filename with non-normal characters replaced by ?

    (my $name = $filename) =~ s/[^\w\d.\- ]/?/g;

    print "File#$file_count name: $name\n";



    # print out the numerical values of the characters in $filename

    # Note that I am avoiding the use of functions such as 'split'

    # since this script is to be used for diagnosing problems

    # with strange characters in filenames

    my $char_count = 0;

    my $hex_values = "";

    my $octal_values = "";

    while ($filename =~ /(.)/g)

    {

        my $c = $1;

        ++$char_count;

        $hex_values   .= sprintf("%3x ", ord($c));

        $octal_values .= sprintf("%3o ", ord($c));

    }

    print "File#$file_count name has $char_count characters:\n";

    print "Hex:   $hex_values\n";

    print "Octal: $octal_values\n";

}

closedir(DIR);

xApple

01-15-2006 05:30 PM

Hmm... these scripts don't seam to be working as they should...
Maybe it's just me that isn't running them properly.

I created 2 files containing the scripts exactly as they appear in your last post.
The two plain text files have their respective script names, i.e. "listdir" and "listdir_diag" and are located in a sub folder of my home directory.

I chmod'ed them for them to be executable by me and also created a "test" folder (containing 3 small files) to test the scripts on.
But when I try to execute them I get the following:

Code:

xApples-PowerBook:~/scripts xapple$ ls -al

total 16

drwxr-xr-x    5 xapple  xapple   170 Jan 15 03:08 .

drwxr-xr-x   28 xapple  xapple   952 Jan 15 23:17 ..

-rwxr--r--    1 xapple  xapple  2251 Jan 15 02:51 listdir

-rwxr--r--    1 xapple  xapple  1371 Jan 15 02:51 listdir_diag

drwxr-xr-x    5 xapple  xapple   170 Jan 15 23:20 test

xApples-PowerBook:~/scripts xapple$ ls test

aaa     bbb     ccc

xApples-PowerBook:~/scripts xapple$ ./listdir test

use: bad interpreter: No such file or directory

xApples-PowerBook:~/scripts xapple$ ./listdir_diag test

xApples-PowerBook:~/scripts xapple$

The first script seams to have a bug in it and the second one produces no output...
What am I doing wrong ?

hayne

01-15-2006 05:35 PM

The "bad interpreter" error message probably indicates that the line-endings in the script file are not correct. They need to be Unix-style line endings. This is covered in the FAQ that I linked to above. (see the section on end-of-line characters)

What editor did you use to create the script files?

[edit]
I see that there is indeed a bug in the scripts - they don't properly handle the case where you supply the directory as a command-line argument.
I only tried them without any arguments - when they list the contents of the current directory.
[/edit]

hayne

01-15-2006 05:52 PM

I have now edited the above post to correct the error in the 'listdir' script so it now works correctly if you give it a command-line argument (the directory to be listed).
The other script ('listdir_diag') didn't suffer from this bug. (I.e. the problems you had with 'listdir_diag' must have been solely due to the line-endings.)

xApple

01-15-2006 07:17 PM

Ok, I got the scripts working !
(this time I used BBEdit to save them instead of the generic TextEdit)

And your prognostics were right: the first one fails and the second one seams to do what it is made to.

Here are the results:

Code:

xApples-PowerBook:/Volumes/HardBox/toKill xapple$ ls

Bene

xApples-PowerBook:/Volumes/HardBox/toKill xapple$ ~/scripts/listdir Bene

Can't stat file Greystoke.la.légende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi: No such file or directory

Can't stat file Le Seigneur Des Anneaux - La Communauté De L'Anneau CD1 .avi: No such file or directory

Can't stat file Le Seigneur Des Anneaux - La Communauté De L'Anneau CD2.avi: No such file or directory

Can't stat file Sin.City.FRENCH.SVCD.TS.DiVX.REPACK-BlueeeMasKKK.vérifié.par.divxorama.net.avi: No such file or directory

xApples-PowerBook:/Volumes/HardBox/toKill xapple$ ~/scripts/listdir_diag Bene

File#1 name: Greystoke.la.l??gende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi

File#1 name has 72 characters:

Hex:    47  72  65  79  73  74  6f  6b  65  2e  6c  61  2e  6c  c3  a9  67  65  6e  64  65  2e  64  65  2e  54  61  72  7a  61  6e  5f  46  52  2e  63  64  31  5f  70  72  65  73  63  72  69  74  2d  70  61  72  2d  6c  65  73  2d  72  65  66  72  61  63  74  61  69  72  65  73  2e  61  76  69 

Octal: 107 162 145 171 163 164 157 153 145  56 154 141  56 154 303 251 147 145 156 144 145  56 144 145  56 124 141 162 172 141 156 137 106 122  56 143 144  61 137 160 162 145 163 143 162 151 164  55 160 141 162  55 154 145 163  55 162 145 146 162 141 143 164 141 151 162 145 163  56 141 166 151 

File#2 name: Le Seigneur Des Anneaux - La Communaut?? De L?Anneau CD1 .avi

File#2 name has 61 characters:

Hex:    4c  65  20  53  65  69  67  6e  65  75  72  20  44  65  73  20  41  6e  6e  65  61  75  78  20  2d  20  4c  61  20  43  6f  6d  6d  75  6e  61  75  74  c3  a9  20  44  65  20  4c  27  41  6e  6e  65  61  75  20  43  44  31  20  2e  61  76  69 

Octal: 114 145  40 123 145 151 147 156 145 165 162  40 104 145 163  40 101 156 156 145 141 165 170  40  55  40 114 141  40 103 157 155 155 165 156 141 165 164 303 251  40 104 145  40 114  47 101 156 156 145 141 165  40 103 104  61  40  56 141 166 151 

File#3 name: Le Seigneur Des Anneaux - La Communaut?? De L?Anneau CD2.avi

File#3 name has 60 characters:

Hex:    4c  65  20  53  65  69  67  6e  65  75  72  20  44  65  73  20  41  6e  6e  65  61  75  78  20  2d  20  4c  61  20  43  6f  6d  6d  75  6e  61  75  74  c3  a9  20  44  65  20  4c  27  41  6e  6e  65  61  75  20  43  44  32  2e  61  76  69 

Octal: 114 145  40 123 145 151 147 156 145 165 162  40 104 145 163  40 101 156 156 145 141 165 170  40  55  40 114 141  40 103 157 155 155 165 156 141 165 164 303 251  40 104 145  40 114  47 101 156 156 145 141 165  40 103 104  62  56 141 166 151 

File#4 name: Sin.City.FRENCH.SVCD.TS.DiVX.REPACK-BlueeeMasKKK.v??rifi??.par.divxorama.net.avi

File#4 name has 80 characters:

Hex:    53  69  6e  2e  43  69  74  79  2e  46  52  45  4e  43  48  2e  53  56  43  44  2e  54  53  2e  44  69  56  58  2e  52  45  50  41  43  4b  2d  42  6c  75  65  65  65  4d  61  73  4b  4b  4b  2e  76  c3  a9  72  69  66  69  c3  a9  2e  70  61  72  2e  64  69  76  78  6f  72  61  6d  61  2e  6e  65  74  2e  61  76  69 

Octal: 123 151 156  56 103 151 164 171  56 106 122 105 116 103 110  56 123 126 103 104  56 124 123  56 104 151 126 130  56 122 105 120 101 103 113  55 102 154 165 145 145 145 115 141 163 113 113 113  56 166 303 251 162 151 146 151 303 251  56 160 141 162  56 144 151 166 170 157 162 141 155 141  56 156 145 164  56 141 166 151 

xApples-PowerBook:/Volumes/HardBox/toKill xapple$

Does this help us in finding a solution ?

hayne

01-15-2006 10:39 PM

Quote:

Originally Posted by xApple

Does this help us in finding a solution ?

Well, it confirms that the filenames are using a character encoding that is apparently not allowed in OS X. I have been trying to create filenames with characters shown by the above, but have failed to do so.

One possibility is to try using the utility 'convmv' to change the encoding of these filenames. You can download this utility (which is a Perl script) from here:
http://j3e.de/linux/convmv/man/
Then you'll have to fool around with it a bit to see if you can convert the filenames.
Basic usage is like this:
convmv -f name_of_existing_encoding -t name_of_desired_encoding name_of_file
where name_of_existing_encoding is one of the encodings that you can list by running the command:
convmv --list
Which one should be used is something I don't know. It seems that the troublesome characters are (in hexadecimal) "c3 a9" which I've seen listed as being the Unicode for é
For the name_of_desired_encoding, I think you want to use UTF-8 or UTF-16

By default, the convmv command just tells you what it is planning to do - it doesn't actually do it. To get it to actually change the filename, you need to add the "--notest" option:
convmv -f name_of_existing_encoding -t name_of_desired_encoding --notest name_of_file

-----------------------------

Another way of deleting these files that would avoid the above "fooling around" might be to turn on file sharing (in Sharing preferences) and then access your drive from some other (non-OS X) machine. You might be able to delete the files from that other machine.

TrumpetPower!

01-15-2006 11:40 PM

I'm getting a serious filesystem corruption vibe here.

Boot off the install disk, use the Tools (?) menu to get to Disk Utility, and verify and / or repair the disk.

Or you could boot into single-user mode and run fsck, if you're comfortable with that sort of thing. It does the same as Disk Utility.

Cheers,

b&

weltonch777

01-15-2006 11:55 PM

I managed to successfully create the file Greystoke.la.légende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi by using the hex values you provided to form a string for the C function fopen(const char*, const char*);

Bad news is, I had no problem deleting the file with that name. Deleted it from the command line using "rm Gre*", and from finder using Cmd-delete.

They aren't "Suggested" filename characters, but they don't appear to corrupt the file system on there own.

hayne

01-16-2006 12:15 AM

Quote:

Originally Posted by weltonch777

Are you sure that the file got created with the character values you supplied?

In my experiments (I've been trying with Perl), I can create a string with those hex values, and then create a file (using sysopen(FILE, $filename, O_CREAT)) but then when I check what the filename of the newly created file is (e.g. with the 'listdir_diag' script above), I find that different characters are in the filename.
I.e. it seems that the file creation routines do some translation of the characters in the filename.

weltonch777

01-16-2006 12:16 AM

I'll check
_______

ok, you are correct

stdio changed

Code:

0x6c, 0xc3, 0xa9, 0x67,

Code:

0x6c, 0x65, 0xcc, 0x81, 0x67

Code:

File#2 name: Greystoke.la.le??gende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi

File#2 name has 73 characters:

Hex:    47  72  65  79  73  74  6f  6b  65  2e  6c  61  2e  6c  65  cc  81  67  65  6e  64  65  2e  64  65  2e  54  61  72  7a  61  6e  5f  46  52  2e  63  64  31  5f  70  72  65  73  63  72  69  74  2d  70  61  72  2d  6c  65  73  2d  72  65  66  72  61  63  74  61  69  72  65  73  2e  61  76  69

hayne

01-16-2006 02:30 AM

Quote:

Originally Posted by weltonch777

stdio changed

Code:

0x6c, 0xc3, 0xa9, 0x67,

Code:

0x6c, 0x65, 0xcc, 0x81, 0x67

Yeah - that's what I'm seeing too.
For future reference, here's the C program I wrote for testing this sort of thing:

Code:

// This C program attempts to create a file with a specified name.

// It then reads the current directory to check if the file got created

// with the desired name.

// It is intended for testing creation of files with strange characters

// in their names. 

// Cameron Hayne (macdev@hayne.net)  January 2006

 

#include <stdio.h>

#include <stdlib.h>

#include <string.h>

#include <ctype.h>

#include <dirent.h>



void printString(char *str)

{

    int len = strlen(str);

    int i;

    for (i = 0; i < len; i++)

    {

        if (isprint(str[i]))

        {

            printf("%c", str[i]);

        }

        else

        {

            printf(" 0x%2x ", (unsigned char)str[i]);

        }

    }

    printf("\n");

}



char *findDirEntry(char *startOfEntryName)

{

    char *fullEntryName = NULL;

    int startLen = strlen(startOfEntryName);



    DIR *dirp = opendir(".");

    if (dirp == NULL)

    {

        fprintf(stderr, "opendir failed\n");

        return NULL;

    }



    struct dirent *dp;

    while ((dp = readdir(dirp)) != NULL)

    {

        if (strcmp(dp->d_name, ".") == 0 || strcmp(dp->d_name, "..") == 0)

        {

            continue; // skip . and ..

        }



        if (strncmp(dp->d_name, startOfEntryName, startLen) == 0)

        {

            fullEntryName = strdup(dp->d_name);

            break;

        }

    }



    if (closedir(dirp) != 0)

    {

        fprintf(stderr, "closedir failed\n");

    }



    return fullEntryName;

}



int createFile(char *filename)

{

    FILE *fp = fopen(filename, "w");

    if (fp == NULL)

    {

        fprintf(stderr, "Failed to create file\n");

        return (-1);

    }

    if (fclose(fp) != 0)

    {

        fprintf(stderr, "fclose failed\n");

        return (-1);

    }

}





int main()

{

    char desiredName[] = "Greystoke.la.l??gende.de.Tarzan_FR.cd1_prescrit-par-les-refractaires.avi";

    desiredName[14] = 0xc3;

    desiredName[15] = 0xa9;

    char *startOfDesiredName = "Grey";



    printf("Attempting to create file named:\n");

    printString(desiredName);

    createFile(desiredName);



    // Now check if the file shows up in the directory listing with that name

    char *entryName = findDirEntry(startOfDesiredName);

    if (entryName != NULL)

    {

        if (strcmp(entryName, desiredName) == 0)

        {

            printf("Directory entry matches desiredName\n");

        }

        else

        {

           printf("Directory entry is:\n"); 

           printString(entryName);

        }

    }

    else

    {

        printf("No entries starting with \"%s\"\n", startOfDesiredName);

    }



    return 0;

}

Mednanu

01-19-2006 06:40 AM

Had this happen before after I had cancelled an rsync in mid-sync. These errors are likely due to "bad text encoding" errors on the files themselves. Telltale signs that you're experiencing this condition ( other than the exact symptoms that you are currently seeing ) are clicking on the file and having it dissappear, then reappear sometime later on, seemingly at random. Also, you will sometimes be able to move them but usually will not be able to delete them through any method whatsoever ( ie - GUI trash, rm, srm, etc, etc ). Root can sometimes rename them in order to rid the filename of pesky invisible characters, but the issue will still persist and the file will still be reported as in use, even after a reboot or if you boot to another System folder or an entirely different CPU.

Solution: Run Disk Warrior.

It will report the damaged text encodings and likely be able to fix them. You should then be able to delete them anyway you wish after that.

xApple

01-22-2006 11:16 AM

Well, in the end, to solve the problem what I did is ask a friend to shortly borrow some of his HD's free space, while I reformatted my external USB drive.
The files are now gone for good ^^.

I didn't own "Disk Warrior"... and the price is expensive..
"fsck" didn't do anything, in fact didn't even report a problem.
and the "convmv" util just seamed as much of a hassle as temporally storing the data elsewhere while the disk was erased...

But I thank you all for your attempts at helping me... the mac community is always so firendly !

I just suppose strange file corruptions will always happen...

neuralstatic

02-02-2006 06:08 PM

was 'rm -rf directoryname' not tried? i know everyone wanted to dissect it, but that would have worked, right?

hayne

02-02-2006 06:41 PM

Quote:

Originally Posted by neuralstatic

was 'rm -rf directoryname' not tried? i know everyone wanted to dissect it, but that would have worked, right?

It was tried and it didn't work. See posts #3 & 4.
The strange characters in the filename apparently cause confusion at the BSD level.

neuralstatic

02-02-2006 06:50 PM

that's exactly why i asked. he said he tried rm and sudo rm. naturally an rm won't work on a dir with files.

he skipped right over rm -rf. according to what he says he did. i ask since this has worked for me before.

hayne

02-02-2006 06:58 PM

Quote:

Originally Posted by neuralstatic

Look at what is in the "code" box - it shows that he tried 'rm -rf bene'
I presume that (according to post #4) he then did 'sudo rm -rf bene'

This is not the only time we've encountered a problem like this with files that are undeletable due to strange characters.
See: http://forums.macosxhints.com/showthread.php?t=15814

neuralstatic

02-02-2006 07:02 PM

oh, how did i miss that.

thanks

TrumpetPower!

02-04-2006 02:16 PM

I'm still getting filesystem corruption vibes. Really, really strong ones, too....

Cheers,

b&

biovizier

02-04-2006 03:02 PM

You might get that impression given some of the "symptoms", but the earlier posts confirm the presence of "odd" encodings in the filenames, and this issue has come up a number of times on all sorts of Mac forums. A few years ago, a simple solution was to boot into System 9 (which seems to be able to handle more odd encodings than OS X) and delete the file from there. More recent Macs can't boot in System 9 so obviously that is no longer an option, though I wonder if attaching it to an older Mac via target disk mode would work... Someone mentioned using Disk Warrior to fix encodings - I haven't heard whether that works or not, but it would be a shame if money had to be spent to fix a problem that can be caused by just transferring a file to your computer...

TrumpetPower!

02-05-2006 11:25 AM

Quote:

Originally Posted by biovizier

Let me re-phrase.

If you have an inode that can't be deleted or renamed, the filesystem IS corrupt. Period.

Now, it may be that, in this particular case, the reason the files can't be deleted or renamed is because their filenames have illegal characters not permitted by the HFS+ spec--an archetypal form of corruption. It may also be that the filesystem is theoretically okay, that the spec permits such filenames, but that all the standard libraries and utilities are broken in such a way that they can't deal with all valid filenames--in which case, one is left to wonder how the file got created with that name in the first place. Either way, it doesn't matter.

What does matter is that this is a very, very serious bug. Filesystem corruption of any kind is serious bad mojo. If this is something that can be reproduced, especially if it can be done without doing anything particularly unusual, it's also a major security hole.

Frankly, it sounds like it'd be near-trivial to fill up a filesystem by creating a super-large file that can't be deleted. Or by creating millions of small files that can't be deleted, either. That in and of itself constitutes a really, really nasty DOS attack that'd be difficult or impossible to protect against in many real-world situations. (Yeah, quotas, partitioning, etc.) Considering how poorly OS X and other systems that allocate swap dynamically handle low disk situations...this is about as bad as they come.

Has anybody bothered to tell Apple about this? Is anybody from Apple listening?

What's the best way to report security problems to Apple?

Cheers,

b&

hayne

02-05-2006 04:22 PM

Quote:

Originally Posted by TrumpetPower!

If you have an inode that can't be deleted or renamed, the filesystem IS corrupt.

You might be under the misapprehension that the HFS+ filesystem has inodes as the fundamental entities. That is not correct. While there is an inode number for each file (e.g. as shown by 'ls -i'), that is merely an attribute added for backward compatibility with traditional Unix utilities. The fundamental handle (means of referring to a file) is the filename, not the inode number.
See:
http://developer.apple.com/technotes/tn/tn1150.html
http://www.usenix.org/publications/l.../sanchez_html/

Quote:

one is left to wonder how the file got created with that name in the first place

It is obviously a bug. Some gatekeeper routine that is supposed to check for valid characters is failing to do its job. Once the file is there, the various utilities are failing because they have built-in assumptions about the possible characters. I think these files are getting in via some sort of filesharing from a non-OSX system.

Quote:

it's also a major security hole

I don't think it's a security hole since you would need permission to create the file in the first place.

Quote:

Has anybody bothered to tell Apple about this? Is anybody from Apple listening?

It is unlikely that anyone in Apple engineering reads these forums.
It would be nice to report this bug but I think first what we need is a way to reproduce it. That is why I was trying to find out exactly what characters were in those troublesome filenames and why I tried (without success) to create a file of that name on my system.

Security issues, like other bugs, can be reported via Apple's bug report form:
http://developer.apple.com/bugreporter/bugrptform.html

TrumpetPower!

02-05-2006 05:41 PM

Quote:

Originally Posted by hayne

I don't think it's a security hole since you would need permission to create the file in the first place.

There're lots and lots of situations where semi-trusted users have exactly that permission. Labs in servers, for example, as well as most Web, ftp, etc. servers that accept files for upload. You can also perhaps create archive files with bad names in them, email things to people....

Quote:

It would be nice to report this bug but I think first what we need is a way to reproduce it.

Yeah...that's the kicker. I'm sure sending proof-of-concept code would get their attention much better than just ``files with weird characters in their names can't be deleted,'' which is what we're stuck with currently.

Hmmm...here's a thought. You (or somebody else?) mentioned that you could use OS 9 to delete the files? Perhaps they're easier to create in OS 9. If so, here's a suggestion:

Create a file on a computer running OS 9. If possible, get that file onto a read-write disk image that has the problem. If not, get the file onto a cheap thumb disk. Confirm that OS X has the problem. Then, send get the disk image or the thumb disk to Apple, along with an appropriate write-up. That should get their attention.

I'd offer to help...but I don't have anything that'll boot OS 9 anymore.

Of course, it could still just be ye olde fashioned filesystem corruption....

Cheers,

b&

PS Perhaps manually creating a UU-encoded file would be an easy way to trigger this? Editing a tarball or a zip file? b&

All times are GMT -5. The time now is 05:28 PM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
Site design © IDG Consumer & SMB; individuals retain copyright of their postings
but consent to the possible use of their material in other areas of IDG Consumer & SMB.