Many, many duplicate files in Spotlight searches
 

OS 10.4.10
Ibook G4
1.5 GB Ram

Hi,

I see that Spotlight is a System Preference

I have so many duplicates of files and folders in every Spotlight search I make that it would take forever to delete, and in Spolight there is no way to select more than one item at a time.

I have had this problem for some time.

Why are so many duplicates created; how can I stop this from happening; and how can I get rid of the duplicates that are already there?

Thanks, Rafael

Are there actually duplicate files, or just duplicate results? If there are duplicate files, you have other issues. However, if the problem is just with the Spotlight results you can tell Spotlight to re-index. Open the Terminal utility and type the following:

sudo mdutil -E /

Press return, then enter your user's password. You'll be informed that the index has been erased. After a short delay, re-indexing will commence.

Do you mean that you have duplicates of files on your system - and therefore that Spotlight is (correctly) showing these duplicates when you do a search?

I.e. are you asking about a problem with SpotLight, or a problem with the existence of duplicate files ?

Many, many duplicate files in Spotlight searches
 

These are just duplicate results.

When you write "sudo mdutil -E /" there's a space between ...mdutil and -E and a space between -E and /, right?

Terminal: Here's what I got. It wouldn't allow me to enter my user's password. By password it means my main OS password, right?
--------------------------------------------------------------
Last login: Sun Sep 23 12:46:26 on ttyp1
Welcome to Darwin!
adsl-75-18-196-169:~ rafaelmontserrat$ sudo mdutil -E /

WARNING: Improper use of the sudo command could lead to data loss
or the deletion of important system files. Please double-check your
typing when using sudo. Type "man sudo" for more information.

To proceed, enter your password, or type Ctrl-C to abort.

Password:

It would be good (to be completely clear) if you gave us a specific example of a file that shows in duplicate.

Correct

It is normal that it doesn't show the characters that you type when you enter the password. Is that what you meant? If so - i.e. if you didn't get an error message saying that something was wrong - then the command executed okay and hence your Spotlight database should now be reset.

"It would be good (to be completely clear) if you gave us a specific example of a file that shows in duplicate."

OK. Attached is a picture (.png) that I took of one screen. To conform to this site's attachment rules, I cut the .pngs in half. It is the same screen.

"It is normal that it doesn't show the characters that you type when you enter the password. Is that what you meant?"

Just after the last letter of the text that asked to enter password was a black rectangle as high as a capital letter, and as wide as a character (a-this wide). I think the flashing cursor? was there, but it was obviously the place I was to enter password. Nothing happened. the cursor didn't move to the right. I couldn't enter a password.

In RE: "... it doesn't show the characters that you type when you enter the password." There was no typing. The cursor didn't move to the right.

"If so - i.e. if you didn't get an error message saying that something was wrong - then the command executed okay and hence your Spotlight database should now be reset."

No error message. I don't think the command executed OK because as you can see from the attachment, the duplications are the same as always. Nothing's changed in Spolight as far as all the duplicates go.

Sherlock Duplicates
 

I don't see the attachment on my posted reply, or don't know where to look for it. Here it is again.

What you describe is precisely the way it is supposed to work. The black rectangle doesn't move at all when you type your password. Just type it in blind and press Return.

I see the duplicates in your screen capture.
One possibility is that there is something wrong with the source data - e.g. in the case of the bookmarks that your bookmarks file is corrupt.

The 1999 date on the archives is a bit suspect too.
Is the date on your Mac correct?

You should definitely try doing the 'mdutil' command recommended above for resetting the Spotlight database. Note that it may take several hours after you run that command for the database to be re-created. So it is best to leave your Mac on for at least a few hours after running that command.

Another thing you should try is doing "Repair Disk" (different from "Repair Disk Permissions"!) in Disk Utility when booted from the Install CD/DVD. I.e. you restart from the Install CD/DVD, then run the version of Disk Utility that is available from one of the menus there.
See this Apple doc:
http://docs.info.apple.com/article.html?artnum=106214

Many, many duplicate files in Spotlight searches Reply to Thread
 

OK. I did that. filled in my PW and pressed 'Return'.
This what I got (below). Can I work along with this process going on?
Volume index removed: What volume? Is this process restricted to Sherlock alone? Should I wait until I get notification before I use Sherlock again? This is a Unix thing, Right? In the very bowels of the system where I have never gone. It's dark here. I've put my computer's destiny in your hands, Mr Haynes.


Password:
/:
Volume index removed. Index will be rebuilt automatically.
adsl-75-18-187-59:~ rafaelmontserrat$

Heh. :-) All you did was what we suggested originally. No big deal.

The volume "/" is the root volume, otherwise known as the boot volume, or the disk that contains the currently running system software. On most computers, there's only one volume on the internal hard disk.

If you click the Spotlight icon, you'll see the progress indicator for the index rebuilding. The process keeps your computer quite busy, but you can (slowly) do other things at the same time.

Cooel. I just looked at the Spolight message. It said only 102 hours left. OK. I'm patient.

Many, many duplicate files in Spotlight searches Reply to Thread
 

Here's what Spolight Indexing is doing after four hours: "Calculating Tme". A while back, instead of the barber pole was a horizontal bar, moving from the left, but it just seemed to be standing still. That was when it said 105 hours.

RE: "The process keeps your computer quite busy, but you can (slowly) do other things at the same time."

I have /not/ been doing other things 'slowly'. I guess I'll stop the process and run it overnight. How do I stop the process? I looked around in Help and couldn't find out. Can I simply re-start the computer?

just let it keep running, and try to ignore it; it will finish when it finishes... :)

Don't stop the process - just leave your Mac on and let it do what it wants.
It will (with the usual settings) go to sleep after a bit. The indexing will continue when it wakes up - i.e. when you use the computer next.
The indexing should not interfere significantly with normal use of the Mac.

That's a bit excesssssssssssssssssssssssssssssssssssssssssive.

I've deleted many a Spotlight index/database, and 2 hours was
the average rebuild time. Of course, it will depend on the amount
of data files one has, etc., but I think I've got my share. But to me
it sounds like your Mac has some issues. 4 hours should be plenty.

Perhaps running DiskWarrior might de-gunk your disk directory
and speed up things. Or possibly there's some flakey 3rd-party
Metadata importer (.mdimporter) file in either the global library,
or a user library:

/Library/Spotlight
/Users/*/Library/Spotlight

Maybe your volume is just too full (or fragmented, or both).
What does df -lh return?

100 hours just sounds abnormal.

-HI-

I agree that 100 hours is excessive - maybe just the estimate is wrong.
But you definitely should run some filesystem-repair utility (like Disk Utility as I recommended above, or DiskWarrior as recommended by Hal Itosis)

You might also find important messages appearing in your system.log.

I left me computer on overnight and turned the sleep command off, on the assumption that with my not doing any work at all, the indexing would have a better chance of working.

This morning when I clicked Spolight, there was only the customary blank Spolight fill in window. The informaation window as per my attachment of Yesterday, 05:42 PM was no longer there. I put a search in the Spotlight, and when the results showed, there was no change in the problem of multiple duplicates.

I'll start the indexing process again now, but I would appreciate any further feedback. I have a feeling that Sherlock indexing may have become stuck (stopped) with the appearance of the window in the attachment.

I have to work today on the computer today. In regards to what you say above: "... but you can (slowly) do other things at the same time." and "... normal use of the Mac", I believe that the work I'm engaging could be considered as pretty intense, or not slowly, or normal. I don't know. What I'm doing is the whole IPhoto process from downloading pictures from my camera, to culling out pictures I don't want, to then exporting pictures to the 'Pictures' folder in my home folder. During the day, I'm also active on entourage and Safari and on word. Is this normal or beyond normal use? I can work at present without solving the Sherlock problem, so that's not a problem. One thought I have is that 'indexing' is being attempted at the same time that I am adding all this work to the computer. Would that throw indexing off. Again, if what happened yesterday is that indexing got stuck, and stopped, when that attached window appeared, then either attempting to run indexing today, or starting it over tonight might be more successful.

How is indexing stopped, if I want to stop it? I saw warnings telling me to be cautious with any commands that included "sudo".

Of course the 105 hours is excessive, but after that appeared, and when there was still some movement in the indexing process, that hour figure reduced dramatically, I recall to 2 to 5 hours.

I don't know how to look for "messages appearing in my system.log".

"What does df -lh return?" I looked for that in my posts but can't find where I wroted that.

Your detailed feedback is greatly appreciated. This problem goes way back. None of the newsgroup people knew to go as far as to bring terminal into play. As for the dates, the example I showed you had 1999 as the date of the files. Very likely so. Other duplicates have other dates, and I know many of them are accurate; maybe some aren't. Then again, 'time' is not something I pay too much attention to on the computer. My work doesn't need it, except for "insert time" in word (which works)". Otherwise I use my watch.

So I'm starting the indexing now and going to work. Any further feedback is appreciated, and I'd expecially like to know how to turn off indexing since I'm now involved with Terminal and sudo and don't want to screw anything up. Finally, is it so that Transfer is the access to Unix? As though I knew the first thing about Unix. I have eBay to post on, and a novel to complete.

Thanks, Rafael

I started the indexing. As for 'hours remaing' it's started at 21 hours, but falling moment by moment, so I don't think there's anything to worry about there.

Rafael

I believe sherlock is now indexing because the blue horizontal bar at the bottom has moved 3/4" and the time says 3 hours. I've also done iPhoto work. I think the other day I did something that stopped the process.

Sherlock went back to 'Calculating Time' (the attachment) for a couple of hours. I let an hour go by, I opened Sherlock, and it showed it's regular window. I didn't make a search. It must have been like that for an hour, because just now I opened it and it's back to 'Calculating Time'.

The plain sherlock window is there. I put in a search word, and the same multiple files appeared. Nothing's changed. What do you think I should do? the rest of the computer is working fine.

Did you do any filesystem repair ? (as recommended in posts 8 & 17)

By the way, why do you refer to the search as "Sherlock" ?
That is the name of the older search utility in pre-Panther (I think) versions of OS X.
Does the word "Sherlock" appear anywhere on the user-interface?

I have Disk Repair in OS X. I don't have Disk Warrior. I'll read the article. I'll rund Disk Repair. Sherlock? From old habit. Besides, I think it's a better name, /and/ I'm a fan of him and Dr. Watson.

> there was no change in the problem of multiple duplicates.

Then either there are dupes laying around, and you just don't know it.

Or, maybe there are directory errors creeping into the index.
Why you haven't responded to anyone's suggestion about running
a disk repair utility is beyond me. (oops sorry; missed page 2).

I also mentioned mdimporter files. Did you look in those
folders? Was there anything in them worth mentioning?
I wonder if multiple mdimporter files could produce this?

[By the way, when I go from this page 2 of the posts, back to page 1, then return to 2, I lose what I've written. No big deal. I've copied the text.]

There's no way I would ever made that many duplicates. Absolutely. No way.

"maybe there are directory errors creeping into the index."
I don't know what this means.

"Or possibly there's some flakey 3rd-party
Metadata importer (.mdimporter) file in either the global library,
or a user library:"

/Library/Spotlight
/Users/*/Library/Spotlight"

I did overlook this, not quite understanding it. Why, within one of my Mac libraries, would there be "some /flakey/ third party Metadata ... "? Flakey?

What do I /do/ when going into /Library/Spotlight
/Users/*/Library/Spotlight?

"...volume is just too full (or fragmented, or both)."

Don't know what this means either, but I do know that my computer—most of the programs—are running slower than common. For instance, in closing a screen in any window, the red button, there's no instantaneous closing. Takes a bit of time. Annoying. Also in all programs, it's hard to get the cursor only in place, between two letters of a word, for instance. What happens is either the whole word is selected, or a whole paragraph. Have to go slowly to get the cursor to stay without select appearing.
-----------------------------------------------------------------------
Hal Itosis, coming back to your post just now, I think part of it got cut off. The post ends with, "I wonder if multiple mdimporter files could produce this?" For instance the part, "... some /flakey/ third party Metadata ... " has disappeared.

I'll go through all your suggestions. May take some time due to my other work.

Thanks for your patience.

Now is probably the time to restart you computer from your restore DVD and run Disk Utility on your harddrive. It really sounds like some file system corruption.

I've go the same Problem: Every contact in my address book shows up twice in the spotlight-search-results. I've tried everything: re-indexing, booting from install disk and checking the startvolume. The duplicates stay. Any other idea?

Command-click each result to bring up the source of the result. I'm betting that one is from your Address Book and the other is a card from your Dot-Mac attachments folder.

If it turns out they are from the same source, run 'Disk Repair' from the Utilities menu after booting your OS X DVD. This is a sign of a corrupt directory structure.

No – one result leads to the Adressbook, the other one to no folder or application at all.:confused:

I already did this – run disk repair from the OSX DVD twice – without any effect. The duplicates stay where they are.:(