Lion/IMAP issue
 

I've been having an issue with Lion's mail.app since I began using it about 2 months ago. Spent many hours talking with Apple Senior advisors who exhausted their ideas for fixing it, and it's now in the laps of the Apple engineers. I have no idea what the engineers are doing to solve it. I am told that there are other, similar issues with Lion Mail, but none quite like mine.

Nutshell: Lion Mail will not correctly show all of the email which the web server says is there. Yet, lions appears to "know about" the correct number: when I make the accounts fresh and force mail to pull in the email, mail.app will show the correct number in the little activity bar, but after it's done syncing, it won't actually show ALL of those emails.

Mail.app in Snow Leopard on the same computer, same mail accounts, does show the correct number, as does a G4 Mini running Tiger.

And, a friend with a similar computer as mine, running Lion, was able to duplicate my situation on his machine, accessing my email using Mail.app on his rig.

My question is: based on this data, can anyone offer clues as to what the problem and solution might be?

I can function just fine in Snow Leopard and there's no huge burning need to be in Lion, but eventually it will become more of a practical hindrance not to be in Lion.

thanks for any thoughts.

a

Do you have more than one IMAP server to test this on? I had serious problems with IMAP on Godaddy servers which they refused to even acknowledge so was forced to switch to another vendor's server whence all issues disappeared. In that case, it was not limited to Mail.app but also appeared intermittently on Thunderbird Linux as well. If the IMAP server is not implemented exactly right, it may work on some clients but not others. The fact that, in your case, only Lion is affected, suggests bad serendipity with something not quite Kosher on both sides of the equation.

Sorry to not be of more help.

I could probably create an email account through my ISP to test things out. My hosting company, BlueHost hasn't given me the feeling that they are really engaged with the issue. I've asked them more than once, more than one person about it and I get extremely short answers, and the feeling they can't be bothered.

they all claim that there are Zero issues with OS X and IMAP at Bluehost, yet one person flat out said that he doesn't know what OSs he's dealing with.

So...apathy? Lights on/nobody home? who knows?

I don't know whether I have enough/any information of the kind where I could tell them "Look, here's the evidence, please deal with it."

But I will make some time to try your suggestion with another account.

thank you!

a

Hmmm, interesting, Bluehost is the place I went to solve my Godaddy issues. Sorry to hear they were not helpful. I guess Apple is interested, but also not helpful (yet). Maybe Apple needs to create an account on Bluehost to allow them to test the issue themselves since it seems reproducible. Then they could mutually debug the issue.

P.S. I am NOT using Lion... although I tested it a while back... did not see this issue, but then, I was not looking for it either.

Funny you should say that...I created such a test account for them for just that purpose—at their request—about 3 weeks ago. I see no evidence that it's been used by them, but the senior advisor helping me with the problem says they're testing in ways that I wouldn't be able to see...whatever that means.

I do not know what all is involved in getting clients and servers to interact properly, but my fear is that this is an elusive problem which is destined for the Void of Space where it will turn into a little ice comet and leave the galaxy.

a

How many messages in the folder(s)? I find that when a folder gets too large (over 2GB) you will start to see problems. The Inbox in particular needs to be as clean as possible.

Next is your schedule. Are you checked ever 2 minutes? Most providers like you to use 10-15 min to check. If you check too often the handshake will hardly complete befoer the next schedule runs.

Have you run rebuild on the Inbox?

Have you tested in a new User? This would eliminate any issues in your User's folder.

I use Hostmonster and they are very similar to Bluehost and have no problems.

pretty huge. many messages.

I have it set to every minute in Snow Leopard and Tiger without issue.

Yes...several times. Also started from scratch rebuilding the entire email accounts letting it pull them in. same results.
Yes..Several times.

switching from BlueHost is on the A-List. I don't like how they are very tightlipped as they say "no problems."


BTW..thank you for making all of those suggestions..even tho I've tried them all several times and several combinations thereof, they are most worthy suggestions anyone in my situation would need to try.

a

I would definitely work on folder sizes. Especially the Inbox.

your feeling is that Lion may be more particular about folder sizes?

if so, the remedy would be to make chunks of that email local, "on my Mac" email?

I don't know if this is a Lion thing, but I can say from experience that users with too many messages in a folder have issues.

Following up on dianeross' suggestion: in Snow Leopard (where mail.app does accurately reflect what's on the server) I severely trimmed my mailboxes, making each one mostly "On My Mac" mail, so that the live on-the-server mail counts were only a few hundred emails per, at most.

I then rebooted into Lion, let mail.app there sync up with the server, and the problem persists. Where the number of emails is well under 100, Lion mail gets it right, but 100 or more, it doesn't.

thank you for any more clues or ideas!

a

Same issue, more info...
 

I'm glad to see others with this issue, I've submitted feedback on this issue with each Lion version, including the current one (10.7.2).

I can confirm that the size, hosting, etc. is not the problem. My mailboxes show correct unread messages on Thunderbird, and webmail, and did on Snow Leopard, as well.

On my iMac (2007 model), the machine is slow enough to show that the unread messages count is being written correctly as the mail is downloaded, and then overwritten with 0 or, in some cases with 1, no matter how much mail is in the box.

In addition, one of my boxes had the problem, but, when I renamed the box (?), the problem went away for that box. When I named it back again, it reappeared.

This is a very obvious and repeatable bug in the software. For those who have a small number of boxes, the possibility of renaming the boxes until you get correct behavior exists - I have too many boxes within a multilayer directory structure, just thinking about it makes me tired.

For now, I boot Thunderbird to see where I have new mail, and then use mail to look at/reply to the messages. It's idiotic to resort to that, and, in the presence of bugs...

Hope this helps.

Since posting, I have learned or discovered that some of my email which presented no issue in Snow and Tiger Mail.app, nor in Thunderbird in ANY version of OS X, may have had some quality about it that prevented Lion from counting properly.

Now, to my reasoning, the presence of "bad" email doesn't let Lion or Apple of the hook: either there are x-number of emails in an account, or there aren't and bottom line is that Lion mail can't get it right when other mail programs/OS X combos do get it right.

I have no clue what explains the disparity, but I've already invested an embarrassing amount of time on just this one issue, several folks here have lent heroic levels of help and suggestions to me trying to solve it.

On the one hand, this is the most serious Mac OS problem I've had in 20+ years using Macs...that's not too shabby..

On the other hand, it was a black hole of time and I will never go down that road again.

I feel that, cool as it is, Lion isn't "done." There are serious interface issues which made me go back to Snow.

We're already hearing about the next cat: Mountain Lion.

regular Lion needs more time in the oven, IMHO, but I'm not running the show...

those are my 2 rupees...


a

That comment got me thinking.

What if there are duplicate emails in the account? That is, for whatever reason, the exact same message was placed into your mailbox twice. (I can think of a number of ways this could happen, from timing issues to lost acknowledgements. For example, the message comes in by two different routes. Or a receiving server says "I got that" but the sender doesn't hear the "I got that" and sends it again.)

Which would raise the interesting question: if the exact same message was sent once but received twice, is it two emails or one? Maybe Lion is the one that takes the time to notice the error, and the only one to compensate.

I'm not saying that's what happened in this case. It's just that things are not always as straightforward as we might naively assume, and x does not always equal x.

Let's assume that's the cause here. acme has both Lion and Snow Leopard available, with the respective Mail versions showing different mail counts. If one of the ~/Library/Mail folders were copied to the other OS so both the 10.6 and 10.7 folders are on the same volume, would a file-comparison tool (e.g., TextWrangler, etc.) enable acme to finally see where the discrepancies are?

while I did a lot of side-by-side numeric comparisons of server-versus-Lion mail, I didn't do in-depth comparisons of individual email.

Some, yes, but we're talking a total email count in the realm of 145,000 emails, so...not something I was up for..

but I did look at individual subject lines, and, as I recall, there were non-duplicate discrepancies.

IOW, discrepancies that couldn't be explained by duplicate emails, but ones in which some emails which were on the server side but were not on the client side in Lion.

as NaOH points out, Snow Leopard mail got the count right, Tiger mail did, and Thunderbird got it right in both Snow Leopard and in Lion!

When I finally discovered that the culprit was likely these "bad" emails, I'd racked up as I said an embarrassing amount of time on this problem. Working with a senior tech advisor, we talked about some more experiments I could do if I really wanted to get to the bottom of why the "bad" emails were bad, but by that time, I was crispie-fried on experiments, copying, processes of elimination, the try this n that..

If anything, Apple got them a whole LOTTA free field R&D thanks to yours truly. I hope they fix whatever it is.

but I'll tell you what...anytime I even think I smell another problem like that rearing it's ugly head, I'll switch to thunderbird or whatever alternative available so fast that whatever Cat that is will lose its own shadow!

cheers!


a

I think the point that ganbustein was making is that it's possible under specific circumstances that Lion actually has the correct count.

If you want to pursue the file-comparison test I mentioned—yes, I realize you've spent a bit of time already and are reluctant to waste more, but this wouldn't take much effort on your part—it's quite simple.

1. Copy one ~/Library/Mail folder from one OS to another OS. Personally, I'd do this with 10.6 and 10.7 copies, nothing earlier.
2. Use TextWrangler to do a folder comparison using Search > Find Differences.
3. Once it's done, you'll get a list of different files and you can likely use that output to help you get the missing content to Lion (whether by forwarding the messages to yourself, copying the missing files to Lion, etc.).

The bulk of the work is handled by Text Wrangler, a little is spent by you reviewing its results.

OK..that sounds painless enough...lemme set myself a reminder and give that a whack tomorrow.

thanks!

a

If you're not already familiar with TextWrangler, just be certain when you pull up the Find Differences dialog to select the option for Compare Folders (rather than files). For all the other options, I would enable Case Sensitive and disable all others.

One other idea occurs to me to simplify the results you'll see. In the Find Differences dialog, select the Use File Filter pop-up menu and choose New. You'll be prompted to give a name for your filter, then in the following dialog create one for File Name Contains emlx. This will make it so only email messages are being evaluated in the selected folders and will save you from having to review results for non-message files.

NaOH; does this experiment's success depend on which way I go? IOW, is it better to copy from Snow and view in Text Edit folder comparison in Lion, the other way around, or is it inconsequential which way I go?

thanks!

a

As far as I'm aware, it shouldn't matter which way you go. I did suggest taking 10.6 and 10.7 simply because they're OS-adjacent versions and I figured that setup was less likely to have differences between message formats, etc.

Right on...ok...shud be able to do that this morning.

Now, I've since cleared out the "bad" emails by making them "On My Mac," so they are no longer in the Server/Client chain of communications.

Does that fact obviate this test?

a

It's possible that it will negate the test. I just ran a simple test under Lion Mail. I got the Finder ID of an email in my inbox. The ID was 17012, so that means the file name is 17012.emlx. I moved the message to an On My Mac mailbox and the ID changed, so it's Finder name also changed. I moved it back to my inbox and the ID and name changed again.

Here's what I'm thinking: Try the comparison test. If the results are unwieldy and your not inclined to wade through them, then you can abandon the test without much effort having been spent. If the results aren't off-putting, you can proceed.

Here's another idea to hopefully mitigate the issue of moved messages: Create a new user account in Lion and set up the Mail account while importing it from 10.6. If I'm understanding you correctly, that should replicate the counting discrepancies you'd been seeing. You could then run the file-comparison test in that account. This assumes I'm properly understanding your current setups. The point is that, if possible, you want to re-create the 10.6/10.7 message-count issue before running the file-comparison test.

aight..

will do