Mail doesn't agree with server: 10.6, 10.7
 

I've been having an issue with Lion Mail.app since September: the local client doesn't show the same mail count as the web server.

IMAP set up.

I set up a test email account for Apple Engineers to use to try to re-create the problem. They say they can't but when I compare the numbers of test emails I have in both Lion and Snow Leopard email, both are less than shown on the web mail account for that email address.

Last time I checked, Snow leopard was accurately showing numbers of emails for other accounts and the various folders.

Any clues on what's up here? I can't get my host to look at the issue. I describe the problem and troubleshooting I've done, and they push it back onto Apple.

Apple says they've exhausted the limits of what they can do.

Yet, here I sit, with a dubious email situation. I would like to return to Lion but I can't until I have functional, accurate email.

Thoughts or clues would be most appreciated.

thank you!

a

You're leaving out lots of essential details... :D
Just some things that I thought about while reading your post, so others might have some better ideas if you can answer these queries...
You said that both SL and Lion Mail show different amounts of mail, and both are less than the webmail - yet you also say that SL Mail number is accurate.... If SL also shows fewer mails, then which is correct, and why do you trust SL to be more accurate, when you say it also shows fewer emails?

I suppose that you have determined that there is nothing in common with any 'missing' emails - like a tighter spam filter in Lion, for example?
Are you losing emails that YOU sent, emails that you received, or emails that you have stored off the server, etc?

DeltaMac; My bad for typing such a squishy accounting..I'll tighten it up here.

1. Snow and Lion agree on how many test emails there are in the test account being used by Apple, and both are less than the Webmail for that account says...by quite a lot; probably about a thousand or so.

2. I trust Snow more because, after discovering and going through exhaustive (and exhaustING) troubleshooting, I decided to make Snow my main working OS, and did a side-by-side comparison of my email accounts and their folders, and found Snow to be in agreement with the web server. Doing the same procedure in Lion showed that Lion is not even close, in some cases, it's off by thousands of emails.

3. I don't believe that any spam control on my end could be at fault...I have things set up to appear in the junk folder, then I delete or rescue manually. One folder was off in Lion by about 1000 emails; they weren't anywhere, including "Junk," and many hundreds of other emails that should be in that particular folder WERE there, and all from the same mail list, so having the same "From" and "Reply-to" info. Besides, actual spam does squeak through, despite the list mom's best efforts...

I don't use any spam doo-dads...just whatever Mail.app itself provides.

4. I don't know whether I'm losing emails that I send..it is possible, but I do frequently check my "Sent" folder to make sure that this or that email actually did go out. I haven't had anybody squawk that I haven't answered them. Not scientific, but...that's the best I have at the moment.

When I say the troubleshooting was exhaustive, Apple support and I did virtually everything short of nuke and paving. We trashed the email and settings on the Lion user several times and started all over, did so in a new user, did so again in another new user. Same results every time.

Weird thing, when Lion mail was pulling in mail into this particular folder, the progress bar "knew" how many emails were on the server: 1580 IIRC. but after synching was done, Lion mail would only show about 600. Snow, at that same time, would show all 1580. The number has since changed of course, but Snow got it right.

I remember the numbers because tech support and I were using that and a few other mail folders to gauge our success.

If there are other details needed, let me know...

I have talked to my host, but it feels like they are stonewalling. They won't look at it, they won't check anything out. They hear my story and as I said, blame it back on Apple.

It's a situation of being dependent on 2 entities which have more power and knowledge about a situation than you do, and if you can't come up with a compelling enough technical argument, you get shuttled back and forth like a ping pong ball.

thanks for any help!

a

You have of course Synchronised your mail fully, rebuilt the mailbox, and also dumped the /Users/username/Library/Mail/Envelope Index which invokes a complete re-indexing of local content.

I have also found Lion Mail not to handle IMAP/POP well as Snow.This is very true if you have a huge number of mails per folder on IMAP server. I have users with 10-20k in their inbox and things are not working right. I expect that as most other sites who have controls have a max of 5k messages per folder which inproves overall performance IMHO.

PS during big syncs you have to stop the activity, shut mail then reopen and let it sync.
When i pulled down 80k messages for a user the other day it took along time to come into check and even now is off by a bit but certainly not 1000s.

The user is currently moving to no more than 5k per folder and has informed me that these folders have synced with exact message counts with webmail.
I must admit we use special batch scripts to move mail on the server side but depending on how good the webmail system is you could use that and let folders sync down.

Yes. Several times, and in 3 users in Lion.

Another OS Hints poster said similar, and advised me to reduce the number of emails online. I did so, by making huge chunks of my mail local, "On My Mac" mail.

It didn't fix the issue. Many/most of the mail folders in Lion Mail.app were off compared to the web server.

For Lion, is the solution to simply not use Mail.app? I don't want to get dependent on Outlook...I'd rather go open source, really...have you or anyone else discovered whether, say, Thunderbird is accurate with IMAP?

a

Now, that I am not sure I did exactly as you describe above.

what do you mean by "stop the activity?"

do you mean...set check mail manually, and not at the pre-set intervals?

a

And yes sometimes things on Server Side do need doing (indexing, rebuilds, optimisations, restores etc). But most cheep/free ISPs will understandably not do anything ! If you have a paid service then they should be able to support you.

I can send a ticket to most of my suppliers they will look into it and sort it quickly and efficiently.

You did not have to make them local ideally, they should have been filed into folders on server on webmail. And doing that does not sort the issue as the mails stay on server unless you delete them ?

OK..good to know...I'm in the process of shopping for a new provider. There've been a couple of things I've chafed at but this is really over the line.

Not sure what the line is between cheap and not cheap. I think we're paying $/6 mo and some others that come highly recommended are similar in cost, or a bit more. Pair is $10 for a corresponding level of hosting...

a

I am no mail whiz, but my understanding of IMAP is this...if it's on the server, it's there until you delete it either locally or on the web mail side.

so my interpretation of "reduce the number of mails" was to delete them from the server using Mail.app, and since I wanted to keep the content, I simply made them local.

beyond that, I am in Unknowne Waters...



a

Stop the activity ( open up mail activity monitor - stop the sync )

Thunderbird/Outlook are pretty good with IMAP but most programs will when peopel abuse mailbox count. Exchange 2007 admins pretty much limit mailbox count to 5k for a reason as Outlook 2007 starts to choke syncing up and puts more load on server.

OK..I did not do that with the sync. Basically what we did (with an Apple mail specialist on board) was to wipe the Mail accounts and settings and set them up again, and when set up, let Mail do what it does.

that 5K limit...is that all mails, or 5K per mail folder?

if it's 5K total emails, then I am WAY beyond that, even now!

a

Yes. Your understanding of IMAP is correct.

I would never just start deleting stuff of IMAP server to troubleshoot, that is your master content ! which all other devices sync from.

you did not need to "reduce the number of mails" by deleting just by filing into smaller more manageable folders. Each folder should sync better and quicker with less mails. Each folder has its own index rather then one huge master index for most of your mail.

To make it clear, either do your filing on a known working machine or from webmail interface. Not on your troubled machine. Once you have the IMAP side in order start fresh and sync down, be patient I have seen it take days to sync up fully with some forced sync action now and again + message rebuilds + envelope index trashing.

Also do not try and move too many mails on client side fine on server side from webmail.
ie. do not select 10k mails and move them into a new folder. I would say 100-500 any more "can" choke the IMAP connection depending on how server is setup. But test with a few batches first.

5000 emails per folder. This is not a precise science just how the EXCH admins have always worked. things like Speed of machine, connection, client program, server all are factors in this equation.

But overall first sync can always be a bit sticky so i am not surprised.
Sync up outlook and remap IMAP folders correctly and see how that fairs.

OK..excellent points to keep in mind. Since I did move vastly more than 500 emails at a time, have I "broken" something?

This is now going back to late September/Early October of 2011...

To clarify, I didn't "delete" email so much as I moved it to "On My Mac" folders, thereby, I presume, causing it to become removed from the server.

In future, I can make other folders of still-on-server email, but out of the main folder in order to reduce the per-email load.

One factoid I did notice when doing the side-by-side comparison of Lion mail to web server: after reducing the amount of email in the various folders, I noticed the amounts to be in agreement in folders below 100 emails! I can't be sure what the magic number is, but closer to and above 100, the counts were off.

100 is a pretty anemic figure for email in this day and age...


a

same results obtainable with something other than Outlook?


FYI, the total numbers of emails I recall being moved, as the numbers whizzed by was in the neighborhood of 145,000 emails.

Not sure where that puts me in the universe of email, but I know there are some with vastly more email than I have.


a

Of course people have lots if it. I am a sysadmin and I am seeing people with 500k messages stored 28gb of mail ! in reality it is digital hoarding ( I am guilty too). I mean after 6 years all my paper stuff gets shredded don't do enough digital shredding. I keep saying "do you really need all of this ?" I get a firm yes !

The key problem with Email is the insesenent internal company chit chat ping pong file sharing ineffencies. I did an attachment test on mailboxes and one 8mb attachment was taking over 1.25gb on server so much replicated data !

People should be using a mixture of comms IM,vid, audio, phone calls, face to face as email has become a bloat of long signitures HTML newsletters with offers that have expired....you get the picture. Al this data being backed up too......bloat swell.....

what a headache that must be for you!

times I was hooked up to the mothership and the IT czar said "clean up your emails, everybody" I did it, and stuff that I just HAD to have, I saved it out as text.

..which is basically my vision for saving old email as "On My Mac" mail. I get the benefit of search within mail, plus if I need to reply to something in 2006, I have that person's email address right there.

mainly, it's an informational resource..I belong to a few lists and software forums..I view it as a library...

truth be told there are a number of "I love you, sweetie" and "what's for dinner" emails that probably aren't critical to life itself.

;-)

a

Just a little rant.....overall I do not get that upset....freedom and all that.

Downloaded thunderbird 8.0 and set it up for the email account where Snow mail and Lion mail agree, and which is less than what web mail shows for that account.

Thunderbird also shows what Snow and Lion show, which is less than what web mail shows.

Does this help point a finger at the host?

or, what DOES it point to as an explanation?

a