Mail.app ignoring POP3 -ERR response
 

Help! :)

After months of travail, and awaiting for a fix in either 10.4.1 or 10.4.3, I have recreated a problem I have had for a while in Mail.app:

Network:

- Linksys WRT54GS with Parental Controls Enabled (OEM from NetOpia)

Mac System:

- eMac G4 1.25Ghz with 768MB wired to above WRT54GS.

Now, the Parental Controls work excellent in all web based access, and is uniform between both my Wintel machines and the eMac. However, when it comes to eMail, there is a serious discrepancy.

In Microsoft Outlook on Windows, as well as PowerMail 5.x, GyazMail 1.3 and Thunderbird 1.0.x on Mac OS X 10.4, whenever you are NOT logged into the Parental Controls, the receipt of emails via POP3 is blocked, and returns a "-ERR Your use of this service has been blocked by Parental Controls." error in a pop-up window.

However, on Mail.app 2.0+ on Mac OS X 10.4, the client continues with the connection, and the Parental Controls, acting as it should, blocks the emails being received and deletes them from the POP3 account on the server. (This is correct functionality, as it is intercepting a connection for which a user account is not associated on the Parental Controls server, and thus is considered not authorized to receive emails from anyone (think of it as a "limited guest" account)).

This "side effect" on the Mail.app, however, is detrimental to implementing my Mac as the primary mail client at home, since my other family members may not always remember to "login" to the Parental Controls prior to opening Mail.app and/or clicking on Send/Receive, and thus all of their pending emails to be received will be ERASED.

Technical Observation:

Using the Terminal to telnet into the server on port 110, here is an example of the POP3 conversation, which is equal on both the Windows and Mac systems (username and passwords have been generalized for security purposes):

+OK <25783.1136258034@pop08.mesa2.secureserver.net>
USER me@somewhere.com
-ERR Your use of this service has been blocked by Parental Controls.
USER me@somewhere.com
+OK
PASS something
+OK
STAT
+OK 1344 79987762
QUIT
+OK Sayonara

So, as it seems, since the above is the same on both the Windows machine and the Mac machine, I can only summarize that Mail.app "retries" the authentication conversation again, and as you can see, will be authenticated by the POP3 server the second time around. Hence, it "ignores" the initial ERR response.

Can anyone in this forum report this to Apple, to see if they could sneak a "fix" into 10.4.4, or possibly 10.4.5?

Or, perhaps, does anyone know of a work around that will make Mail.app honor the initial ERR response in the POP3 conversation?

Thanks,

Clik

I don't think you will have much luck in convincing anyone that retrying in the face of an initial error is a bug.
I would instead suggest that the bug is in the design of the parental controls system on the Linksys.

I would agree with you, except that as I mentioned above, PowerMail v5.x, GyazMail 1.3 and Thunderbird 1.0.x all honor the -ERR returned. On the Windows platform, Outlook, Thunderbird and OE (which is all I have tested), honor the error as well.

Now, to your point, I looked into the log file that can be generated by Outlook, and it states (after receiving the initial error) "Retrying Authentication"....BUT, it goes through the whole reconnect again, which in essence results in the same -ERR again. Then it reports it to the user.

I would bet that the other Mac OS X email clients also exercise a similar methodology, so again, there just may be something with the Mail.app where it just tries to authenticate again, in the same session, and perhaps the other 6 clients mentioned above, all try a whole new connection.

Anyway, thanks for your feedback, as it clarifies the issue further for others as well. :)

Clik

<<bump back to top>>

<<bump back to top>> Please help, anyone.

Mail.app POP3 skips -ERR response -- HELP
 

I own a Linksys WRT54GS with Linksys Parental Controls enabled on the router. The Parental Controls "intercept" web, email and IM requests, and validate them against certain acceptable rules (as assigned by the Parent).

This solution offers a uniform control across my Windows PCs and my Mac.

I love my Mac, which is an eMac 1.25Ghz with OSX 10.4.4. I would like to have my whole family start using it as the primary email, web computer at home, and perhaps make a transition to a full Mac house within the year.

However, there is one thing impeding this: Mac Mail.app v2.0 will not play nice and respect the Parental Controls.

Last time I posted this (early Jan) I had only executed a simple 'telnet' exercise from my Windows PC and Mac PC to show the interaction with the Parental Controls via a simple POP conversation. I then 'speculated' that Mac Mail.app was skipping the '-ERR' that was being returned by the Linksys Parental Controls system.

However, I am now armed with concrete evidence that I have gathered utilizing 'tcpflow' (http://www.circlemud.org/~jelson/sof...tcpflow.1.html) at the terminal level, to capture the POP conversation of several Mail applications that WORK, and the Mail.app application which doesn't.

Here are the results:

Mail.app v2.0.5

064.202.165.092.00110-192.168.001.103.49955: +OK <23274.1137248072@pop11.prod.mesa1.secureserver.net>

192.168.001.103.49955-064.202.165.092.00110: APOP me@mail.com 1c2b6a9b0539875b663e9ea25aff804b

064.202.165.092.00110-192.168.001.103.49955: -ERR Your use of this service has been blocked by Parental Controls.

192.168.001.103.49955-064.202.165.092.00110: USER me@mail.com

064.202.165.092.00110-192.168.001.103.49955: +OK

192.168.001.103.49955-064.202.165.092.00110: PASS mememe

064.202.165.092.00110-192.168.001.103.49955: +OK

192.168.001.103.49955-064.202.165.092.00110: STAT

064.202.165.092.00110-192.168.001.103.49955: +OK 1355 85250174

192.168.001.103.49955-064.202.165.092.00110: UIDL 1

064.202.165.092.00110-192.168.001.103.49955: +OK 1 1108412777.32256.smtp04.prod.mesa1.secureserver.net,S=1970

192.168.001.103.49955-064.202.165.092.00110: UIDL 1355

064.202.165.092.00110-192.168.001.103.49955: +OK 1355 1137013402.537860.smtp04.prod.mesa1.secure.3007060912

192.168.001.103.49955-064.202.165.092.00110: QUIT

064.202.165.092.00110-192.168.001.103.49955: +OK Sayonara

Quick observation: As you can see above, after sending an 'APOP' (Authenticated POP) command as per RFC1939, Mail.app seems to 'perhaps assume' that the error means that APOP is not supported, so drop back to the normal USER/PASS clear text authentication. That's bad.

GyazMail v1.3.4

064.202.165.092.00110-192.168.001.103.50035: +OK <24832.1137249090@pop05.mesa1.secureserver.net>

192.168.001.103.50035-064.202.165.092.00110: USER me@mail.com

064.202.165.092.00110-192.168.001.103.50035: -ERR Your use of this service has been blocked by Parental Controls.

192.168.001.103.50035-064.202.165.092.00110: QUIT

Quick observation: Notice that GyazMail 'respects' the initial '-ERR' that is returned (as per RFC1939) and subsequently QUITs the connection.

GyazMail v1.3.4 with APOP Enabled (This is a setting that can be turned on and off, unlike MAIL.APP which seems to have it as the default)

064.202.165.092.00110-192.168.001.103.50101: +OK <12562.1137250331@pop06.prod.mesa1.secureserver.net>

192.168.001.103.50101-064.202.165.092.00110: APOP me@mail.com a70ea1a90fb413e0f06a5f344d0a0c20

064.202.165.092.00110-192.168.001.103.50101: -ERR Your use of this service has been blocked by Parental Controls.

192.168.001.103.50101-064.202.165.092.00110: QUIT

Quick observation: GyazMail continues to work as it should.

Mozilla Thunderbird v1.0.2

064.202.165.092.00110-192.168.001.103.49964: +OK <26903.1137248764@pop05.mesa1.secureserver.net>

192.168.001.103.49964-064.202.165.092.00110: CAPA

064.202.165.092.00110-192.168.001.103.49964: -ERR authorization first

192.168.001.103.49964-064.202.165.092.00110: USER me@mail.com

064.202.165.092.00110-192.168.001.103.49964: -ERR Your use of this service has been blocked by Parental Controls.

192.168.001.103.49964-064.202.165.092.00110: QUIT

Quick observation: Thunderbird apparently tries to check what all of the CAPAbilities are of the mail server, but then when trying to authenticate, still fails and 'respects' the '-ERR' response, as per RFC1939.

Mozilla Thunderbird v1.0.2 with APOP Enabled (This is a setting that can be turned on and off, unlike MAIL.APP which seems to have it as the default)

064.202.165.092.00110-192.168.001.103.50109: +OK <28428.1137250549@pop03.mesa1.secureserver.net>

192.168.001.103.50109-064.202.165.092.00110: AUTH

064.202.165.092.00110-192.168.001.103.50109: -ERR Unrecognized authentication type

192.168.001.103.50109-064.202.165.092.00110: CAPA

064.202.165.092.00110-192.168.001.103.50109: -ERR authorization first

192.168.001.103.50109-064.202.165.092.00110: APOP me@mail.com e8e90f493db724dff1af43f6939766eb

064.202.165.092.00110-192.168.001.103.50109: -ERR Your use of this service has been blocked by Parental Controls.

192.168.001.103.50109-064.202.165.092.00110: QUIT

Quick observation: Thunderbird continues to work as it should.

Now, I also downloaded PowerMail and was going to use it for generating a similar capture, but my previous "test trial" period had run out, and the application continued to be expired, even though I uninstalled and reinstalled it. Nonetheless, I know from previous testing (without the packet captures with tcpflow) that PowerMail also stopped the POP3 session with the prescribed '-ERR' listed.

You may say "then why not use one of the apps above?" Well, I like the integration of Mail.app with iPhoto, the cost (FREE) and the integration with the remainder of the OS. I would like to see the above problem, which is now listed in detail, hopefully resolved in a 10.4.5 release (?).

Anyway, thank you all for your review of this posting, and if you have any workarounds, it would be greatly appreciated. (I would like to get a Mac Mini this month for my wife, but I can't unless this works.)

- Clik

I think this is a misstatement of the situation.
As you have shown in your tcpflow output, what is happening is that Mail.app is not issuing a QUIT when it gets an ERR response from the APOP command. Mail.app knows nothing of the "Parental Controls" - it only knows that it got a -ERR response. The question is what should a mail client do in this case. As you have shown, Mail.app proceeds to issue a USER command while other clients just QUIT.

In my reading of RFC1939, it is not at all clear that the behaviour of Mail.app is incorrect. RFC1939 does say that the POP server should not respond to both APOP and USER and that the client can detect if APOP is supported by means of the <timestamp> in the banner greeting, but it doesn't say that the client must QUIT when it receives a -ERR from an APOP.

I think you should submit a bug report to Apple:
http://developer.apple.com/bugreporter/bugrptform.html
and see what sort of response you get.
Please report back here so we will know if Apple considers this "working as designed" or not.

If the server-side is supposed to be disallowing a login, it should always be returning a -ERR when a client attempts to login; Mail.app is trying APOP, gets told there's an error, so assumes APOP isn't supported, so tries USER/PASS.

Note that in section 13 of RFC1939, it states
so there's no way a client could know that APOP is right or wrong when it tries and fails.

Definitely a server-side issue.

Would agree with you, but Mail.app is working the same way whether it is a BELLSOUTH Mail Server, GoDaddy.com Mail Server, HotMail POP Server, and i even tried my buddy's account on his work's Public Exchange mail server, all with the same result with the Mail.app client.

Hence, the following clients are all working as they should:

- GyazMail on Mac
- PowerMail on Mac
- Mozilla Thunderbird on Mac/Windows
- MS Outlook Express on Windows
- MS Outlook on Windows
- Eudora 7 on Windows/6 on Mac

All of them recognize and respect the -ERR, and in addition, nearly all of them offer the "option" to enable APOP, and when enabled, also work as prescribed, and stop on the '-ERR'.

I believe that because, somehow, Apple has integrate APOP as the "default" command, with an automatic fallback to "USER/PASS", that it is directly contrary to the RFC1939 quote you found earlier.

I just believe that Mail.app should:

A> Provide APOP support and usage is an option that can be turned on/off

B> Have USER/PASS and APOP both respect an initial -ERR and QUIT

C> Have the APOP command not try to revert back to USER/PASS if an -ERR occurs, since the expectation would be that the server is following RFC1939, and will not allow both authentication types in a single user session.

Again, I am not trying to invent something new. Just trying to have
Mail.app behave like its other pay to play and free counterparts.

Thanks for your response. :)

The above seems to be saying that you think Mail.app is behaving in a way which contradicts the paragraph from RFC1939 quoted above. But the paragraph from RFC1939 is stating what is expected of the POP server - it does not apply to the client.

True. The paragraph in RFC1939 is dictating the behavior of the server, not the client.

However, I will stand by my statement that "knowing" that the server should or would behave in a particular fashion should preclude the client from acting in a contrary way. So, all of the clients I listed above do not fallback to USER/PASS if APOP returns an error, probably in adherence to the RFC since the server shouldn't allow for it anyway. Mail.app does.

All of the other clients have an option to enable/disable APOP, and once enabled, only it is used. Mail.app assumes APOP first, and then falls back to USER/PASS. To me, this is an inappropriate behavior.

(BTW...thanks for the ADC link. I posted the details of the issue, and will post here as to Apple's findings or resolution, if any.)

Thanks again. - Clik

The thing is, in the case where a server does support both APOP and PASS, and you are using Mail.app with a user which doesn't have server support for APOP but does for PASS, how else would Mail.app know to fall back to PASS when APOP fails?

Good point, and although I have not researched it further with the existing clients, how do all of the other Mac Mail clients I listed previously do it successfully, and still honor the -ERR message returned by Parental Controls?

Perhaps by having the APOP usage be a Preferences toggle, and when you turn it on, if an error concerning 'APOP support' is returned from the server, the client must actively toggle it off (?) and try again (this is the way it works with everyone else I listed earlier).

Also, the Mail.app client could still achieve the same effect of 'auto-switching' from APOP to POP (USER/PASS) by QUITing and then re-establishing a USER/PASS session, and subsequently honoring any -ERR message that is returned, just like its counterparts I listed above.

Again, I am not trying re-create the wheel, or provide logic that hasn't already been implemented and tested. At this point, it is obvious the 'auto-switch' functionality that Apple chose to implement for APOP to POP fail-over is what is not cooperating with the -ERR logic of how the client reacts and handles the -ERR.

All I can say is that all of the other respected mail clients for Mac work, and perhaps someday Mail.app will to. :)

- Clik

That may well be what Apple will decide to do in response to your bug report.
However, I'm curious (and the answer to this is probably something that you should include in your bug report as well) why you don't think that the bug isn't with the Linksys Parental Controls implementation. I.e. can you explain why its behaviour is reasonable? I would have thought that any parental controls should work properly independent of whatever strange behaviour the client programs have.

Note in this regard that "honoring any -ERR message that is returned" is not the correct wording to describe what client programs should do. A client program gets a -ERR message and then it acts on that information. It doesn't "honor" it. The security of the email transaction is not reliant on the client upholding some sort of contract - it is up to the server to provide or to deny service.

My guess is this may be a usability issue, someone at Apple may have thought that simply trying PASS when APOP fails is simpler for the user than having a pref and possibly making users wonder what's APOP?; but of course, this is all conjecture.

Since, when -ERR is returned from APOP, the server is still in AUTHORIZATION state, the USER command is supposedly still valid. Of course, the RFC (1939 at least, I haven't checked some of the others for specifics on this) doesn't explicitely state how this kind of thing should be handled...

I do agree with you here, and have wondered "why" does the Parental Controls engine simply allow you to resend the "USER" command, and then get right through.

As a quick tangent, to note, the reason why I am emphatic in getting this fixed/changed, is because I want to use Mail.app as my mailer (I like it), and because this "issue" (if we don't want to call it a bug) actually causes data loss. The reason:

- Mail.app sends APOP, and gets a -ERR, then send USER/PASS and gets in.

- Parental Controls do not see an authenticated user, so use "guest" instead

- Mail.app sends a RETR command, and the Parental Controls (seeing me as a "guest" with no email addresses in the accept list) intercepts the emails, and deletes them from the server, while returning an "Administrator Blocked" email for each that it blocked.

Result: Emails are gone forever, simply because the Mail.app/Linksys Parental Controls combination didn't work well together to stop the process early when the initial -ERR was received.

Although that may seem destructive, it is the correct way to function, because my expectation as a Parent is that if any of my children receive a porn spam, that it will be erased from the server, and no longer available for retrieval by other mail clients or via a web interface.

Anyway, back to your point, I am pursuing the Linksys route as well. However, I still see the problem as specific to this client due to the percentage of clients that work (99%) as they should. To prove the point, my buddy came over with Microsoft Entourage yesterday, and I tested it, and it worked.

So, here is an updated list of additional clients that work:

- Microsoft Entourage for Mac
- GNUMail.app for Mac (downloaded yesterday)
- Mailsmith by Bare Bones with Mac (from my buddy's laptop)

At this point, every other client works with the Linksys Parental Controls functionality, and most of them offer APOP as an "option", and the others don't offer it at all.

I believe that Mail.app is the only mail client with the auto-switch from APOP to USER/PASS feature, and this is (at an industry level, perhaps not an RFC level) is non-standard functionality.

Understood. Perhaps the better word is "aknowledge" and "accept".

Thanks - Clik

Waitress: Good morning.

Customer: Good morning. One coffee please.

Waitress: Sorry, we don't serve coffee here.

Customer: (pauses)

Department of Homeland Security (waiting in the shadows): Hmm, why isn't he leaving? He was told they don't have what he asked for. Suspicious behaviour - must be a terrorist!

Customer: Okay, then I'll have a tea.

Waitress: Here you go. (puts a cup of tea on table)

Department of Homeland Security: We'll take that! (grabs tea, pours it on floor, ...)

Funny how sarcasm even translates well electronically. ;)

Anyway, I will hold on to my belief that the Mail.app client should conform to industry norms, even if those norms are not full possible standard behavior. :)

"11 out of 12 mail clients acknowledge the -ERR message and stop, while the 12th simply drinks tea..."

- Clik

Seems there might be other ways to resolve your issues with the Parental Controls and "guest" logins causing problems. On a setup such as mine, for instance, all the mail is pulled onto the local system via fetchmail, so regardless of the mail client being used, the only issue would become whether fetchmail plays nicely with Parental Controls. Unfortunately, this is rather difficult to set up and get working correctly.

Can the login to Parental Controls be scripted in some way? If so, it should be possible to create a Login Item script that automatically logs the user into Parental Controls when they log into their user account. This might be the simplest workaround for the problems you describe.

I like your idea, and I tried looking into it previously. Let me ask you this, is there a Mail.app "started" event that I could have trigger an AppleScript or Automator script to check the existence of a valid connection to mail server, otherwise prompt for a Parental Controls login?

- Clik

I'm not sure of the benefit of delaying logging into Parental Controls until Mail.app starts, though I assume there's good reason for this.

What might be most useful is if you could describe exactly how you log into Parental Controls as opposed to simply logging into the mail server (I'd think the mail login would be in each user's keychain). How do you invoke the PC login? Can you do this login from within Terminal? (If so, it can probably be scripted.)

I'm imagining your router is basically acting as a mail proxy, but this may not be a good model. If you can give us some details about the steps required to perform both PC and mail server logins, as well as about how your users mail accounts are set up relative to the computer's user accounts, some one here might be able to piece together a solution.

In answer to your question, I don't know of any Mail.app startup handle you could use for this (I use GyazMail btw, so my familarity with Mail is a bit minimal), but you could easily set up a script to periodically check whether Mail.app is running and trigger the required login script.

I agree - it would seem to me that this would be better done as a login item - so it happens automatically at login.

Anyway, one way of running an AppleScript when an app starts up is to replace the app with a wrapper AppleScript that does whatever you want and then launches the (renamed) app.