Google is Messing with SMTP Headers and it is screwing up HTML Emails (SOLUTION BELOW)
Our web server sends out tons of emails to customers and employees. We
use Google Apps for our email and iPhones at times to read them too.
Something strange started happeningon 4/25/2012. HTML Emails started
not working on 4/25/2012. Nothing has changed on our servers, emails
juststarting displaying headers and not looking right. Keep reading, we
know why it is happening...
  
The first image show a daily email that started displaying funny on
4/25/2012. Notice that the "From Address" shows "<>". The middle
screen grab show what the email looked like before 4/25/2012, email
looks fine. Then for the last few days, the email looks like the image
on the right. Starts off showing the header: "Content-type: text/html;
charset=iso-8859-1" and then a bunch of email headers. If I scroll
down, I will see the raw HTML of the email, but why? Nothing changed on
our end. Something did change on Google's end.
Email Headers before 4/25/2012
To: xxx@3gstore.com
Subject: Router Advisor Click Log 2012-04-23
MIME-Version: 1.0
From: 3Gstore.com <xxx@3Gstore.com>
Message-Id: <20120424050102.8871E4B05A3@mdg-web.localdomain>
Date: Tue, 24 Apr 2012 00:01:02 -0500 (CDT)
X-pstn-neptune: 0/0/0.00/0
X-pstn-levels: (S: 1.35367/99.90000 CV:99.9000 FC:95.5390 xxx )
X-pstn-dkim: 0 skipped:not-enabled
X-pstn-settings: 2 (0.5000:0.5000) s cv gt3 gt2 gt1
X-pstn-addresses: from <xxx@3Gstore.com> forward (user good) [2172/88]
X-Original-Sender: xxx@3gstore.com
X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain
of xxx@3gstore.com designates 74.125.149.61 as permitted sender) smtp.mail=xxx@3gstore.com
Precedence: list
Mailing-list: list xxx@3gstore.com; contact xxx@3gstore.com
List-ID: <xxx.3gstore.com>
X-Google-Group-Id: 725066998195
List-Help: <http://www.google.com/support/a/3gstore.com/bin/static.py?hl=en_US&page=groups.cs>,
<mailto:xxx+help@3gstore.com>
Content-type: text/html; charset=iso-8859-1
X-pstn-neptune: 0/0/0.00/0
X-pstn-levels: (S:59.71620/99.90000 CV:99.9000 FC:95.5390 LC:95.5390 R:xxx )
X-pstn-dkim: 1 skipped:not-enabled
X-pstn-nxpr: disp=neutral, envrcpt=xxx@3gstore.com
X-pstn-nxp: bodyHash=df69e09be9f37e07b8380517f9e5f7c11537df73,
headerHash=62a12f02680bb6d95f35bf5420a5a43fbcb7e920, keyName=4,
rcptHash=01f0a189360825ca14f163766b98db238208c87a, sourceip=74.125.149.244, version=1
<strong>Total Routers Clicked: 8</strong><table
border='0' cellpadding='0' cellspacing='3' width='100%'><tr>
Here is exactlty the same email that is having the display issues:
To: xxx@3gstore.com
Subject: Router Advisor Click Log 2012-04-25
MIME-Version: 1.0
X-Original-Sender: xxx@3gstore.com
X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain
of xxx@3gstore.com designates 74.125.149.55 as permitted sender) smtp.mail=xxx@3gstore.com
Precedence: list
Mailing-list: list xxx@3gstore.com; contact xxx+owners@3gstore.com
List-ID: <xxx.3gstore.com>
X-Google-Group-Id: 725066998195
List-Help: <http://www.google.com/support/a/3gstore.com/bin/static.py?hl=en_US&page=groups.cs>,
<mailto:xxx+help@3gstore.com>
X-pstn-neptune: 0/0/0.00/0
X-pstn-levels: (S: 3.77844/99.90000 CV:99.9000 FC:95.5390 LC:95.5390 R:xxx )
X-pstn-dkim: 0 skipped:no-from
X-pstn-nxpr: disp=neutral, envrcpt=xxx@3gstore.com
X-pstn-nxp: bodyHash=a4ea0efb001cfaf7335141c3df807a506f7bbc3d,
headerHash=3151a48c99d25e8180b6aa61d8fdd1ba72e93b47, keyName=4,
rcptHash=01f0a189360825ca14f163766b98db238208c87a, sourceip=74.125.149.240, version=1
Content-type: text/html; charset=iso-8859-1
From: 3Gstore.com <xxx@3Gstore.com>
Message-Id: <20120426050102.9E5BE4B03E9@mdg-web.localdomain>
Date: Thu, 26 Apr 2012 00:01:02 -0500 (CDT)
X-pstn-levels: (S: 4.43673/99.90000 CV:99.9000 FC:95.5390 LC:95.xxx )
X-pstn-dkim: 0 skipped:not-enabled
X-pstn-settings: 2 (0.5000:0.5000) s cv gt3 gt2 gt1
X-pstn-addresses: from <xxx@3Gstore.com> forward (user good) [2172/88]
X-pstn-nvpr: envrcpt=xxx@3gstore.com,disp=neutral
X-pstn-nvp: sourceip=208.67.207.101,bodyHash=xxx,headerHash=xxx,sig=xxx
<strong>Total Routers Clicked: 9</strong><table
border='0' cellpadding='0' cellspacing='3' width='100%'><tr>
Notice that the "Content-type" and "From" headers are in different
positions and there is an extra carriage return/line feed after the line
(ie. a blank line). According to the RFC for sending emails, when you
have a double CRLF, that is the end of the headers. For some reasong,
Google is re-arranging the headers and adding extra returns. WHY?
When the exact same email is sent to Hotmail or others, the headers are received exactly as they are sent.
Anyone else seing this same issue? We have confirmed with another user
that started experiencing this same issue yesterday as well.
Friday April 27, 2012 11:45am CST UPDATE WE HAVE A FIX, NO NEED TO WAIT ON GOOGLE TO FIX
Something
changed with Google's Email servers and it seems to impact incoming
email that is generated from the PHP MAIL() call. Many servers use this
(including the one at 3Gstore.com). Emails that are sent in plain text
are fine. Emails that are sent as HTML emails are not displayed
properly - all the HTML code is displayed. However, the same email sent
to Yahoo or Hotmail or others is fine, so the problem would appear to
be something on Google's end, since emails have been sending perfectly
for a few years.
This information is for the programmer/developer at your site.
They can fix
the issue in 5 minutes. In a nutshell, you have to replace the PHP
MAIL() call with a direct SMTP Call.
More info on Mail call at http://php.net/manual/en/function.mail.php
Here are the steps to following:
- Download SMTP mail package from http://3gstore.com/smtp-fix.php
- This
package is slightly upgraded (and simplified) version of the email
package originally developed by Manuel Lemos. The original package can
be downloaded from http://poss.sourceforge.net/email/get/
- Unzip and open test.php in a text editor.
- Modify line numbers 4 to 7 with appropriate "from", "to", "subject" and "message".
- Save and upload all the files to your web server.
- Run test.php
- Enjoy!
- PS: The debug mode is enabled by default, you can disable it in class.smtp.php on line number 24.
This is what the php mail() called looked like before:
$from="info@domain.com";
$to="you@domain.com";
$subject="This is an HTML email";
$message="<h1>Sending
HTML email via local server using SMTP
method</h1><b>This</b> <i>is</i> <a href='http://google.com'>a</a> <u>test</u>";
$headers = 'MIME-Version: 1.0' . "\r\n";
$headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";
$headers .= "From: ".$from . "\r\n";
mail($to,$subject,$message,$headers);
Here is the updated php to use the new SMTP mail package:
require("class.smtp.php");
$SMTP=new SMTP();
$from="info@domain.com";
$to="you@domain.com";
$subject="This is an HTML email";
$message="<h1>Sending
HTML email via local server using SMTP
method</h1><b>This</b> <i>is</i> <a href='http://google.com'>a</a> <u>test</u>";
$SMTP->send($from,$to,$subject,$message);
So, when sending emails from SMTP vs PHP MAIL, Google does not change the headers and emails will be delivered without any issue. We have spent the last 24 hours tracking this down and testing and we hope that this helps others.
4/27/2012 4:37pm CST - Email we just received from Google:
Sorry for the regression, though the fault doesn't lie entirely with Gmail.
The
problem is with the PHP mail class. If you specify headers to add,
then they get added with the wrong line ending. Then, your helpful
local mailer you're using to send (postfix, for example, in our
testing). SMTP mail is supposed to all be sent with CRLF line endings,
but apparently postfix only checks the first header to see whether or
not it needs to do the conversion. This means that if the first line
has only an LF, and some other line has a CRLF, you'll wind up with
CRCRLF on the wire.
Our code interprets CRCRLF as CRLFCRLF, ie two lines, which will end your headers early.
GIGO,
as they say. Unless there's something in one of the RFCs I missed
about how to correctly handle this type of messed up data.
Anyways,
we've had a long time bugfix for this broken condition, but we recently
had a regression where the bugfix wasn't being applied. Its already
been fixed and should be out in the next release next week.
I
should point out that this obviously doesn't require PHP, it just
requires sending mixed ending headers to some sendmail clones.
Sorry for the trouble.
So, it appears that Google will fix this next week - THANKS GOOGLE!
|
