My company’s largest form of communication is via email. We use email to notify our subscribers that a new content is available. I am not talking about spam or marketing mailings, our customers opt-in to get these emails and rely on them as a means of notification. When a large number of customers don’t get an email from us, we are inundated with customer support requests to figure out why this happened. For the most part, it’s a good system. We follow all the best practices for sending bulk email and have had very few issues over the past year with one exception: yahoo.com.
Best Practices
One of the most reliable ways to determine if you are doing a good job following best email practices is to view your sender score. Return Path, who are experts in email delivery say this:
What is a “good” Sender Score? At a basic level, high is good and low is bad. We have found that the best email senders regularly maintain a Sender Score above 80. And anything below 30 should be cause for alarm.
Our current senderscore is 95. I won’t get into any categorical breakdown of the score here, but this means that we are generally doing all the right things. There is always room for improvement, but overall our reputation is excellent. This is probably why we don’t see the kinds of problems exhibited by yahoo with any other domains.
Symptoms
One of our clients publishes content daily and we have about 35,000 subscribers with yahoo.com email addresses. The entire mailing list is over 150,000 addresses long and we have not had any issue with non-yahoo domains blocking us for any period of time in a very long time. On a semi-regular basis, all email we send to yahoo.com will be blocked with this message (from our smtp logs):
[Message Expired] <banner> failed – 421 4.0.0 Message temporarily deferred – 4.16.52. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html
These blocks last for 4 hours. If you look at the url listed above, you see that we are being blocked for 1 of 2 reasons:
- the message you attempted to send exhibited characteristics indicative of spam,and/or
- emails from your network have been generating complaints from Yahoo! Mail users.
Combatting Yahoo
We’ve taken several measures to combat this problem. First, we implemented domain keys. This verifies our DNS domain and should help message integrity. As you see in the screenshot below (notice the envelope with key icon), yahoo.com is able to verify our domain keys.
Next we signed up for yahoo.com’s feedback loop program. This allows us to receive notice when someone marks one of our messages as spam so we can remove them from future mailings. It also gives us data as to how many people have marked a particular message as spam and the number of emails marked as spam in given time period. These messages appear in standard ARF/rfc822 MIME format, making them easy to parse.
It should be noted that yahoo’s feedback loop differs from those of other large providers such as Comcast and AOL. Other providers tend to excise the recipient’s email address who marked you as spam from their feedback loop message, so you can only get a copy of the original message and a notice that *someone* marked it as spam. This allows you to aggregate the data to tell if a certain message caused problems. Yahoo seems to use it as another way for users to opt out of a mailing.
(FYI – you can build a nice little feedback loop parser using Perl and the Email::Folder, Email::Simple, Email::MIME and Email::Valid modules by Ricardo SIGNES.)
Why yahoo sucks
The CAN-SPAM rules for unsubscribing to a mailing list state each mailing must include:
- A visible and operable unsubscribe mechanism is present in all emails.
- Consumer opt-out requests are honored within 10 days.
- Opt-out lists also known as suppression lists are only used for compliance purposes.
As you may have guessed by our sender score, we are including a link in every email. This is something spam filters typically check for. For the end user however, these rules don’t matter. Users are not interested in WHY they aren’t receiving a message anymore, just that they are not seeing it anymore.
If indeed the information yahoo gives us above as to why they have blocked is accurate, it would seem Yahoo places far too much weight into their users marking a message as spam. Look at the image below. Notice the prominent “Spam” button.
If I’m a user and I want to stop receiving a mailing, it’s far easier to click the “Spam” button than it is to look through an email to find a way to opt out of receiving it. Since we’ve seen an increased block rate since we signed up for the feedback loop, it makes sense that yahoo’s algorithm takes into account users that have already marked you as spam that you resend to. Signing up for yahoo’s feedback loop seems to adversely affect your reputation with them if you DO NOT purge the recipients who mark you as spam from your list.
We are currently logging our feedback loop entries and have just released a script that parses it and removes people who use the “mark as spam” button from future lists. I plan to write another post detailing our findings.
I’m currently torn between the principles of good science (change one thing at a time) and the fact that I need to make this problem stop quick, but I’ll try to get as much good data on this issue as I can since pretty much every bulk mailer has had this issue.
The Data
Disclaimer: It should be noted that most email providers are constantly tweaking their rules so any conclusions may or may not still be valid at the time you are reading this.
Let’s look at a couple weeks of data where we had problems. Here’s a table that relates the number of users marking us as spam as reported by our feedback loop to whether or not we were blocked.
| marked as spam | Block? |
|---|---|
| 21 | N |
| 17 | N |
| 10 | N |
| 5 | N |
| 2 | N |
| 31 | Y |
| 58 | Y |
| 36 | Y |
| 9 | Y |
| 1 | N |
| 21 | N |
| 3 | N |
| 8 | Y |
| 280 | Y |
| 30 | N |
Remember that there are roughly 35,000 emails being sent to yahoo.com email addresses every day. These numbers (with perhaps the exception of the day we got 280) seem to be within a reasonable percentage. There also does not appear to be a direct correlation to the number of messages we see in the feedback loop to the days where blocks were applied.
Since we know our message format is good (Thanks, Spam Assassin), the blockages DO appear to relate to the dreaded “Spam Button”. Today is the day we start purging the recipients in our feedback loop, so I will continue to collect data for the next several weeks and see if it makes a difference.
