When I started researching this, I thought it would be kind of a nightmare to integrate a web service into our current web architecture based on struts2. It turns out it’s really easy to build and maintain a solution that gives you the ability to use both REST and JSON types dynamically. I used the following pattern:

• WSBaseAction – uses XStream to determine request type and serialize beans
• WS*Action – action classes that extend WSBaseAction

WSBaseAction:

package com.example.actions;

import javax.servlet.http.HttpServletRequest;

import com.example.view.RequestView;
import com.example.util.RequestUtils;
import com.thoughtworks.xstream.XStream;
import com.thoughtworks.xstream.io.json.JsonHierarchicalStreamDriver;
import org.apache.struts2.interceptor.ServletRequestAware;

public class WSBaseAction extends BaseAction implements ServletRequestAware {

private String alertMessage;
private Object payloadBean;
private String requestType;
private String alias;

public WSBaseAction(String alias) {
super(true, null, new RequestView());
this.setPublicCache(false);
this.alias = alias;
}

public String getPayload() {

if (getRequestType() == "json") {
return getJsonPayload();
}

return getXmlPayload();
}

private String getXmlPayload() {
XStream xstream = new XStream();
xstream.alias(alias, payloadBean.getClass());
return xstream.toXML(payloadBean);
}

private String getJsonPayload() {
XStream xstream = new XStream(new JsonHierarchicalStreamDriver());
xstream.alias(alias, payloadBean.getClass());
return xstream.toXML(payloadBean);
}

public Object getPayloadBean() {
return payloadBean;
}

public void setPayloadBean(Object payloadBean) {
this.payloadBean = payloadBean;
}

public String getAlertMessage() {
return alertMessage;
}

public void setAlertMessage(String alertMessage) {
this.alertMessage = alertMessage;
}

public String getRequestType() {
if (this.requestType == null) {
String url = RequestUtils.getRequestUrl(request);
if (url.endsWith(".json")) {
this.requestType = "json";
} else {
this.requestType = "rest";
}
}
return requestType;
}

public void setRequestType(String requestType) {
this.requestType = requestType;
}

public String getAlias() {
return alias;
}

public void setAlias(String alias) {
this.alias = alias;
}

private HttpServletRequest request;
public void setServletRequest(HttpServletRequest arg0) {
request = arg0;
}
}

WSExampleAction:

package com.example.actions;

import com.example.WebReaderConstants;
import com.example.services.DocumentMetadataService;
import com.example.view.RequestView;
import com.example.actions.data.*;

public class WSExampleAction extends WSBaseAction {

	private ExampleService exampleService;

	public WSExampleAction() {
		super("example_response");
	}

	@Override
    public String execute() throws Exception {

        RequestView view = (RequestView) this.getModel();

        exampleService.populateMetadata(view);
        ExampleBean bean = new ExampleBean();
        bean.setNumPages(view.getNumDocumentPages());
        bean.setTitle(view.getTitle());
        bean.setShortTitle(view.getShortTitle());
        setPayloadBean(bean);

        return getRequestType();
    }
}

struts.xml:

...

	<!-- Allows for rest and json webservice requests -->
	<constant name="struts.action.extension" value="action,xml,json" />
...
        <!-- Web Service Calls -->
	<action name="**WS*" class="com.example.actions.WS{2}Action">
	    <interceptor-ref name="webreaderMobileStack"/>
	    <result name="rest" type="freemarker">/templates/containers/restContainer.ftl</result>
	    <result name="json" type="freemarker">/templates/containers/jsonContainer.ftl</result>
	    <param name="alertMessage">${example.web.alert.message}</param>
	</action>
...

jsonContainer.ftl

${payload}

restContainer.ftl

<?xml version="1.0" encoding="UTF-8"?>
${payload}

So it works like this:

1. Struts sees a request whose action starts with WS (for webservice) and redirects to the appropriate action
2. The action gathers data from the service layer and populates our bean
3. Struts fills out the template by calling getPayload
4. The base action serializes the payload bean in XML or JSON depending on what the url extension was (WSExample.json or WSExample.xml)

My company does digital editions of magazines. You know it’s going well when you see slick movies like this:

I’ve had a pile of electronics sitting around the house for a couple years now (2 old desktops, 4 CRT monitors, various speakers, cell phones, and a 47″ projection TV) that need recycling and I’ve found that there are not too many places around that make this easy for you, or that they are too expensive. I saw this recently and it’s so wildly inexpensive (free for most things, $10 a piece for monitors and you get a $10 gift card back) I thought I’d give the monkeys in the blue shirts a go.

http://www.bestbuy.com/site/olspage.jsp?type=category&id=pcmcat149900050025

I went in with 2 monitors and walked out in 5 minutes with 2 $10 gift cards to replace my $20. No personal information, no bullshit at customer service (or perhaps their bullshitting tactics have advanced to such a level that I can no longer detect them). The only time I knew I was at bestbuy was when an employee gave me the double-finger-snap-and-point-with-both-hands-like-a-gun and asked “Are you all set there chief?”

NOTE: they don’t do TV’s so I’ve still got to save room in the garage, but this is still excellent

It always ends with LOLCATS

Somehow I never came across this problem before. I had a table that looked like this:

| links | CREATE TABLE `links` (
`link_id` int(5) unsigned NOT NULL auto_increment,
`link` varchar(250) NOT NULL default ”,
`x` float unsigned NOT NULL default ‘0′,
`y` float unsigned NOT NULL default ‘0′,
`h` float unsigned NOT NULL default ‘0′,
`w` float unsigned NOT NULL default ‘0′,
`link_type_id` int(5) unsigned NOT NULL default ‘0′,
`alt` varchar(255) default NULL,
`color` varchar(8) default NULL,
`style` varchar(16) default NULL,
PRIMARY KEY  (`link_id`)
) ENGINE=MyISAM AUTO_INCREMENT=65123341 DEFAULT CHARSET=latin1 |

A program I was writing needed to check if a link already existed in this table so I created this in my dao layer:

public List<Links> findByParsedLink(Links instance) {
log.debug(”finding Links”);
try {
Query q = sessionFactory.getCurrentSession().createQuery(
“from com.texterity.webreader.data.Links l where ” +
“l.x = ” +
“and l.y = :y ” +
“and l.h = :h ” +
“and l.w = :w ” +
“and ” +
“l.link = :link and l.style = :style”);
q.setFloat(”x”, instance.getX());
q.setFloat(”y”, instance.getY());
q.setFloat(”h”, instance.getH());
q.setFloat(”w”, instance.getW());
q.setParameter(”link”, instance.getLink());
q.setParameter(”style”, instance.getStyle());
List<Links> results = (List<Links>)q.list();
log.debug(”find by example successful, result size: ”
+ results.size());
return results;
} catch (RuntimeException re) {
log.error(”find by example failed”, re);
throw re;
}
}

For some reason, none of the floats would match. Didn’t matter if I was using setFloat, setParameter, etc. I changed the ‘=’ to ‘like’ and everything seemed fine until I encountered a ‘0′  value for one of the floats in which case ‘like’ would not match, but ‘=’ would. Turns out that floating point values are difficult to match in SQL when they lack precision.  To make this work, I altered the table to look like so (note the precision values for the floats):

| links | CREATE TABLE `links` (
`link_id` int(5) unsigned NOT NULL auto_increment,
`link` varchar(250) NOT NULL,
`x` float(10,2) unsigned NOT NULL,
`y` float(10,2) unsigned NOT NULL,
`h` float(10,2) unsigned NOT NULL,
`w` float(10,2) unsigned NOT NULL,
`link_type_id` int(5) unsigned NOT NULL,
`alt` varchar(255) default NULL,
`color` varchar(8) default NULL,
`style` varchar(16) default NULL,
PRIMARY KEY  (`link_id`)
) ENGINE=MyISAM AUTO_INCREMENT=835 DEFAULT CHARSET=latin1 |

Now I’m matching on ‘=’ with all values.

Plan A

Mow the lawn, hang out with Liam while Melissa runs errands, put Liam down for a nap

Problem with Plan A

Rain in the forcast early

Plan B

Move Liam’s old toys and clothes to the attic, hang out with Liam while Melissa runs errands, put Liam down for a nap

Problem with Plan B

Dryer stops drying clothes fully

Plan C

Fix dryer, Move Liam’s old toys and clothes to the attic, hang out with Liam while Melissa runs errands, put Liam down for a nap

Problem with Plan C

Can’t figure out what’s wrong with the dryer and upon taking it apart I short the electrical component

Plan D

Buy a new dryer, put Liam down for a nap

Execution of Plan D

$1500 later, I have a brand new washer and dryer

It’s rare I deal with any kind of MS product. On occaision, when I develop a UI one of the last steps I take is to “debug” it in IE, or work around the general IE idiocy issues. That’s about it. My Xbox 360 has worked well for the last several years but I don’t really do anything to exciting with it. I know the folks over at MS aren’t too sharp in the software design area, but they still never cease to astound me.

Today I signed up for Netflix again after a long hiatus. One of the things I was looking forward to was live streaming. I figured I’d just use my 360 to stream. So here’s how the process went:

1. Start my Xbox, which auto-signs me into my current Silver membership
2. Navigate to the netflix icon, which prompts me to download the netflix streaming app
3. Download the netflix streaming app and start it
4. I’m informed that I need to upgrade to a gold membership to stream netflix
5. I try to upgrade my account from the xbox and I get a “8015d080″ error, which leads me here. Turns out the Windows Live ID’s expire after a period of inactivity. Evidently, logging into your xbox live account does not count as “activity.”
6. I find out that you can sign up for a new Windows Live account, then transfer your gamertag to the new account via the Xbox Live dashboard. Great!
7. Sign up for the new Windows Live account, navigate to the “Change my Windows Live ID” option and I’m greeted with “Can’t connect to Xbox Live. Please retry later.” I’m already signed in mind you.
8. Google a bit to find out others have this problem and they had to go through MS Support to resolve it.
9. Send an email to support (the form on the xbox support page doesn’t work correctly either – maybe just in firefox, or firefox on Linux – the dropdowns don’t update when you change topics from “hardware issue” to “Xbox Live issue”. I’d guess the chances of my email getting to the proper support place are roughly 1/1,364).

If I had the choice of taking a job programming in .NET, Silverlight, etc or eating out of the garbage, I’d be dumpster diving in no time.

My company’s largest form of communication is via email. We use email to notify our subscribers that a new content is available. I am not talking about spam or marketing mailings, our customers opt-in to get these emails and rely on them as a means of notification. When a large number of customers don’t get an email from us, we are inundated with customer support requests to figure out why this happened. For the most part, it’s a good system. We follow all the best practices for sending bulk email and have had very few issues over the past year with one exception: yahoo.com.

Best Practices

One of the most reliable ways to determine if you are doing a good job following best email practices is to view your sender score. Return Path, who are experts in email delivery say this:

What is a “good” Sender Score? At a basic level, high is good and low is bad. We have found that the best email senders regularly maintain a Sender Score above 80. And anything below 30 should be cause for alarm.

Our current senderscore is 95. I won’t get into any categorical breakdown of the score here, but this means that we are generally doing all the right things. There is always room for improvement, but overall our reputation is excellent. This is probably why we don’t see the kinds of problems exhibited by yahoo with any other domains.

Symptoms

One of our clients publishes content daily and we have about 35,000 subscribers with yahoo.com email addresses. The entire mailing list is over 150,000 addresses long and we have not had any issue with non-yahoo domains blocking us for any period of time in a very long time. On a semi-regular basis, all email we send to yahoo.com will be blocked with this message (from our smtp logs):

[Message Expired] <banner> failed – 421 4.0.0 Message temporarily deferred – 4.16.52. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html

These blocks last for 4 hours. If you look at the url listed above, you see that we are being blocked for 1 of 2 reasons:

the message you attempted to send exhibited characteristics indicative of spam,and/or

emails from your network have been generating complaints from Yahoo! Mail users.

Combatting Yahoo

We’ve taken several measures to combat this problem. First, we implemented domain keys. This verifies our DNS domain and should help message integrity. As you see in the screenshot below (notice the envelope with key icon), yahoo.com is able to verify our domain keys.

Next we signed up for yahoo.com’s feedback loop program. This allows us to receive notice when someone marks one of our messages as spam so we can remove them from future mailings. It also gives us data as to how many people have marked a particular message as spam and the number of emails marked as spam in given time period. These messages appear in standard ARF/rfc822 MIME format, making them easy to parse.

It should be noted that yahoo’s feedback loop differs from those of other large providers such as Comcast and AOL. Other providers tend to excise the recipient’s email address who marked you as spam from their feedback loop message, so you can only get a copy of the original message and a notice that *someone* marked it as spam. This allows you to aggregate the data to tell if a certain message caused problems. Yahoo seems to use it as another way for users to opt out of a mailing.

(FYI – you can build a nice little feedback loop parser using Perl and the Email::Folder, Email::Simple, Email::MIME and Email::Valid modules by Ricardo SIGNES.)

Why yahoo sucks

The CAN-SPAM rules for unsubscribing to a mailing list state each mailing must include:

• A visible and operable unsubscribe mechanism is present in all emails.
• Consumer opt-out requests are honored within 10 days.
• Opt-out lists also known as suppression lists are only used for compliance purposes.

As you may have guessed by our sender score, we are including a link in every email. This is something spam filters typically check for. For the end user however, these rules don’t matter. Users are not interested in WHY they aren’t receiving a message anymore, just that they are not seeing it anymore.

If indeed the information yahoo gives us above as to why they have blocked is accurate, it would seem Yahoo places far too much weight into their users marking a message as spam. Look at the image below. Notice the prominent “Spam” button.

If I’m a user and I want to stop receiving a mailing, it’s far easier to click the “Spam” button than it is to look through an email to find a way to opt out of receiving it. Since we’ve seen an increased block rate since we signed up for the feedback loop, it makes sense that yahoo’s algorithm takes into account users that have already marked you as spam that you resend to. Signing up for yahoo’s feedback loop seems to adversely affect your reputation with them if you DO NOT purge the recipients who mark you as spam from your list.

We are currently logging our feedback loop entries and have just released a script that parses it and removes people who use the “mark as spam” button from future lists. I plan to write another post detailing our findings.

I’m currently torn between the principles of good science (change one thing at a time) and the fact that I need to make this problem stop quick, but I’ll try to get as much good data on this issue as I can since pretty much every bulk mailer has had this issue.

The Data

Disclaimer: It should be noted that most email providers are constantly tweaking their rules so any conclusions may or may not still be valid at the time you are reading this.

Let’s look at a couple weeks of data where we had problems. Here’s a table that relates the number of users marking us as spam as reported by our feedback loop to whether or not we were blocked.

Remember that there are roughly 35,000 emails being sent to yahoo.com email addresses every day. These numbers (with perhaps the exception of the day we got 280) seem to be within a reasonable percentage. There also does not appear to be a direct correlation to the number of messages we see in the feedback loop to the days where blocks were applied.

Since we know our message format is good (Thanks, Spam Assassin), the blockages DO appear to relate to the dreaded “Spam Button”. Today is the day we start purging the recipients in our feedback loop, so I will continue to collect data for the next several weeks and see if it makes a difference.

I still fucking hate you Microsoft, but if I have don’t to code for your shitty standards in IE6 and IE7 for much longer, I will hate you slightly less.

http://blogs.msdn.com/ie/archive/2009/04/10/prepare-for-automatic-update-distribution-of-ie8.aspx