July 12 – My third year anniversary!


Today is a special day at Microsoft, it is the three-year anniversary of the day I joined Frontbridge (now Microsoft Exchange Hosted Services) as a spam analyst.  Ah, what a memorable three years it has been.


On our first day on the job, me and three others (the Fantastic Four) went down to Los Angeles for four weeks of training.  We met the other lone spam analyst and we spent the next two weeks learning about spam and how to fight it and then the subsequent two weeks doing that over and over again before returning north to Canada.


I have processed a lot of spam in my time but for the first two years my main focus was false positives.  I used to process about 90% of the FPs we saw and I became incredibly good at predicting which spam rules were going to perform well in the field and which ones were not.  In those days, our spam team's primary tricks of the trade were writing regular expression spam rules on the contents of the email message.  I would process all of the false positives and then go on to spam.  Whenever I came across a legitimate false positive (which wasn't often) I could often look at the message and predict what part of the message was tagged as spam by our spam rules.


Some time passed and we added on another spam filtering service (component) which was automated.  I was responsible for setting up the false positive process, and I became good at predicting what FPs were caused by this new component and which ones were caused by our spam rules.  Time passed but the spam stayed the same.  In those days, pornographic spam was one of the most common types of spam and obfuscation of words was the preferred filter-evasion technique.  We saw image spam back then, but it always was embedded in a link.


In 2005, we continued to process spam but we started seeing some more foreign stuff (due to our customer base).  Still, not much changed.  We saw stock spam, pharmacy spam, 419s, and so forth.  All the while I was still handling false positives.


In summer 2006, we saw a sudden shift in spam tactics.  Image spam hit our networks.  I had seen image spam before, spammers sometimes used it in their CAN-SPAM boilerplates in the footers of their messages.  But, this was a new tactic for which we were ill-prepared.  Spammers were inserting gif and jpg images into their spam messages and delivering mail that way.  At the time, there was a new outbreak every week and I was working six days a week trying to handle all of this stuff.  However, time passed, we got some new features implemented and the image spam problem started to drastically reduce.  My own personal image spam rules have blocked over a billion messages since they were implemented back in September.


Time passed and 2007 has rolled around.  There's a new breed of spam floating around, pdf spam and "gift-card" spam (which isn't new, but the payload to a virus is).  I don't process much spam anymore these days, but I still troll through our various inboxes to get a feel for what's going on.  Now, I am a Program Manager of (anti) spam effectiveness, which means I am in charge of collecting various measurements on our networks.  Furthermore, the scope of my duties has greatly expanded in the past three months so now I have a great deal of influence into driving and defining new antispam features.  In my opinion, this is a very natural progression because I felt that as a rule writer / spam analyst, I was getting close to the end of how far I could go and the logical next step was to move beyond spam rules.  I had to become familiar with a whole variety of anti-spam techniques.  This is not to say that we did not have techniques other than spam rules (far from it), but now I have a great deal of influence of reshaping the process of how we do it.


So, it's been an interesting three years.  Hopefully the next three are just as interesting.

Comments (9)

  1. Congratulations, keep up the good work and the great blog!!

  2. Al Iverson says:

    Congrats! How come you are not here at the FTC spam summit in DC?

  3. Norman Diamond says:

    Oh, since you’re Canadian, might you perhaps know the answer to one of my questions.

    Around 30 years ago, computer abuse was prosecuted as wire fraud, because there were no other applicable laws and in that particular case the abuse was performed by a connection from an on-line terminal.  (If the abuse had been performed by a program submitted on punched cards or whatever, I guess it wouldn’t have been prosecutable.)  Anyway, as I recall, the prosecution was successful.  The abusers were convicted of wire fraud.

    Anyway, in this millennium, tons and tons of computer abuse take place over telecommunications lines.

    Why doesn’t the RCMP prosecute these as wire frauds?

    Now for a couple of tangents on other issues:

    > […] more foreign stuff […] pharmacy spam […]

    For you, isn’t the pharmacy stuff domestic?  Now for me the pharmacy stuff is foreign, just as I’m a foreigner myself, and I kind of doubt that Canadian pharmacies have licences to sell their crap in Japan.

    But speaking of foreign stuff, 95% of spams could be blocked by blocking all English-language e-mail.  There are clowns who block e-mail from Japan or with Japanese encodings, who succeed in blocking 2% of spams and blocking tons of legitimate e-mails.  If they really wanted to do their clowning job they’d do it to English instead.  Anyway, blocking needs to be done on content not on language.  I’m glad you understand that, but this is just a tangent on the issue of "foreign" stuff.

  4. redgreen_x@163.com says:

    Congratulations. Infact I read your blog everyday quietly. I’m a spam fighter too, you are my idol. haha  

  5. tzink says:

    Thank-you, Josh, for the kind comments.

  6. tzink says:

    Thank-you, Al.

    I thought about attending the FTC summit, but the timing doesn’t work out for me.  I’m currently overseeing several projects here that are due for release in September.

    While I’m not at the FTC conference, I will be at the CEAS next month.  Should be fun!

  7. tzink says:

    redgreen_x,

    I am flattered by your comments.

  8. tzink says:

    Norman,

    > Why doesn’t the RCMP prosecute these as wire frauds?

    I have no idea.  My member of Parliament used to be the Justice Minister for Canada (he’s now the president of the Treasury) so I guess I cuold ask him.  But it wouldn’t be the RCMP prosecuting these as wire frauds, it would be the Crown.

    > For you, isn’t the pharmacy stuff domestic?  I kind of doubt that Canadian pharmacies have licences to sell their crap in Japan.

    They don’t really have licenses to sell their stuff in the United States, either.  😛

    > Anyway, blocking needs to be done on content not on language.

    That’s actually an issue that has been discussed here recently.  For people in east Asia, for example, they would never receive English language mail so for them, blocking all English mail would be an effective spam filter.  Similarly, a lunch cafe in Italy would also rarely receive English language mail.

    Microsoft has some engines written for detecting foreign languages so in the future, what would be a nice option for users is to block certain languages if they know they never receive mail in it.

  9. Norman Diamond says:

    > But it wouldn’t be the RCMP prosecuting these as

    > wire frauds, it would be the Crown.

    Oh yeah, how could I forget.  The RCMP arrested the abusers, the Crown did the prosecuting, and I don’t know who decided what the charges would be.

    > For people in east Asia, for example, they would

    > never receive English language mail

    Well, I thought that part of my argument was to be a sort of "proof by contradiction", showing that language is not a useful criterion for filters, only content is.

    People in Singapore, Malaysia, the Philippines, and some other countries get quite a lot of English language mail.  One person in Japan received English language mail from you, some others receive legitimate English language mail occasionally, and some receive it quite often.  "Most" English language mail from abroad to Japan is spam, but it would not be solved by filtering out all English language mail, it can only be solved by filtering out spams.

    Even in discussions that are conducted in English, if someone doesn’t understand an error message that someone roughly translated to English, it might be necessary to quote the exact error message in its original language.

    A lot of people don’t bother changing Outlook Express settings before and after sending each e-mail, so it uses the default language settings of the country where Outlook Express was purchased (concurrently with a new computer).  When spams are reported to the administrators of spam senders, some spam sending administrators bounce the reports because the reports came from countries where they sent the spams to.  If I were in charge of a blacklist, those spam senders would never get off the blacklist.

Skip to main content