Spam analyst performance
I was recently reading Dr. Brett Steenbarger's blog and started reading some of the historical research he has between peak performers and traders. If you are a stock trader I would highly recommend you read his blog, it's one of the best I have come across. Steenbarger is releasing a book that studies the performance of top achievers and athletes in their field, and how stock traders can learn from them and include their habits into trading habits.
Steenbarger has found that top performers become very, very good in their fields by repetition of what they do. This may seem obvious to us, but he further finds that top performers enjoy doing the repetitive tasks in order to become good. In other words, cellists like practicing scales and top quarterbacks enjoy throwing footballs through tires. He also finds that in order for something to be engrained into us, a person needs to practice something over 1000 times.
1000 times of doing something is a lot, but when I look at spam analysis this is a pattern that holds true. I no longer process spam the way I used to because a few weeks ago I moved into a research role combined with a put-out-daily-fires role (ie, when a spam emergency comes up I am on the front lines of making recommendations and solving the problem). However, the reason I got this role is because I understand the nature of spam. I was responsible for processing around 90% of the false positives we received in a week, and I had to do them by hand. There was no automation other than the scripts I wrote to move messages around. On a weekly basis, I had to process about 6000 false positives. Multiply that by 45 weeks a year (allowing for holidays and other tasks that take time) and that is over a quater of a million pieces of mail I processed in 2005-2006. Furthermore, in December 2005 and February 2006, I had to assemble a corpus of messages to do some testing - I had to sort 25,000 messages in two days (each time). That is a tremendous amount of mail processing, but it reinforced how quickly I could process spam.
When I first started the job I had to be careful to make sure I wasn't too overzealous in what I was doing and it took me time to go through messages. Towards the end, I could fly through messages simply by looking at the subject lines of messages. I estimate that I could work 3x faster at the end then when I started.
This came in handy the other day when we received a new type of obfuscated stock spam. Some of the other analysts who hadn't processed the spam before wrote rules for it, but because I recognized the pattern as pharmacy spam we had received weeks earlier, I instantly knew how to block it and corrected some other analysts' "rules" for blocking the message. It was a new take on an old trick.
It takes time to learn how to recognize spam patterns and to be able to mentally form a diagnosis of how to block it. If it takes an athlete 1000 repetitions to become an expert at it, I would say that it takes a spam analyst 25,000 repetitions of processing mail to become a very good spam analyst. I think it would take double that to become an expert. An analyst needs to live through a few cycles of spam before their training is complete. Indeed, some new people we hired a couple of months ago have become powerful, but they are not Jedis yet.