A Common Set of Antispam Metrics, part 2

  • Article

2. Definitions

The email industry needs to converge on a set of standards around metrics. Specifically, while we all think we know what we mean, what we don’t know is what others think they mean. So, let’s define them:

  • Legitimate mail (ham) – legitimate mail, or good mail, is email that an average user would expect, or want, to receive in their inbox, all things being equal. This includes personal mail between users, mail in a business environment, and opt-in newsletters.

  • Spam - There are many ways to define spam and the simplest is that it is unsolicited commercial email. This definition can be extended to contrast with legitimate mail; spam is mail that an average user would not want to receive. This disqualifies newsletters that a user has opted into but no longer wants to continue receiving.

  • False positive (FP) – This is a message that the spam filter says was spam but the end user (legitimately) says is not spam.

  • False negative (FN) – This is a message that the spam filter says was not spam but the end user says is spam.

  • True negative (TN) – This is a message that the spam filter says was not spam and the user agrees.

  • True Positive (TP) – This is a message that the spam filter says was spam and the user agrees.

Give these definitions, how do we know how good our spam filter is performing? How do we compare each other in an apples-to-apples comparison?