A Common Set of Antispam Metrics, part 3

3. Measurements

The first way to do this is by way of Catch Rate. Catch rate is defined by the following:

Catch rate =
= Spam correctly identified / (Spam correctly identified+missed spam)
= TP / (TP+FN)

This Catch rate gives us the effectiveness of a spam filter on known spam. Given 100 messages and a filter that tags 75 of them as spam and 25 as non-spam, the catch rate is 75%.

The next metric is the false positive rate. Catch rate is only one half of the equation. The false positive rate is best measured by the following:

False Positive Rate =
= Messages incorrectly flagged as spam / Total legitimate messages
= FP / (TN+FP)

There is some ambiguity around false positive rate. It can (and has) been identified the following way:

False Positive Rate=
= Messages incorrectly flagged as spam / Total messages
= FP / (TN+TP+FP+FN)

When the amount of spam isn’t very large relative to the non-spam, this metric is easier to acquire. However, the first measurement is more reflective of reality. When we think about false positives, we think of them relative to our legitimate email stream, not relative to our total email. Why would we want it relative to our total inbound email if we don’t expect to see any spam anyhow?

Comments (2)

  1. Justin Mason says:

    hey, did you get my comment about the SpamAssassin wiki documentation on accuracy-measurement methods?  I don’t see it here…

  2. tzink says:

    Yeah, I got it.  I’ll post a link later on.