Why people keep proposing a Final Ultimate Solution to the Spam Problem (FUSSP)

In the antispam world, from time to time somebody new likes to come in and propose a solution that will wipe out spam: Email authentication! Statistical classifiers! Blacklists! User education!

These terms are derisively referred to as the Final Ultimate Solution to the Spam Problem. It’s a term that industry veterans give to ideas that have been considered but abandoned because they are unworkable or don’t address the full problem while still leaving large gaps for spammers to exploit. This is summed up at this blog post (click link for the full list, I have pruned it):

Your idea advocates a

(x) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam.

Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck

Furthermore, this is what I think about you:

( ) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!

Of course, this is tongue in cheek but it is not far off from the truth. Unless you have been fighting spam for a long time, there is no simple solution. It is only the people who haven’t been doing this for very long who propose these ideas without understanding that there is an existing huge investment in the existing email protocol and people won’t move off it quickly if it is less convenient than their existing, insecure system.

So why do people keep proposing it?

It turns out that this is explained by science: the Dunning-Kruger effect. The Dunning-Kruger effect was published in the late 1990’s by researchers at a university. It is a cognitive bias where unskilled people overestimate their ability in something. But not only do they overestimate their own abilities, they fail to recognize how poor they are at it, nor can they recognize actual in others who really do have it.

For example, suppose I went and took a couple of badminton lessons and learned the basics. I then went and played against all my friends and beat them soundly. Feeling pretty confident about myself, I enter a tournament and get destroyed by all of my opponents. My basic knowledge after a few lessons greatly increased my confidence but I was still a terrible player compared to people who were very good at what they do.

The Dunning-Kruger effect also finds that people with genuine skill tend to underestimate their abilities in something. They think that if they find something easy, others do, too. Therefore, they have no advantage. This is not true, they really are good, but skilled people don’t think so.

Finally, the Dunning-Kruger effect is present only when people have some ability in an area. For example, in our above example, the beginner badminton player may overestimate his abilities at badminton, but at horseback riding he knows he is an amateur and is unlikely to do well in competition. Thus, a little bit of knowledge goes a long way to giving you false confidence in your abilities.

This brings us back to the FUSSP.

Relative newbies to the industry know a little bit about fighting spam and online abuse. They know about filters and blacklists but then falsely extrapolate that it is much simpler than it really is. This is wrong, spam filtering is very complex but because of Dunning-Kruger, newbies think they know more than they do and they fail to recognize how little they know. Furthermore, they fail recognize that others with far more experience have never proposed nor implemented what they think will solve the problem. It is not the complete outsiders (like friends or relatives in different professions) who make these proposals, but industry newcomers with a little experience.

This also explains why experts never propose an actual FUSSP, they only propose managing the spam problem. Experts know that spammers are actively trying to subvert filters; they also understand how this can be done and if they can do it, then so can spammers. Therefore, they are far less assertive in what they do and do not claim.

So why do people keep proposing Final Ultimate Solutions to the Spam Problem?

Because of cognitive bias.

Comments (0)

Skip to main content