The efficacy of anti-virus

Brian Krebs has a good post up on the efficacy of anti-virus products and how A/V should not be relied upon as a substitute for common sense (not opening untrusted attachments, not clicking on links in spam, ensuring you have up-to-date software, etc).  The reason, says Krebs, is that most A/V products are not very good at detecting zero-day malware.  Below is a chart showing how good certain products are (names removed) at catching pieces of malware that A/V products would traditionally be expected to detect:

This chart tends to confirm what I have read and heard a couple of places elsewhere – traditional A/V products only catch about half of new viruses that appear in the wild.  In other words, when a virus writer writes a new piece of malware and releases it to the general public, either via spam or some other mechanism, the average A/V product has a 50/50 chance of catching it on the day it was released.

This begs the question of how independent tests can assert that A/V products catch 99.5% of viruses?  If what we see above indicates that it is no better than a flip of a coin, how do we come up with the stats that A/V is really, really good?  Here are some possible explanations:

  1. The tests are biased.  In order to test a company’s A/V efficacy, a tester has to collect a corpus of malware.  They then run this malware through the A/V engine and see what gets caught.  Of course, it takes time to generate a corpus and so much of it is historically archived.  Some of it is engineered/created by the testing organizations.  What is happening is that the corpus is filled with known viruses that everyone has blocked a long time ago, and also filled with viruses that are not seen in real life. 

    It would be kind of like doing the Pepsi Taste Test on a bunch of Pepsi drinkers.  Yes, Pepsi will come out ahead but by biasing your sample, you can engineer the results that you want.  But it isn’t representative of real life.

  2. It isn’t easy to acquire fresh malware in numbers.  Once a corpus has been built, it’s not easy to acquire fresh malware in any large numbers.  After all, in order for a test to be statistically significant, you need a lot of samples in order to get around the margin of error.  Assuming that a pre-built corpus has been assembled previously and contains 1000 pieces of malware, and then 20 new pieces of malware show up via honeypots or some other acquisition mechanism, then even if half of the new ones are caught and all of known ones are, then it is still a 99% catch rate.  That looks pretty good, but the problem is that half of the new ones – the ones that matter – haven’t been detected.

Yet in spite of these glaring flaws in A/V, it is still an essential product.  The fact is that while malware is being created every single day, it isn’t distributed to the entire Internet world at the same time.  It hits some people, but it doesn’t hit most others.  And of those that it does hit, it doesn’t get delivered to every one at the same instant.  It arrives in phases, and by the time one phase hits one set of users, the A/V signatures will have caught up and the A/V software will prevent the user from getting infected.  The early users who don’t take precautions will get infected… well, yeah, that happens.  What are you going to do other than stop clicking untrusted links and opening untrusted attachments?  But the downstream wave of users will be protected from the new round of malware.

In addition, some malware out in the wild is older and floats around for a bit of time.  A/V does protect against that even if it isn’t the most dangerous threat out there.  So, the bottom line is that while users have to take basic security measures to keep from falling for bad things, you should still make sure your software is up-to-date and running the latest definitions.

Comments (2)
  1. tonyr says:

    I think I've read  this somewhere and its that the hackers just decompose the security fixes and then  try to out race the update deployment.  I think that ms should in cooperation with all the av vendors should put out the signature 1st then the security hotfix.   Of course somehow the hackers would get that and decompose it … oh well we are screwed..

  2. Jeff K. says:


    Depending on how we define the term "A/V software", I would heartily disagree that it is "an essential product."  Pure A/V, intended to be faced only towards viruses, offers little in terms of benefits (as you well show) while demanding much in terms of resources.   Norton/Symantec's antivirus product comes to mind.  I used to work at a large University that had a site license for A/V, Client Security, and (eventually) whatever the newest catch-all offering was whose name I'm now drawing a blank on.  When talking with end users, I would be frank about my opinion that I would rather have most viruses (*not* worms/trojans/rootkits) than I would have Symantec A/V installed on my system.  Client Security offered A/V+firewall, so I pushed them towards that.  

    I started working at the University in 2006, having left my job to return to school and finish my degree, and at the time I was still running Win98 SE on my PIII 450 at home.  I ran that machine as my exclusive home system until 2008, never had an issue even though I did not have any A/V installed.  That's because you and Brian are absolutely correct, there is no substitute for "common" sense.  Using it mitigates the need for A/V, and not using it mitigates the usefulness of A/V to the point of making the cost/benefit tradeoff of A/V a bad one.  When I first supported user machines as a local/network/server admin (small company, I was the web guy) in 2001, half of them ran ME and all the users POPed their email, and I fought Gator like I was Steve Irwin.  Those machines needed A/V.  Now, with the introduction of XP SP2's firewall, mail being web access for most, and the enlightenment of users on dodgy shareware, pure A/V just requires too much.  Any realtime protection that isn't a firewall (which should ideally be implemented elsewhere in the OSI, anyway) has to prove its worth via marginal protection added against marginal resource use.  A/V doesn't, in my opinion.

    What *is* needed, and what is installed on far too few machines, thanks to the brainwash of the A/V industry simultaneously convincing users that they *must* have A/V and that A/V is *all* they need, is proper malware software.  A proper Ring-0 capable rootkit detector.  Something to monitor attempted changes to startup items/scheduled tasks/helpers etc., vectors for the actual impact of malware to manifest.  I personally have found that a combination of WinPatrol (automated monitor of key entry points), MBAM (without the protection module, just a weekly scheduled scan), RootkitUnhooker (one of maybe two rootkit detectors that can actually live up to its name and protect against serious rootkit methods), and Windows Firewall works for me.  This is a system that doesn't solely rely on signature-based detection to succeed, which reliance causes the issues you note with A/V such as time lag for protection.  Next to no resource suck, and best of all, it's 100% free.  

Comments are closed.

Skip to main content