Lies, Damn Lies, Information Leaks, and Statistics

Robert Hensing posted some criticism of a study that purported to analyze how many users are at risk due to using out of date or unpatched browsers. Rob rightfully points out that you can actually be running a very old version of IE (depending on OS), and still be patched against current attacks.

A flaw that IE doesn't have is advertising to the server the exact minor version of the application. People often underestimate the value of information leaks – advertising the exact minor version is basically saying "Hello, you may attack me with these exploits, but I'm patched against those exploits." You can often figure this out with various fingerprinting techniques, but sometimes you can't. As it turns out, Safari, Firefox and Opera all have information disclosure flaws, and these were used to estimate the number of vulnerable browsers by examining Google's server logs. Because IE doesn't advertise this information to the server, they couldn't do a valid comparison, and dropped to a different data set.

Rob admits to not having much statistical training, but having spent far too much time in graduate school, I've had quite a bit. First thing to consider is the sample size. As it turns out, you don't need that many samples to be valid. Secunia's sample was around 500,000, which is more than adequate. The next thing to consider is whether we're really dealing with 2 different populations. To their credit, they do call this out:

Secunia [21] identified (for the month of May 2008) that 4.4% of IE7, 8.1% of Firefox, 14.3% of Safari (Windows only), and 15.2% of Opera users have not applied the most recent security patches available to them from the software vendor. In comparison, we discovered that 16.7% of Firefox, 34.7% of Safari (all OS), and 43.9% of Opera Web browser installations (using our Web server log-based measurements) had not applied the most recent security patches. We found that our Firefox, Safari, and Opera results were higher than those of Secunia's, differing by a factor of 2.1 (Firefox), 2.4 (Safari), and 2.9 (Opera), and attribute this difference to a probable bias for more security aware users to take advantage of Secunia's security scanner PSI than the average global community.

First of all, we can clearly establish that we're dealing with 2 distinctly different populations. The assertion that IE6 is insecure is invalid, as it is still in support and gets security patches just like IE7, and while IE 7 is doing a bit better on bulletin count, it isn't a huge difference, and as I've noticed with several other products, there is a grace period where attackers don't bother until there's enough adoption. While it is interesting that around twice as many Firefox users aren't fully patched as IE7 users, this might be an artifact of release timing. The authors of the study then attempt to deal with these different populations by comparing the Google results to the Secunia results, but there's a lot of variance between the browser types – if Firefox users going to Google are 2.1x less likely to be secure than Firefox users identified by the Secunia study, but Opera is different by a factor of 2.9x, then the difference between IE users overall vs. Secunia is really anyone's guess. Is it 2.9x? 4x? 1.5x? No one really knows.

It's an interesting thing to try and study, and the hypothesis that different patch delivery mechanisms might make a difference in how many users are at risk is also interesting, but data on IE users who are the majority of the population, and could behave differently as a group than users of other browsers, is really not available which makes the conclusions very questionable. Another factor that they appear not to have considered is that the number of browsers missing patches is going to be a function of how often you see patches. Something patched once a year is more likely to be patched than something patched 25 times a year.

Interesting paper – too bad their conclusions aren't supportable for the bulk of the users who are using IE.