Lies, Damn Lies, Information Leaks, and Statistics

Robert Hensing posted some criticism of a study that purported to analyze how many users are at risk due to using out of date or unpatched browsers. Rob rightfully points out that you can actually be running a very old version of IE (depending on OS), and still be patched against current attacks.

A flaw that IE doesn't have is advertising to the server the exact minor version of the application. People often underestimate the value of information leaks – advertising the exact minor version is basically saying "Hello, you may attack me with these exploits, but I'm patched against those exploits." You can often figure this out with various fingerprinting techniques, but sometimes you can't. As it turns out, Safari, Firefox and Opera all have information disclosure flaws, and these were used to estimate the number of vulnerable browsers by examining Google's server logs. Because IE doesn't advertise this information to the server, they couldn't do a valid comparison, and dropped to a different data set.

Rob admits to not having much statistical training, but having spent far too much time in graduate school, I've had quite a bit. First thing to consider is the sample size. As it turns out, you don't need that many samples to be valid. Secunia's sample was around 500,000, which is more than adequate. The next thing to consider is whether we're really dealing with 2 different populations. To their credit, they do call this out:

Secunia [21] identified (for the month of May 2008) that 4.4% of IE7, 8.1% of Firefox, 14.3% of Safari (Windows only), and 15.2% of Opera users have not applied the most recent security patches available to them from the software vendor. In comparison, we discovered that 16.7% of Firefox, 34.7% of Safari (all OS), and 43.9% of Opera Web browser installations (using our Web server log-based measurements) had not applied the most recent security patches. We found that our Firefox, Safari, and Opera results were higher than those of Secunia's, differing by a factor of 2.1 (Firefox), 2.4 (Safari), and 2.9 (Opera), and attribute this difference to a probable bias for more security aware users to take advantage of Secunia's security scanner PSI than the average global community.

First of all, we can clearly establish that we're dealing with 2 distinctly different populations. The assertion that IE6 is insecure is invalid, as it is still in support and gets security patches just like IE7, and while IE 7 is doing a bit better on bulletin count, it isn't a huge difference, and as I've noticed with several other products, there is a grace period where attackers don't bother until there's enough adoption. While it is interesting that around twice as many Firefox users aren't fully patched as IE7 users, this might be an artifact of release timing. The authors of the study then attempt to deal with these different populations by comparing the Google results to the Secunia results, but there's a lot of variance between the browser types – if Firefox users going to Google are 2.1x less likely to be secure than Firefox users identified by the Secunia study, but Opera is different by a factor of 2.9x, then the difference between IE users overall vs. Secunia is really anyone's guess. Is it 2.9x? 4x? 1.5x? No one really knows.

It's an interesting thing to try and study, and the hypothesis that different patch delivery mechanisms might make a difference in how many users are at risk is also interesting, but data on IE users who are the majority of the population, and could behave differently as a group than users of other browsers, is really not available which makes the conclusions very questionable. Another factor that they appear not to have considered is that the number of browsers missing patches is going to be a function of how often you see patches. Something patched once a year is more likely to be patched than something patched 25 times a year.

Interesting paper – too bad their conclusions aren't supportable for the bulk of the users who are using IE.


Comments (3)

  1. Alun Jones says:

    There are some issues with your assertion that disclosing minor version numbers is a security issue.

    1. For some time, generally after the major versions are first released, the major version number _is_ the minor version number.

    [dcl] We can’t avoid all information leaks, just the ones that aren’t needed.

    2. Have you seen attacks that query for the version number and use that to select which exploit to use? I realise that browsers are clients, and I spend much time in server-land, but I’ve seen exploits attempted that aren’t even for the same protocol or platform, let alone the same version number, as the server they have reached. Clearly, the majority of attacks simply try “whatever exploit we have on hand”, without regard to version number checking. (This is a slightly circular argument, since of course they can’t check version numbers if they don’t have them)

    [dcl] Yes, I’ve written them. Some people write really dumb exploits. Personally, I prefer to only attack things when I know it will work. Sets off fewer IDS sensors that way.

    3. Attackers are aware that, even when a version number is given them, it may be a lie designed to conceal weaknesses. Therefore they ignore version information, and use fingerprinting instead.

    [dcl] Sure, but if they’re lying to you, they’ve probably applied all the patches, and you’re not getting anywhere anyway, AND they watch their logs. Fingerprinting doesn’t always work – there has to be some side-effect of the version that can be noticed.

    4. Version information can be useful to a peer, to allow them, say, to generate responses that will be parsed correctly on different versions that contradict one another in their interpretation of standards.

    [dcl] So then you need a document version, not an app version. Document versions change less frequently than patch levels.

    I’m sure I could come up with more, given time.

    [dcl] Keep them coming – I’m not convinced by any of these.

    Of course, if you have a flawed version of a program, the best answer is not to hide the version information, but to fix or work around the flaw.

    [dcl] Well, sure, but you wouldn’t want a big neon sign on your front door stating locked/unlocked, would you? Don’t give attackers information that isn’t absolutely required.


  2. Posting for Larry Seltzer:

    Feel free to post this as a comment to the entry. You may have seen my own column on this (

    My own IE7 USER-AGENT, as I describe it in the column, is this:

                   Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506; InfoPath.2; MS-RTC LM 8)

    There’s no browser minor version, but there is a lot of detail not usually necessary for a web server, such as the security components thing, the Live Meeting version, the CLR version, including minor version. Isn’t Microsoft being inconsistent here?

    [dcl] This is an area where you will see some inconsistencies. As you may have noted from Alun Jones’ comments, not everyone here agrees with me on this. Even if they did, it would take some time to get all of them fixed.

    You might note that the Office component only advertises major version.

  3. Alun Jones says:

    Please don’t take David’s note “As you may have noted from Alun Jones’ comments, not everyone here agrees with me” as a suggestion that I work at Microsoft – I don’t.

    But yeah, people inside and outside of Microsoft disagree with David on this and other issues. On most of those issues, David’s spot on – and I certainly hide minor versions from unauthenticated connections to my server.

    I don’t see it as necessary, but I do it because it’s easier to satisfy a customer’s request than it is to try and persuade them they’re wrong.

    I don’t think it’s likely to hurt much, but there’s always the possibility that I’ve changed some piece of functionality in a patch release that might warrant having a client detect the change by querying the minor version number.

Skip to main content