A HTTP Detective Story by Eric Lawrence

As a little kid, my dad read with me a lot; we usually read detective stories.  While of questionable literary merit, those books developed in me a burning desire to figure stuff out, to pull back the curtain, to understand the mysterious.  I still maintain that curiosity today-- I joined the IE team because I wanted to learn the browser inside out, and I developed Microsoft Fiddler to expose the secrets of HTTP in a user-friendly way.

I recently came across a bug in the bug database for IE7 which was intriguing— logins for Internet Explorer 7 and IE6 on XPSP2 had started failing for subscribers at a popular magazine’s website.  The site didn’t have any recommendations on how to fix the alleged “cookie flaw in SP2”, though the issue was discovered by their users during the SP2 beta.

Now, as most of you know, XP SP2 introduced major security improvements for users, while causing some issues for web developers.  Before we shipped SP2, we fixed a ton of compatibility issues to ensure that IE continued to work on the broadest range of sites without making security sacrifices.  It looks like we missed a compatibility break; this bug still hadn’t been resolved, so I dove in.

I fired up my trusty HTTP debugger, Microsoft Fiddler.  Some developers in the audience might be scratching their heads-- I’ve got the full source for IE on my computer, so why start by looking at the traffic?  Simple—it’s usually the clearest way to see what’s really going on.  HTTP is beautiful that way— if you can see both the traffic of a working scenario and the traffic from a broken scenario, you can resolve most problems of this nature. 

The site noted that subscribers using Mozilla FireFox or Opera could log in just fine, so I hooked FireFox up to Fiddler and watched the traffic flow by.  Nothing seemed amiss, and the login worked as expected.  Okay, now to try out IE on XPSP2…  I performed the same login procedure, using the same credentials, and… hrm.  That’s odd.  The login silently failed and I was left sitting at the “Please sign in” button on the homepage.  Very mysterious-- my curiosity was piqued. 

I started by filtering the Fiddler session list to ignore all of the image and CSS requests, since they’re unlikely to be related to login problems.  This left two HTTPS pages and two HTTP pages. 

  1. https://example.com/loginform.asp submits to
  2. https://example.com/loginverify.asp which then 302 Redirects to
  3. http://example.com/auth.asp which then 302 Redirects to
  4. http://example.com/index.asp

Using the Microsoft WinDiff feature inside Fiddler (select any two sessions, right-click and choose WinDiff from the menu), I found the body of the POST was identical in both cases, so that wasn’t the culprit.  I also determined that there are no extra cookies being set or sent to FireFox, so the magazine’s hypothesis that this bug is caused by cookies is rejected. 

The only cookie being passed around is a standard ASP Session ID cookie; I eventually decided that this cookie was probably associated with a serverside session variable set inside of auth.asp, e.g. IsLoggedIn. 

I carefully looked for differences in the headers… Nothing jumped out at me.  On a hunch, I added the following rule to my Fiddler Rules file:

static function OnBeforeRequest(oSession:Fiddler.Session)


     oSession.oRequest["User-Agent"] += " Gecko";


This rule will append “Gecko” to the browser user-agent header.  I used Fiddler’s menu options to clear my cache and cookies and then retried the login using IE. And this time, login succeeded.  What?!?  That can’t be right.  I disabled the rule, cleared the cache and cookies, and tried it again.  Login silently failed.  Hrm.  Let’s try something else:

static function OnBeforeRequest(oSession:Fiddler.Session)


     oSession.oRequest["User-Agent"] = "Opera/8.0 (Windows NT 5.1; U; en)";


Login was also successful using “Opera” as the user-agent string.  Interesting.  Someone doesn’t want IE to log in.  But why?

As with many mysteries, even though I thought I had collared the culprit, it turns out that something far more interesting was afoot.  A single fact kept tickling at the back of my mind… The bug says this site used to work before XP SP2 was released.  How could that be? I don’t have an unpatched copy of IE handy, but I might be able to fake it…

static function OnBeforeRequest(oSession:Fiddler.Session)


     oSession.oRequest["User-Agent"] = "Mozilla/4.0 (compatible; MSIE 5.01; Windows 98)";


Login still failed.  There must be something else that differs…  I then recalled that I’ve got a rusty old Windows 98 image on a Virtual PC.  I fired up the Virtual PC, pointed the proxy configuration at my XP SP2 box running Fiddler, and ran the login.  It worked.  Bingo.

With the click of a mouse, I performed a WinDiff of the Windows 98 login traffic against the XP SP2 login traffic…  And I had my man.

You see, the Windows 98 request for Auth.asp carried the header

Referer: https://example.com/loginverify.asp

This violates the RFC which states “Clients SHOULD NOT include a Referer header field in a (non-secure) HTTP request if the referring page was transferred with a secure protocol.”  We fixed a bug for XP SP2 where if a HTTPS page 302 redirected to a HTTP page, the Referer was incorrectly being sent.

Could it be so simple? I added the following rule

static function OnBeforeRequest(oSession:Fiddler.Session)


     oSession.oRequest["Referer"] = "https://example.com/loginverify.asp";


And login now worked on Windows XP SP2. 

Wow, that’s really silly.  Rather than developing a real security mechanism, they’re using a crusty old bug to check to see if the user is coming out of the secure login page, under the assumption that if they’re getting to Auth.asp from loginverify.asp, the login info must be correct!

It’s been said before, but I’ll say it again… never fully trust header data coming from the client, including the Referer.  It’s horrendously insecure; as we’ve just shown, it can be bypassed in a single line of code. 

Without ever looking at the source for IE or the source for auth.asp, I was able to deduce:

  • The Auth.asp page logic expected a security flaw in Internet Explorer 6.
  • Opera and FireFox never sent the referer, so any request carrying their user-agent was automatically permitted.
  • Microsoft fixed the flaw for XPSP2, causing Auth.asp to fail to work as the author expected.

I sent off my analysis to the web developer for the magazine and as I type this, logins now work with XPSP2 and IE7.  Case closed.


PS: The second installment in the Fiddler on MSDN series is almost ready… The new article is all about improving your websites’ performance.

Updated: changed "site.com" examples to be "example.com", which is correctly reserved in RFC2606 for this purpose.

Comments (60)
  1. Anonymous says:

    Nice story — good to see another anti-IE argument debunked.

    Also good to see regular posts to this blog. Looking forward to hearing more about the features of IE7.

  2. Anonymous says:

    Good work, this is the sort of article that I expect to read here. No hazyness of upcoming things and nice clear adventure with even some morale to it. Now, if someone tried to do the same sort of article about transparent png – it’s haze all over again. Stick to the present and past and suddenly everything clicks. Good read.

  3. Anonymous says:

    Great read, thanks!

    It’s scary what some web developers do in the name of "security".

  4. Anonymous says:

    Gazzilions of sites (or rather uneducated webmasters) use browser detection ;(

    These are biggest cause of "bugs" in Firefox and Opera. I didn’t suspect that it can occur to IE too πŸ™‚

    Maybe IE7 will have to have User-Agent like:

    "Mozilla/4 (MSIE 6.0) Opera/7, but really MSIE/7.0 (not Gecko or KHTML)"


  5. Anonymous says:

    Great read! πŸ™‚

    Btw, what would you do with Developers who has designed application in a way that a 7 year old can *hack* it. Read http://blogs.law.harvard.edu/philg/2005/03/08


  6. Anonymous says:

    Thanks for this! What’s great is that even though I’m no developer, it reads as a detective story. Keep up this kind of writing!

  7. Anonymous says:

    This is a great article, it’s definitely a *must-read* for any wannabe web developer. haha… brought tears to my eyes (good ol’ days of hacking through a friend’s web-based game with Proxomitron)

  8. Anonymous says:


  9. Anonymous says:

    A great article. Thoroughness and engaging the brain instead of the mouth win the day!!

  10. Anonymous says:

    Fun story… and *thank you* for informing the developer instead of <a href="http://blogs.msdn.com/ie/archive/2005/02/01/364581.aspx">taking out the bug fix</a>! πŸ™‚

  11. Anonymous says:

    Great story — just the kind of content I was hoping to see on this blog. Keep it up!

  12. Anonymous says:

    It’s good to see you chose standards compliance with HTTP over bugwards compatibility, and it’s also good to see the web developer fixed the problem so quickly and easily after being notified. I wonder how many other broken sites could be fixed so easily, if the bugs they depended on in IE were fixed like this one was.

  13. Anonymous says:

    LiveHTTPheaders extension for Mozilla is not such a resource hog:


    That Fiddler has some nice features though.

  14. Anonymous says:

    That’s a great story showing the scientific method in action to solve problems.

  15. Anonymous says:

    Great post.

    When you think about it, this is a great example of why IE should STOP supporting quirks mode. Not that it is a security issue or anything. But some stuff in IE6 (and IE5 etc etc) is broken and web developers should know this.

    Backwards compatibility isn’t always a good idea.

    Keep up the good work though πŸ™‚

    *is hoping IE7 will support standards*

  16. Anonymous says:

    via1="Anne van Kesteren: HREF":http://annevankesteren.nl/archives/href/2005/03

  17. Anonymous says:

    I don’t understand why the web developer hadn’t debugged this himself. He would have found that the reason the logon was failing was the lack of that header, and maybe before contacting Microsoft PSS he would have investigated and found that the RFC said it shouldn’t even been there. He would then (hopefully) arrived at the conclusion that MS stopped IE from including that header as a bugfix.

  18. Anonymous says:

    It’s Firefox, not FireFox.

  19. Anonymous says:

    Best post on this blog ever!

    Netscape used to employ an evangelism team where they’d contact webmasters whose sites would not work properly in Netscape or other Gecko browsers and help them get their sites working. It’d be great if Microsoft had a team of people doing what you just did to help web developers write to the standards and then you could trim a lot more of this backwards compatibility.

    The fiddler tool sounds very powerful, I find the Live HTTP headers extension in Firefox invaluable and fiddler sounds like it’s even better.

    I’ve been spending most of my time in Linux or a Mac lately but will check out that tool next time I’m in Windows.

    Well done on your efforts to get to the bottom of this, if it was me doing the investigating I’d have come to the conclusion that the webmaster didn’t like IE after it worked when changing the user agent.

  20. Anonymous says:

    Dave: keep in mind that its a lot more than just web developers who depend on the backwards compatibility. There are probably thousands of programs out there that embed the WebBrowser control or MSHTML and would have potential compatibility issues.

    As I’ve suggested before, MS could keep IE6 components exactly as they are now, and use Side-by-Side components for all future versions, such that apps use exactly the version they need. This would enable them to make radical changes to IE without having these huge compatibility issues.

  21. Anonymous says:

    "Wow, that’s really silly.  Rather than developing a real security mechanism, they’re using a crusty old bug "

    Sure, but what’s even sillier: apparently, you are not even surprised or sorry that the bug has been out there for such a long time (Win98 or even earlier? And YOU only were able to fix that in XP SP2??? Geeez, what have you been doing all the time?)

    Given these facts, I smell more than just a little bigotry when Microsoft comes along now and points at RFCs and tells all the (good or bad does not matter here) web developers how silly they are after all the years that they have had to live with all those stupid bugs in your engine / browser – you certainly must be kidding if this is all their fault now. Develop your software standards-compliant from the very start and this won’t happen, simple as that.

    Even the worst web developer would develop standard-compliant sites if he had to. Microsoft, in contrast, has introduced proprietary hacks (NTLM and many more), tolerated stupid HTML and even offered authoring tools for stupid, non-standard HTML (Frontpage). So who are you to point your finger?

  22. Anonymous says:

    Heh. Fascinating and hilarious. I’m sure we’ve all met horrific hacks like that one!

    Fiddler looks really fantastic BTW (although it’s a minor shame .NET 1.1 is required).

    I am personally sick to death of User-Agent sniffing/faking, and would personally donate UKP100 to Microsoft to simultaneous sabotage and fix it, by making the UA string for IE7 simply "IE/7 (Win32)". πŸ™‚

  23. Anonymous says:

    Here here ralph.

    Whats worse is when MS claims they can’t fix the bugs because too many people use them or have implemented hacks to get around them.

  24. Anonymous says:

    <<you certainly must be kidding if this is all their fault now.>>

    Ralph– I think you’re a bit confused. Sure, Microsoft had a bug… But trying to exploit that bug to build a login mechanism is just dumb.

    The website ~must~ have known this behavior was a bug because they disabled this dubious security speedbump for Firefox and Opera.

    And then to go so far as to claim there’s some mysterious "cookie flaw" in XPSP2 and advise people not to upgrade to SP2 because of it is just nasty.

  25. Anonymous says:

    Even though the other browsers still do it wrongly I am glad to see you guys not eating up the Microsoft ‘it has to be backward compatible’ cake when something needs to be fixed.

  26. Anonymous says:

    Nice article, although I echo Tim’s sentiment that the web developers should have been perfectly capable of fixing this themselves (and that they shouldn’t have been doing this in the first place!).

    Please refer to RFC 2606, sections 2 and 3 though, please. You don’t own the site.com domain, and example.com etc are created especially for this type of situation. You have broken links in your post – any decent weblog system should be able to avoid auto-linking example URIs like this.

    Making up domains for examples instead of using the ones reserved for such purposes only makes it more difficult for automated tools to pick up broken links, both for you and for the owners of the domain you are abusing, since it introduces unwanted noise.

  27. Anonymous says:

    Oh, so I should remove the section of my code that checks for IE and serves it text/html instead of application/xhtml+xml? I mean, if I’m not to trust header data πŸ™‚

  28. Anonymous says:

    And Clover:

    > I am personally sick to death of User-Agent

    > sniffing/faking, and would personally donate UKP100 to

    > Microsoft to simultaneous sabotage and fix it, by

    > making the UA string for IE7 simply "IE/7 (Win32)".

    Of course Microsoft were the ones that seem to have invented deliberately confusing user-agent strings.

    The original Netscape UA was like follows:

    Mozilla/version (OS; I)

    Mozilla was the codename of Netscape.

    Then IE decided it wanted to work with all sites that allowed Netscape so they became

    Mozilla/x.x (compatible; MSIE version; OS; etc)

    The Opera had to work with sites that only allowed IE so it defaulted to

    Mozilla/4.0 (compatible; MSIE x.x; OS; etc) Opera/version

    Although opera allows the spoofing to be turned off.

    Firefox doesn’t pretend to be any other browser but it has one of the longest user agents that details the version of Firefox, the build date and the version of Gecko it’s based off.

  29. Anonymous says:

    Boys and girls … If you want similair functionality to "fiddler" in Firefox check out: http://livehttpheaders.mozdev.org/

    Don’t want to be rude by posting this in an IE blog, I’m patiently waiting for IE7 … IE6 incompatability is HUGE pain, I don’t develop in it and thought other people don’t aswell.

  30. Anonymous says:

    >I mean, if I’m not to trust header data πŸ™‚

    I assume you’re just teasing, but do note that I was saying that trusting header data ~coming from the client~ when making security decisions is a bad idea.


    Interesting note on RFC2606. Thanks.


    Vis-a-vis LiveHTTPHeaders– Yes, it’s an cool little tool, although I haven’t encountered anyone who has used both LiveHTTPHeaders and Fiddler that considers their functionality that similar. The original blog posting for Fiddler has a discussion about the additional power in Fiddler.

    There are a number of LiveHTTPHeader-like plugins for IE, including ieHTTPHeaders.

  31. Anonymous says:

    "I have the full source code of Internet Explorer on my computer".

    Very good! How come you avoid being hacked? Having the IE source code and browsing the dangerous Internet trying out even older IE versions on the same machine. Aren’t you afraid? There is always something unpatched in IE, even in the latest version, so one day that source code might be stolen. Don’t you protect against this? I thought that the code was safely stored on a special server disconnected from the Internet, not on every developer’s machine.

  32. Anonymous says:

    Its great that IE isn’t worried about backwards compatability when it comes to things like security. I do wish, however, that this mentality would transfer over to its rendering engine and standards. At some point I would like to see IE say "Ok, well it is time to move forward – sorry all you ancient programs"

  33. Anonymous says:

    > Oh, so I should remove the section of my code

    > that checks for IE and serves it text/html

    > instead of application/xhtml+xml? I mean, if

    > I’m not to trust header data πŸ™‚

    Yes, you should. Use content negotiation instead.

  34. Anonymous says:

    > Use content negotiation instead.

    Actually, you still need to check for Internet Explorer when doing conneg.

    Set your server up so that you prefer to serve application/xhtml+xml over text/html for user-agents that support it.

    This means that Firefox etc will get application/xhtml+xml and Internet Explorer will get text/html, right? Wrong.

    The problem is that when you hit reload in Internet Explorer, instead of sending its usual Accept header, it sends Accept: */* instead, giving equal weight to each media type. Since text/html isn’t mentioned, it doesn’t get priority over application/xhtml+xml.

    The end result is that you set up your server, check it in Internet Explorer and find nothing wrong, make the updates to the live server, and get bombarded with complaints from end-users, since it breaks for them when they hit the refresh button.

    Yet *another* bizarre Internet Explorer bug making things difficult for us!

    Hey, Internet Explorer developers, there must be a good reason for you transmitting a dummy Accept header on reload, but I can’t think what it is for the life of me. Care to enlighten me?

  35. Anonymous says:

    I agree with creativeHavoc. A significant portion of the Internet would probably be "broken", but it would be a push toward proper implementation of Web standards. The current XHTML 2.0 draft seems to be aiming toward Web authors who are serious about following the standards. However, without a *decent* rendering engine, those standards are useless.

    Also, why must IE be "hardwired" into the OS? It should be an optional component when installing, separate from the Windows Explorer interface.

  36. Anonymous says:

    <<Internet Explorer developers, there must be a good reason for you transmitting a dummy Accept header on reload, but I can’t think what it is for the life of me. Care to enlighten me? >>

    I wondered about this a lot before I joined the IE team. I’ll take a look.

  37. Anonymous says:

    I am curious about this line in the story:

    &ldquo; Opera and FireFox never sent the referer, so any request carrying their user-agent was automatically permitted. &rdquo;

    Do you mean to say that this site has such horrendous security that anyone going to http://example.com/auth.asp”>http://example.com/auth.asp without a Referrer: header is automatically logged in? As in, anyone can fire up any browser they want (before you sent in the fix, only Firefox or Opera, but now, apparently, any browser at all), type http://example.com/ in the browser bar to set the ASP session ID, and then type http://example.com/auth.asp”>http://example.com/auth.asp in the browser bar, and be logged in?

    I could actually believe this, given some of the horrendous web application security I’ve seen elsewhere, but if so then there really should be an MSDN article describing how to do this correctly – that is, how to start a session over http, accept the login credentials over https, and then continue a logged-in session over http without requiring a common back-end database for the http and https portions of the website. (I can think of at least one method, but yo no hablo asp. It does require a shared secret phrase between the two halves, but I think that’s acceptable.)

  38. Anonymous says:

    <<Do you mean to say that this site has such horrendous security that anyone going auth.asp without a Referrer: header is automatically logged in?>>


    There are actually quite a few ways of doing this securely; lacking a common backend does make it more challenging, but certainly not impossible. One thing to keep in mind is that compared to the current approach, pretty much anything is an improvement.

    I’m not an ASP pro myself anymore, but the following shared secret approach would seem to work…

    When redirecting the user out of the SSL tunnel back to the HTTP site, send along two query string parameters, the username and a hash:

    SHA1(Secret + Username + Date)

    And when the HTTP page gets the request, rehash the Date, Username, and Secret; if they don’t match, then don’t log the user in.

    The two servers need to share the secret; the date is to make replay attacks less useful.

    Of course, you could also just move the login to the HTTP frontend and use JS on the client to MD5 hash the credentials before submitting. Then, use a webservice on the HTTPS site to verify the credential hash. This is how, for example, the Yahoo mail login appears to work.

  39. Anonymous says:

    while you are on the topic of the user string is there any prblem with MOCA (Microsoft Online Crash Analysis) on a WIN 98 machine with IE 6.0 SP 1 NET Passport Unavailable at This Site Help

    The .NET Passport service is currently unavailable at this Web site for one of these reasons:

    .net passort not avaible on a Microsoft site. HAHA

  40. Anonymous says:

    Nice article.

    However, I am amazed at how you can fix a bug years old in XP2, and then say how people were using that "bug" in order to perform some action? You joking?

    So, let’s assume that Mozilla/Opera did not implement that ‘feature’ which was present in IE for ages, and were following RFCs properly (which they usually try to do). Then, magazine site would only work with IE, and Mozilla/Opera would NOT work. What would happen? Everyone would blame Mozilla/Opera for not working properly, even if they followed RFCs. Granted – it’s developer error, but you’ve placed mechanisms (not standard compliant) in IE which allowed him to do that.

    Man, how many things did I have to change in my VALID HTML/CSS code in order to make it work under IE…

    Now, I can expect SP3 which will fix some more bugs. Then, I will realize those "IE fixed" pages won’t display in IE anymore, since you’ve finally fixed bugs which were years old.

    And then you will blame ME for that?

    I am sure Microsoft has tons of good coders, and I respect them. But don’t give me this crap about you being ‘standard compliant’ when we all had to break standards numerous times (for YEARS) in order to make your products work properly.

  41. Anonymous says:

    Another site (http://www.meryl.net/css/) for CSS inspiration that lists Big Companies and Non-D (non-designer related sites).

  42. Anonymous says:

    Interesting finds this week

  43. Anonymous says:

    Very intersting. Hope FF gets a crack at it….

    Very nice how you thought of it.


  44. Anonymous says:

    very scary that there are developers out there (quite possibly developing a site that’ll be storing my personal data) that "code" garbage like that. Ugggg… coders should be freakin licensed to do this job.

  45. Anonymous says:

    <<having gone through the much vaunted "compulsory security retraining" still advocate the use of techniques like MD5 or SHA1 hashing >>

    1> I’m not a Microsoft developer.

    2> What algorithm do you suppose SSL uses for hashing?

    3> SHA1 has been "broken" insofar as you can find a collision in 2^69 cycles instead of 2^80. MD5 was broken worse, but it’s still non-trivial. We’re talking about protecting login to a magazine site; you could buy a subscription for ~way~ less than you could purchase the grid of computers needed to crack the hash. And don’t forget, we’re changing the hash every day.

  46. Anonymous says:

    I think it’s despicable that so called Microsoft developers, having gone through the much vaunted "compulsory security retraining" still advocate the use of techniques like MD5 or SHA1 hashing which are proven to be flawed not only in theory, but in practice also.

    No wonder your products are so buggy.

  47. Anonymous says:

    I don’t get it:

    > I joined the IE team because…


    > I recently came across a bug in the bug database for IE7…


    > I’ve got the full source for IE on my computer…


    > I wondered about this a lot before I joined the IE team. I’ll take a look.


    All of this I understand. Until I read this:

    > 1> I’m not a Microsoft developer.


    Was somebody else posting as Eric?

  48. Anonymous says:

    That comment was from Eric and yes, Eric isn’t a developer, his position is "program manager".

    You can read more about program management at Microsoft at http://www.microsoft.com/careers/careerpath/technical/programmanagement.mspx

  49. Anonymous says:

    Hey, InterNet ExPlorEr GuYs*, it’s "Firefox", not "FireFox"… nor "FiRefOx", nor "FirEfoX" etc.

    (* That there be satire.)

  50. Anonymous says:

    So, it needs a direct Internet connection?

  51. Anonymous says:

    I’ve got a good detective start for you:

    what on earth dictates when IE will save my username and password? It seems to happen randomly. Firefox seems a lot easier to use in this regard.

    Also, for IE7 wouldn’t it be awesome to have multiple-login choices for sites, both for the standard login popup box and on sites?

    Often I use 4 or 5 logins, giving me access to different parts of sites.

  52. Anonymous says:

    So FireFox and Opera honour the RFC’s by not sending a Referer to an HTTPS site, and MSIE fails the RFC’s.

    The fix is to make MSIE honour the RFC’s, not to workaround it.

    If MSIE honoured RFC’s, it would not have had these problems in the first place.

    The fact that it used to break RFC’s in one way, and now breaks RFC’s in a different way, is not an excuse. It is not a "detective story". It’s a bug.

    Fix the bug.

    The pleasure about not having to fix the MSIE code I can understand (it must be a nightmare!), but that is what needs to be done.

  53. Great post on using his tool, Microsoft Fiddler , to track down a web site issue by my friend and colleague,

  54. A HTTP Detective Story by Eric Lawrence shows how: 1 – resoundingly useful the Fiddler tool that Microsoft have released is. 2 – resoundingly stupid some people can be. Non-withsthanding the fact that Internet Explorer (pre XPSP2) returned the referer..

Comments are closed.

Skip to main content