IE Code quality commentary…

I just saw this post by Michal Zalewski on BugTraq.  From the post:

It appears that the overall quality of code, and more importantly, the
amount of QA, on various browsers touted as “secure”, is not up to par
with MSIE; the type of a test I performed requires no human interaction
and involves nearly no effort. Only MSIE appears to be able to
consistently handle [*] malformed input well, suggesting this is the
only program that underwent rudimentary security QA testing with a
similar fuzz utility.

I’m wondering when Michael’s post will show up on slashdot.

Edit: Corrected Michal’s name – Sorry about that.


Comments (45)

  1. Anonymous says:

    "I’m wondering when Michael’s post will show up on slashdot."

    It’s now submitted, let’s see what happens.

  2. Anonymous says:

    I have managed to crash IE using pretty straight forward Javascript code, so take these tests with a grain of salt. The tests are ‘targeted’ to non-IE browsers. If one was so inclined, they could do the exact opposite – write tests targeting IE and comment ‘Non-IE browsers have much better code quality since they do not crash on these tests but IE does’.

    This is not to say the browsers which did crash on these tests don’t have bad code, but so does IE. Given so many IE exploits, one can’t say IE code base is very high quality, it may still have a plethora of holes of which no-one knows yet.

  3. Anonymous says:

    Michael wasn’t testing the javascript interpreter, he was testing the HTML renderer. I’m sure that his results would be different when looking at a different component.

    But in this case, he wasn’t testing something that was targetted at non IE browsers. As far as I know, all browsers CLAIM to handle HTML (if any of the tested browsers don’t claim to support HTML, please let me know).

    In this case, he was simply performing a basic security test that should be performed by EVERY test department: Fuzzing the input.

    In other words, he took valid inputs and made them invalid in various ways, and tried to see what would happen when the browser tried to render the HTML.

    Remember – the bad guys don’t write valid HTML. They write INVALID HTML. So if all your security testing is done with valid HTML, you’re not thinking like a bad guy.

  4. Anonymous says:

    I simply won’t consider FireFox until they implement a security manager. Currently XPCOM binary extensions have no security model at all. My complaint about IE’s security manager is simply that it’s pretty hard to say ‘open this link in Restricted Sites’.

    However, they’re doing better than the Linux kernel – they actually have smoke tests and test plans. I cannot *believe* the amount of praise that Linux gets when it’s such an unknown quantity. Lest you suggest I have no experience, I was an active Linux user four years ago in the late 2.0.x/early 2.2.x days, and I clearly recall the regular disk-trashing bugs that appeared in the early 2.2 kernel series. The clean-room journalled filesystems are still a joke – your data is safer with ext2 than with ext3 or ReiserFS. If you want a journalled filesystem, go with SGI’s XFS or IBM’s JFS. Linux sites are deluding themselves that there are no problems in the OS. You have literally no way to know whether a new kernel release will work correctly on your system.

    Mozilla/Firefox smoke testing still appears to be post-checkin, though, not pre-checkin as I believe has become common practice at MS. It’s not automated, which suggests the software wasn’t designed-for-test.

    If you choose Microsoft software (and to a greater or lesser extent commercial software in general), you have to believe that Microsoft have tested the software to the best of their ability, and that the build/test labs that are mentioned genuinely exist, and that Microsoft personnel, and consultants hired by MS, have performed the security reviews they say they have and that they’re skilled to do so. With Open Source, at its worst you have to believe that a nebulous collection of unknown people, of unknown size, of unknown skill, review all changes made to the software, typically with no release plan of features that are to be included.

    For serious business purposes, I know which one I choose.

  5. Anonymous says:

    Hole is a hole – no matter how it was exploited – thru architectural ignorance or otherwise. So your argument that HTML parser is significant than other portions of a browser is not valid. IE has bad code inspite of Microsoft’s so called testing efforts and non-IE browsers too have bad code (arguably in different places) inspite/despite of their testing efforts. So what are we so happy about?

    Lets see in how many *days* Mozilla issues a fix and find out how what’s the least it took Microsoft to issue fix for any of the previous exploits.

    Piece of software as complex as browser is going to have bugs – What matters is how soon it gets fixed and how many users does it affect. IE is _bad_ in both cases – it affects hell lot more users and Microsoft wasn’t anywhere near quick to issue fix to known exploits.

  6. Anonymous says:

    Doesnt Matter: No, I’m NOT arguing that an HTML parser is significant.

    But I AM arguing that if their HTML parser failed this basic test, what will happen to their JavaScript interpreter? What does this say about the methodologies used to test the components that make up their system?

    If extensive regression testing of a fix isn’t a criteria, then your time to fix can be way smaller than if you have to run large regression suites.

    I don’t have numbers, but I’m wondering how many fixes to Mozilla have to be revised after they were "fixed"? I also don’t have numbers for MS products, but I suspect (with no strong evidence) that it’s somewhat lower.

  7. Anonymous says:

    Of the reported 3, 1 didnt crash on anyone, 1 was already fixed in Dev builds and I fixed the 3rd one myself – a simple NULL pointer deref. Within _hours_ everything suddenly feels safe, without having to hopelessly depend on the vendor to fix the problems. Isn’t this magical compared to what would have happened with a closed source product?

  8. Anonymous says:

    Larry – Extensive regression testing applies to only such things as architectural changes. For instance – Previously you used to allow to run a active X control if you thought you are in Local zone. Now some one is able to trick you that you are in Local zone even though you aren’t. Then you got to fundamentally change the way you arrive at what is Local zone. That’s going to break may be dozen things that rely on the original buggy way of your thinking and then yes – you need to regression test it. If it breaks – you need a ugly workaround instead of an elegant fix.

    Why would someone need to worry about regression in case of NULL pointer deref?

    It’s altogether a different and easy game with OSS – If a elegant fix breaks something then you can easily/elegantly fix the source of the problem and all other dependents who rely upon that bug – no ugly workarounds and hell lot of regression testing is necessary.

    And I don’t understand what you said in your last statement – I haven’t heard Mozilla had to fix their fix any time – but I definitely remember couple such things happening with MS fixes.

  9. Anonymous says:

    I don’t know if all bug fixes made with Mozilla have been error free. I do know that on other OS projects, it has taken several revisions to create a security fix that didn’t itself introduce new bugs.

    In <i this /i> case, the fix may have been simple and clean and easy. In other cases, it’s not at all as clear.

  10. Anonymous says:

    10/18/2004 2:38 PM Mike Dimmick

    > I clearly recall the regular disk-trashing

    > bugs that appeared in the early 2.2 kernel

    > series.

    OK, I don’t because I didn’t experiment with Linux in those days. I remember Windows 95’s disk-trashing bugs from those days, and I remember Windows Server’s 2003’s disk-trashing bug from a few weeks ago. And Windows 2000’s disk-trashing bug from a time midway between those two.

    > Linux sites are deluding themselves that

    > there are no problems in the OS.

    I haven’t seen that, unless you mean some of the marketing pages on commercial vendors’s sites. If you hate the marketing more than I do, you’re in luck: you can buy a computer, even a notebook computer, without paying for an unwanted copy of Linux. I’ve seen lots of sites reporting problems in Linux. My own opinion also is that there are two essential differences between Linux and Windows:

    (1) With Linux you DO get what you paid for (except if you paid for it).

    (2) With Linux if something needs fixing, and if you’re a programmer, then you DO have a snowball’s chance in hell of fixing it.

    > If you choose Microsoft software […] you

    > have to believe that Microsoft have tested

    > the software to the best of their ability

    No way. Things as trivial installing Windows 98 Service Pack 1 (onto Windows 98 first edition), rebooting, and clicking the Start menu; or as trivial as installing Word 2000 upgrade on an existing Office 97 installation and clicking the Start menu, etc., pretty clearly demonstrate that Microsoft never tested them. Sometimes Microsoft tests the US versions of their products, but the vast majority of their products don’t benefit from that.

    And then last weekend I tried installing .NET Framework 1.1 SP1 onto Windows Server 2003. There’s a special version of that service pack for Windows 2003, separate from the version for the rest of Microsoft’s OSes. And it doesn’t even install, it tries to dereference a null pointer during installation. That’s quite a reassuring security fix eh?

    This doesn’t mean Linux is better, it just means Windows isn’t.

  11. Anonymous says:

    To summarize – Software is hard to get right, _humans_ code software as of now and thusly there is every chance that it is not perfect – But you are better off when you have the source with you. You can fix it by some means if nothing else works out. You don’t have to be at anyone’s mercy.

    And most importantly if everything in open and out there, you get elegant fixes instead of mere workarounds and you have the ability and capacity to correct the design if need be, without having to worry too much about how many other closed things it might break. (Linux USB API is a good example of this – they changed it thrice and they fixed all the drivers dependent on it – no ugly workarounds and bloat.)

  12. Anonymous says:


    The teardrop TCP security fix took Microsoft two attempts to get right back in the day (and significantly longer than the equivalent Linux kernel patch).

    Software development in general is beginning to wake up to the needs of security and the basic truth that pretty much any bug can be a security hole.

    It’s grossly unfair to state that the open-source world is radically worse than commercial vendors at this – security has taken a back burner for a lot of people. However in general, the people crying out at the beginning were much more able to work on open-source projects. If you look at the age of bounds-checking patches for gcc, at anti-stack-smashing approaches for the Linux kernel, amongst other things, these were all done before the big recent stink about security.

    I think it’s a good thing that Microsoft have ‘gotten’ security – I think a lot of people underestimate what Microsoft can do, but to state that the open-source world is significantly worse is to do large portions of it a disservice. I was very impressed having had a quick look at some functions in MSDN that they have security notes accompanying them (strcat, strtok, sprintf), and I hope that this makes commercial vendors take notice.

  13. Anonymous says:

    didn’t crash firefox for me,

    Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20040913 Firefox/0.10.1

  14. Anonymous says:

    That’s quite interesting!

    However: One of the biggest flaws with IE is that it approves also broken code (html etc) for rendering and renders it. EVEN when the rules of a technology explicitly say that the parsing MUST be stopped at the first error.

    The fact that IE has been always so forgiving on the code has contributed greatly to the fact that most of the code on the Interweb is just plain crap in quality.

    I would personally like to drag every IE developer behind the sauna to be put out.

  15. Anonymous says:

    I really dont know what is wrong here. Tested on firefox PR1.0 XP SP1 fully patched and there is is no crashing evidenced, even after several refreshes.

    However, I know that even a ‘hardened’ IE is no where near as safe as firefox in regards to viruses and spyware. I guess your not cleaning PC’s for a living?

  16. Anonymous says:

    Based on the specific URL’s he has provided it appears that FireFox PR1 crashes on Mozilla-Die1 and 2 but it ok on all others.

    I agree that the provided tool should be run agains Mozilla and firefox for a considerable time to determin anyother code errors.

    As for the issue of rendering bad HTML I am against it, however old sites should not be shunned. I think that if a browser incounters a doctype in the HTML header then it should be enforced. and an error presented about bad html, with an option to do a best effort.

    If all browsers did this then all webdevelopers would produce valid code.

  17. Anonymous says:

    The live ‘lite’ script didn’t kill my Firefox 1.0PR, but the natty Javascript console popped up to warn me of the illegal character. I do like Firefox’s debugging tools.

    I’d say Firefox, in the build up to their first full release have been doing exactly the kind of tests Larry claims they don’t do *

    * not based on any knowledge of any kind whatsoever, this is me speculating.

  18. Anonymous says:

    Fun for Larry (just hover mouse over the link in IE – how did MS testing miss that one out? sorry if you already read slashdot 🙂 –

  19. Anonymous says:

    [quote]I really dont know what is wrong here. Tested on firefox PR1.0 XP SP1 fully patched and there is is no crashing evidenced, even after several refreshes.[/quote]

    Same here. It would be nice to know what versions he was running… I didn’t see anything about that in the article… but I could have missed it.

  20. Anonymous says:

    DoesntMatter: Works just fine for me in IE6 on SP2, no crash here.

  21. Anonymous says:

    I find this one also (also shown in Slashdot) really funny:

    I think it doesn’t break Explorer now but it lasted quite a long time. My AVirus detects that web page as a Trojan when seen in Explorer. I find it strange that an AVirus does the work that the explorer should be doing.

    This errors are just HTML errors, no javascript involved. Are those workers paid for NOT doing what they should do?

    > you have to believe that Microsoft have

    > tested the software to the best of their

    > ability

    The thing is that I don’t believe that any more (and many people agree with that). Users were doing beta testing for them. Now they are starting to get better but … sorry, too late.

  22. Anonymous says:

    OK it doesn’t crash in Explorer XPSP2 … please, when will I have it for my Windows98, 2000, ME, …?

    I find it funny that MS is saying that they have improved many things in Explorer XPSP2 when most Windows users can not use it (just because they are not using WinXP).

    Sorry, that’s not an answer.

    Oh, if you would like to know, I have XPSP1 fully patched … and that web page crashes the browser.

  23. Anonymous says:

    IE with tabs : (pre-empting any firefox zealots screaming about IE having no tabs. Also features adblocking, google bar support.

    Incidently, I wonder if firefox PR 1.0 still has the proxy bug that means you get an authorisation dialog for every resource.

    I’m not anti-firefox, I use it myself, it’s just people moaning about IE when it was a leader for years gets a tad laborious.

  24. Anonymous says:

    Strange, the first example crashes my IE with XPSP2. My IE version string is 6.0.2900.2180.xpsp_sp2_rtm.040803-2158

  25. Anonymous says:

    incidently doesn’t crash with myie2/maxathon

  26. Anonymous says:

    Something great to mention: none of the tests seams to crash Konqueror 🙂

  27. Anonymous says:

    sounds like many of those bugs have already been fixed on various platforms/updated versions of FF. and for those that haven’t, i’m sure they will be fixed quickly enough. i hope Zalewski continues to find bugs so that the quality of the code will continue to be improved by open-source developers worldwide.

    really, who needs IE anymore except MS to try to trick/force users into being locked into their proprietary stuff.

  28. Anonymous says:

    The Mozilla code-base has never impressed me in the first place.

    I ran a Cyber Cafe for a year on GNU/Linux server w/ X Terminals and Mozilla was by far the most unreliable application–freezing entire user sessions.

    Firefox is a huge improvement and now seems reasonable. IE on Windows XP and Win2K3 Server also crashes a lot for me–do not know why.

    BUT–once Konqueror is properly configured (cause it never is, out of the box), it’s highly reliable. In the past, it still had rendering issues but nothing significant any longer. Even when it did crash, it didn’t freeze up a user session like Mozilla or, sometimes IE. Konqueror, after all, isn’t a browser–it uses the khtml kpart to render (and the most recent versions of khtml also enables wysiwyg editing capabilities).

    But Safari on Macintosh uses khtml and is pretty well configured from the start.

    To be honest, I’ve been long impressed with khtml’s ability to render malformed html. And it’s light and quick, too. It uses full C++ and thus largely avoids the tendency of C to have buffer overflow errors, and numerous other kinds of errors.


  29. Anonymous says:

    I used to work with you at msft on Exchange I was in QA for backup/restore. I then worked on MS Agent and I recall we spend time testing our API’s for buffer overload and rnd crap being sent to them. Recently I started a project to extend IE by adding lots of features like tab browsing, memoing, blogging, etc… and was planning on giving it away for free. The thing that killed that project was how buggy and some cases incomplete the IE API calls are in how they work togather. In the end I think all the functionality I wanted to add could be added but IE would be very unstable and thus no one would use my extensions. Its great IE is more stable than the other browsers, my own experience agrees with that, but I use it 50/50 with Firefox and I still get IE hanging often but not as often as Firefox. For me its a mixture of features vs. stability. I think msft really missed the boat on not making IE a killer app when they had the market and redifining what a browser is. I guess it didn’t bring in direct revenue and was thus expendable.

    Torr Randell

  30. Anonymous says:

    I totally confused Michael Zalewski with Mark Zibowski

  31. Anonymous says:

    In a recent blog entry in Larry Osterman’s WebLog he explains various browsers other than MSIE have trouble with malformed HTML markup. He claims they have a security problem while MSIE is essentially bulletproof. He cites Michael Zalewski with an…

  32. Anonymous says:

    Welcome to the Slashdotting pal!

    The page didn’t crash my browser, FireFox 1.0 PR running on Windows 2000 SP4

  33. Anonymous says:

    To Mike Dimmick:

    "Currently XPCOM binary extensions have no security model at all."

    It’s because they don’t need any security model. XPCOM is not ActiveX. You install XPCOM components into your system like any other dynamic libraries – you have to know beforehand if you trust them or not. This is no different from installing any IE addons to your hard drive.

    "Mozilla/Firefox smoke testing still appears to be post-checkin, though, not pre-checkin as I believe has become common practice at MS. It’s not automated, which suggests the software wasn’t designed-for-test."

    Smoke testing is generally post checkin (for some big changes the developers make test releases that are tested before the checkin is made). However, there are automated tests happening after checkin – page load tests, new window test, startup test, footprint, … Also, every day thousands of volunteers download the nightly builds and use them and report bugs.

    "If you choose Microsoft software … have performed the security reviews they say they have and that they’re skilled to do so."

    It’s in the company’s best interest to claim so, but since it’s closed source we have no way to know what they have done and how good their people and processes are. With open source, we at least can find out this information. Several types of reviews, including security reviews by security professionals, have been done on Mozilla source code.

  34. Anonymous says:

    I hope those who tried this and didn’t get a crash immediately closed and restarted the browser. It might corrupt memory and not crash until much later, possibly causing additional data corruption along the way. (The same could be true with IE, but I assume the original tester knows what he’s doing.)

    What this really makes me wonder about is something like… 🙁

  35. Anonymous says:

    Is there some mis-understanding. The tool used, which it seems like everyone is pointing to is brute force test. This means that it produces random html code 99.99% of the time, the browser will cope, but when it refreshes the codefor the 1 millionth permutation, the browser may, or may not crash. Using the script that he was useing, it took him 2 hours, to find those 2 "security" problems, (browser crashes). and thats with the code running on the local machine and auto-refress on. So… it probably wont crash the time that you use it, unless your realy un-lucky. oh, and to find bugs to crash firefox, all you have to do is to search beleve me, you can find plenty. shame we cant see Micro$ofts bug racker, sure to be large i would imagin …


  36. Anonymous says:

    Larry – Why do you advocate Microsoft and closed source software? You know what people like you are? Insane. Why would poeple write OSS in the first place if existing solutions weren’t sufficient? Of course the Firefox team knows what they’re doing. Have you ever visited even? If you believe that MS software is better, hey, go ahead. May your system be riddled with spyware/malware and many a virus. But preaching crap should not be tolerated. Does microsoft pay you for this? Throughout their existence they’ve lied and cheated their way to the top. They’re not on top because they make good software. A lot of companies are not on top becasue of quality software. This isn’t hard to see. Many people know this. You’d have to be blind not to see how big of a problem microsoft is. This is capitalism. You’re advocating shit, pal. People like you are partly the reason computer technology is behind at least 10 years.

  37. Anonymous says:“> caused 3 IE windows to close, among the 7 that were open at the time. If processes were properly isolated from each other then only 2 IE windows should have closed, not 3. The reason for generously allowing 2 was that I had opened“> by right-clicking a link and selecting "open in new window", which sort-of implies that IE’s bugs will kill both windows together.

    Of course I told the crash reporter to send a report to Microsoft, but I kind of doubt that the report includes the above facts.

    As usual I can’t use the mouse to copy and paste from a dialog box, such as the dialog box displaying IE’s version number.

  38. Anonymous says:

    I don’t know if Heikki will come back here or not, but anyway: XPCOM is a binary interface, XPInstall is an installer solution, you can script XPCOM and XPInstall in Mozilla and FireFox using XPConnect. The net result is that – just like ActiveX – a site author can cause an unmanaged binary component to be downloaded by the browser, installed, then scripted by the page.

    Firefox 1.0 PR does now have an information bar similar to IE 6.0/XP SP2’s, when a site tries to install a component. Unlike IE’s, the user can only opt to allow sites to install components, rather than allow this single installation. If the user chooses to allow, and tries again, you get a dialog with ‘Install Now’ and ‘Cancel’ options. There’s a timeout to stop you just pressing Enter, but the default is still ‘Install Now’ – unlike IE’s, which is ‘Don’t Install’.

    Once the component is installed, if the component’s interface is marked [scriptable], it can be scripted. There’s no equivalent of ‘Safe for Initialization’, or the IObjectSafety interface, where the object can participate in safety decisions. The only option is to either enable or disable JavaScript. I saw mention that only scripts from the same site as the page will be able to script objects in the page, but that’s not much hardship for a determined spyware-injector.

    It’s another case where Microsoft’s actual security *model* is stronger, but has been let down – in the past – by a weak *implementation*.

  39. Anonymous says:



    Larry: Maybe you need an "About Me" link on the left.

  40. Anonymous says:

    Rendering bad HTML is probably the only thing IE is good at (it sure can’t render correct HTML properly)


    IE with tabs : (pre-empting any firefox zealots screaming about IE having no tabs. Also features adblocking, google bar support.


    MyIE 2 is about twice as buggy and random-crash-prone than regular IE. And it eats up a rediculous amount of memory when you have lots of tabs open.

  41. Anonymous says:

    Let me be perfectly clear. I never said that IE was perfect. The IE team doesn’t say that IE’s perfect (

    But I AM saying that we tested against fuzzed input, and that testing against fuzzed input is necessary.

    People need to get away from the idea that just the input is syntatically incorrect it can be ignored.

  42. Anonymous says:

    SiEd blog &raquo; Testing