“if Microsoft is so good at testing, why does your software suck?”


What a question! I only wish I could convey the way that question is normally asked. The tone of voice is either partially apologetic (because many people remember that I was a major ask-er of that same question long before I became an ask-ee) or it’s condescending to the point that I find myself smiling as I fantasize about the ask-er’s computer blue-screening right before that crucial save. (Ok, so I took an extra hit of the kool-aid today. It was lime and I like lime.)  


After 27 months on the inside I have a few insights. The first few are, I readily concede, downright defensive. But as I’ve come to experience firsthand, true nonetheless. The last one though is really at the heart of the matter: that, talent notwithstanding, testers at Microsoft do have some work to do.


I’m not going down the obvious path: that testing isn’t responsible for quality and to direct the question to a developer/designer/architect instead. (I hate the phrase ‘you can’t test quality in,’ it’s a deflection of blame and as a tester, I take quality directly as my responsibility.)


But I am getting ahead of myself. I’ll take up that baton at the end of this post. Let’s begin with the defensive points:


1.       Microsoft builds applications that are among the world’s most complex. No one is going to argue that Windows, SQL Server, Exchange and so forth aren’t complex and the fact that they are in such widespread use means that our biggest competitors are often our own prior versions. We end up doing what we call “brown field” development (as opposed to ‘green field’ or version 1 development) in that we are building on top of existing functionality. That means that testers have to deal with existing features, formats, protocols along with all the new functionality and integration scenarios that make it very difficult to build a big picture test plan that is actually do-able. Testing real end-to-end scenarios must share the stage with integration and compatibility tests. Legacy sucks and functionality is only part of it…as testers, we all know what is really making that field brown! Be careful where you step. Dealing with yesterday’s bugs keeps part of our attention away from today’s bugs.


(Aside: Have you heard that old CS creationist joke: “why did it take god only seven days to create the universe?” The answer: “No installed base.” There’s nothing to screw up, no existing users to piss off or prior functionality and crappy design decisions to tiptoe around. God got lucky, us…not so much.)


2.       Our user-to-tester ratio sucks, leaving us hopelessly outnumbered. How many testers does it take to run the same number of test cases that the user base of, say, Microsoft Word can run in the first hour after it is released? The answer: far more than we have or could hire even if we could find enough qualified applicants. There are enough users to virtually ensure that every feature gets used in every way imaginable within the first hour (day, week, fortnight, month, pick any timescale you want and it’s still scary) after release. This is a lot of stress to put our testers under. It’s one thing to know you are testing software that is important. It’s quite another to know that your failure to do so well will be mercilessly exposed soon after release. Testing our software is hard, only the brave need apply.  


3.       On a related point, our installed base makes us a target. Our bugs affect so many people that they are newsworthy. There are a lot of people watching for us to fail. If David Beckham wears plaid with stripes to fetch his morning paper, it’s scandalous; if I wore my underpants on the outside of my jeans for a week few people would even notice (in their defense though, my fashion sense is obtuse enough that they could be readily forgiven for overlooking it). Becks is a successful man, but when it comes to the ‘bad with the good’ I’m betting he’s liking the good a whole lot more. You’re in good company David.


But none of that matters. We’ll take our installed base and our market position any day. No trades offered. But still, we always ready to improve. I think testers should step up and do a better job of testing quality in. That’s my fourth point.


4.       Our testers don’t play a strong enough role in the design of our apps. We have this “problem” at Microsoft that we have a whole lot of wicked smart people. We have these creatures called Technical Fellows and Distinguished Engineers who have really big brains and use them to dream really big dreams. Then they take these big dreams of theirs and convince General Managers and VPs (in addition to being smart they are also articulate and passionate) that they should build this thing they dreamt about. Then another group of wicked smart people called Program Managers start designing the hell out of these dreams and Developers start developing the hell out of them and a few dozen geniuses later this thing has a life of its own and then someone asks ‘how are we going to test this thing’ and of course it’s A LITTLE LATE TO BE ASKING THAT QUESTION NOW ISN’T IT?


Smart people who dream big inspire me. Smart people who don’t understand testing and dream big scare the hell out of me. We need to do a better job of getting the word out. There’s another group of wicked smart people at Microsoft and we’re getting involved a wee bit late in the process. We’ve got things to say and contributions to make, not to mention posteriors to save. There’s a part of our job we aren’t doing as well as we should: pushing testing forward into the design and development process and educating the rest of the company on what quality means and how it is attained.


We can test quality in; we just have to start testing a lot sooner. That means that everyone from TF/DE through the entire pipeline needs to have test as part of their job. We have to show them how to do that. We have to educate these smart people about what quality means and take what we know about testing and apply it not only to just binaries/assemblies, but to designs, user stories, specs and every other artifact we generate. How can it be the case that what we know about quality doesn’t apply to these early stage artifacts? It does apply. We need to lead the way in applying it.


I think that ask-ers of the good-tester/crappy-software question would be surprised to learn exactly how we are doing this right now. Fortunately, you’ll get a chance because Tara Roth, one of the Directors of Test for Office is speaking at STAR West in November. Office has led the way in pushing testing forward and she’s enjoyed a spot as a leader of that effort. I think you’ll enjoy hearing what she has to say.

Comments (13)

  1. Paul Darby says:

    There is a bit of a well known qoute that testers cannot ensure quality – they can find the bugs but they can’t make people implement a fix.

    I don’t entirely subscribe to this view:  there are ways and means of pushing for fixes.

    However, the question I’m slowely heading towards is this "Of the bugs identified within Microsoft, what sort of fix ratio does Microsoft have?"  Do they attempt to fix everything?  Is the majority of things found in the field that joe public whing about already on the Microsoft defect management system?

    I think it’s marginally more forgivable for an organisation to know about bugs and not have fixed them, than for the bugs to come as a complete surprise to the organisation.  Of course, I’m a tester so I would aire more towards this viewpoint.

  2. Check out James’s recent post with the eye-catching title “if Microsoft is so good at testing, why does

  3. wtroost says:

    At project wrap-up, a surefire conclusion is that "more people should’ve been involved earlier".  Kinda like "more documentation was needed".

    But the more people you involve, the smaller the chance your project goes anywhere. 🙂  If you require testing involvement in an early stage, you’d probably stifle innovation.

  4. Excellent post James (as always (and yes I like parentheticals too (really! 😛 ))).

    One further point I’d make is that geting the smart SDET’s into the process sooner isn’t enough. Getting the support organization folks in sooner can pay dividends. Someone in the MS third tier of support has been fixing, deploying, and reverse engineering these products for years – often for much longer than the dev’s, testers, or PM’s and through far more versions. And yet they are usually added to the mix right before the public beta. Far too late to have said ‘that new feature as designed is unsupportable by us or customers and here’s why’.

    But that’s an old beef – I definitely think that the closer testers get to the design/spec phase, the better. I’ve known some darned smart testers and seen them catch some ridiculously insidious problems before they got out the door. That’s the worst part of your job – the better you are, the less anyone knows how great you are. It’s like all the goofs who think that Y2K was some made up non-event; they didn’t see a tens of thousands of people preemptively fix the issues for 3 years. 🙂

    – nedpyle

  5. SRLTeam says:

    I was reading the post on “if Microsoft is so good at testing, why does your software suck?” Written

  6. aramirez says:

    I liked your posting but I have a slightly different view about the ‘testing quality in’ perspective.

    Testing is a practice of damage control for trying to find the bugs that have been already designed and developed into the product.

    As we are greatly outnumbered by the ratios you mentioned, all we can do is focus on "customer tolerance limits" by targeting those defects with the highest impact to the user base first and work our way down the list.

    Certainly the solution is not for test to become a quality "police" but like you well said it, to make sure testing and all stakeholders get involved in the design phase so that our combined skills and customer understanding are used  to prevent those defects from being introduced in the first place.

  7. Добрый день. C момента публикации знаменитой статьи…

  8. I was triggered to read a blog post by James Whittaker, software architect at Microsoft, through an article

  9. [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

  10. qwerty says:

    part 1

    "applications that are among the world’s most complex"

    I'll say this. A "good" programmer finds simple ways to tackle complex problems.  This has obvious advantages in too many areas to number.

    There is also the area of unneeded complexity.  I am forced to use MS Word in order to interact with everyone else because everyone else uses it because everyone else does (it is a bit of a collective action problem).

    What strikes me about Word is that in the effort to make "hard things easy" they have not succeeded and instead have made "easy things hard".  The product would like to accommodate a 70 year old technophobe in crafting a nice letter with bullet points in a way they can pick up and also format complex documents for publication in scientific journals.  It is very hard to have it both ways, and they have not achieved it.

    I am very computer savvy and program in multiple languages and have taught myself more software package than I can remember.  Still, after years of use, MS Word does seemingly random and unexpected things and I waste a lot of time trying to coerce it to do things that should be trivially easy.  They simply need to adopt a simpler approach to the problem that can be easily understood and used, not all of the 100s of hierarchical and interacting ribbons and lists and options.

    To change focus to charting to explain my point, compare chart making in Excel vs a much better software, SigmaPlot.  In sigmaplot, you can simply click on any element in the graph (lines, tick marks, anything) and it is its own object.  You can then move it around, type in an exact position for it, or change any of its properties.  Guess what?  You can make extremely impressive graphs that look exactly as you'd envision with little effort and it took me maybe two days exposure to the software to be able to get results that far exceed Excel.

    That is an example of a simple, yet powerful approach that does not treat the user like an idiot that needs 1000 built in behaviors that cause the product to do what MS thinks we really should want.

  11. qwerty says:

    part 2

    The same approach can be applied to word processing and several open source desktop publishing programs use it to good effect.  I use these in my work I do in isolation, but I cannot use them to collaborate with everyone else that is using Word.

    So, please, don't be defensive.  Just admit that MS products are not nearly as good as they should be and that  they are enjoying their market position and don't need to be all that great to be rich.

    I read somewhere else, may not be true, that MS farms out code writing to private contractors all over the world.  A common practice is to get the code to function anyway possible, hit compile, and then celebrate "yeah, I'm done and get to cash a check".  I am not a trained computer scientist, but I've had to learn the basics of programming for my work and I produce very impressive ecosystem simulations for research purposes.  Even I know that this is not the correct way to write programs that other will be using.  I've also read where people have reversed engineered some MS products to look at the code and they were shocked at how poorly written and inefficient the code was.

  12. Matt says:

    The fact that your software is overly complex and full of features and bloatware no one would ever want or use is no excuse for it to have the same bugs for years and years spanning version after version.

Skip to main content