Insightful graph: The ship date predictor


The best graphs are the ones that require no explanation. You are just told what the x- and y-axes represent, and the answer just jumps out at you.

One of the greatest graphs I've seen at Microsoft is this one that a colleague of mine put together as Windows 95 was nearing completion. He took each email message from management that changed the Windows 95 RTM date (also known as the ship date) and plotted it on a chart. The x-axis is the date the statement was made and the y-axis is number of days remaining in the project, according to the email. The dotted line is a linear least-squares fit, and the green star is the actual ship date (July 14, 1995).

600 
500 
400 
300 
200 
100 
 
 
 
 
 
 
 
 

Apr

1992
Jul
Oct
Jan 1993
Apr
Jul
Oct
Jan 1994
Apr
Jul
Oct
Jan 1995
Apr
Jul
Oct

What's so amazing about this chart is that the linear approximation predicts the actual ship date with very high accuracy. The slope of the line is 0.43%, which means that if you took the predicted "days remaining before we ship" and multiplied it by around 2.3, you'd be pretty close to the actual ship date.

In other words, management fairly consistently underestimated the number of days until RTM by a factor of 2.3. (Another way of looking at it is that the development team consistently underreported the number of days to completion to management by a factor of 2.3.)

Bonus amusement

Here is a pull quote from each of the announcements, lightly edited.

Date Revised RTM Remark
February 1992 June 1993 "Ready to RTM 6/93. Otherwise, I'll be applying for a job at McDonalds."
April 1992 September 1993 "This is a critical release."
July 1992 March 1994 "The feature set will NOT be expanded to fill the new schedule."
September 1992 December 1993 "This product must RTM by the end of 1993. If we miss this window of opportunity, then the value of this product goes way down."
January 1993 March 1994 "I recently learned that Team X was planning around a Q4 94 ship date!" (Team X provided code to Windows 95.)
March 1993 April 1994 "We need to formulate plans which get us there."
August 1993 May 1994 "It's really important for the company that we make this date. This must be our last slip."
December 1993 August 1994 "This is about as late as we can go without incurring big financial problems for the company."
February 1994 September 1994 "What determines the ship date is the team's commitment to a ship date. We must make our RTM date."
May 1994 November 1994 "Software and hardware vendors are counting on us."
August 1994 February 1995 "Completing this milestone by the end of the year is absolutely critical to the product gaining quick success."
December 1994 May 1995 "People all over are planning their business on when we release. We must make our current date."

Today marks the 20th anniversary of the public release of Windows 95. Just one more year, and you'll be old enough to buy a drink!¹

Bonus reading: Start Me Up (again): Brad Chase (who ran the worldwide launch of Windows 95) tells the story of how Start Me Up became the anthem for Windows 95, and addresses the legend that that it cost $14 million to license the song. (Spoiler: It was more like $3 million.)

Bonus chatter: The ticket price for the Windows 95 team reunion party is $47.50. This seems like an odd number, but it makes more sense when you buy two tickets (one for you, and one for your partner).

¹ In the United States, the age at which it is legal to purchase alcohol is 21.

Comments (51)
  1. Sockatume says:

    "Another way of looking at it is that the development team consistently underreported the number of days to completion to management by a factor of 2.3."

    They've got a lot to learn if they want people to think of them as miracle workers.

  2. SS says:

    So, did he/she get the job at McDonalds?

  3. Ross Presser says:

    You really hate RSS, don't you, Raymond? Your "graph" is completely destroyed in the RSS feed. What was wrong with an image? It's not like the text characters you used have any real meaning outside their precise placement. An image would have been semantically more appropriate.

    [Is it my fault that your RSS reader strips HTML tags? Images have their own problems: What DPI should I generate them at? Should I anti-alias them? Sometimes they get lost. And sometimes someone asks for a higher resolution version, and if it's an image, I don't have one. Scalable graphics FTW. -Raymond]
  4. AC says:

    Reading the remarks section in the table convinces me that your first interpretation of "managers underestimate work" is the correct one.

  5. acq says:

    @Ross Presser

    I like Raymond's chart in HTML.

  6. Muzer says:

    @Ross Presser Raymond has previously reported that the blog engine isn't very good at supporting images. And I for one enjoy everything being in HTML, because it means that in a decade's time all the "images" will still be visible as today (assuming MS haven't taken down the blog by then), unlike if you look at some of Raymond's older articles imported from elsewhere…

  7. Alex says:

    Was terrified at what's greeted me when opened RSS reader. Turned out it was HTML chart. Well, okay.

  8. Gabe says:

    I find it quite odd that http://win95anniversary.com/ claims that new features in Win95 included "Minimize, maximize, and close buttons on each window" and http://www.geekwire.com/…/party-like-its-1995-microsoft-vets-to-host-epic-party-to-celebrate-20-year-anniversary-of-windows-95 claims that "minimize buttons on each window" were a feature.

    As far as I can tell, the minimize button goes back to Windows 2.0 (it was an arrow then, and a triangle in 3.x, so Win95 only brought us a different icon on the button). The close button is the only one that was new for Win95 (previously you had to double-click the system menu).

    Or am I missing something? Did Win95 add a minimize button to windows that didn't previously have them?

  9. John says:

    So if it had made one of the earlier RTMs we could have had Windows 93 or Windows 94?  The horror!

  10. Boris says:

    @Ross Presser: but isn't RSS somewhat last decade, and why would you need it anyway when you can expect a daily post precisely at 14:00 GMT? The whole point of RSS was to avoid useless page visits.

  11. Don Reba says:

    Here is the linear approximation's quality using information available up until each of the points: http://i.imgur.com/Gu1Ijy0.png

  12. Don Reba says:

    @Boris, myself, I am subscribed to 200 feeds and have no desire to know at what time each of them updates.

  13. jnm236 says:

    All his graphics are messed up on RSS. If I care enough to see what it looks like, or it overlaps and interesting part of the text, I'll open up this page. Raymond's is the only blog I've ever come across with this issue. Personally I'd kick up a fuss until I was able to use images, but it's up to him

    @Boris As far as I can tell RSS is used as much as ever. There's nothing quite like it. I don't have the time or the memory to click a bookmark for every blog I read- why should I waste all that scanning if I can see it all in one place? Plus the easier formatting, lack of ads, ability to track what I've missed without even more scanning, all synced online or on my phone. I'd rather spend my reading time reading than managing reading.

  14. acq says:

    Rally a nice candidate for "First world problems" meme: I have to *click on the link* to see the graph in Raymond's instead to see it in a RSS reader.

    Ruins the day, right?

    andrewchen.co/the-death-of-rss-in-a-single-graph

  15. alegr1 says:

    The slope should be either 0.43 or 43%, not 0.43%

  16. John says:

    Time to play the "Where's Raymond" game with that Geek Post ;)

    Congratulations on 20 years, I still remember my first Windows 95 machine with many fond memories, it was a Sony Vaio PCG-90 picked up in December of 95 it had a screamin 90Mhz Pentium and I want to say 32mb of RAM with a 2 or 4GB Harddrive, I shutter to think what my parents must have paid for it back in the day, but it was an amazing machine for the time, I have many fond memories of Microsoft Games' Movie Studio and Windows 95 Plus!

  17. Klimax says:

    Graph shows correctly in RSS feed in IE11 alias RSS Platform.

  18. Mark S says:

    Please ignore the whiners.  Works fine in IE9 and Live Mail.  Boo hoo, an extra click to see the graph if it doesn't.

  19. Mike Dunn says:

    I feel old now, thanks. :P

  20. MJP says:

    It's disappointing that if you search for "evidence-based scheduling", you get Joel Spolsky's 2007 essay and that's about it. There should be hundreds of articles discussing different techniques, dozens of tools to help you do it, it should be a fundamental part of project management. Instead managers are sending out the same kind of deluded emails today that they were 25 years ago. And I'm still hopeless at estimating costs.

  21. 12BitSlab says:

    Raymond, did you or your team feel any of the pressure to get Win95 out the door?  Any stories you can share?

  22. Antonio Rodríguez says:

    Come on, guys, this is Raymond's blog, and he can do whatever he likes with it, be it using HTML art or writting about the CLR or the Itanium for a week. If he decides not to use images, whatever the reasons, we can not complain. I have to add that I'm a fan of Raymond's HTML art, but I don't think that changes anything.

  23. Mark says:

    @Gabe: yeah, only close was new

    Back to today's topic: it seems your statistician could have saved a bunch of managers a bunch of trouble if he had publicized this graph around the middle of '94.  By then he had enough data to confidently announce "We ship in one year and I can prove it!"

    :)

  24. Engywuck says:

    I recently tried to install Win95 in a VM (had to look something up). Now I know why I like modern OS more :-) (the problems started with the CD image not booting and me not having a boot diskette image. All the ones on the internet seem to be .exe that want to directly write to a floppy drive).

    But for the time it was quite nice. For the (affordable) hardware of its day, anyway. Plus it made the move away from DOS far easier. Congrats to 20 years (and a safe retirement).

  25. KrzaQ says:

    SVG seems like a viable solution to both rss and vector graphic problems.

  26. Guest says:

    @KrzaQ

    But not the getting lost problem.

  27. Karellen says:

    Graphs work fine in Akregator (KDE RSS reader) also.

  28. JulioG says:

    Wow. Twenty years after Windows 95, and I still double-click the top left corner of windows to close them.

  29. Joshua says:

    [Is it my fault … Scalable graphics FTW. -Raymond]

    There's the Raymond we know and love.

    @Mark: They don't listen.

  30. Josh B says:

    @Ross & jnm236: Looks 100% in every RSS reader I have. Update yours or move on to something that isn't broken and abandoned. Don't complain to the blog that you can't see perfectly standard HTML in your RSS reader, complain the to RSS reader devs.

  31. acq says:

    The release of Windows 95 was an important moment in my life too! And Michael Dunn is not the only one feeling old.

    Congratulations, Raymond. And, in my opinion, your blog isn't less significant feat than participating in the development of Windows 95.

  32. Antonio 'Grijan' says:

    @Engywuck: I have Windows 3.1 and 3.11, a couple of alphas/betas of Chicago, and Windows 95 RTM, each in its own virtual machines, and it's really interesting to play with them and see what changes and what stays. It has made me appreciate Windows 95 in ways I didn't understand back then.

  33. Gabe says:

    Wikipedia says that Windows 3.1 RTMed in April 1992. If it's to be believed, the estimate that said Chicago would be released in June 1993 was made while Windows 3.1 was still in development!

  34. Michael Entin says:

    That was actually very good graph. I remember another Microsoft product which was in a state "6 month from the ship date" for three years in a row.

  35. Boris says:

    @Don Reba: OK, but 200 feeds sounds like your entire history over the years, including dead, inactive or rarely active feeds, as opposed to the top ten you'd be aware of on a daily basis.

    @jnm236: there's Twitter and Facebook, so RSS would be useful mostly for blogs which don't automatically advertise every post via those channels.

  36. Jonathan says:

    Awesome graph, awesome Rendering. Yes, it does get munged by TheOldReader, but I already know I should open the real page.

    @MJP: I too was going to link to Evidence-based scheduling, as the 2.3 coefficient is exactly what that was supposed to capture.

  37. Neil says:

    @JulioG: Twenty years after Windows 95 and I still press Alt+F4 to close windows…

  38. @Guest: That can be solved with inline SVG images.

    [Wouldn't have helped. The problem is with RSS readers that strip attributes. Stripping attributes out of SVG makes it pretty useless. -Raymond]
  39. M says:

    If team X was planning a Q4 1994 ship data, a revision to March 1994 was overly optimistic.

  40. @M: Financial quarter, maybe?

  41. Michael Dudley says:

    Raymond–it would be great if you could add an "M=1" line to show us where "perfect" estimates would be.

  42. DWalker says:

    Yes, Win95 was great (for its time)!  

    Raymond, you said "…the green star is the actual ship date (July 14, 1995)."  I thought the actual ship date was August 24, 1995.  That's what one large search engine says.  And THAT date is 20 years ago (from yesterday); July 14 1995 is not.  (Depending on the definition of "ship date".)

    [There are two dates in play. Ship date usually refers to RTM date, the date that bits were released to manufacturing. The other date is GA (General Availability), when the product was made available to customers. Development stops at RTM, so that's the date developers care about. But the general public cares about GA, since that's the first day they can get their hands on the product. -Raymond]
  43. Yuhong Bao says:

    Someone should do something similar for Exchange Server 4.0.

  44. boogaloo says:

    I wonder who made this statement

    December 1993 August 1994 "This is about as late as we can go without incurring big financial problems for the company."

    Whether they believed it at the time and how they feel about it now?

    "Wikipedia says that Windows 3.1 RTMed in April 1992. If it's to be believed, the estimate that said Chicago would be released in June 1993 was made while Windows 3.1 was still in development!"

    @Gabe It's not uncommon to start planning your next project while the previous one was being bug fixed. They shipped some of the Windows 95 features as Windows 3.11 in November 1993. I've always been curious how similar the code for Windows 3.11 + Win32s and Windows 95 are (Raymond may have covered it and I may even have read it but I don't recall it).

  45. DWalker says:

    Thanks, Raymond.  You mentioned both the ship date and the public release date, and I forgot that they could be different.

  46. Anon says:

    Even on a blog with a target audience for developers, people can't grasp the difference between "your HTML is broken" and "the thing I use to read the HTML is breaking the HTML"

  47. Sean Liming says:

    Was the same plot taken for all the operating systems that followed?

  48. Orion says:

    Sweet, does that mean the Windows 10 reunion party will have a ticket price of just $5? That would really make the Windows 2000 alumni envious.

  49. Boris says:

    @Orion: no, you just won't be admitted without having brought at least four people with tickets.

  50. georgie says:

    Was this the same group of executives who made those statements?

    I am curious if there is some general pattern of 2.3x overestimation for projects.

  51. Yukkuri says:

    @Georgie — Software development has had this pattern since forever. Brooks' Mythical Man Month is still cited routinely even though it was published 40 years ago…

Comments are closed.

Skip to main content