The last consumer operating system written in my lifetime…


Way back in the day, I was chatting with Tom Miller, one of the NT filesystem developers – he and Gary Kimura were the guys who designed and built NTFS.

At some point, the conversation drifted around to our motivations for working on NT, and Tom made a comment that’s stuck with me for the 15 years since he made it.

“I work on NT because it’s an opportunity to work on the last new PC operating system to be written in my lifetime.”

 

It’s true that many of the low level concepts embodied in NT originated in the operating systems that Dave Cutler and the core NT team developed when they were at DEC, but the entire NT operating system was written from the ground up – with the exception of the command interpreter, much of the code in USER, the TCP/IP stack (licensed from Spider systems) and a couple of the networking utilities, every bit of code in NT 3.1 was brand spanking new.  And there were plans in place to replace the TCP/IP stack with an in-house TCP/IP stack (delivered with NT 3.5), so even that was written from the bottom up.

When I relate Toms comment to others, they invariably respond “But what about [Linux|OSX|Be]”.  Taken in turn:

Linux was started in 1991, after the fact, but it’s not a complete OS – Linux is the kernel, but the other parts of the OS date from other projects started long before Linux was started.  The kernel is new, but the entire OS isn’t – all the user mode components come from previous operating systems, as does the networking stack.

OSX is based on FreeBSD, which was in turn based on 386BSD – large parts of the OS have been reengineered, but again, it wasn’t a bottom-up attempt to write an entire OS.

Be’s OS might have qualified as a new OS, since it was written from the bottom up, but unfortunately it didn’t succeed in the marketplace.

There have been a number of original OS’s built for various small devices since then (Palm, Symbian, WinCE, etc), but none for PC’s. 

In reality, this isn’t really surprising, it takes a HUGE amount of effort to launch a complete operating system – hundreds of millions of dollars just in direct development costs.  I still remember when we had the first big scheduling meeting when we laid out the list of things that we had to have for NT – many of the developers in the room were aghast at the number of components that simply HAD to be present to be considered a complete operating system: 16 bit application compatibility (which was a real issue for our non x86 platforms), printer drivers, video drivers, printing UI, multimedia hardware support, networking support, filesystem utilities, the list went on and on.

Nowadays, the list is even longer – for instance, back in 1989 when that scheduling meeting happened, you didn’t need a web browser, because the web hadn’t been invented yet (the first web page was written in 1990), but today, clearly you’d have had to have a web browser on the list.  Similarly, today you need to have camera support, digital media player support, instant messenger support, etc.

My suspicion is that Tom’s statement will continue to hold true – the amount of effort required to build an entire operating system from the bottom up is daunting, and it’s unlikely that anyone is likely to embark on such an effort in the foreseeable future.

Comments (40)

  1. Anonymous says:

    The last operative system in his lifetime? Well, I wouln’t bet it…who knows what we’ll see in 20 years O_o

    While maybe it can’t be considered a example of a PC operating system, plan 9 is a beautiful example of an OS written from scratch (and some would say that it’s dead, but the fact is that today’s OSes are still far of doing some of the things plan9 can do, so…)

  2. Anonymous says:

    I think your defense of Tom’s quotation is a little too aggressive. You dismiss BeOS on grounds that are irrelevant to the rest of your argument, and I suspect (not having the context) to Tom’s motivations in making his comment. An OS’s lack of commercial success does not retroactively change how its development was begun, and I think the developers of BeOS deserve as much credit for setting out to build a completely new OS as the original developers of NT, especially as they didn’t have revenue from very successful existing products to support their work.

  3. Anonymous says:

    "The [Linux]kernel is new, but the entire OS isn’t – all the user mode components come from previous operating systems, as does the networking stack."

    I thought the Linux networking stack was new code. Care to be more specific?

    http://www.tldp.org/HOWTO/NET3-4-HOWTO-4.html

    "Developing a brand new kernel implementation of the tcp/ip protocol stack that would perform as well as existing implementations was not an easy task. The decision not to port one of the existing implementations was made at a time when there was some uncertainty as to whether the existing implementations may become encumbered by restrictive copyrights because of the court case put by U.S.L. and when there was a lot of fresh enthusiasm for doing it differently and perhaps even better than had already been done. (…)"

  4. Anonymous says:

    But aren’t you blurring the line between OS and application a little?

  5. PatriotB says:

    Strangely, there’s people out there who claim that Vista and/or WS 2003 are completely new OSes, written from scratch.

    "Vista isn’t simply a super-patched version of XP, It’s been practically written from scratch. While initial versions of Vista were based on XP, the developers realized that this constant system of patching just wouldn’t work with what they wanted to do, and decided to base it instead off of Windows Server 2003, which had been written from the ground up for stability and security." (http://www.zdnet.co.uk/talkback/?PROCESS=show&ID=20051632&AT=39236741-39020330t-10000003c)

    "You DO realize that Vista is built from scrach?? Windows Vistas is rebuild from ground up." (http://www.zdnet.co.uk/talkback/?PROCESS=show&ID=20051655&AT=39236741-39020330t-10000003c)

  6. Anonymous says:

    Which is it: Consumer OS or PC OS? If we’re talking consumer OS, then perhaps the entire handheld market would qualify (Palm, Symbian, etc.). If we’re talking about the PC OS market, I would submit that the Hurd, Be, and the other flamebait candiates would indeed qualify. If we’re talking about Consumer PC OSes, then you may have a point, although the line between the embeded PC market and the traditional PC market is blurring lines with every release.

  7. LOL… 🙂

    Vista is based on Win2K3, not XP, but…

  8. Anonymous says:

    What about MinuetOS?

    http://www.menuetos.org/

    Os written completely in assembly.

    Sure, you can’t go to Best Buy and pick up a MinuetOS PC. But it fits your definition, as does BeOS but you dismiss Be since it failed in the marketplace.

    Oh and I think Minuet OS was written by 3 guys without hundres of millions of dollars. 😉

    Does NextSTEP count?

    I think Tom needs to amend his statement a little. I’ve taken the liberty of re-writing it. 🙂

    "I work on NT because it’s an opportunity to work on the last new PC operating system to be written at Microsoft in my lifetime."

  9. Anonymous says:

    "Strangely, there’s people out there who claim that Vista and/or WS 2003 are completely new OSes, written from scratch. "

    A lot of people made the same claim about Windows 95, which was far from the truth.

  10. vince says:

    as someone else posted, the Linux networking stack was written from scratch.

    And just about all of the Linux infrastructure was written from scratch as well. True the GNU project is about 10 years older than Linux, but that still counts as within your lifetime unless you are fairly young.

    Just about all of the U*IX utils had to be re-written from scratch; it’s not like Sun or HP or IBM or Microsoft were going to just hand over the source code to their implementations, even if they were allowed to.

    The great thing about Linux is that anyone can have the opportunity to work on it, wheras with NT only a "chosen few" get the honor (or the blame).

  11. For the purposes of this argument, the "start of life" is 1989 (that’s what makes it the "last"). The GNU project started in 1984 (I remember back when Jim Goslings Emacs was ported to gnuEmacs).

  12. Anonymous says:

    I can’t think of any really convincing arguments for an OS having to include a web browser, digital camera support, digital media player support, instant messaging support, etc.

    These can all be provided by applications (and even – shock horror – by third-party applications).

    It’s this sort of "WINDOWS HAS TO INCLUDE EVERYTHING" mentality that has got Microsoft into trouble more than once.

  13. Anonymous says:

    From http://www.skyos.org/ :

    The Sky Operating System, or SkyOS, is an operating system written for x86-based personal computers. SkyOS was created in 1996 by Robert Szeleney as a small bootloader. In the past 8 years, SkyOS has evolved into a full-featured, modern operating system, with a goal to be the easiest to use desktop operating system available for the average computer user. The development staff has also increased to include business, software, and graphics developers.

  14. vince says:

    > For the purposes of this argument,

    > the "start of life" is 1989 (that’s

    > what makes it the "last"). The GNU

    > project started in 1984 (I remember

    > back when Jim Goslings Emacs was

    > ported to gnuEmacs).

    Well yes. And by 1989 the GNU project basically had gcc and Emacs and that’s about it. Were Visual C and Notepad also re-written from scratch for NT?

    MS has such an odd view of what encompasses an "operating system".

    You’d have a much more effective anti-Linux argument if you pointed out that the X Windowing System project dates back to 1984.

  15. Visual C’s not a part of NT, and never has. And Notepad dates from Windows 3.1, which was 1990.

    This wasn’t supposed to be an "anti Linux" post, more of a "cool quote" post, and a commentary on the amount of effort needed to write something that includes all the pieces that people consider to be a part of an operating system.

  16. vince says:

    > and a commentary on the amount of

    > effort needed to write something

    > that includes all the pieces that

    > people consider to be a part of an

    > operating system.

    Yes, I agree. And those of us Linux developers who have done it w/o the backing of a multi-billion dollar corporation know how hard it can be to get all the pieces in place, maybe even moreso.

    The Linux OS was started in 1991, after your deadline, and was writen entirely from scratch.

    The Linux Operating System doesn’t include userspace. So no compilers, no windowing systems, no web-browsers no web-servers.

    So by the above non-MS definition of operating system, Linux is a more recent from-scratch than NT. Just being pedantic.

    I probably wouldn’t have even bothered to post at all if it weren’t for the innacurate "borrowed network stack" comment in your original message.

  17. Anonymous says:

    I really like this post because he was right that the ability to write a new OS from scratch is really getting more and more insurmountable. But at the same time, your claim is indefensible and very revealing about your MS-centricity. I love Microsoft and I love your blog, don’t get me wrong, but to imply MS has got this one sewd up is ridiculous.

  18. Anonymous says:

    Re Vista being built on Windows server 2003: I thought Windows server 2003 was the server version of Windows XP, with much of the same code base?

  19. Anonymous says:

    "I can’t think of any really convincing arguments for an OS having to include a web browser, digital camera support, digital media player support, instant messaging support, etc. "

    It’s this sort of EVERYTHING BELONGS IN THE APP mentality that will hold back Linux until it’s miserable death in a decade or so.

    The kernel should support hardware – including cameras, media players and graphics cards. One of the primary responsibilities of the kernel is to mediate access to the hardware and the Linux kernel just doesn’t live up to those responsibilities.

  20. Anonymous says:

    Although there is some nitpicking about all this, I think it is fair to say that the original quote is fair and that Apple and the Linux/BSD community are striving to make it false.

    Several years I dealt with a really smart guy who was doing big picture important techie stuff for Microsoft. He had previously come from a UNIX company and "got" UNIX. (I believe he was a co-founder). He was asked my he was working for Microsoft. The answer: The ability to do things that end up on hundreds of millions of desktops. They did.

  21. Anonymous says:

    I think Tom is right — nobody will be able to build a successful PC OS from scratch anymore. Linux and OSX are hodgepodges of components from all over the place. The standard libraries are from one place, file sharing is from another place, GUI from some other place, kernel from somewhere else. That doesn’t make the OS any worse necessarily, it just doesn’t mean they fit Tom’s criteria.

    Windows NT was not just a kernel and filesystem designed and written from scratch, it was the whole OS and user experience. For example, things like printing, OpenGL, filesharing, and windowing also had to be included and written from scratch. Of course in 1989 they didn’t even know it would be Windows — it was going to be OS/2 3.0 — so back then it only had to be designed to run POSIX, OS/2 1.x, and DOS programs.

    Regarding the other applications, I’m pretty sure that although COMMAND.COM was included for compatibility, a friend of mine told me it was written in assembly, so a command processor would have to have been rewritten in C. That also explains why it’s called CMD.EXE. OTOH, I’m pretty sure I remember seeing something like "Copyright 1983" at the top of notepad.c, meaning that it is directly related to the code that shipped in Win 1.0’s NOTEPAD. I always wondered about other programs, such as File Manager, though. FileMan on NT acted like a native NT app (properly supporting 64-bit file sizes and such), whereas Explorer on NT still used 32 bits for certain file sizes as late as Win2k, indicating it had been ported instead of rewritte.

  22. Anonymous says:

    The last _commercial_ consumer os…

    where commercial means primarily income oriented, this bringing the ‘bundle (our stuff) as much as possible with the os’ and the ‘buy/destroy the competition, especially if their products are competitive, spread fud if it’s all that remains’ mentality

    I’m not bashing ms, I like some of the products, I use some regularly, I believe they have some very bright people working on the software, and I’m sure the company isn’t evil, not even BillG and his close underlings.

    Bottom line, Microsoft is an very, very, _very_ greedy company, and they sacrifice on lots of ocasions the quality of the software and the sentiments of the users in order to increase their profit. I’m sure they aren’t the first/only company working like this, the problem is that they have a pretty big impact on our lives nowadays…all this might change in a few years for better (think ibm) or for worse 🙂

    It very much depends on the users. In fact, all the eforts made by ms up to now to be perceived more by the marketing-created image and less by the technical achievments might end up working against the company if that image starts to degrade. This is contrasted by solutions like Linux that rely more on the technical/tried and proven aspects that, while still the dominating issues for the server market, slowly started permeating to the desktop users…

    As for Linux as an os, this (http://lists.ssc.com/pipermail/suitwatch/2005-November/000098.html) puts it better than I could 🙂

  23. Anonymous says:

    I really enjoyed the post, and agree the original quote is insightful and largely true. There are lots of OSes out there that may technically be counted as a desktop OS, but I don’t see the value in arguing about what is and isn’t an OS. In the end, the excitment is working on an OS that will be widely adopted as the major, or a major OS. And to write it from scratch, what a dream.

    So the only OSes other than NT that I see as major on the PC desktop are OS X and Linux. And while every line of code in these may have been written in the last 15 years, they both started as copies of much older Unixes, using the same designs and ideas. They are evolving, obvously, but there was never that clean slate. That option to rething everything from scratch. Man, would that be an exciting oppertunity.

  24. Universalis says:

    It strikes me that there are two ways of having Tom Miller proved right. One is for no new PC operating system to appear. The other is…

  25. vince says:

    I often see the argument put forth in some of these posts… that Linux is just a retread of *NIX but NT is "something new".

    Can someone explain to me what is so amazingly new about NT? How did it break from tradition and do something truly innovative that wasn’t already done before? How is it not just a souped-up VMS with lots of *NIX ideas thrown in?

    I know I’ve heard the HAL and also maybe file-forks before, but I am pretty sure earlier OS’s had done those already.

    If you want a truly innovative OS, look at Plan9, or to a lesser extent BeOS.

    Linux has never pretended to be a novel OS. It was designed to take the best things from *NIX and make them better, something I think it has succeded at. Being "new" or "from scratch" doesn’t always mean "better".

  26. jeuge says:

    FYI

    http://www.microsoft-watch.com/article2/0%2C2180%2C1882174%2C00.asp

    Another OS being completely written from scratch by MSR.

  27. Anonymous says:

    Just to add jetfuel to the fire, I would say that market share matters for the point of this. If you took the applaudable effort to make an OS, but have less than say, 1000 customers, then you don’t count HERE IN THIS THREAD. Vista is not rewritten; the NT code base "branches off" to create a product. And sorry Linux; a kernel with not even a shell does not count, and GNU was not new in 1991.

  28. Anonymous says:

    I really disagree that a new OS would require a web browser, although it is definitely a very Microsoft thing to say. Yes, there needs to be a web browser in the box but that’s a different thing.

  29. Anonymous says:

    The last consumer operating system?

    Could very likely be the case, so long as

    Microsoft continues to be the monopoly that

    it is, and continues to squash any upstarts –

    DRDOS comes to mind, for example – that are

    perceived by Microsoft as being a "threat."

    Somehow, though, I don’t find that to be

    very heartening.

  30. "I really disagree that a new OS would require a web browser, although it is definitely a very Microsoft thing to say. Yes, there needs to be a web browser in the box but that’s a different thing. "

    You just contradicted yourself. You said that a new OS doesn’t need a web browser, but it has to be in the box. That doesn’t make a lot of sense. What’s in the box is what’s in the OS.

    An OS has ALWAYS been more than just a kernel, who would distribute a *nix distribution without a shell (pick your shell, but without shell)? How about a *nix distribution distributed without rm or mkdir?

    Oh, you say, but those are necessary utilities. Well, so’s a web browser.

    Microsoft has chosen to package its web browser in a fashion that allows it to be used by any application installed on the system, and for an operating system, that’s a GREAT design decision – having an HTML renderer as a part of the platform means that any application can assume that the platform contains an HTML renderer, and thus the applicaiton doesn’t have to include one.

  31. "Can someone explain to me what is so amazingly new about NT? How did it break from tradition and do something truly innovative that wasn’t already done before? How is it not just a souped-up VMS with lots of *NIX ideas thrown in? "

    I never, ever said that NT was a break from tradition or particularly innovative. Heck, I even mentioned that in the article.

    But NT WAS written from scratch (with the exception of the command interpreter (which was actually written in 1985 for DOS 4.0 and later adopted for OS/2) and a number of the networking utilities).

    That’s the differentiating factor that makes this "interesting" – nobody’s written an operating system <i>from the bottom up</i> since then.

    (btw, I hate the lack of rich comments :()

  32. vince says:

    Why do you persist in thinking an OS has to have apps?

    It doesn’t.

    And just because most people run GNU on top of Linux does not make GNU some sort of inseparable part of it. I can write a simple assemly-language program from scratch that prints "Hello World" and have it be /sbin/init. If Linux boots and runs this program, would you deny that my app is running under the "Linux" OS?

    Even if no GNU utils are in sight it’s still Linux.

    > That’s the differentiating factor that makes this

    > "interesting" – nobody’s written an operating system

    > <i>from the bottom up</i> since then.

    Yes, but whenever we point out that someone has, you change your definition to exclude it. All you are proving now is that "No one has written Windows NT since we wrote Windows NT".

  33. Anonymous says:

    Saturday, November 19, 2005 9:49 AM by LarryOsterman

    > "I really disagree that a new OS would

    > require a web browser, although it is

    > definitely a very Microsoft thing to say.

    > Yes, there needs to be a web browser in the

    > box but that’s a different thing. "

    >

    > You just contradicted yourself.

    I think he distinguished the OS from the box, which might mean he equates the OS to the kernel. It’s not necessarily a self-contradiction, even if it differs from the opinions of you and me.

    > You said that a new OS doesn’t need a web

    > browser, but it has to be in the box. That

    > doesn’t make a lot of sense. What’s in the

    > box is what’s in the OS.

    Well, Internet Explorer is a critical component of Windows OSes, including Windows 95. And in Windows 95, Internet Explorer wasn’t in the box. Later versions of Windows 95 had the entire OS in the box, but earlier versions of Windows 95 had only a partial OS in the box. Do you still think that what’s in the box is what’s in the OS?

  34. Anonymous says:

    Larry, I think vince has you. The traditional definition of "operating system" has always been limited to the parts that manage the resources (memory, disk, cpu, …). That would be the kernel, the device drivers, the file system, and all that non-user level stuff. Everything else is an application.

  35. Dave, if that were true, then why does every single operating system shipped today contain all that application stuff?

    Try to find a Linux distribution that doesn’t include at least two browsers, three shells, two or three text processing applications, three or four different text editors, etc.

    In the classical sense, an operating system is just the kernel and memory management, filesystems, etc. But that classical sense has been completely meaningless for at least the past twenty years.

  36. Anonymous says:

    CraigM: I’m not sure about Symbian, but IIRC from my Palm developing days, the Palm OS wasn’t completely written from the ground up. It used the Kadak kernel under a specific license which (again IIRC) prohibited things like multithreaded apps, etc.

  37. Anonymous says:

    I just 2 questions

    1. How old was Tom then?

    2. Is he still alive?

  38. Anonymous says:

    Hi Larry,

    Just to clarify one point. While Tom was the principle architect of NTFS, Brian Andrew (who you didn’t mention) personally contributed the largest amount of code to NTFS, and eventually owned it — more or less — until he retired last year. His departure was a true loss to Microsoft.

    David

  39. David, of course you’re right. But I primarily associate Brian with CDFS, since he did all the design and implementation of it.

    And yes, I miss Brian 🙂 I miss lots of people 🙁