Bugs, how I love them …


I love hunting them down and killing them.  Hunting down and killing the developer who caused the bug would be just as much fun if it weren’t illegal in every country in the world, that and programmers would probably be an extinct species if every programmer who had a fix a bug killed off the guy who created it … and of course if they introduced the bug … well you get the picture, it’s not pretty. 


The thing is, if you haven’t debugged a bug in an emulator, you haven’t really had to debug.  Oh, I know people say debugging hardware is tough and I’ve had to do that, but debugging software that emulates hardware or even virtualizes hardware is an art and you really don’t know the how hard hunting down a bug can be until you’ve gotten an emulation bug.


The first hard bug I ever had to find was back in my Apple days (think System 6 … yeah I’m that old :-).  Essentially if the user created a new folder on the desktop and renamed it you corrupted the desktop database which essentially wiped your hard drive clean.  I didn’t track it down but figured out a way to reproduce it.  The engineer who came and looked at it introduced me to one my favorite programs:  Macsbug .


From then on it was assembly or nothing.  For me starting at the bottom was the way to go; who needs source code.  Understand the machine, the compiler and the rest is pretty easy … until I met an emulator.  Don’t get me wrong, I know that developers all day deal with tough bugs.  Code is filled with them, but hands down in my 16 years of developing software the emulator bugs are the toughest.  In the six years I worked on Virtual PC and Virtual Game Station I would say on AVERAGE my bugs took over a week or more to fix.  That’s PER bug, at least 40 hours of my time, not working on 5 or 6 bugs at once, focusing my entire attention at one issue.  On Virtual PC 7, in the 16 months I worked on it I think I only fixed 30 some odd bugs. 


The doozy  of them all has to be the malformed TLB entry.  For those who need a brush up, Virtual PC was a full blown PC motherboard emulator.  We emulated a Triton-FX/BX chipset from the Pentium 3 chip down to the A20 interrupt.  In between all of that was memory management, disk, networking and host of other integration features.  In the memory management unit for our emulator we dealt with page faults and memory page mappings … all of this to properly emulate the MMU and let Windows (or other x86 based OS) believe it was running on real hardware.   TLB stands for Translation Look-aside Buffer, and it is used in managing page faults.  Most of the code that did this was written back in the early days of Virtual PC by Eric Traut for when we have the original PowerPC processors and G4 machines (circa 1998-1999).


Okay, so cutting to the chase, when the G5 processor was introduced we had to do a great deal of work to re-write our low-level emulator to deal with a number of issues brought up by the G5.  What we didn’t know is that lurking deep in the guts of the memory management was a very subtle malformed TLB bug.  So here we are, 14 months of development, we’re ready to ship with XP SP2 (we held up shipping to make sure we could ship the latest service pack) and wouldn’t you know, 20% of the time on DUAL G5 processors, when running through the OOBUE (Out of box user experience) on XP SP2 Professional, Windows would crash, but somehow manage to recover.  With XP SP2 Home it was only 10% of the time and when we ran XP SP2 Professional with a stripped down OOBUE, we got the failure rate down to 5%.


As anyone who has ever developed software would say… SHIP IT!  Okay, no that’s not really what they would say.  Usually it’s something like this:  “ $*&^!@%@!#$!@) !(!*@#&@!#*(“ and if they are from New Jersey the explicatives go on for quite a while.  Needless to say we finally did ship VPC 7.0 and found a fix for the TLB bug.  It only took me 9 weeks and 100s of hours trying to get the bug to reproduce though automation and manual means, as well as having the smartest emulator gurus at Microsoft looking over my shoulder at the source code and helping me debug the problem.  


When I thought I had the fix, we ran all the lab machines through an automated test script overnight.  Each machine ran through the OOBUE about 100 times, and we had maybe 20 machines going, turning our lab into a sauna.  I came in the next more and found out that we had a 0% failure rate.  I can’t even put into words the feeling I get when I solve these tough problems, anyone who has tackled a tough bug knows what I am talking about.


That was the thing I loved the most about working on Virtual PC, no matter how good you thought you were there was always a bug like that waiting to challenge you.  In some ways that challenge is what I am going to miss the most too.

Comments (28)

  1. Hansköbi says:

    Perhaps you can fix some challenging bugs in msn messenger;-) I’m glad there’s going to be a new version by the end of the year! Please make it better!

  2. Anand says:

    If the ppl over at the MSN Messenger department worked as much as you do, we’d have a decent chat application.

    Maybe.

  3. Eric Albert says:

    Do you remember what the actual bug was?  I’m curious.

    And I know exactly what you’re talking about regarding the feeling of checking in a fix for some very tough to track down bug and suddenly seeing occurrences of the bug completely vanish.  It’s great.

  4. James says:

    If Microsoft is abandoning Virtual PC because of Apple’s plan for bootable Windows, that is a mistake. Because some things, such as outdated drivers, prevent certain things from working in the OS X classic environment, about several times a month I have to reboot into OS 9. If there were a solution that allowed me to have a fully functioning (Classic is not) OS 9 emulator, it would be worth having to avoid OS9 reboots.

    Even though the OS 11 will allow the MacOS and the Windows System to run off the same computer, my understanding is that you have to keep rebooting to go back and forth.

    Virtual PC eliminates this problem. Ceasing devekopment on Virtual PC is a mistake.

  5. khalidur says:

    you guys should think about moving the "posted by" part to the top of the post, it would make it easier to identify who is writing what post.

    i’ve only been reading the blog for a day and already i’ve recognised various distinct writing styles and it would be nice to put a name to the style :).

  6. Dr.T says:

    "Hunting down and killing the developer who caused the bug" is illigeal in every country – thats right. It’s also right, that killing the developer who caused the bug would decimate the count of developers down to zero.

  7. Alex Blewitt says:

    Why are you using a smaller, less readable, font in your postings with relation to the other posts on the Mac Mojo feed? Frankly, if you’re just going to post miniscule fonts then I’ll just skip over your posts in the future.

  8. Mark says:

    Hey… just found the blog via http://www.tuaw.com/

    Any chance on increasing the default font size? it’s a bit hard to read on my 15"powerbook in firefox.

    (i know, i know… i can change the font myself, but the site is still hard to read in its default settings)

  9. Rob says:

    Messenger requires a lot of work to be honest, it’s one of the worst IM clients for Mac due to the fact that it lacks so much in terms of functionality compared with other IM clients for both Windows and Mac.

  10. Tk2K says:

    Wow man, you really should write more!

    I really love your style of writing! you state some of the problems with software coding very very well, and I enjoyed reading this a huge amount!

  11. TK2K says:

    wow man, your writing style is fantastic!

    please, please, tell us more about the work you do, the way you describe your inner passion for this stuff is amaszing! i really love your writing style!

  12. Stephen says:

    Are you planning on adding the features that are in Windows Live Messenger like webcam video to the Universal Mac version.

  13. Glenn Kelly says:

    Hello.

    Maybe you guys can fix a bug in Entourage, when composing an email, you click the drop down to change the font type, and it take forever and a day for the list to appear. Can you make the formatting window like you do in Word, which does not hang around when you want to change the font.

    Also, not a criticism, but why is Office for Mac way better than Office is on a PC!! Im not complaining. Entourage is miles better than Outlook. Keep up the good work. Oh, and will there be a new release of Office for Mac anytime soon?

  14. commenter says:

    Hi Bugs are more apparent when you check things on other systems. Being a PC user I don’t have Calibri installed on my system and the text size of 10pt is way too small for my old tired eyes 😉

  15. Marcelo R. Lopez, Jr. says:

    I’ve been reading up on the MacBU blogs, and came across yours, and your post about the sad ( I’ll refrain from drama and withhold the "tragic" card. Oops, too late. ) loss of VPC PPC.

    This is particularly sad for me because up until now I’d been doing something only a mad man would conceive of. Actually trying VS2005 on a 1.67Ghz AlPB. Yes, I know, with the effectiveness of a 450Mhz P3, but you’re talking to a man who only recently and begrudingly parted with both of his Apple ][+’s ( If you’ve ever seen people use the "][" when referring to an Apple, that’d give you an indication of what an old coot I am. ).

    Anyway, I’m writing with some small glimmer of hope that you might know someone who might have encountered what I’m encountering within VPC 7.02 ( I’ve applied every patch I could find ). Essentially the problem is that while I can run VS2005, I’m unable to install SqlServer 2005 ( Express or Standard, doesn’t matter ). After the actual install has completed copying files, the install actually tries to start the SqlServer service. It’s here that I run into a problem. The service dies ( simply terminates ) with an exception, and then the installation gives up, and removes anything that it had copied onto the drive. I’ve read through blogs, forums, high and low to try to find out why it throws this exception. And since I don’t have the symbols, it’s not like I could fire up WinDBG, attach to the process myself, and trample through the startup to figure out what in blazes is going on.

    I’m sure you’re pretty busy, and off chasing the next version of Office:Mac, but if you find yourself feeling generous, and know someone who could lend me a hand, you’d have my sincere thanks and gratitude.

    If only knew where to email this to….

  16. ExCntx says:

    Sorry about the font size.  I’m pretty newbie when it comes to posting.  I wrote it in Word and did a copy paste, so that’s why the font’s so small, of course the lines I added at the end are the right size.  Maybe we can fix it…

  17. IRMUG says:

    One thing that puzzles middle east Mac user is the strange position which is taken by MacBU in not supporting the RTL script system(Arabic, Persian Hebrew, Urdu….). While the Windows version support many of these languages fully the Mac version developer has categorically has refused to add RLT support to it. I would like to ask people here why the middle east Mac users are discriminated.

    Thanks for your answer

  18. Singerboy says:

    Probably, not the best forum…please direct me to the best one:

    Office saving issues(several different errors) on network mounted home directories…this is a widely published issue on the net. Apple knows about it, MS knows about it. Ever since 10.4 this has been an absolute nightmare. 10.3 was the last good implementation of these 2 products working seamlessly together. How do we get you guys to work together to resolve this? We’ve even opened apple server support cases on this ($$$).

    Where do I send this to on the MS side to get the ball rolling in a more collaborative way with Apple.?

  19. Mr. Smith says:

    MacBU, can we have MSN Messenger for Mac with all the features that Windows Live Messenger has now (videochat too)?? Messenger for Mac is now an horrible and terrible application… It’s disgusting and rubbish comparing with other IM software…

    Yeah, I think that you can do it for the next Universal Binary release… didn’t you think so?

    And can we have an Microsoft Office Publisher edition for Mac? Maybe when the UB software will come out…

    I hope that you keep in touch this: it’s what Mac-users would like to have.

  20. Skrud says:

    Great post. I think the feeling that you get after overcoming an exceptionally challenging problem — whether it’s a bug or just a tricky logic problem — is what makes software worth doing.

  21. Andreas says:

    I hate bugs. I hate them. I just want things to work. But, well, I’m not a developer.

  22. Azzam says:

    One thing that puzzles middle east Mac user is the strange position which is taken by MacBU in not supporting the RTL script system(Arabic, Persian Hebrew, Urdu….). While the Windows version support many of these languages fully the Mac version developer has categorically has refused to add RLT support to it. I would like to ask people here why the middle east Mac users are discriminated.

    Thanks for your answer

  23. <rant>Bugs . . .  hmmm, I’m not sure this is the right place for this comment, but you mentioned System 6, and that caught my eye. I too am old enough to remember System 6 (and before–all the way back, even to MultiPlan). In the old days, Excel had a preference much like it has now that permitted users to specify what the behaviour after a "RETURN" should be (down, up, left, right, no movement). Somewhere along the line, this got bundled in with the ENTER key. IMHO this is a bug, or maybe at least a bad idea. ENTER used to be "no movement" and RETURN was specified by the preference. Can we please return to that early Excel behaviour? It’s really a better idea. Especailly after years of doing it, I’m still frustrated (as an old fart) that I cannot retrain myself to expect the same behaviour out of the two different keys.</rant off>

    Thanks for listening.     randy

  24. Bart Hanson says:

    You’re on the inside man… I thoroughly enjoyed your post. I have always thought that as computers get more powerful convergence between the platforms would occur via emulation etc. and this seems to be what is happening. Along with this convergence, I hope that app developers see the importance of open file formats. (Publisher anyone?) Users will need to be able to read their files in Mac OS 21.5.4 and to pass data between apps. I have always been a Mac man, (since System 7.1) and have never felt that Microsoft is the enemy. I even like some things about PC’s but the personality of the Mac shines through every time. Good luck and success to you at the MBU.

  25. neo says:

    I wish Messenger can be fixed by you.

  26. So Totally True says:

    Great Thread!

    I found a BUG in Entourage for Classic, where for no reason at all, after opening TOOLS ACCOUNTS, right there in the listing for accounts, where it should say Untitled E-mail account 1(POP), it says "U t tl d E- a l a  o  t 1(POP)" .. wtf?? Is this a bug, or possibly a Mac virus ?

  27. foundation for california community colleges