Dogfood Doesn’t Always Taste Good

A while back someone asked which phones the Windows Mobile team uses.  While I will end up answering that question, in typical form, I’m going to be really verbose about it.

In Windows Mobile, we’re very strong proponents of what we call, “Eating our own dogfood.”  What this means is that we use the code we’re developing on our primary phones.  If you’ve ever been on a “Beta test,” you know that beta code isn’t always very stable or bug free.  Well, we frequently use builds that aren’t even up to beta quality.  When internal people complain about this, my response tends to be, “They don’t call it dogfood because it tastes good.”  We dogfood (it’s a verb too 🙂 to find bugs.  Better it be unstable for us than for you….

Many developers do a lot of their work on the same emulators you use in the SDK.  Emulators are nice because we own all the code for them and can easily make them do whatever we want.  The downside, of course, is that you can’t put an emulator in your pocket and carry it around.  Microsoft doesn’t make phone hardware, so to do actual dogfooding we need to work with our partners to get a device to use.

We don’t make the chicken or the egg

One of the huge challenges we face in Windows Mobile is finding acceptable devices to use for Dogfood.  Because the device we need to use hasn’t been created yet, we almost always develop the next release on a device created for the previous one.  This causes serious problems (that lead to clever solutions) when the previous hardware doesn’t have features we’re trying to develop.  For example, it was hard to add QWERTY keyboard support because no device had a QWERTY keyboard.  And no one was going to make a QWERTY device until we added QWERTY support.

Another issue is that a huge amount of the code in a WM phone isn’t written by us.  It’s written by the OEM who manufactures the phone.  Frequently, to develop a new version of our software we need to modify the OEM parts of the code.  The trouble is, the OEMs generally don’t want to give us their code.  They’re worried that we might give it to their competitors. 

For these reasons, our choice of development platforms is limited not only by finding devices that will do what we need, but also by finding OEM partners willing to either let us use their code or make changes for us when we ask for them.  The “make changes for us” idea is a pain for the OEM, because they don’t sell phones based on that work.  Any time they spent making something work for us is time they didn’t spend making a phone they’re actually going to sell. 

These things contribute to the “dogfood not tasting good” situation.  Frequently our internal development builds are done on inappropriate hardware with driver modifications that were made by an overworked team with higher priority things to do.  As cool as this place is, it’s not always glamorous.

Dogfood through time

We’ve actually been doing this for ten years now, but I’ll just go back to the PocketPC 2000 release.  One of the main development platforms for that was the Compaq Aero 2100.  In these fun times, the only way to update a device was to physically remove the ROM chips and replace them with new ones.

When PocketPC 2000 came out, the most popular device created for it was the Compaq iPaq 3600.  The iPaq was a natural choice for our development/dogfood platform, but maybe not for the reasons you’d think.  First, we already had a relationship with the OEM (both the iPaq and the Aero were made by HTC).  That helped us convince them to give us access to its driver code.  Second, it ran on an ARM processor.  This was before we had standardized on ARM, but that was the direction we were heading.  Third, but maybe most importantly, it had Flash memory that let us install new builds without physically replacing ROM chips. 

While working on that, we also had a group doing the first Smartphone (for the Smartphone 2002 release).  There we had a huge chicken and egg problem.  There was no previous device that was anything at all like what we were trying to build.  Our division actually has a small hardware team that designed and built a few prototypes for us to use.  The first was called “Avenger” and the second was “Avenger 2.”  We don’t have the manufacturing capability to make a lot of these, so we worked with an OEM to make a version of Avenger 2 for us.  While this was supposed to be called “AV2,” a language mishap turned the name into “AR11.”  We also did a ton of work with a small phone company, but that didn’t end well.  I’ve got plenty of stories to tell there, but … won’t. 

We had a similar chicken and egg problem with the next PocketPC release (2003).  It needed to be a phone but there were no previous phone PocketPCs.  Thankfully, we were able to work with HTC (who had made the best Smartphone 2002 device) on what would become the XDA (aka the Wallaby).  For development of non-phone PocketPCs, we continued to use the iPaq.

For Smartphone 2003 we continued to use the AR11.  We also used the HTC SPV (aka the Canary) for a while but switched to the Motorola MPX200.  There were a number of reasons for using the MPX200, but one of the biggies was that the OEM who made it was willing to give us code for the drivers.  Our work on the AR11 had made HTC worried about giving us the SPV driver code.

Our next release was WM 2003 Second Edition.  One of the features in this release was QWERTY support in PocketPC.  To do this, we first paid a company to make a QWERTY sled for the iPaq.  We had a grand total of two of those, and I believe they cost us more than most people’s salaries.  We also found a company that was planning to make a keyboard attachment for the XDA, paid them to put a different layout on it, and bought a ton of them.  These were cheaper because the company was already manufacturing the keyboards.

We needed a way to do 240×240.  We did that by taking a 320×240 iPaq and just making its display driver only draw the first 240 pixels.  We needed a way to do 640×480.  For that we used an XDA (which had a 320×240 screen), made its display driver tell the system it had a 640×480 screen, and then “pixel quartered” the output (average pixels together) to make it fit.  It was ugly, but if you squinted, you could almost pretend it was the higher resolution.  And it let us test to make sure that text fit on the screen correctly, etc.

The most funky thing we needed to do was 240×320 for Smartphone.  We couldn’t use the pixel quarter trick there because the MPX200 was 176×220, which isn’t an even multiple.  Instead we made the Smartphone version of our code run on the XDA.  To simulate the number pad, we overlaid a number pad on the screen and let you tap it with your fingers.

For Windows Mobile 5, we had a number of devices.  On the PocketPC side we had three: the HTC XDA II (aka Himalaya), a prototype HTC CDMA device that didn’t end up shipping, and the Toshiba E750.  Although HTC had reluctantly given us source code for the iPaq and the first XDA, they wanted it more restricted this time.  So we worked out a deal where only two people in the entire company could see the code for those platforms.  These two developers were tasked with all driver development for the release.  Thankfully they were both extremely good developers (and one of them is a superhuman who typically gets more done than any two or three normal people).  On the Smartphone front we started with the MPX200.  Trouble was, it didn’t have enough ROM to hold WM5.  After a lot of herculean measures to keep it going, we finally switched to using the HTC SMT 5500 (aka the Typhoon).  We never got source code for the 5500, but HTC was using it to practice for their upcoming WM5 device and we were able to ride on those coattails.  

Windows Mobile 6 continued to use the 5500 for a while but switched to its replacement, the HTC SDA (Tornado).  We also used the HTC MDA (Wizard).  At the tail end, we got some HTC Dashes in.  In all of these cases, we didn’t have access to any of the source code.  An HTC engineer made changes for us. 


Not everyone in Windows Mobile works in the main feature team.  Some people work on focused items for OEMs so they see and use more devices than the main group does.  For instance, we worked really closely with Motorola on the development of the Q.  As a result, many people dogfooded Qs.  Etc.   But, for the majority, the above pretty much tells the tale.

Mike Calligaro

Comments (31)

  1. Mark says:

    Oh Mike you cannot forget the lovely HTC Universal. It would be unfair not to mention this device, which new employees were often stuck with, as well as all of our our MVP’s and many of our external beta customers grew either to love or hate .

    I am considering building a small fort in my office with the remaining and returned Universals.

  2. jayongg says:

    You also forgot that little bird-like clamshell Smartphone that never shipped :). Personally I still like them for development!

  3. K says:

    >> Windows Mobile 6 continued to use the 5500 for a while but switched to its replacement, the HTC SDA (Tornado).  We also used the HTC MDA (Wizard).  At the tail end, we got some HTC Dashes in.

    How come the SDA and MDA aren’t getting WM6 ROM updates then? =

  4. MikeCal says:

    Mark, you’re right.  I did forget about the MDA IV (Universal).  That was also used during WM6 development.

    Jay, I’d say that device falls into the "Completeness" paragraph at the end.  While you and I use it (it’s currently my primary device) the team at large didn’t use it for any of the major releases I mentioned.

    K, I can say, "We don’t call it dogfood because it tastes good" to other developers.  I can’t say that to end users.  The quality of the device-specific driver work to get WM6 on those devices was good enough for us to do development, but nowhere near good enough to release to the public.  See: for more information.


  5. scyost says:

    K – to follow up to what Mike is saying, I believe the Tornado might not meet the hardware requirements for WM6. Near the end, it became very painful for our Tornado dogfooders – the downlevel devices that we use for dogfooding internally typically don’t have enough CPU power or memory to handle the new platform. It’s not an experience I would wish on our customers.

  6. Albert Rosa says:


    This was very informative and a great peek into the lifestyle that Is a Windows Mobile Developer!

    Thanks for post!


  7. Zack says:

    Do you think it gives Apple iPhone software/OS developers an advantage to have ready access to the target device hardware as compared to Windows Mobile developers who have to work on a device designed for the previous version of WM?

  8. MikeCal says:

    Zack, there are pros and cons with our way vs Apple’s.  Our way is to create a platform and have a large number of partners work with and profit from that platform.  Apple’s way is to do almost everything themselves with as few partners as possible.  

    A pro to their way is definitely that they fully control their own destiny.  The chicken and egg problem we deal with so much will be smaller for them, since they can always just make whatever platform they need and develop for it.  And they shouldn’t have to worry about the source code to their drivers since they own everything through and through (at least, I assume that’s the case).

    The downside is that every last idea has to come from them.  Most of the best designs for our phones have come from our partners.  Sometimes they’ve taken features we were doing anyway and used them in clever ways.  For instance, the MDA (Wizard) was only possible because we added keyboard support and the ability to rotate the screen.  But we didn’t have the Wizard in mind when we added those two features.  That was HTC’s innovation.  And, sometimes, our partners come to us with compelling desgins that require changes from us.  For instance, the primary reason we have qwerty keyboard and landscape support in Smartphone was that Moto showed us what they wanted to do with the Q, and we said, "Wow, then we should add these two things to make that possible."

    It remains to be seen which model will end up working better.  In the PC space, having a lot of partners was the better solution.  In the music player space, doing everything yourself was better.  In the end, will phones be more like computers or music players?  I can’t say for sure, but, despite the challenges I listed in the main entry, I believe we’ve made the right choices.


  9. Patrick McDonald says:

    You mentioned during WM5 testing a HTC CDMA device that never shipped.

    Was this a upgraded version of the Harrier, aka the Sprint PCS PPC-6600 or the Verizon XV-6600?  I’m really interested to know if it was or if it was an entirely new phone.

    Can I get a leftover Universal? 🙂

  10. MikeCal says:

    Patrick, it predated both of those devices by a large amount.  It was more like a CDMA version of the original XDA (Wallaby).


  11. Matthew says:

    "the downlevel devices that we use for dogfooding internally typically don’t have enough CPU power or memory to handle the new platform. It’s not an experience I would wish on our customers."

    Err, none of the production devices have sufficient resources to run the fat sow that is Windows Mobile either, yet you don’t seem to object to subjecting your customers to those.

  12. Fred says:

    Matthew, I take it you don’t actually have a Windows Mobile device?  

    Windows Mobile devices are computers, albeit with cell phone functionality and in a nicely-sized package.  If you load a bunch of extraneous software on your computer, eventually you’ll run out of memory and other resources.  Windows Mobile devices are no different.  

    On my Cingular 3125 (aka HTC Star Trek), I’ve got ten applications loaded in addition to the ones included by Microsoft, HTC, and Cingular.  My device is rock solid.  I turn it off/on when I fly, but otherwise I don’t need to reboot my phone.  It’s very responsive and reliable.

    Obviously YMMV, but if you have problems with a computer do you immediately blame the OS?  Or could there be other factors involved?

  13. mamaich says:

    About Universal:

    "Jay, I’d say that device falls into the "Completeness" paragraph at the end.  While you and I use it (it’s currently my primary device) the team at large didn’t use it for any of the major releases I mentioned"

    Why don’t you consider all xda-developers and other forums community as a team of beta testers? That community is much larger than your development team in MS, and can provide not only the list of bugs, but can also give solutions or workarounds.

    We can give a review of WM6 usability from an end-user’s point of view, point you to interface bugs that are still left unfixed, give you some hints over what should be improved in newer builds. Just give us an opportunity to do so.

    You cannot do anything to stop people from using a beta/leaked/cooked ROMs on their own risk. So why not to make a profit on this by somehow legalizing these ROMs and providing a place where bugs/wishes can be posted so that they’ll be read by your development team?

  14. msbuild says:

    Mike left out a few classic devices that I fondly remember from early Smartphone days.

    Prior to the Avenger and AR11s we had the Hornet development boards. They were massive slabs of metal that had boards and screens and such glued on them. Hand-made in our hardware lab out of off-the shelf parts, you couldn’t carry them around but you could certainly run Smartphone on them. They had the footprint of a legal sized sheet of paper! I have two of these suckers sitting on the shelf behind me as a type. I bet if I could find a power cord they’d even power up. I used a Hornet for easily six months before we had portable hardware.

    When PocketPC Phone Edition started up someone figured out how to make a "portahornet". Basically cram the Hornet parts into a plastic box the size of a shoebox. Downside? The antenna stuck out right where your ear was supposed to go. Did a good job of getting ear wax out though!

    Ah, memories.


  15. scyost says:

    Hey Mamaich,

    This is my opinion and may not be endorsed by my employer – here are some of the factors involved:

    – you’re flashing devices that, in most cases, were sold subsidized by an operator. They don’t like those devices to be flashed and modified with arbitrary external builds – they incur a supportability cost among other things. Without consent and agreement from an operator, we wouldn’t be able to get permission to do what you’re saying. We internally have to pay a premium to get these devices with unlocked bootloaders so we can flash development images onto them and they don’t like it when those images leak.

    – there are marketing and competitive reasons to keep the product under wraps until it is close to release. The issue is more severe for our product because we license Windows Mobile to OEMs and then operators sell the devices. So there’s typically a lag time of weeks to months between when we finally sign off on a device and when it gets to shelves. If we were to disclose all our features at beta time, it might be a year or more before customers could even get their hands on the features that we’ve disclosed. In this time the momentum and demand for the features could cool off, and it gives our competitors a long lead time to know our plans. Marketing likes to carefully choose when the new feature set becomes widely public. (I’m glad that is their job and not mine)

    – like Mike says above, hardware incompatibilities and other problems frequently lead to a bad experience on devices that aren’t designed and tested against the new platform. Having external customers bumping into these problems would probably sour several of them quickly. Angry beta testers tend to give less useful feedback, and additionally these bad experiences can get spread around to give the product a bad name, even though they won’t be present in the final retail copy. Sometimes it takes a strong stomach to be a good beta tester and stick with it through the rough times.

    I’m not saying that we couldn’t work out a way to get value from an external open beta – Vista and Office and Visual Studio are all embracing that concept. There are good arguments for going in either directions. I personally as a tester would love to have that feedback coming in about my features before the release. It’s just a complicated topic. I realize most of the issues I describe above probably don’t have a lot of weight for you as an individual, but they are important to others that would have to be involved in the process. We’re definitely interested in getting more feedback earlier in the development process for the reasons you describe and more. I think it will just have to maybe move more slowly than it does for some of Microsoft’s other products because our business has different challenges than theirs.


  16. mamaich says:

    "They don’t like those devices to be flashed and modified with arbitrary external builds – they incur a supportability cost among other things"

    We flash such ROMs on our devices on our own risk. And if it would make a brick – it would be whole our responsibility, and we would try to reanimate them ourselves via JTAG soldering or pay the full price for a new motherboard. And in that case OEM would get an extra money!

    Just an extra line in a support agreement that sounds like "flashing anything to your device would cause warranty loss" and minor hardware changes (like a jumper on motherboard that should be soldered to allow bootblock flashing) would stop 99% of users from doing such things at home.

    Unlocking bootloaders and using undocumented commands somehow reduces an amount of support calls to operators. Lots of the failed ROM upgrades on HTC Universal can be fixed by "task 28 55aa" command and similar ones.

    "I realize most of the issues I describe above probably don’t have a lot of weight for you as an individual, but they are important to others that would have to be involved in the process".

    You are right. All of us understand that such wishes cannot be carried out due to marketing reasons 🙁

  17. MikeCal says:

    mamaich: to be clear, my message to Jay wasn’t about the Universal.  It’s referring to another device most of you have never seen.

    As for beta tests, we do hold them, but they’re with closed groups of people under NDA.  Scott touched on some of the reasons we need to do that.  As for people understanding that they’re using the beta "at their own risk," we’ve generally found that not to be case.  Yes, some people like yourself will understand what it means.  But enough wouldn’t that we would get deluged with support.  Case in point, we once had a closed beta for a particular device.  We made it very clear upfront that, because the drivers were out of date, camera wouldn’t work on this particular build.  Despite that, we still had a ton of people claiming publicly that the next version of Windows Mobile wasn’t going to support cameras.  When things like this happen with pre-screened groups of people who have actually done beta tests before, we’re extremely wary about opening betas to the public at large.  

    I suspect that most of the people active on XDA Developers would be great beta testers.  But it’s an open forum, and putting betas there would result in everyone using them.


    Hey, it was good that the portahornet’s antanna kept you from putting the device to your head.  There was no shielding in that device and long phone conversations probably would have fried your brain.  (-:


  18. buzz_lightyear says:


    "They don’t like those devices to be flashed and modified with arbitrary external builds – they incur a supportability cost among other things"

    On other side we (community forums) give much more support for any device, than operators, who subsidize their devices. Our support doesn’t cost them any money.


    "Angry beta testers tend to give less useful feedback, and additionally these bad experiences can get spread around to give the product a bad name, even though they won’t be present in the final retail copy."

    I haven’t seen any angry "community testers" yet. I totally disagree.


  19. scyost says:

    buzz – You don’t have to convince me, you have to convince the operators. 🙂

    See Mike’s comment above re: cameras for an example of widespread but incorrect negative feedback.

  20. So there is still no sharing of the driver code from the OEM to Microsoft? I would think it would be in their best interest to do so.


  21. The Shonko Kid says:

    Mike, great story.

    I’m a big fan an proponent of dogfooding, can you tell us if you think WM is a better platform because of it, or do you think that it probably wasn’t worth (the considerable) pain and effort?

    Having used an SPV for a few months back when they were new, I think you could tell that the developers had been using it as their primary phone for a while.

  22. MikeCal says:

    Michael Sainz:

    As with most things, this is a complicated issue.  "Best Interest" is hard to quantify.  There are definite benefits to giving us the source code to a device.  We give back any changes we make, so if we find and fix bugs, the OEM will get those improvements. And, when we update a platform for the next version of the OS, they get a great example on what they need to change for their own devices.  

    But technical interests need to be weighed against business ones.  If we only work with company A, then that company can be fairly comfortable sharing stuff with us.  But what if we work with two companies, A and B, and they are competitors?  If A is more successful than B, A might worry that if they give us their code, we might give it to B to give them a leg up.  

    At that point they have to weigh the benefit they get from us working with their code against the risk of us using their code to improve their competitors.  

    Now, we as a company would never do that.  If we say we’re not going to give their code to their competiors, we won’t.  Doing so would be immoral.  But there are 1000 people working in Windows Mobile.  What if one of them is less scrupulous than the others?  That’s where these restrictions come from.  They know they can trust ME.  They work with me all the time.  And, for similar reasons, they know they can trust two of the engineers who work for me.  So they say, "Okay, we’ll give you the code, but only Mike’s two engineers are allowed to see it."


  23. MikeCal says:

    The Shonko Kid:

    There’s no question in my mind that dogfood improves the WM platform.  It’s definitely worth the pain and effort to make happen.  I’ll go so far as to say that I’m not fully sure it would be possible to ship a good product without some amount of dogfood.  Abstract thinking about how things should work is fine and all, but you’ve got to actually validate your theories through use to make sure you’re right.


  24. Mick says:

    Mike,there’s probably a good answer for this but what I’d like to know is … if Nokia, RIM and soon Apple can design, build and market their own phones and mobile OS’s, why can’t Microsoft? Wouldn’t that solve a heap of problems?

    Cheers for the insight!


  25. MikeCal says:

    Mick, see my response to Zack above.  


  26. Midget_1990 says:

    I aggre with mamaich and buzz_lightyear here, although some of the builds i have ‘found’ recently for the universal have been fairly new, it would be nice if even a small group of the more experianced members were allowed to partake in beta’s.

    just out of intrest whats the cureent developement build for WM6 on the universal (i’d like to know how far behind i am 🙂 ) dw if thats not something you can say…


    Midget_1990 (and yes, I am only 17)

  27. MikeCal says:

    We will only use a platform for Dogfood if we have the ability to do nightly builds on it.  During development of a product, our build lab builds every night and we have the ability to flash a new build every morning.  

    Various people flash their devices at different frequencies.  Testers tend to flash more frequently than program managers, for instance.  

    Also, the frequency that we tend to update our devices changes at different points in the development cycle.  Early on things are changing frequently and we need to do more frequent updates.  Later in the process, things are more stable and we’re better served by dogfooders doing less frequent updates.  That helps them find longer term bugs.  You’ll never find the "it stops working after 3 days of use" bug if you flash it every day.  Etc.

    But, internally, we have nightly builds available to us.

    As for opening up betas to small outside groups, we do that sometimes.  They’re always people who sign NDAs, etc, though.  We’re unlikely to put a beta on an open website for anyone to download anytime in the forseable future.


  28. mirekluza says:

    I have my MDA Vario (HTC Wizard) for a few months and my guess is that the ability to "hack" devices or to use unofficall firmware increases attractivity of the platform. At least for technically oriented people who like to "play" with their devices (perhaps a minority, but I guess it includes most of actuall or potentiall programmers – so it benefits far more than numbers would say).

    Before buying my device I was deciding between Windows Mobile device and a Symbian device (I was quite happy with my normal Nokia phone). What decided was the available software. But another factor came in later – I mean ability to use "unofficall" firmware… I doubt I would have so much fun with any Nokia. I went through 4 versions of WM 6 (all usable, I was just always curios what’s new 🙂 ). I guess I am now hopeless fan of Windows Mobile… And I know that as soon as my contract runs out, I will buy another Windows Mobile device – better processor and VGA display would be nice… I guess Nokia has lost me as a customer for ever…

    But back to the dogfood. It can be quite tasty. What irritates me is that I know there are officiall Microsoft working versions of WM 6 for the HTC Wizard, but if all kept rules they would be just lying in Microsoft. For some marketing reasons… Even though I would be prepared to pay a resonable price for upgrade. The WM6 is implemented for Wizard, so there would be no additional expenses…

    Fortunately this is real world and thinks leaks out. So I am running a very well working copy of WM 6 based on a dogfood version from August 2006 (made *specifically* for HTC Wizard). I am thankful to people who made it. I will not give here any details, I guess people from Windows Mobile team know very well about these things…

    One would think that in a market economy when the producer has a product and a customer is willing to pay, it will lead to deal.. But I guess the world is more complicated… I guess it must be even for Microsoft programmers a bit frustrating to have a product and not being able to deliver… But I guess marketing people do not understand this…

    In conclusion – I know it is not going to change. I am happy with my MDA Vario with WM 6. Though I would be happier if I could use an officall way to get it.


  29. I read Joel Spolsky’s blog from time to time. I am a little embarrassed to admit that I took the following

  30. One of the most interesting stories at Macworld hasn’t gotten a lot of attention in the larger press – namely that Google was around at Macworld a lot more than most people realize. It’s not just that they have…

  31. GUEST says:

    seems like wrote part of a busines case on why microsoft should create de hardware, instead of answering what mobile phones you test on.