Hi, I’m Eric and I’ll be your software developer this evening

other day I mentioned my worst customer-impacting mistake ever — marking a heavily
used object with the wrong threading model.  A
number of people commented to me that it was unusual to see a developer own up to
such a mistake in a public forum.  "urn:schemas-microsoft-com:office:office" />(Surprisingly,
no one pointed out that it was also odd to do so while talking like a pirate.)


it’s because of restaurants, believe it or not.  My
father has been in the restaurant business for many years. Something he taught me
at an early age is that one measure of the quality of a restaurant is how
few mistakes they make
, but a more important measure is how
they treat the customer once a mistake has been made
. Do they apologize,
take responsibility, and immediately act to correct the mistake
, or do they engage
in cover-ups, blame-shifting and foot-dragging?  I
don’t go back to the second kind of restaurant, no matter how good the food is.


software industry is no different. Here are four aspects to consider:


Customer focus: Most importantly, when a customer-impacting mistake is made it is
necessary to inform customers of the problem, take responsibility and if possible
fix the problem, period.  If that means
they think I’m a bozo, I’m cool with that.  Preventing
customer pain is a whole lot more important than anyone’s opinion of me.


Public image: Just because I don’t walk up to you and say “Hi, I’m Eric and I’ll be
developing your application infrastructure and development tools this evening” doesn’t
mean that said tools aren’t developed by humans with names and faces!  Too
often companies such as Microsoft are perceived as faceless monoliths, a few dozen
six-story sea-foam-green glass boxes that mysteriously transform money into code.  This
misperception belies the reality; I work with dedicated and talented people who care
deeply about making their customers more productive.  All
those people — all the testers, developers, managers, writers, etc, — deserve credit
for shipping great software.  And when
someone really screws up, well, adult humans should be capable of the occasional mea


Evolution of the industry: The engineers who sign off on a bridge are quite literally
putting their names on the line.  If software
engineering is ever going to evolve from a craft to a fully-fledged engineering discipline,
we need to start acting like engineers.  />(And
as a

math major, that’s hard for me to say with a straight face, believe me!)  If
I’m not comfortable with signing on the line and saying that this code is ready for
prime time, then I shouldn’t be shipping it.


Diffusion of knowledge:  Finally, the
best mistakes to learn from are other people’s
.  Learning from your own
mistakes sucks, not to put too fine a point on it.  If
I can help other people learn from my mistakes, so much the better for the world as
a whole.


soon: Eric’s least customer-impacting mistake ever.  But
that will have to wait until this evening, because my band has a gig at our Product
Unit meeting this afternoon.  Yes, the
Trinity Team has their own rock band!  We
are “Red Pills Reloaded“, aka “Rage Against The Blue Pills“.

Comments (6)

  1. Every time I look at the Citicorp building I think that the <a href="http://deadprogrammer.livejournal.com/2003/04/15/">only thing that is holding it up</a> is the integrity of it’s architect.

    By the way dictionary object thing is nothing compared to ::$DATA thing. I had a lot of fun looking at other people’s code when that happened.

  2. Blake says:

    Despite how easy it was to exploit, ::$DATA was really a pretty subtle bug to catch. The fact that NTFS supports multiple named streams per file has always been little known or understood. The fact that the ‘main’ stream is named :$DATA was never documented at all as far as I’m aware. I’m sure the people on the IIS team who wrote the URL cracking code had no reasonable way to know that it was named such.

  3. Eric Lippert says:

    The fundamental problem wasn’t the $DATA vulnerabilty. The fundamental problem was that the guys who wrote the URL cracking code _failed to an insecure mode due to a canonicalization error_. The fact that $DATA happened to be the vulnerability that was found is actually kind of irrelevant — when you make decisions based on noncanonical data you invite these kinds of problems. I talked about this quite a bit in my book actually.

  4. Blake says:

    I agree whole heartedly about ‘fail safe’, in the original sense of the phrase. The importance of that as a design point can’t be overstated.

    That said, I don’t follow why that was strictly a canonicalization error. They are valid path characters per rfc1738 and it’s updates. It’s a valid Win32/NTFS path as well. Why _should_ they be canonicalizing it?

    Hmm, on the other hand, what is broken is when I think about it is extracting the extension and hence type. Actually, there’s lots of interesting questions there… clearly the streams within a file have different types. The stream containing document properties shouldn’t be passed to the ASP engine even thought it’s file extension is .ASP.

    Okay, I’m just rambling now, ignore me.

  5. Peter Torr says:

    I thought we transformed code into money…? 🙂

    As to the :$DATA thing. I was actually thinking the other day (for reasons that are beyond me) that it would be "cool" to get IIS to scan the entire wwwroot folder (and any other vroots) and keep all known filenames in memory. Then whenever it got a request, it would merely check if the name was in its list, return it if it was, or reject it if it was not. This (in theory) would stop many information leakage errors, because (i) there’s no attempt at canonicalisation, and (ii) IIS never says "Oi! Mr. Windows, Can you load this ‘ere file for me?" It only loads files that it knows are good and safe, using the exact same filenames it retrieved from the filesystem search itself. Just a thought, although the perf / memory impact is probably quite bad..