Whimsical embarrassment as a gentle form of reprimand


A few months ago, I messed up a cross-component check-in and broke the build. I'm not proud of it. (In my excitement over finally having passed a few weeks' worth of testing requirements, I absently submitted only one of the components for check-in! My change was 99% within one component, and I forgot about the other 1%.) My submission cleared the "single-component check-in" queue at around 4:30am, and before I got a chance to fix the problem at 8am, a complex job was submitted into the "multi-component check-in" queue. That job failed, of course, because I neglected to update the second component.

A few hours later, I was greeted with a large inflatable bunny rabbit (looks similar to this guy) in my office. His name is "Bug Bunny", and it is my lot to be Bug's keeper until somebody else breaks the build. (But hey, at least I fixed it before 5pm. At 5pm, my team's lab kicks off its nightly builds, and if you break those builds, the next morning's "official team build" doesn't get released, and testers don't have anything to install.)

I suspect many groups have an object with a similar purpose, namely to be "bestowed upon" the person who most recently messed up.

Comments (39)
  1. Ytram says:

    At one of my previous companies, we had something we called the Suicide Bunny. It was a rabbit stuffed animal toy with a noose around its neck. It wasn’t really for breaking builds(we didn’t do source control or daily builds at this company), but more for when someone was given a particularly hard task that usually meant staying late and/or re-writing some of the current code base.

  2. Fat Albert says:

    Here, breaking the build obligates you to provide a tasty and nutritious (not really) breakfast for the team the following workday. Generally, that amounts to 3 dozen donuts. So, there’s financial incentive to be careful.

  3. Gord says:

    Our most recent build-breaker must proudly hang this poster:

    http://despair.com/mis24x30prin.html

  4. Bruce Morgan [MSFT] says:

    On the Encarta team back in 1995-98 or so, we had "the clown". This was a rather scary looking stuffed clown figure, plastic face & cloth body, with long red hair etc. You earned the clown if you broke the build or if you significantly broke someone else’s feature due to your error. You couldn’t pass it along until some other clown did the same thing.

    This bit of negative reinforcement had some interesting side effects. Some people actually felt so shamed by having the clown, they went to extreme and productivity killing efforts (multiple builds on separate machines, excessive unit testing, etc) to ensure they never earned the clown.

    Other devs took pride in their "push the envelope" ways. After a while, there was a sense of "real devs aren’t afraid of the clown". Somehow he also gained a little plastic cigarette glued to one hand, and a little bottle of whiskey glued to the other, and sometimes a jaunty little hat. Now people proudly showed him in their office, occupying their guest chair or a place of honor on the desk.

    Eventually we retired the clown. I wonder what become of him…

  5. anon says:

    It’s sane to break a build once in a while, that’s human.

    I am just as suspicious about someone who never ever ever breaks the build as someone who routinely does…

  6. Dan says:

    Where I work, we have an item that we call the "Fork of Mediocrity". It is a wrapped plastic fork. Whenever one of us does something bone-headed, that individual gets the fork for an unspecified period of time.

    The whole ritual has gained a fairly complicated rules structure. Only the person who has the fork can bestow it upon another, but the transfer is only allowed if a sufficient period of time has passed since the mistake, based on the severity of the mistake.

    However, if a more serious mistake than the current mistake is made, the fork can be transferred immediately.

    I’ve had the fork for over a month.

    And don’t even get me started on the "Ice Cream Scoop of Excellence"…

    (And no, I’m not making any of this up.)

  7. Tam says:

    We have the Big Rigs box at our studio. Perhaps one of the most excellent games ever made.

    http://www.gamespot.com/pc/driving/bigrigsotrr/review.html

    Whoever breaks the build will find themselves the owner.

  8. Jay B says:

    I don’t know if you ever make wide sweeping changes across many files/modules, I certainly do. I work in the Tools and Infrastructure team and frequently make changes to large amount of files (a recent conversion from OWL on BC 5.02 to OWLNext on VC++2003 across our entire code base resulted in thousands of files changed), and I’ve had to start record keeping on files I’ve touched. As soon as I start making changes in a file, I put its path in an excel spreadsheet and later when it’s time to check-in, I do a BeyondCompare diff on those files to review the changes, if any. I also check files to see if any ReadOnly flags have been removed on files I didn’t have in my spreadsheet. It’s made a pretty big difference for me, but I know that each developer deals with this differently, and their processes aren’t always good. Even with all the checking I do, I still sometimes screw up the check-in, since it’s a manual process for us.

    I have yet to find a solution that is error-proof. Source control integration can help to some degree, but I have to question whether it would allow you to check-in changes you didn’t intend to.

  9. Andrew Ho says:

    The first place I’ve heard of with a totem of shame for breaking the tree was at Netscape, where they had "the lemon." This was a plastic lemon that you gained by making a bad checkin. You could only lose it by giving it to someone else with a bad checkin.

  10. daveidixon says:

    At a former workplace we had the "Devil Monkey". It was a cute stuffed monkey with a knife glued to one hand. He was not handed over conspicuously, but hidden in your office waiting for you to stumble across it. Be good or suffer the wrath of the Devil Monkey!

  11. Jay B says:

    ^^^Also, in addition to the previous post…

    We have Promotion Groups (PVCS) and breaking the Dev level build (check-in sanity check) isn’t really a big deal. Breaking the Public build (testing level) isn’t such a great thing though. So after I check files in, I add them to a separate spreadsheet that I use to track promotions. Once I see a successful Dev build, I promote the files listed in the promotion spreadsheet, and then remove them.

    It’s a lot of "stuff" to do, but it’s better, in my opinion. Keeps me sane.

  12. squid says:

    At Turbine we originally had a jar of pickled cow brains someone found in a specialty market. If you broke the build, you got the brains.

    Oddly enough, the brains dissapeared one day, no one was sure what happened to them. They got replaced with a picture of one of the leads wearing liederhosen.

  13. David Grant says:

    "real devs aren’t afraid of the clown".

    True in many ways.

  14. worddev says:

    The Word team has "the hat", a green hard hat with a "be nice to nerds" sticker on it given to those that break the build. The hat recently gained a body, a fairly shapeless black mannequin with a green head. For a while, everyone that got the hat added to it: there were mouse cords noosed around the neck, a power cord, an attached slinky, a trade show badge, a plastic lizard, etc. The hat itself had a popcorn bucket glued to it. I think most of these have been removed lately, but the mannequin and the hat itself remain. Lugging the thing to the new owner can develop into quite a procession.

  15. Jon says:

    I actually relish getting these items so I can put them in my drawer and stop that retarded practice forver.

  16. JenK says:

    Once upon a time I worked on a website project. It was small enough to do a full, automated build for each checkin that is automatically propped to the test server.

    Each build generated a automated build email.

    So the reward for breaking a build was to have email to the entire team with

    Subject: Build nnnn FAILED

    Build due to: Checkin by DevName

    Errors: Blah blah blah

    Granted, most people had those emails shunted to a folder. The QA Manager (me) and Dev Manager, tho, had our rules setup to build failures showed up in our inbox…

  17. Kelli Zielinski says:

    Should I consider myself really lucky?

    When I managed to tank the build in two branches a couple of weeks ago, all I got was a mail saying, "We backed out your change, this is what you need to do to fix it, don’t do that again."

  18. Tim says:

    Spooky about the bunny – at the last place I worked, we used one of these:

    http://www.snugglechums.co.uk/products/detail.asp?id=131

    Then again, we also had the ‘Cow of Success’, which was passed on (awarded, really) to any developer who had checked in a particularly cool feature. It was just a small fresian cow toy. But it made it all seem worth it.

  19. Paolo Marcucci says:

    We had the "hubcap of shame". A dirty, rusty, heavy and extremely dangerous hubcap we found on the street one day.

  20. A dev group I belonged to had a large #10 sized can of soup (turkey soup perhaps) with a very large feather attached to it for this purpose. The rules said it had to be displayed where it could be seen above the cubicle walls. it never graced my cube I am pleased to say but I did have a close call or two.

  21. Anon says:

    One company I worked for used the "hairbag" for the build break award. A zipper-lock clear food-grade bag filled with one shaggy-headed dev’s haircut trimmings. Kind of disgusting, says I.

  22. Derek says:

    So, Jon, got a whole drawer full, huh?

    Anon, that’s not really humorous. It’s just gross.

  23. Eric says:

    Well, no wonder why you made the mistake. You were working at 4:30 AM!!!

    I am sure you didn’t stop until you cleaned up your mess. I find myself usually doing dumb mistakes that early in the morning… :D

  24. Anon, again says:

    Anon, that’s not really humorous. It’s just gross.

    Well, one man’s whimsical is another man’s… whatever. :)

  25. Les C. says:

    I don’t know if it is a cultural thing (I’m Australian) but if someone in a company tried to publicly embarrass me like this for a genuine mistake, I would (a) complain to senior management and/or (b) resign. (Gentle ribbing from colleagues UNofficially is a different thing)

    Or maybe i’m just sensitive :)

  26. android says:

    We had the Homer award for bone-headedness. A plastic statue of Homer Simpson that said, "Mmmm, donuts" and stuff like that. I never got it, but my boss got it for when he left his passport in his hotel room when he was supposed to be flying back from Bangalore.

  27. Norman Diamond says:

    I enjoyed this posting, really I did, I like it, I’ve earned my share of bug bunnies too, etc. Maybe we need to get some live bug bunnies, and if they reproduce more reliably than average bugs do then we’ll even have enough of them to go around ^u^

    But please, could someone explain why breaking the build is such a big deal?

    There’s no end to the discussion that would properly arise over this question, so I’ll try to shorten it by giving two examples that maybe aren’t major enough but they’re still examples.

    Prefast is a really useful tool. If Prefast error messages are treated as being build-breaking, then builds that formerly were considered unbroken are now considered broken. Excellent. It is far better to have bugs found by breaking builds than by releasing them.

    Sometimes a compilation breaks due to a reason that is too obscure for the author, and the author finds that adding a cast makes it compile. So the build is no longer broken, but lying about the calling convention results in worse disasters later. Mr. Chen kindly blogged about the issue maybe around two years ago. I’ve also been distressed to see the same kind of cast used in order to lie about calling sequences, in a famous book on DEVICE DRIVERS.

    How and why did it come to be more important to avoid breaking builds than to investigate and solve the root causes for the bugs? What is the big deal about a broken build?

  28. Lars says:

    He.

    _Of course_ we have a token of shame (or daring, depending on your view) as well: Gerdie Gorilla. She even has her own badge and keyboard.

    But then we had a major infrastructure upgrade, including changing our compiler from a pre-ISO to a ISO-C++ version. My suggestion for handling the necessary code changes was to create a number of sed scripts to handle the mechanical changes (like adding ‘std::’ qualifiers), which of course meant that for a while the files were even more broken as the sed-scripts evolved to handle all circumstances (but after these teething pains, the scripts saved us a lot of development time).

    Gerdie took up permanent residence in my cube back then, and then stayed there for the next couple of years. Only recently she had moved on to other daring developers, confusing me to no end as I now have to look for other landmarks in order to locate my cube.

  29. clockwise says:

    Or maybe i’m just sensitive :)

    Definitely. I’m Australian and I can assure you that at quite a few different offices people get embarassed regularly for unintentional mistakes. In the military it is practically your duty. Hey, it was unintentional right? So why do you care? Ok obviously you shouldn’t give the person a complex so that they never want to do any work again… hey maybe that’s a better solution.

  30. Cooney says:

    Speaking of bunnies, did it look like this?

    http://store.slavelabor.com/FBTinfo.shtml

  31. Bruce Morgan [MSFT] says:

    Norman Diamond: The reason why build breaks are so terrible is that they kill productivity. The source tree is supposed to be sacred – at any point, I can sync my machine to source control and have a working instance of the product.

    If a dev checks in code that crashes when he/she runs feature X, then that dev and others working with feature X are affected. But if someone checks in code that flat out doesn’t compile and link to produce the set of executables, well, anyone syncing will be dead in the water until the build break is fixed. People depending on other changes to test against or to make their own changes against are now delayed. On a team with dozens of developers, one mistake can turns into major downtime.

    Worse, if the project has things like daily automated builds (and virtually projects of significant size do) then that day’s build will be delayed or skipped. Now people across the team (or even entire company) who were for the next daily build to test new features or verify fixes are now delayed.

    Most build breaks are not of the "the compiler can’t compile my code and a cast will fix it". Someone who checks in code that didn’t compile on their own machine deserves worse than a clown in their office. Most build breaks in my experience are from mismatched header files or two devs making conflicting changes that failed to be caught by the checkin system.

  32. Jonathan says:

    Norman: Also, when making sweeping changes (replacing compiler versions, adding PREfast, changing platform SDK etc.) you make sure you can get a build out of it, and then you checkin the new tool + your fixes in one atomic change. I’ve done this sort of thing in our project (>100 devs) several times already. And if it’s too much for one dev to do (like changing PREfast version, which netted us ~600 new warnings), you stage it so first PREfast warnings are just warnings, let everyone clean up their code, and then make PREfast warnings break the build.

  33. Norman Diamond says:

    Thursday, December 08, 2005 1:31 AM by Bruce Morgan [MSFT]

    > at any point, I can sync my machine to

    > source control and have a working instance

    > of the product.

    I want to say just sync to the latest working version, but if there are too many broken versions in a row then obviously this isn’t a solution.

    For testers, if there are no worthwhile repetitions of yesterday’s tests because the build hasn’t changed, imagine if testers would create some new tests instead. Some bugs that weren’t tested for yesterday or last year are still waiting.

    > Someone who checks in code that didn’t

    > compile on their own machine deserves worse

    > than a clown in their office.

    Agreed.

    Thursday, December 08, 2005 6:40 AM by Jonathan

    > Also, when making sweeping changes

    > (replacing compiler versions, adding

    > PREfast, changing platform SDK etc.)

    Yes, you do that work in parallel on separate machines until you get that working. That’s not the same as a developer breaking a build though. Even in days[*] when a company might be able to afford one whole computer, they scheduled production runs and test runs at different times.

    [* More accurate would be nights, but "even in nights when" doesn’t sound right ^_^]

  34. Apache Gump (http://vmgump.apache.org/gump/public/buildLog.html) does a clean build of all the OSS Java projects, four times a day.

    Your team breaks the build, you get a red bar on your project, everyone that depends on you gets an orange (prereq failed), and some recrimination begins. If you are a root project (xml parser, Ant), then everybody breaks.

    Over time the reliability of your project becomes a public statistic; you can also subscribe to a project’s RSS feed to track its builds.

    Gump3 proposes rollbacks on a failure to track down which particular SVN checkin transaction caused the breakage. This not only enables more accurate email messages, it will enable public logging of developer’s commit success/failure ratio. This will be public, it will be online for all to see -not just your current team, but the rest of the OSS community, maybe even your next employers.

  35. JDM says:

    Where I worked, we had the "Wheel of Remorse". When someone screwed up they were handed a .38 revolver that only had one bullet; they would then spin the chamber and pull the trigger once, while the barrel of the firearm was pointed at their temple.

  36. anon says:

    we have a monkey too

  37. AndyB says:

    We had a "wheel of blame" at the last place, that was populated with a single person’s name..

    Once when the product was first designed, someone thought it’d be a good idea to put an expiry date on all the parameters in the database. It just so happened this person created the rows in the DB, and set the expiry date to several years in the future. Over time, no-one thought twice about them – they were in all subsequently created databases, and no-one thought they were used.. until we received a call from salesmen who were presenting a demo to a major Hong Kong bank… they’d run the demo through in the morning, all was fine, but when they came to present in the afternoon, nothing worked…. yup. several years later, the expiry date (that was entered around lunchtime…) expired!

    When the issue was tracked down, one Andy Staley got the blame for the disaster, and it stuck. I can only remember one of the names on the Wheel – Admiral Staley of HMS Blame, but there were about a dozen entries.

    I thought it was a good thing really, when a junior did something wrong, we could blame Mr Staley regardless which defused any public humiliation to the junior offender, whilst still keeping the concept.

  38. Stars says:

    How/Why is this article rated 4 stars? Explain please.

Comments are closed.