First, a caveat: I have nothing but respect for the people who run build and BVT labs. It’s a thankless task that requires obscene amounts of time and energy.
Now on with the story J…
My favorite example of things that you shouldn’t do in a build/BVT lab: Page a developer in on a weekend to look at a BVT failure when the developer’s got an 18 month old son and his wife’s off enjoying a “Mom’s Day Out”.
I was watching Daniel one weekend when he was 18 months old (back in 1993ish) when I got a call from the build lab to come in to look at a BVT failure. In those days, the BVT lab and the build lab were in a single lab – the BVT machines on one side of a partition, and the build machines on the other side. The BVT and build labs were filled with lab shelving – a desk-like bench that held the monitors and a shelf above that held computers. All the machines were connected to power strips (and network hubs) connected to power supplies run through poles that fed into the ceiling. A typical lab setup here in Redmond (it still is). The builds ran all day, so the build team had people in the lab basically 24/7.
I told them that I was watching Daniel, so it would be really inconvenient for me to come in, but they indicated that they couldn’t find anyone else on the team to look at the problem. And what the heck, they’d met Daniel before, they knew he was a good kid, so what the heck, they’d keep an eye on him while I looked at the dead machine.
So Daniel and I came into work to look at the problem. The build guys took Daniel to their side of the room, and started playing with him (as much as you can “play” with an 18 month old) and I started working on the machine that had failed BVTs.
Things went just fine until all of a sudden, I heard an exclamation coming from the build lab side of the wall:
Kyle: “Hey, what the heck?”
Orson: “What just happened? My machine just went dead.”
Kyle: “Wait, the MIPS machine just went dead too”.
I felt that familiar feeling you get when you realized that your kid’s probably just gotten into trouble and came running.
Yup. The build guys had decided that Daniel was safe exploring the area behind the lab shelving – looking at all the cool wires, stuff like that.
They had forgotten about all the really pretty red lights attached to the power strips that were supplying power to the build machines.
You see, Daniel’s a really curious kid. It didn’t take him very long to realize that when you pushed the little red light, it turned off. And if you pushed it on the other side, it went back on. So he turned it into a game and went down the entire rack of build machines flipping the little light on every one of the power strips.
And every build machine quietly turned off, one by one.
The good news is that the build guys took it in stride, the better news was that since we were running on our own Dogfood, while the builds that were running had stopped, the disks weren’t corrupted – we had to run chkdsk on one or two of them, but the builds restarted without a hitch.
But they never did leave Daniel alone in the build lab again.