The cries of "Oh no!" emerge from each office as the realization slowly dawns


Today is the (approximate) 15th anniversary of the Bedlam Incident. To commemorate that event, here's a story of another email incident gone horribly awry.

Some time ago, an email message was sent to a large mailing list. It came from somebody in the IT department and said roughly, "This is a mail sent on behalf of Person X to check if your XYZ server has migrated to the new datacenter. Please visit http://blahblah and confirm that your server name is of the form XXYYNNZZ. If not, please Reply to All."

Uh-oh.

The seasoned Microsoft employees (and the new employees who paid attention during new employee orientation) recognized the monster that was about to be unleashed, and the cries of "Oh no!" could be heard emerging from each office as the realization dawned.

And then it started. All the replies from people saying "I'm still on the old datacenter." And then the replies from people saying, "Stop replying!"

What's frustrating is that you can't do anything about the catastrophe that is unfolding. Any attempt to reply to the message telling people to stop replying only makes the problem worse. All you can do is stand back and wait for the fire to burn itself out.

Ten minutes later, Person X sent a message to the mailing list. "Please DO NOT Reply all to this email thread. I am working with the IT department to see if there is another way to get this information."

It took another ten minutes for the messages to finally stop, but that seems to have shut things down. Now it's time for blame and speculation!

We were never told whose brilliant idea it was to try to gather information by sending mail to 7000 people telling them to reply all. One theory was that Person X went to the IT department saying "Hi, I'd like to collect information XYZ from this large list of people. Can you help?" And some new hire in the IT department said, "Sure, I can get that information for you. I'll just send everybody some email!"

After the dust settled, somebody made a tally of the damage.

Number of people on the mailing list: around 7000.
Number of replies: 70.
Of those, number of replies saying "stop replying": 17.

To commemorate this event, a colleague of mine who maintains a popular internal tool pushed out an upgraded version. The new version had a checkbox on the main page:

I have not been migrated to the new datacenter.

Bonus chatter: It so happens that the message was sent at the beginning of the summer, and most of the "I'm still on the old datacenter" replies came from summer interns. Maybe it was a test.

Comments (35)
  1. Brian_EE says:

    We had a very similar incident where I work about 8 years ago, except it was much larger ( >> 10,000 users ). The corporate IT team had to put a block on the subject line to kill it before the servers buckled under the pressure.

    The message was sent to the all-employee Exchange list and was of non-business-related content (something about getting some people to volunteer for a local charity).

    In the end, the employee was reprimanded, and a global email went out about proper (business) use of email and judicious use of the "reply-all" button vs. the "reply" button in Outlook.

  2. Paul says:

    With this email specifically it seems Reply-To or BCC would have prevented most of the problems, but nobody knows how to use BCC :-(

    [Any number of things would have prevented this problem. But the original sender explicitly said, "please Reply to All," which strongly implies that they had no idea what they were starting. -Raymond]
  3. Tom says:

    I haven't been migrated, so I clicked the check box but nothing happend.  How do I get migrated?  /sarcasm

  4. @Tom says:

    Send a mail to "AllInternet.Users@theworld.com".

  5. JDP says:

    Metafilter presents: "Free bananas in the kitchen!!!". http://www.metafilter.com/…/PLEASE-UNSUBSCRIBE-ME-FROM-THIS-LIST

  6. Gabe says:

    Couldn't the original sender have just said "Please reply to me" instead of "Please reply to all"? Or at least the follow-up message could have said something to that effect. Are we just supposed to assume that the originator was really naive/stupid?

  7. Aaron says:

    Are you conflating two different events?

    I was there for "Bedlam DL3".

    For many years, I had the T-Shirt that was made afterwards.  ("I Survived Bedlam DL3" on the front, and the back had fine-print of many of the "Me Too" messages and the more amusing posts).

    However, the way I recall is quite different from your post, and much more in line with the article you linked to.  It started with "Why am I on this mailing list?  Please remove me from it.", not "XYZ server has migrated to the new datacenter.".

    Also, as its mid-October, there's no way this was sent at the beginning of summer.   The summer interns had all left by the time this happened.

    [This is a separate incident from the Bedlam Incident. I linked to the Bedlam Incident for context. The point is that everybody who had gone through Bedlam realized what was about to happen. Hence the article title. (And as I noted, this particular story occurred "some time ago", not "yesterday".) -Raymond]
  8. Random832 says:

    "With this email specifically it seems Reply-To or BCC would have prevented most of the problems, but nobody knows how to use BCC :-("

    Except, it specifically said to reply all. I can only assume that the writer of the email thought there would be some benefit to doing so.

  9. Joshua says:

    Do not ever permit 3 different mailing lists implemented as remailers to be subscribed to each other.

  10. Mason Wheeler says:

    Aaron's right.  The way the article is written makes it look like you're describing the Bedlam incident.  It would be more clear if it started something like this:

    Today is the (approximate) 15th anniversary of the Bedlam Incident.  And while it may be the most infamous case of multiple remailings getting out of control, it's certainly not the only time it's happened, even at Microsoft.

    For example, some time ago an email message was sent to a large mailing list…

  11. FRC says:

    I remember something like this happening when I was working as an a- back in 2005 or so? Some HR person accidentally sent an e-mail to an alias that had every contract on the main campus on it. People started replying, then replying to the replies, with the crazy list of removing me from the list and all the don't reply to all messages, sent to everyone.

    Then… it went crazy. People starting to replying with large images, and then quoting those images and replying with other images. You could watch the server bog down more and more until they finally killed the list and dev/nulled the messages. After that almost everyone group involved got some retraining on the proper use of e-mail.

    Except our group… I had the same "Oh, no!" Reaction (thanks to reading your past posts about it), so I told my office mate, also a contract, not to reply to it. I think after that it went down the chain so instead of making to worse we got to sit and watch it. The net result, are group (which was like 4 contractors) got called out for not making it worse.

  12. Evan says:

    @Random: "Except, it specifically said to reply all. I can only assume that the writer of the email thought there would be some benefit to doing so."

    My guess? Multiple people needed to see the answer. So why not just email them as well, and tell people to reply-all? That way the others would see it too! It makes perfect sense.

    (Disclaimer: It does not make perfect sense.)

  13. No One says:

    "[H]ere's a story of another email incident gone horribly awry."  (Emphasis mine.)  I think that this clearly indicates that this story is not Bedlam DL3 but another email storm story to commemorate Bedlam DL3.

    I can't believe I had to say that?

  14. Aaron says:

    In reply to "No One", who wrote "I can't believe I had to say that?"

    I believe that text was recently added it.   Unless I totally overlooked it, it wasn't present when I wrote my first message and "Mason Wheeler" agreed.

  15. @Mason No, Aaron's not right. It's just that neither of you read and understood the second sentence "To commemorate that event, here's a story of another email incident gone horribly awry"

  16. Mason Wheeler says:

    @Paul, @No One:

    The article has been edited since it was first posted.  Originally, the first paragraph only contained the first sentence.

  17. No One says:

    @Aaron, Mason Wheeler:  I apologize, you are correct.  A cached version does omit that second line.  Damn me for being too snarky today.

  18. Not Al says:

    Maybe they meant reply to Al? Was there a sysadmin named Alfonse?

  19. @No one

    It's Monday. Who isn't being too snarky?

  20. Adam Rosenfield says:

    My university has these so-called spam wars every so often.  Someone will decide they need to email the entire student body, so they will email all of the individual dorm mailing lists (the all-students mailing list is moderated, and rightfully so).

    More often than that they will not use BCC, and if the email is ridiculous enough ("I lost my super-important fraternity pledge pin on a sweatshirt somewhere on campus!"), people will reply-all with snarky comments, and then it spirals out of control.  Some people will reply-all saying not to reply-all etc.  There are internet memes and ASCII art.  Someone will say they forgot everything they know about airports, and someone else will send the entire text of the Wikipedia article on airports.  The Bible and Quran get thrown around as well.  Typical college student trolling.

    The email servers could handle it just fine AFAIK, but it was an interesting phenomenon.

  21. Andrew says:

    I passed the test :) That was a ridiculous thread.

  22. Joshua says:

    sudo rm -rf /

    Just displays an error message these days.

  23. KC says:

    Isn't this how Exchange got the feature to have access rights on distribution lists? My last company had several that only admin staff could send to.

     KC

  24. Mott555 says:

    Perhaps they were trolling, or was just joking and assumed everyone knew about the Bedlam incident and wouldn't *actually* do a reply-all. Sort of like some linux communities where asking for help will get you a few people (either trolling or joking) who suggest running "sudo rm -rf /". It's never resulted in dead servers before, but I know I've suggested someone do something only to have them actually go and do it because they didn't understand I was jokingly referring to some previous event where it was proven to be a horrible idea. So anymore I make my suggestions so over-the-top that no one should take them seriously, i.e. "fixing" a crashed workstation with liberal amounts of diesel fuel and an ignition source.

  25. Drak says:

    @Joshua: Just wondering, how would you test that it actually shows that message? :0

  26. Ted says:

    I knew a fair number of interns on the thread, and it was interesting to watch and see as each of them decided, "okay, enough FTEs have replied that I should actually respond to this." They were all suitably embarrassed afterwards.

  27. Eric says:

    We had a few of these "Bedlams" at a large company I worked for.  I loved watching the chaos unfold.  The best part was when an employee in Europe ripped the "stupid Americans" for using replying all and of course he hit reply all on his email.

  28. @ Drak

    % rm –help

    […]

         –preserve-root   do not remove '/' (default)

    There.

  29. Joshua says:

    @Drak: What do you think chroot() is for?

  30. Evan says:

    @Kwpolska

    It's a good thing all versions of 'rm' have that. Oh wait, only GNU's does. :-)

  31. James says:

    A year ago I trolled my company of several thousand people to try to catch people who inadvertently use reply-to-all.  I set up a mailing list with nobody subscribed to it, and I then sent out email addressed to it but BCC'ing almost everybody in the company.  The email explained that the entire company was subscribed (which wasn't true) and solicited some useless information from everyone.  Anyone who used reply-to-all sent mail to both myself and to the empty mailing list, and I had configured the mailing list to auto-respond with a message explaining that the person had just failed a reply-to-all test.

    (The failure rate was actually kind of low (~1.6%), and a few people got really upset at me, so it probably wasn't worthwhile in retrospect.  Most of my colleagues enjoyed it, though.)

  32. cheong00 says:

    @Evan: Btw, I think the "current" trick is "rm -rf ~/", not "/" anymore.

  33. Drak says:

    @Kwpolska, Joshua: I believe it says it will not work, but have you actually tried if it really does not work?

    My favourite of these 'helpful suggestions' used to be on windows when people were using telnet to ask people to swith to tty4 (alt-f4 ;)

  34. Kevin says:

    @cheong00: Of course, in most sensible shells, *this* still works just fine:

    rm -rf /*

    (You could also type out the '–no-preserve-root' switch, but it's long and ugly.)

  35. cheong00 says:

    @Kevin: Actually this new variant is to play with students that happens to have Unix cluster account (usually without root access), so "rm -rf /" would have no effect anyway even without that protection.

    And ruining the users own account for his/her bad behaviour/temper serves them just right.

Comments are closed.

Skip to main content