The details of the major incident were not clearly articulated, but whatever it is, it’s already over


When a server is taken offline, be it a planned unplanned outage or an unplanned unplanned outage or something else, the operations team send out a series of messages alerting customers to the issue.

Some time ago, I received a notification that went like this:

From: Adam Smith
Subject: Nosebleed Service : Major Incident Notification - Initial
Date: mm/dd/yyyy 1:16AM

Major Incident Notification

dfdsfsd

Affected Users

fdfsdfsdf

Start: mm/dd/yyyy 12:00AM Pacific Standard Time
mm/dd/yyyy 8:00AM UTC
End: No ETA at this time.

Incident Duration: 1 hour 15 minutes

Impact

fsdfdsfsdf

Continued Notifications

fdsfsdf

Information & Support

  • Other Support: Please send questions or feedback to

Thank you,

Adam Smith
IT Major Incident Management

Well that clears things up.

Curiously, the message includes an incident duration but doesn't have an ETA. Thankfully, the message was sent one minute after the incident was over, so by the time I got it, everything was back to normal.

Comments (15)
  1. David Crowell says:

    fdsfsdf

  2. Mark (The Other Mark) says:

    If I had gotten that email, with no more context than appears in this post, I would have read it as "The incident has lasted 1 hour and 15 minutes so far, and we don't know how much longer it will last".

    Further, since it's an initial incident going out over an hour after the Start Time, and it's going at at just after 1 AM, they are probably still at the phase of waking up the escalation points, etc, and getting them to all join the bridge and argue… I mean, discuss what went wrong. That 12 AM Start Time (… Nice, round number) is a guess.

    I'd expect an actual, real update to follow shortly.

  3. Matt says:

    I've seen this kind of report quite often when there are systems in place which constantly badger anyone working on an outage.  You can see the well meant intentions behind them, but when it gets to the third or fourth, "please respond to acknowledge you are still working on the issue" message, someone with their head deep in trying to bring a server back up is going to respond with whatever gets the annoyance to go away quickest. dsasddfs usually being the thing that does the trick.

    (Incidentally, we ditched one of the popular alerting systems a while back due to this combined with its extremely unreliable handling of acknowledgement messages; rather than one person working on the actual problem, it ended up with two or three people running round sending, "yes we're working on it, the notification system's just gone crazy" e-mails and text messages as the system happily trundled on towards the world-is-ending levels of its escalation list.)

  4. Nathan_works says:

    In a totally unrelated emergency outage — I didn't see the 2015 "completely arbitrary NCAA tourney picks" post this year ? :(

  5. Mike Dunn says:

    Come on, fdsfsdf!

  6. Roman says:

    Major Incident Notification: Cat walked over the keyboard.

  7. @Roman: Actually, Windows Event Viewer is full of these, all of them tagged as critical error. So, is Reliability Monitor. My personal favorite (not) is "Windows Explorer has stopped working", especially when Windows Explorer is working as usual and nothing is wrong.

  8. @Fleet Command: You sure it's not the desktop instance that's crashing?

  9. SeanBa says:

    @Mark: I'm pretty sure your interpretation is correct. I found a set of major incident notifications and the duration went up every hour as they sent out status updates. The resolution notification had the final duration. The system my group uses just has a start time listed while the incident is active. The resolution notice has start time and end time and you have to do the math for duration yourself. The notices are also color coded. So if you get a red or yellow one you know the issue is ongoing. If you get a green one the issue is resolved.

  10. James says:

    My interpretation is that someone was trying to test the notification system (especially since it was about an incident that perhaps never even existed) but possibly didn't realize that it was live and would spam the whole company.

  11. Klimax says:

    @MNGoldenEagle:

    Also it could be the other instance. Its windows would appear as unresponsive and after some time they would start to react once again. (Have very recent observation…) It'(s classified as crash.

  12. Boris says:

    But surely this calls for a support incident regarding the company-wide, uninformative major incident, in order to ensure that it will never be sent again?

  13. T. West says:

    Somehow I'm reminded of the numerous instances that I've scalded my tongue tasting some food that turned out to be fresh off the stove and had the response to my shriek of pain be "Careful, that food is still hot!".

  14. Derlin says:

    I remember getting a letter in the mail from my company's business class internet provider informing us that our static IPs would be changing.  Based on when we received notice, we had less than three days to prepare our systems.  Since we ran email in house at the time, that translated to a potential for email to go missing for the next week or two as our changes propagated.  I called our sales person with the provider to complain that more time would have been appropriate, and he said he was glad we had a few days notice, since some other customers had received their formal notice AFTER their static IPs had changed.

Comments are closed.

Skip to main content