Le Chatelier’s principle in action: Announcements


As I noted some time ago, one of the most important lessons I learned from Systemantics is Le Chatelier's Principle for complex systems which states, "Every complex system resists its proper functioning."

At Microsoft, there are processes galore. For example, when a server is taken down for planned maintenance, you can expect a series of messages, perhaps following this pattern:

  • Server X will be unavailable from time A to time B. (Sent a few days before the planned outage.)
  • Server X will be unavailable today from time A to time B.
  • Server X is going down now. It should be back at time B.
  • Server X is back.

The problem with this is that you often receive notifications for servers you have no interest in, indeed whose mere existence you were previously entirely unaware of. The notification messages don't include instructions on how to stop receiving the notifications, or if they do, the instructions are incorrect (no points for guessing how I discovered this), so people set up email rules to auto-delete the notifications instead of trying to figure out how to unsubscribe.

The result is that we now have a system in place which generates large quantities of email which are automatically deleted.

Comments (32)
  1. kog999 says:

    This sounds like a case of higher up people "complaining" the IT department doesn’t communicate with its users and the users complaining they dont give a crap about what IT does.

  2. Joe Dietz says:

    I’ve long found it a deficiency that you could not mark internal emails as spam as a not-so-stubble way of communicating back to its author that you didn’t find their email useful in the least.  Alas I just setup auto delete rules for most mail prefixed with ‘MB-‘

  3. I don’t really see this as a problem.  I would much rather be warned about a system that I may be using, then not be warned at all.  Let my computer filter out what messages I care about or not.  While some sort of centralized "system downtime notification service" that people can subscribe and unsubscribe themselves to (and with which they can see what systems they have been auto-subscribed to receive notifications for) might be useful, it really wouldn’t save much time at all.

    (besides, looking at the "To:" line in the email typically gives a good hint at how one was qualified to receive the notificaiton!)

  4. Boris says:

    That reminds me of a (fictional, I think) horror story. It goes something like this:

    A man receives a note in his mailbox with just one line on it "one month from now". Well, he dismisses it as a prank, but a week letter he receives another note: "three weeks from now", then "two weeks from now". The man starts to worry something horrible will happen two weeks later. When only one week is left, the notes become daily. "In three days". "In two days". "Tomorrow". Finally the day arrives and nothing happens. The next day the man receives another note: "yesterday"…

  5. Voo says:

    @Boris: Is it only me or does that story miss a punchline or something?

  6. J says:

    Can those students just not delete the email?

    Or does the university seriously place such tiny caps on downloads from their own email servers?

  7. a random passerby says:

    @Voo: The punchline was… oh, nevermind. You’ll find out a month from now.

  8. mfigura says:

    At my company, there is a similar process for servers that have crashed and need to be reset.

    When the email server crashed, we knew it was back up when we started receiving notifications that IT found the server was down, was about to reset it, and that it’s now back up.

  9. Boris says:

    @Voo, The last line is the punchline. Maybe it wasn’t as funny as I thought.

  10. Voo says:

    @Boris: Sorry over here it’s kinda early and I probably wouldn’t pass a Turing test right now.. Or maybe I just don’t understand humour anymore, who knows :(

  11. -dan says:

    At work I use Thunderbird as an email client, which allows you to filter mail based on words present in the subject.  

    Since messages like those are always standardize, it’s not hard to send them straight to the trash.

    I’m sure you use one of Microsoft’s email clients, it’s been years since I’ve used outlook express so I have no idea but I’m guessing it has a similar feature.

  12. mikeb says:

    @NT:  at least those Foo Updates mailing lists  ensured a certain amount of dogfooding of Exchange and Outlook, so it’s not all bad.

  13. Bob says:

    @Boris

    It was funny, and I want to do that to someone, now.   :)

  14. kog999 says:

    @NT,

    Sadly, sometimes the point isn’t to keep people informed but to keep people especially management from complaining there not informed. nothing gets them to shut up faster then sending them 100 emails a day 95 of which they dont care about.

  15. As long as they send all server outage announcements to Bedlam DL 3, your personal productivity for the day is guaranteed to skyrocket since you won’t be able to use email for the forseeable future.

  16. Anonymous Coward says:

    I would really appreciate it if my company’s IT sent these out. All I get is the occasional "email was broken this morning due to an update, we apologise for the inconvenience"

    Yeah, because an update isn’t something you can easily warn about before, is it now?

  17. JohnCKirk says:

    When I send out messages like this, I try to mitigate it by putting in an expiry date – that way if someone’s off on holiday for a week, they can tell at a glance that they don’t need to read this email when they get back.

  18. curiouscat says:

    @JohnCKirk

    how would the message look like so that they can tell at a glance? would the expiry date be in the title?

  19. Cheong says:

    Perheps Microsoft should, instead of sending emails, setup a web page (Sharepoint?) listing all proposed maintenance schedules. So someone who want to access the server but discover it goes down can check.

    [This lets you discover why a server is down, but it doesn’t let you plan ahead for the outage. (You’d have to check the Web page every morning. And then people would ask for a way to be notified via email when the Web page changes, and then you’re back where you started.) -Raymond]
  20. Gabe says:

    The solution, of course, is to have a way to subscribe to notifications only for the servers that one cares about.

    [That’s how it works, except that (1) various actions will auto-subscribe you, (2) it is not clear from many of the messages that you can unsubscribe; (3) the instructions for unsubscribing often don’t work. -Raymond]
  21. Bob says:

    I just worked it out.

    The solution is to stop using email, and start using Google Wave.

    No, seriously, email has its uses, but forcing all communication to have the benefits as well as the drawbacks of email is dumb.

    So, instead of "you have to check the web page every morning", you have "well, every morning you’re already opening your communication program to retrieve messages, so why not have that subscribe to notifications about servers you use as well as downloading your spam and your jokes and the occasional business email?"

    I bet this suggestion will be very popular with Microsoft system administrators.  ;)

  22. The servers where I work are similar, notifying everyone who is logged in if any attached file system runs out of space. They then repeat this notification every minute or so until it is resolved. With resources being scattered across many servers, this tends to mean I get dialog boxes appearing frequently for drives on which I have no data, no users and only the most casual of interests.

    It’s possible to disable these notifications – but only by disabling *all* server notifications for *all* servers, which is rather bad as a sysadmin. One of these days it might be a system I actually care about which chokes…

    The estates/buildings maintenance people are notorious for sending out frequent email notifications to the entire staff+student body – with the entire recipient list embedded, bloating a simple "Road X will be closed from 10am to 11am tomorrow to move a crane" to several megabytes. Irritating for staff with 300 Mb quotas, downright destructive for students with 40 Mb!

  23. fred says:

    @Arto

    Hey, me too.  How did I get on this list?  Could someone please take me off?

  24. NT says:

    Regarding the original story, I was always baffled during my time at Microsoft that more people did not understand this principle: habitually sending information to people who aren’t interested in it is actually a way to ensure that *fewer* people, not more, get your message.

    I had many arguments with people (usually PMs or GPMs) who decided that the best way of ensuring that everyone who needs to know about Foo gets all the latest Foo Updates was to put their entire group on a “Foo Updates” mailing list — not as individuals who could remove themselves, but as a group.  All it really did was ensure their whole team knew how to use mail filters.

  25. Cheong says:

    [This lets you discover why a server is down, but it doesn’t let you plan ahead for the outage. (You’d have to check the Web page every morning. And then people would ask for a way to be notified via email when the Web page changes, and then you’re back where you started.) -Raymond]

    Sure, just also put the "subscribe server maintenance schedule email" link there and let people subscribe/unsubscribe per server schedules. So they’d have to ask where to check/subscribe/unsubscribe at most one time. They can even produce consolidated "Maintenance schedule for servers you care" email notification email for users each week for maintenance 2 – 4 weeks ahead.

  26. JohnCKirk says:

    @curiouscat: In Outlook 2003/2007, expired messages are greyed out and crossed out. (This only happens after the expiry date/time.) You can still view the messages if you want to, or just delete them unread. This is also useful for messages like "I’m out of the office tomorrow" or "We’re testing the fire alarm at 11am".

  27. Wang-Lo says:

    @Bob — I want to do it too, but my time management skills are so poor I’ll probably forget to send the "Yesterday…" message until two days later.

    -Wang-Lo.

  28. mikeb says:

    @JohnCKirk: thanks for the tip!

  29. Cooney says:

    [This lets you discover why a server is down, but it doesn’t let you plan ahead for the outage. (You’d have to check the Web page every morning. And then people would ask for a way to be notified via email when the Web page changes, and then you’re back where you started.) -Raymond]

    No you aren’t – you subscribe to the servers you care about. If you own a piece of appX, then you subscribe to appX_servers and probably also common_servers and you get updates tailored to your needs, while people who work on these things simply enter maint. info for servers as they are messed with.

    Or you can filter the messages into a bucket and look at it when server_x dies to see if it’s getting messed with deliberately. At least then you know wheter to bug people about it.

    [That’s exactly the problem! I’m subscribed to appX_servers even though I care only about appX_server5. -Raymond]
  30. Cooney says:

    [That’s exactly the problem! I’m subscribed to appX_servers even though I care only about appX_server5. -Raymond]

    Then why’d you subscribe to the whole group? Just sub to server5 and be done with it. The truth of it is that if you care about server5, you probably care about server 1-8, because they’re all the same anyway. If they were different, they’d be in a different group.

    [Wait, you didn’t say that there was an appX_server5 subscription. You only said that there was appX_servers. (And I only care about server 5 because the other servers hold databases I don’t care about.) -Raymond]
  31. SD says:

    "Improvements made to

    Rogers Email services

    January 15, 2002, may

    cause problems when

    sending mail."

    ahh the improvements…

  32. Cooney says:

    [Wait, you didn’t say that there was an appX_server5 subscription. You only said that there was appX_servers. (And I only care about server 5 because the other servers hold databases I don’t care about.) -Raymond]

    I made no representations about how the namespace was organized, I only included appX_servers as an example. Personally, I wouldn’t stick the DB in a group like that – groups are usually best as homogenous things, so appX_servers only has servers for appX – the DB is a different beast.

    So, if it isn’t abundantly clear, this requires a somewhat sensible method to your madness when you set up the server groups, and you can pick groups or single machines (not recommended – DB guys might migrate your DB to another machine and not tell you – your app still works, though).

    [And I believe that’s how the system works. The catch is that some groups autosubscribe you to notifications when you use their services, without telling you how to unsubscribe. -Raymond]

Comments are closed.