I no longer have to remind the payroll department to gear up for annual raises


September 15 is the date on which annual raises take effect. This means that on the 15th, the payroll servers are swamped with people eyeballing their new paycheck breakdown to see what has changed.

And every year, on September 15th, the servers would become overloaded and people would be unable to connect. Instead they'd get a message like this:

Due to unexpectedly high demand, the payroll servers are unable to service your request. Please try again later.

We apologize for the inconvenience.

Unexpectedly high demand? This happens every year on exactly the same date! How could it be unexpected?

For a few years, I set myself a recurring task for August 15th that said, "Remind the payroll department to increase capacity for September 15th."

I don't know if my reminders helped any, but it appears that the payroll department finally figured out on their own the elusive pattern to their high demand days and added additional capacity accordingly. I no longer have to remind them.

Comments (23)
  1. Gabe says:

    Is the increase in capacity from the previous year insufficient for the current year, or do they remove the excess capacity once the rush is over? Otherwise it’s hard to imagine that they would need to be reminded every year.

  2. alegr says:

    In the Soviet Union, we used to joke:

    "Unexpectedly, the fall came, and all the people were battling frantically to save the crops".

  3. Josh says:

    Gabe:  I think the assumption here is that the capacity increase is a temporary measure.  In other words, the capacity needed around Sep 15th is probably way more than what is typically needed the rest of the year.  So, to avoid wasting resources a temporary increase is made.

  4. Leo Davidson says:

    This reminds me of sitting in a phone queue waiting for support on a product/service and being told by the automated system that it is "experiencing unusually high demand…" just like it was yesterday and the day before, and every other time I’ve called it.

  5. Ilia says:

    Every year, about the same time, snow falls in Moscow, Russia. Every year we are told that the snow was unexpected. For everybody.

  6. Ken Hagan says:

    Wouldn’t it have been cheaper to change the error text?

    "Hey, folks, it’s September 15th, you’re all hitting this server at once and so there’s no chance of you getting through. Try again next week. It’s not like you won’t get paid until you’ve checked the new figures."

  7. JamesNT says:

    Payroll is the only other department that has a decent chance of being as horrific as HR.  And these morons handle our money.  

    Gag me.

    JamesNT

  8. gerleim says:

    Hey, this must be a hint "unexpectedly high demand" is the answer for the raise. Any amount of raise is an "unexpectedly high demand".

  9. James says:

    It’s the same here in Scotland, Ilia: every year the roads are blocked with snow, which apparently comes as a nasty shock to the council every single year.

    It would be nice to think there’s a big pool of servers with flexible roles, so in mid-September Finance gets some extra intranet servers, then they shift to be download servers to handle the next Service Pack release or whatever. They probably just got a rack of extra servers which sit taking up space the other 364 days, though.

    Still, it could be worse. I can imagine Raymond either getting a reply saying "OK, to even out the peak load we’ve delayed your rise to October 15 instead" – or sending an e-mail to a few dozen users telling them their raise has been delayed to comply with Raymond’s request to manage the peak better. I imagine that would get Raymond’s reminder cancelled instantly, without a penny spent on extra capacity!

  10. Evan says:

    I don’t know if this occurs when they release other large groups of students, but almost every semester in undergrad, the honors college crashed the class registration servers by about 12:02am on the first day we could register.

    The recovery times decreased with each crash; the first time it was basically down until the next morning, while by senior year it was only down for a few minutes. I think that it even survived us one year.

  11. DriverDude says:

    Raymond, did Payroll ever respond to your reminders? Or did they mutter under their breath, "damn engineer, what does he know"

    "It would be nice to think there’s a big pool of servers with flexible roles…"

    Hyper-V, anyone?

    Or they could spread out everyone’s raises through out the year, or give raises on your anniversary.

  12. Ens says:

    Shouldn’t there be another similar hit at the next pay period, after all bonuses have cleared out?  Although I suppose the take-home from a bonus might be more interesting to people than their new sustained take-home pay.

  13. Ens says:

    Sorry to double-post, but I am amused that some suggested solutions to this are to stagger pay-raise dates.  This is not some insurmountable technical problem that is severely negatively impacting employees.  If they can’t raise capacity — which they can — then the next logical thing to do is nothing at all.

    I know, I know; all suggestions are in good fun.  So in that spirit, let me add the suggestion of a one-off email once per year overnight to each employee detailing the same things payroll can detail, using some fixed dedicated resources over a fixed time.  Microsoft has lots of employees, but not so many that this is impractical.

  14. alegr says:

    “Hey, this must be a hint “unexpectedly high demand” is the answer for the raise. Any amount of raise is an “unexpectedly high demand”.

    This one is a winner.

    Anyway, that begs a question: what kind of crappy server cannot withstand 50 thousands queries per day? Is it 32 MB NT4 system, running on 33 MHz 80486?

    Oh, I guess they share the hardware with blogs.msdn.com.

    [I vaguely recall that blogs.msdn.com is slow because it spends most of its time repelling spam attacks. -Raymond]
  15. Tom says:

    The urban legend is that the Microsoft payroll servers run on OS/2 Warp.

  16. Igor Levicki says:

    Just do it on December 31 at midnight instead and nobody will be around to login and check hence no demand. One server will be enough :)

  17. DeCheung says:

    Why wait until 12/31? I checked at 12:00AM on 9/15 and it loaded just fine. Staying up until midnight isn’t that hard.

  18. Wolf Logan says:

    @Tom:

    I once saw a technician servicing a credit union ATM in the café. As it booted, I saw the splash screen: OS/2 Warp. I didn’t know whether to laugh or cry.

  19. bcthanks says:

    I once saw a technician servicing a credit union ATM in the café. As it booted, I saw the splash screen: OS/2 Warp. I didn’t know whether to laugh or cry.

    Would you rather see Windows NT on the ATM?

  20. Worf says:

    @Wolf: Actually, many ATMs (prior to 2000) ran OS/2, with good reason. It was rock-solid. Most ATM failures were due to trivial things (ran out of money, receipt paper, jammed, etc), rather than the OS or ATM software itself crashing. Most ATMs at the time date back to OS/2 1.x – running so long that their screens got burned in. (This is quite an achievement, given that OS/2 could run in 32-bit mode with 16-bit real mode drivers working alongside 32-bit protected mode drivers just happily – talk about backwards compatibility).

    Then the year 2000 issue became big, and OS/2 1.x and 2.x weren’t Y2K ready. Given a lot of these machines were already, they were ripped out and replaced with new machines, often running Windows (9x, NT, 2000, XP…). Most banks also used OS/2 for their terminals for the same reason – they were rock solid. Of course, by the time Y2K came about, OS/2 was old news, and Windows was everywhere, so they started building ATMs out of Windows, which leads to the many, many, many pictures of ATM blue-screens ( http://images.google.ca/images?hl=en&q=atm+bsod&btnG=Search+Images&gbv=1 )

    OS/2 worked just fine for what the banks needed – a dependable operating system (dating back to 1987!), that got put out to pasture and replaced. It worked, it beat DOS and Windows (3.x) in stability, so the banks used it for its reliability. Machines have been walled up and "lost" while quietly performing their duties and forgotten (Novell servers, IBM mainframes, Sun servers). A few bits of asset recovery done by asking "where’s this cable go?".

  21. Wolf Logan says:

    @bcthanks, Worf:

    It wasn’t so much the OS/2 part that made me wince (though it should have, I’ve done plenty of OS/2 programming in my day — yes, I’m old); it was the "Warp" part. The OS/2 kernel was quite good, but Warp was (to my mind) a waste. I assume the ATM network was originally written using OS/2 back inna day (probably connected to AS/400s somewhere), and the Warp upgrade was due to some support contract somewhere. I’m just imagining the poor sods writing GUIs for OS/2 Warp, though — this wasn’t an old ATM, and its software is recent vintage.

  22. David Moisan says:

    I saw an ATM in my downtown mall a few years ago as it booted.  I can’t quite believe what I saw, but it looked like a Mac Finder screen (pre OSX)!  Sad Mac, and sad you, bereft of your card and your balance…

  23. SuperKoko says:

    From Worf:

    "This is quite an achievement, given that OS/2 could run in 32-bit mode with 16-bit real mode drivers working alongside 32-bit protected mode drivers just happily – talk about backwards compatibility"

    Are you sure? I thought OS/2 1.xx was 16 bits.

    From wikipedia:

    "Until release 2.0 in April 1992, OS/2 ran in 16-bit protected mode and therefore could not benefit from the Intel 80386’s much simpler 32-bit flat memory model and virtual 8086 mode features."

Comments are closed.

Skip to main content