The contractually obligatory beeper, and the customers who demand them


One of the fun parts of meeting with other developers, either at conferences or on my self-funded book tour, is exchanging war stories. Here's one of the stories I've collected, from somebody describing a former company. As is customary, I've removed identifying information.

One day, the engineering team were instructed that the team was being issued a beeper, and that a member of the engineering team had to be on call at all times. This new requirement was the handiwork of the sales team, who landed a big contract with a customer, but the customer insisted on a support contract which included the ability to talk to a member of the engineering team at any time, day or night, to be used when they encountered an absolutely critical problem that the support team could not resolve to their satisfaction.

The engineering team grumbled but knew this was something they had to accept because the customer placed a huge order and the company needed the money. To secure the engineering staff's agreement, management contributed $10 to a pool each week, and whoever was saddled with the beeper when it went off got the pool money as a consolation prize.

The engineering team drew up a schedule, and responsibility for the beeper rotated among the team members.

A week went by. The beeper didn't go off.

Another week. Still quiet.

Months passed. Still nothing.

It was over a year before the customer asked to talk to a member of the engineering team. The engineer who received the windfall understood that the prize was a team effort, and as I recall, he spent it by buying beer and snacks for everyone.

The engineering team figured that the customer had the engineer on call clause on their list of bullet point must-have features, even though they didn't really plan on using it much at all.

Comments (33)
  1. R. Bemrose says:

    "The engineering team figured that the customer had the engineer on call clause on their list of bullet point must-have features, even though they didn't really plan on using it much at all."

    Or they could have had no major problems with it.  I know I'd feel pretty stupid calling "emergency" tech support over a minor issue.

  2. Skyborne says:

    Having worked next to customer service in the past, I am surprised by the ending of this story.  I would've expected engineers getting $10 to handle multiple calls.

  3. Henrik says:

    For a moment I thought I was on another blog. But then everything turned out to work just as it should. I liked that.

  4. Sven says:

    Yeah, this sounded like a set up for The Daily WTF, but then the twist ending was that everything worked out alright. :)

  5. Mott555 says:

    I spend too much time on The Daily WTF. I was expecting a horror story about a member of the team being woke up at 3:00 AM to spend hours trying to fix something that ends up being a minor issue caused by a stupid client not understanding how to use a computer.

    But then everything was fine.

  6. Ace says:

    I would have bribed the customer to page me. :)

  7. Pierre B. says:

    I find it interesting that I seem to be the only thinking that 10$ per week for the new, additional burden of potentially being called at the wee hours of the mourning is pretty cheap. Would you accept a 10$ per week raise for such new responsibilities in your current job?

    (Of course, I expected more than 1 call per year. None of my jobs ever had that low level of escalation.)

    [I made up the $10 amount. It may have been more, I don't remember. -Raymond]
  8. pcooper says:

    Well, they might not have planned on using it much at all, but I bet that one time, they really needed it, and for times like that they were willing to pay for the service. Much like most support contracts (or most insurance), you expect to never need it, but you're willing to pay for those few times you do.

  9. kog999 says:

    "I find it interesting that I seem to be the only thinking that 10$ per week for the new, additional burden of potentially being called at the wee hours of the mourning is pretty cheap. Would you accept a 10$ per week raise for such new responsibilities in your current job?"

    I really dont think it was presented to the engineers as a choice. i doubt management said hey you guys can keep doing what your doing or for an extra $10 a week you can be on call in this rotation and potentally get a call at any time. I suspsect it was more like hey we need to support this customer so you can either carry this pager or find a new job and since we are a fun company we will chip in $10 a week to a pool.

  10. What's a beeper?  Is that like a pager?

  11. Dave B. says:

    That sounds like the pager at my last job. We had a single client that required pager duty for server hardware, and it never went off. We added a few more clients, and still nothing. It ended up that we didn't get a single page until about the third year of carrying it. Our manager joked that it was the easiest $100 a week we'd ever make.

    I think the only time I ever heard it go off was then I did my monthly handover test page, which turned out to be a good thing, since sometime in year 4 the paging vendor discontinued service in our entire state and didn't tell us about it.

    [I wonder how many people remember to perform periodic test pages… -Raymond]
  12. rs says:

    The threat "If you write bad software, we'll wake you at night…" seems to have had the desired effect.

  13. Gabe says:

    Ha, that's nothing! I know of a client who insists on having an engineer in the same city. This means that to satisfy them you would have to hire an enginner in that city, train them to be proficient in that client's project, and then never be able to have that engineer go anywhere or do anything else for fear that the client will call the one day in the year that the engineer is otherwise busy. Or maybe pay engineers to rotate weeks in a company apartment in that city. The only consolation is that this was a major city. If this was in Hawaii, it would have been ridiculous because the engineer would essentially have to be confined to the island that the client is on (due to the need to take a commercial flight back when trouble strikes).

  14. Nawak says:

    Speaking of your book, I ordered my copy today!

    You need to continue the blatant plugs, I think it was the last one (which pointed to an online excerpt) that made me cross the line

  15. tsrblke says:

    @Gabe,

    There are worse things than being confined to a Hawaian island with little or nothing to do ;).

  16. tsrblke says:

    @Gabe,

    There are worse things than being confined to a Hawaian island with little or nothing to do ;).

  17. kog999 says:

    seems like you could assign the hawaian engineer plenty of work if he had a VPN connection and maby some sort of teleconferencing. you would just need to make it clear to him that if this particular customer calls with a problem he needs to drop everything else.

  18. mikeb says:

    "Our manager joked that it was the easiest $100 a week we'd ever make. … I did my monthly handover test page, which turned out to be a good thing, since sometime in year 4 the paging vendor discontinued service in our entire state and didn't tell us about it."

    Which made your company's year 4 contract the easiest $100 a month the paging company made (along with everyone else they forgot to tell)…

  19. Fowl says:

    @Drak, it's possible that the people who would have used Feature X (several levels down and too the left in the customer) had noticed that the feature didn't work when they got the software, and just worked around it because it was too much effort, if not impossible to get that report filtered through back to you.

  20. Worf says:

    Yeah, I also thought it was The Other Site. But two thinhgs were wrong – the story came out good, and the pager pool.

    Most times, it's foisted on someone as unpaid overtime. Even though I think $10/week might be low (I'd suggest a varying amount depending on frequency – maybe $100 the first week, $50 the second, then $20 the third and $10 afterwards…) it's much better than what most on-call people get.

    And yeah, some companies will pay stupefying amounts for on-call engineers in the same city because the cost of downtime can be huge. That company may pay easily $100k/year for that service, because flying someone in may mean a day lost, which can cost the company $1M, say. Or more – there are places where downtime is measures in thousands of dollars per minute (tens aren't uncommon).

  21. hagenp says:

    there are places where downtime is measures in thousands

    of dollars per minute (tens aren't uncommon).

    Most of the "interesting" EDP applications, in fact.

    Like banking, flight booking, etc.

  22. Gabe says:

    tsrblke: Yes, it would work out quite well for the engineer, but would be a bit of a waste for the poor company that would have to employ him. In theory you could give the engineer plenty of work to do, but being 6 time zones away, he would have to be *very* independent. If you ever did need him to meet face-to-face, he'd have to spend all day traveling each way.

  23. Morten says:

    @hagenp: or even some quite boring SCADA systems used in medical companies. I've seen an example (from afar – I wasn't part of that team): 4 times a year the chemicals for a 100 M$ product line (in total around 20 pounds/year) are finalized. If one particular system is down for too long the entire batch is wasted. That customer paid a good price for redundancy and on-call engineers without blinking… :-)

  24. Drak says:

    Sounds a lot like some of our custiomers, who will only buy a product if it has 'Feature X'. We drop everything to build Feature X, then a few years later someone finds a bug in Feature X which means that it never actually did what the client wanted… So the client obviously never used it…  * sigh *

  25. DWalker says:

    Most small to mid-size businesses are not like this, but one large airline claimed (many years ago, when I worked for them) that downtime in their reservations system would cost them $1 million per minute.  They could reboot their 8 "large" interconnected mainframes in 90 seconds, which was considered fast.   (Large at the time.)

    I was standing in line at the airport once to check in, and a manager came out and announced that their main computer system was rebooting and they would be able to check in passengers again in 90 seconds.  He was right.

  26. Poochner says:

    BTDT.  I had a bank VP (I think that means "wears a pressed shirt" at a bank) tell me "This outage is costing us $XX thousand a second." while I was at a terminal working on the system. "So is the time you're talking to me."  OTOH, once a salescritter was walking some prospects through a datacenter for an existing client and saw a bunch of people standing with arms crossed and worried looks staring at the monitors.  He ushered those guys out and returned a little while later with pizzas, since he realized we would have worked through lunch, dinner and breakfast working on it until it was tip-top again.  I miss the days when sales cared about the tech people.

  27. anonymous coward says:

    Your psychic skills are increasing now. I read this article and now I am suddenly and specifically assigned to help one of our customers :)

  28. good software says:

    Clearly this wasn't at MS. Shell developers would (rightfylly) been beeped to death.

  29. CmraLvr2 says:

    @Maurits:  @Andreas Rejbrand: What's a pager?  Is that like a ring tone on your smart phone?.. j/k

  30. Tsk says:

    > Would you accept a 10$ per week raise for such new responsibilities in your current job?"

    In the real world, usually you get forced to do it for free.

  31. JoeWoodbury says:

    When I was at Novell, our team (client APIs) was given beepers so we could be "more responsive." The irony is that they didn't work where our offices were in the Provo complex unless you stood right by a certain window and even then it was iffy.

Comments are closed.

Skip to main content