My phone just DoS’d my office network


The other day I was working in my office minding my own business when I suddenly lost network connectivity. I couldn't contact any machines other than the ones in my office.

When this happens, I go through some basic troubleshooting steps. Is my neighbor's network okay? How about power-cycling the affected machines? Refreshing the TCP/IP security policies? None of that seemed to work, so I called the network helpdesk, on the assumption that my port was shut down because I somehow missed an e-mail message announcing a mandatory security patch that I had failed to keep up with.

My office has an IP phone. It's hard to call the network helpdesk when your network is down and your office phone is network-based. I called them from my cell phone instead.

After giving the helpdesk person the MAC addresses of all the computers in the office, the technician said, "Nope, none of these is on the list. If your network isn't working, it's not because of a port shutdown. Try plugging one of your computers directly into the wall rather than going through the hub."

Bingo, network connectivity restored. The problem was somewhere inside my office. I thanked the technician and set to figuring out what was wrong with my office.

Now, in order to plug the computer directly into the wall, I had to unplug my IP phone, since the office network topology is wall→phone→hub→other computers. The phone goes straight into the wall on the recommendation of the phone people because they argue that not all hubs know how to intermediate the QoS packets that the phone uses to ensure that call quality is acceptable. When I plugged the phone back into the wall and rebooted the phone, I got the message Please wait while updates are being installed. Once the phone finished booting, I found that the rest of the computers in the office had network connectivity.

Basically, my phone downloaded an update and tried to auto-reboot itself, but got wedged, and it took my office network down with it. My phone just DoS'd my office network.

Historical note: This entry was written over a year ago, and the model of IP phone in question has long since been replaced.

Comments (22)
  1. John says:

    Wow, talk about a let down.  I was really enjoying reading this entry and then I got to the last sentence: "Haha sucker, you just wasted your time reading all of that.  Better luck next time, loser."

  2. Someone You Know says:

    We have a similar topology with our IP phones. The phones’ network interfaces only support 100 Mb/s for some reason, leaving our computers’ Gigabit Ethernet cards twiddling their thumbs.

  3. Nick Mason says:

    This made me think about one of my favourite quotes:

    "I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone."

    — Bjarne Stroustrup

  4. Mike says:

    I find the idea that the phone needs an update to be quite amusing.

    And scary. Quite scary.

  5. Nick Mason says:

    I can see it now…

    "Your phone was recently updated!

    PhoneOS recently downloaded and installed an important update to help protect your calls. This update required an automatic restart of your telephone."

    I hope Raymond didn’t have anything important happening on his office network when his phone decided to reboot itself…

    Disclaimer: This comment is tongue-in-check and not intended to cause genuine offence, I hope the following comments don’t degrade into a let’s-moan-about-windows-update forum.  Although if such a forum does exist, please let me know where it is, I have lots of steam to vent on this matter.  And yes, I know you can switch this facility off, which I’ve done now.  Permanently. :-)

  6. Someone You Know says:

    @Mike

    Scary indeed. The last firmware update for our phones made all the phones’ clocks go to UTC-3 (this time of year we’re normally UTC-4) for no obvious reason. They reverted back to the correct time several hours later.

  7. J says:

    My company’s last line of IP phones had a system-on-chip design where the 10/100 network switch was on chip.  This was the switch that routed traffic from your LAN port to the PC port where presumably all your office equipment was connected.  Pretty common setup on IP phones.

    The problem was that the company that designed these SoCs didn’t have the foresight to allow each component (processor, DSP, network, etc.) to be reset individually.  So when your phone would reboot, your entire office network would go down because there would be no way to leave the network switch up while the processor was reset.

    The people who wrote the phone code took the easy route and just reset the phone whenever its IP connection to the phone server was down, and that would start the connection process all over.  That left your PC port without connectivity for about 10 seconds every minute.

    Needless to say, my desk phone is connected to my office, not the other way around.

    (The happy ending is I fixed the code so the only time the phone resets is when it has a firmware update.)

  8. Austin Spafford says:

    Another IP phone horror story:

    The network for the entire company where I work (~200 people) was mysteriously taken down for a few hours. IT finally tracked it down to an IP phone in a conference room that was plugged into a hub via two ethernet cables, likely thanks to a bored designer.

    Why the IP phones have two ethernet ports in the first place is beyond me (daisy chaining?), but the raw danger of whatever effect we had observed was just mind boggling.

  9. Josh says:

    We’re now being required to update to fancy network phones. I really don’t understand the allure. I like a phone with (gasp!) buttons on it, that I can pick up and not worry about if my local network network is up. Instead, we’re getting phones that require our computers to not only be on, but have software installed on them just so we can *painfully* "dial" the phone through a clunky win app. Innovation! /sigh…

  10. Chris Lineker says:

    They have two ports so you can upgrade to VOIP without needing additional network infrastructure

    But just because you can do it that way, doesn’t necessarily mean you should.

  11. Bryan Donlan says:

    Austin, the effect you observed was a broadcast storm caused by a loop in the switched network topology. Essentially, a broadcast packet exits the switch on port A, re-enters on port B, comes back out on port A again, etc. The result is the entire switched segment is flooded by broadcast packets.

  12. Eric TF Bat says:

    It’s not just IP phones, you know.  I was stymied by the fact that our home phones no longer ring.  I tried an isolation test, removing the caller number display doohickey and the ADSL splitters, plugging them into different ports, unplugging one and leaving the other then vice versa, even spraying the ants that were crawling out of the nearby powerpoint.  No dice.  Then I looked at the phone handsets themselves, and found the cause: my two-year-old son had switched the ringers off, on both phones.  I didn’t even know how to do that!  Verily he is an agent of Chaos…

  13. EriKF says:

    @J:

    Hopefully you don’t have one of those wonderful PoE (power over Ethernet) phones.  When they work, they’re great (I don’t have to find room for yet *another* wall wart on my power bar!), but I have found that they can be really finicky.

    I found out about the wonders of PoE in one office where I had to figure out how to plug two phones into a single Ethernet port using just an unmanaged switch.  Needless to say it was an utter failure and I had to wait a week before one of the network people could come and hook up another network drop.

  14. Falcon says:

    @Austin Spafford:

    I’ve heard of similar things happening in rooms where there’s a double LAN point on the wall and a cable running from one point (for people to plug laptops in at the desk).

    The friend who told me this says that cleaners will often plug the other end of the cable into the empty socket, probably thinking that they’ve accidentally pulled it out, and flood the network.

    He adds that if the end of the cable is sitting near the point, it’s guaranteed that they’ll plug it in!

  15. Josie says:

    My corp also has VOIP phones, and it’s amusing to watch the telecomm guy run around rebooting the phones when something goes haywire. So sad…

    MSFT can identify and kill network ports with an exploitable PC? Wow. My corp can’t even find roughe DHCP servers when someone accidentally plugs one in.

  16. Worf says:

    Harumpth. The real problem is the two phone ports-to-switch scenario, but two jacks in the room that run to a managed switch is easily preventable via STP.

    Rogue DHCP servers is easy as well – sniff the network, do a DHCP request, and see the DHCP OFFERs. Remove the legit one(s), and get the MAC of the rogue. Use your managed switch to tell you which port it’s on, administratively bring it down and check the cabling to find the offending network jack…

    I don’t see our office switching to IP Phones anytime soon. I do too much network testing…

  17. Gabe says:

    Based on how many IP phone posts we’ve seen these past few months, it’s no surprise that they were replaced with a different model!

  18. Mahesh Velaga says:

    Thats funny .. how could they possibly be thinking of routing the traffic of all the network through the IP Phone! which can be unreliable .. and a good case being demonstrated as example.

  19. J says:

    @EriKF:

    Actually I’d love to have a PoE switch.  When you develop IP phones and need about 10 of them at your desk, it sucks having only individual power bricks.  Although I’ve managed to collect a few rather nicely designed power strips.

    That being said, our phones had a hardware bug once where they misreported their PoE power requirements to PoE switches.  Oops.  It’s fun debugging random resets.

    Also there was this specific PoE switch that would hum when you plugged our phones into certain ports.  Odd.

    Also a colleague fried a hardware-based debugger by plugging a powered PoE cable into its ethernet port.  Even though I know my laptop is standards compliant and can handle that, I always double check that I’m not plugging in a powered LAN cable whenever I need to plug in the wire just out of paranoia.

    And you’re right about them being finicky.  I don’t know what the quality of switches is now, but half a decade ago there were some cheapies that just flat out lied about how much power they could deliver.

  20. Miral says:

    This wasn’t as exciting as it first sounded.  I came in thinking that the phone launched some kind of packet storm that was blocking stuff, but in the end it was just a loss of power issue.

    The real problem is your topography; you shouldn’t have the hub connected to the phone.  It’s unsurprising that if the phone locked up or lost power then it would stop relaying packets; the same thing happens to hubs.

  21. Neil says:

    I have to admit I don’t understand why hubs would care about QoS packets (they don’t even care what protocol the packet is, do they?); I thought that sort of thing was taken care of at a higher level.

  22. Falcon says:

    @Neil: QoS is part of Ethernet – its use allows the phones to request delay and jitter control at the lowest layers. If a frame containing realtime audio data is excessively delayed during Ethernet transmission, the battle may already be lost, as there’s little or nothing that higher levels can do about it.

    Strictly speaking, *hubs* probably wouldn’t care about QoS, however I’d bet that Raymond and the commenters in this thread are actually talking about *switches*. What are the chances of getting a true hub (repeater) these days?

Comments are closed.