Then again, it might not be overclocking after all


While it's true that there's an awful lot of overclocking out there, it's also true that not everything that looks like overclocking actually is.

Last Thanksgiving, I helped one of my relatives upgrade their computer by scavenging parts from another unused computer (installing more memory and replacing a broken CD drive). When I took the front panel off the machine, I was greeted with a wall of dust. A little wrangling with a vacuum cleaner was called for before I got around to yanking the broken CD drive and installing the replacement.

When we go through the failure reports that people submit, we find a lot of single-bit errors, where the correct value and the actual value differ in only one bit position. If these problems are systematic, then that would be the sign of some sort of software problem, but the ones we saw were one-time events, isolated single-bit errors. We had no proof, but we suspected flakey memory, possibly the result of an overheated machine.

People often stick their computers in out-of-the-way locations, against walls, on a carpeted floor, in a closet. While that's very convenient for home decor, the computer itself suffers because the flow of air through the computer has been impaired. The accumulation of dust impedes the cooling effect further. So make sure the vents on your computer are clean and unobstructed, or you too may find yourself on the short end of a memory glitch caused by overheating.

Jeremy Kelly from the Exchange Server team was part of the team that investigated a crash that looked just like overclocking, except it wasn't. The program crashed on a mov eax, 0x20 instruction, an instruction that merely loads a constant into a register. It doesn't access memory; it's not a privileged instruction; there's no reason why the instruction could fail. Yet it did.

This looked like overclocking, but the problem was consistently reproducible (atypical of overclocking), and besides, companies that pay tens of thousands of dollars for an Exchange server system aren't going to skimp on a few hundred by overclocking it. The Exchange server team were fortunate enough to be able to capture a live debug session, which permitted them to investigate the problem both on the user-mode side and on the kernel-mode side, and that revealed the true cause: The Exchange server had been infected with a rootkit.

The rootkit was lying to the user-mode debugger about what code was executing. It told the debugger that the instruction was the harmless one, when in fact the rootkit was doing something else. Looking at the problem from the kernel-mode side allowed our investigators to identify the true cause of the problem: A crashing bug in the rootkit itself.

Comments (22)
  1. Nawak says:

    Did you inform the rootkit’s vendor about this bug?

    More and more people are using it, it is very important that the bug gets patched!

    ;-)

  2. andy says:

    Or Microsoft should add a compatibility patch to Windows/Exchange for this rootkit? :P :P :P

    (Sorry, it’s Friday and the weather is very nice here today!)

  3. rootbeer says:

    Well till last year December I did not even know what a rootkit was. Granted all operating systems are first-class rootkits it is quite amazing (and terrifying under some circumstances – in my case both) what one can do with virtualization, rootkits and the likes. It was indeed – in my few encounters with the borg, difficult to distinguish between hardware issues, dodgy software, operator issues, etc, etc. I would have to say its more about estimating the contribution of each in disaster situations. I have never had a failed hard disk in my life before – and I used to carry around desktop pc’s on flights. December I lost 4 of them. I have analyzed and subsequently decided that the one SATA was definitely my fault and the other was likely also caused by "User Apprehension". The third is a mystery and the fourth one, well that was not me. I am down to 2 drives now and looking forward to space greater than 40GBs in the near future!

  4. I was just reading Raymond’s latest post, partially relating to computer heat (and how the symptoms it…

  5. kbiel says:

    Since rootkits are becoming more popular, and not just with hackers, I have started using Mark Russinovich’s RootKitRevealer as one of my standard debugging tasks when trying to diagnose a problem with my friends’ computers.

    http://www.sysinternals.com/Utilities/RootkitRevealer.html

  6. Puckdropper says:

    In one class I took, we were told once that sometimes nothing’s wrong with the computer but a cosmic ray just hits it and inverts a bit and crashes…  (Not sure I totally believe it, but it’s just like smoke and dark suckers.)

    I will have to definately start watching for root kits on machines… Spyware and Malware are all too common, and getting users to scan for it is difficult at best.  It’s only a matter of time before they start installing them…

  7. KB says:

    Any hints on which rootkit it was, and how exactly it got installed on a corporate mail server?

  8. oldnewthing says:

    KB: Why ask me? Why not ask Jeremy?

  9. Anonymous Coward says:

    "In one class I took, we were told once that sometimes nothing’s wrong with the computer but a cosmic ray just hits it and inverts a bit and crashes…"

    Believe it:

    http://www.filibeto.org/~aduritz/ecache-sram-data-parity-err.html

    Worked with the affected machines once, Friday/Monday reports included forecasted solar activity (no joke).

  10. Cosmic rays can indeed be a problem.  It’s part of the reason that ECC memory was developed.

    Radiations becomes a really big concern if you make systems for the upper atmosphere or outer space.  

  11. Mike says:

    I read about the exchange server rootkit from another MS blog, the guy goes into much more detail. But I can’t find it now.

  12. Jerry Pisk says:

    That another MS blog, a guy going into much more detail, is the article linked from inside Raymond’s post. D’oh

  13. microbe says:

    Microsoft should patch the system to allow the virus to run, see http://linux.slashdot.org/article.pl?sid=06/04/18/2046203.

  14. 8 says:

    I can’t find the origional page, but i read there was once upon a time a class on programming in C, and one of the students wrote this:

    x = 7;

    x = 7;

    When the teacher asked "Why would you do this?" The student replied "Just to be sure".

  15. Tan Wei Leong says:

    this is an unbelievably scary post.  one presumes that the exchange server as a production machine had various operational precautions re: what was loaded etc. onto it, as opposed to a user’s desktop – and yet a rootkit got there.  

    are there any details as to what rootkit it was and how it got on that machine?  

  16. Anonymous says:

    Instructions that do not access memroy can still fail if Data Execution Protection is enabled and the page being exeucted does not have PAGE_EXECUTE permission.   This can happen if some application or dll is trying to patch the code and forgets to set PAGE_EXECUTE while temporarily setting PAGE_WRITE on the page in order to patch it.

  17. Maurits says:

    Tsk, tsk.  The teacher obviously didn’t teach the "optimize for the most common case" principle.

    /* x will usually not be 7 */

    if (x != 7)

    {

    /* */ x = 7; /* ah, now it’s 7 */

    } else

    {

    /* */ x = 7; /* just in case */

    }

  18. asdf says:

    Puckdropper it is true:

    http://en.wikipedia.org/wiki/Soft_error

    http://www.tezzaron.com/about/papers/soft_errors_1_1_secure.pdf

    one source of error I *still* see coming up is the result of bad capacitors in the power supply:

    http://en.wikipedia.org/wiki/Capacitor_Plague

    usually on crappy OEM computers. The best part about it is that there usually is a sticker on the case and power supply warning of voiding some warranty (which are trivial to defeat if you’re patient).

  19. Jeremy’s post states that the rootkit in question is "Hacker Defender". From http://searchwindowssecurity.techtarget.com/columnItem/0,294698,sid45_gci1112754,00.html it appears that "Hacker Defender" and "Rootkit Revealer" have had an arms race of such in the past. No indication of who won in the end though!

  20. 8 says:

    Sometimes failure doesn’t come from overclocking, not from dust and not from a rootkit.

    I’ve experienced very erratic behaviour from WindowBlinds. I tried to fight it (generically, not specifically), but i quickly saw it would give too much bloat and it just wasn’t worth all the trouble. I’ll dismiss any bugreports showing windowblinds dlls.

  21. ArC says:

    WRT dust, I have a dust filter built into the front of my PC’s case, and then I just taped bits of 3M’s "Filtrete" furnace filter over the remaining vents of the case.  I clean the first and swap out the others regularly and I’ve noticed that the inside of my PC is a lot less dusty than my previous PCs.

  22. John Sab says:

    I’ve been reading here and there and I can’t understand

    Will the new Windows Vista somehow stop Rootkits from spreading?

    For example by dissalowing manipulation of FindFirst or EnumRegistryKeys or something

    more sophisticated?

Comments are closed.