Why does the Windows calculator generate tiny errors when calculating the square root of a perfect square?

Try this: Fire up the Windows calculator and ask it to compute √4 − 2. The answer is not zero. It's −1.068281969439142e−19. "I thought the Windows calulator used an arbitrary precision arithmetic library. How come the answer isn't exactly zero?"

If you read closely in that article, you'll see that it says that the Windows calculator uses an arbitrary precision arithmetic library for basic operations but not for advanced operations, and square-root is called out explicitly as an advanced operation. Specifically, it uses an arbitrary precision arithmetic library for rational operations: addition, subtraction, multiplication, division, and raising to a positive integer power. Other operations use an extended precision library that produces 32 digits of precision.

Even if the value you are taking the square root of happens to be a perfect square, the calculator doesn't realize that. It doesn't try to calculate an integer square root first, and then fall back to the floating point version if the integer square root fails to produce an exact result. It just goes for the square root via the extended precision library.

What you're seeing, therefore, is the usual vagaries of floating point arithmetic, compounded by the fact that the extended precision library does not have a custom algorithm dedicated to high-precision square roots. Instead, the extended precision library calculates the square root via the following identity:

error  if x < 0
x = if x = 0
exp(½ ln x) if x > 0

The value of 1.99999999999999999989317180305609 is good to 20 digits of precision. Not as good as the 32 digits that a custom-written square root algorithm would have produced, but not too shabby.

(An error of one part in 10²⁰ lets you express the distance from earth to Alpha Centauri with an error of less than a millimeter.)

Comments (35)
  1. Sven2 says:

    If the library knows it will produce a result up to n digits of precision, couldn’t it round to e.g. n-1 digits after that? It would bring calculations that happen to end at a nice round number down to that number, while affecting those that don’t somewhere down on the 20th digit only.

    1. Here, n=32, so rounding to 31 significant digits would result in 1.999999999999999999893171803056. No improvement. Noting also, of course, that the rounding is done in binary, not decimal.

  2. French Guy says:

    I got -8.1648465955514287168521180122928e-39 in scientific mode and the same result as you in standard mode. I also calculated exp(½ ln 4)-2 and came up with the same result as √4-2, so my guess is that scientific mode uses higher precision.

  3. IanBoyd says:

    I guess the answer boils down to: *”Because the natural log of four is a transcendental number”*

  4. Antonio Rodríguez says:

    Arbitrary numeric precision isn’t just hard to achieve. Sometimes it is impossible. Transcendental numbers (like sqrt(2) or ln(4)) have an infinite number of decimal places, and can’t be represented numerically. The only way to deal with them is symbolically. And that, of course, is far out of the reach of a “desktop accessory” calculator (where simplicity of operation outweighs almost all other concerns).

    1. bmm6o says:

      Picking nits: sqrt(2) is irrational but not transcendental.

    2. Mark Y says:

      I’m pretty sure sqrt(2) is NOT transcendental, but merely irrational. In fact, it’s about as tame as irrational numbers get.

      1. French Guy says:

        Square roots of rational numbers are indeed not transcendental (because they are roots of polynoms with rational coefficients), but logarithms (and exponentials) of rational numbers are. And even non-transcendental irrational numbers cannot be represented numerically with a finite number of digits, so they have to be handled symbolically for exact calculations.

    3. Persona Non Grata says:

      Hey Raymond, I have a sibling response that is “awaiting moderation”. Do you know why my posts keep getting flagged?

      1. First-time commenters are auto-moderated and I have been too busy to dig through the “awaiting moderation” pile lately. (Note also that I will not approve comments with bogus email addresses.)

        1. Persona Non Grata says:


  5. Dave says:

    *Calculating (in title)

  6. Joshua says:

    Wow! What’s funny is had calc used the actual FPU instructions this problem would not so easily be found because it could do exp2(1/2 log2 x) as that is what the FPU provides.

    1. Steinar H. Gunderson says:

      No, the FPU provides an actual square root instruction (FSQRT). And for the value 4.0, it produces exactly 2.0.

      1. Pseudonym says:

        Of course you meant sqrtss or sqrtsd, because nobody in their right mind would use fsqrt these days.

    2. Given Windows has had a calculator since Windows 1.0, I wouldn’t be surprised if the innards of the current incarnation date back that far. FPUs were an option at least up until 386, I think? Although a lot of software required them to run; not sure if that included Windows or not.

      1. Myria says:

        The 486 was the last Intel CPU that had a model without an FPU built-in. The “SX” model did not have an FPU, but the “DX” and “DX2” models did. Starting with the Pentium, all Intel and clone CPUs thereof have an FPU.

      2. If you read the linked article, it’s explained that the internals of calc have been completely rewritten:
        > The innards of Calc – the arithmetic engine – was completely thrown away and rewritten from scratch. The standard IEEE floating point library was replaced with an arbitrary-precision arithmetic library.

        Besides, I would find really hard to believe that Windows 1 Calc used arbitrary precision arithmetic…

  7. Paul Potts says:

    Hmmm, I thought your statement about the error in representing the distance from Earth to Alpha Centauri might be nonsense, but I did the math myself. It seems to be pretty accurate. Food for thought about just how accurate floating point numbers really can be, at least if they aren’t ruined by algorithms that wind up losing a lot of precision.

  8. Remy Lebeau says:

    This is even worse in Windows 7’s calculator. √4 − 2 produces -8.1648465955514287168521180122928e-39 !

    1. 12BitSlab says:

      I would expect for there to be 2 answers; 0 and -4. But, then again, I attended public schools so…

    2. French Guy says:

      It is actually much better (1e-39 is much smaller than 1e-19). And, like I said earlier, it depends on whether you’re in standard or scientific mode.

  9. Myria says:

    This could be solved by an algorithm that detects when a rational root has a rational answer, but it may not be worthwhile.

    Now if only the Windows 10 calculator could do everything the Windows 7 calculator could, like accept key sequences like “2y3” via copy and paste, I’d be happier.

  10. I wonder why the exp(½ ln x) method was used when it’s straightforward to calculate the square-root directly. The so-called ‘long division’ square-root algorithm, that most of us were probably taught at school, is particularly simple and elegant when implemented in binary. It’s guaranteed to give a precise result when finding the square-root of a perfect square, and is probably how the FPU’s fsqrt instruction does it.

    1. Pseudonym says:

      Nope, modern FPUs use the Goldschmidt algorithm to compute square roots.

      To compute sqrt(x), start with y(0) as an approximation to 1/sqrt(b) (usually calculated using a lookup table, but see also Walsh’s method). Set b(0) = x, and iterate:

      b(i) = b(i-1) y(i-1)^2
      y(i) = (3 – b(i)) / 2

      Then sqrt(x) = b(0) * y(0) * y(1) * y(2) * …

  11. Azarien says:

    I hate the Windows 10 “modern” calculator so much, that I started writing my own.

    1. DWalker says:

      I spent $5.99 and 10 minutes of my time at a store, and I have a perfectly good calculator sitting on my desk.

      It only shows results to 12 digits though, but that’s good enough for most things.

      1. Vince says:

        Does it have copy/paste? ;)

      2. Azarien says:

        Cool. But does it have copy&paste with the PC?

    2. Ray Koopa says:

      You can install the old calculator. Just google for it =]

      1. ender says:

        I copied win32calc.exe from LTSB install.

  12. Ray Koopa says:

    Sigh… guess we’ll never get close to Alpha Centauri.

  13. Random832 says:

    The answer’s actually -8.1648465955514287168521180122928e-39 for me – quite a lot better, but still not exact. Not sure why the difference

    > compounded by the fact that the extended precision library does not have a custom algorithm dedicated to high-precision square roots.

    Here’s the real issue. It’s confusing to people because ‘Everyone Knows’ that “the square root function” (i.e. the regular double precision square root function, as used on many different apps such as Excel, the *old* calculator, etc) handles perfect squares (to include not only squares of integers but also any such number divided by any power of 4) just fine up to what can be represented in a double, and they expect an extended precision library to do the same or better.

    1. French Guy says:

      The difference is which mode your calculator is in, like I said at the beginning of comments. Standard gives you the same answer as Raymond, while Scientific gives you this more precise answer.

  14. MacIn173 says:

    The most funny part is that actually calculates 4^0.5 as 2, but 2-2 != 0.

    In scientific mode: 4 Y 0.5 Enter gives you 2 (not some “almost 2”); – 2 Enter gives you 1,0605907030850721689734498566293e-38
    So, while showing correct “2” on screen, it keeps result internally not as an integer.

Comments are closed.

Skip to main content