The preservation of numerical precision

  • Article

A customer reported a difference in behavior when executing some code in the design time and in a COM dll.

You can run this code to see it: (use the server created from https://blogs.msdn.com/calvin_hsia/archive/2004/06/18/159550.aspx)

?SET("Decimals")

SET DECIMALS TO 15

num=12.45678901234567890

?num

ox=CREATEOBJECT("t1.c1")

*ox.mydocmd("set decimals to 15")

?ox.myeval("p2",num) && just evaluate the parameter and return it

The code creates a real number, prints it, then just passes it to the COM server, which just returns it so it’s printed again.

The output is:

            12.45678901234567890

            12.4567890123

It appears that some precision has been lost.

Internally, VFP keeps track of how much we know about a real number. This can be easily shown by running this code:

SET DECIMALS TO 15

?1.2

?1.20

?1.2000

Indeed, we can demonstrate that the precision of mathematical calculations is propagated from the precision of the operands.

ShowStuff(1.2)

ShowStuff(1.20)

ShowStuff(1.200)

ShowStuff(1.2000)

ShowStuff(1.20000)

PROCEDURE ShowStuff(n as Number)

      ?n, n*2,n*2.0,n*2.00

RETURN

         1.2 2.4 2.40 2.400

         1.20 2.40 2.400 2.4000

         1.200 2.400 2.4000 2.40000

         1.2000 2.4000 2.40000 2.400000

         1.20000 2.40000 2.400000 2.4000000

Try this code to see how VFP uses the DECIMALS setting in calculations. It sums 1/3 j times and compares that with j/3

tryit(2)

tryit(12)

PROCEDURE tryit(nDec as Integer)

      SET DECIMALS TO (nDec)

      ?"Decimals",SET("Decimals")

      x=3

      FOR j = 46190 TO 46195 STEP 1

            s=0

            num=3 *j

            FOR i = 1 TO num

                  s=s+1/x

            ENDFOR

            ?j,num/3,s,s=num/3

      ENDFOR

RETURN

The propagation of precision reminds me of something closely related: probable error propagation.

When I was in physics class in the mid 70’s, I learned about probable error and how to manipulate it. The probable error (PE) is often expressed as a Plus or Minus (±) after a measurement.

For example, if you measure a square object, you might come up with a measurement of 12 ± 1.2 units (± 10 %). The perimeter can be calculated by multiplying by 4, resulting in 48. The probable error is sqrt(4*(1.2)^2) = 2.4

For more information about probable error and measurement uncertainty, and how they propagate through calculation, see Error Analysis