A customer reported a difference in behavior when executing some code in the design time and in a COM dll.
You can run this code to see it: (use the server created from http://blogs.msdn.com/calvin_hsia/archive/2004/06/18/159550.aspx)
SET DECIMALS TO 15
*ox.mydocmd(“set decimals to 15”)
?ox.myeval(“p2”,num) && just evaluate the parameter and return it
The code creates a real number, prints it, then just passes it to the COM server, which just returns it so it’s printed again.
The output is:
It appears that some precision has been lost.
Internally, VFP keeps track of how much we know about a real number. This can be easily shown by running this code:
SET DECIMALS TO 15
Indeed, we can demonstrate that the precision of mathematical calculations is propagated from the precision of the operands.
PROCEDURE ShowStuff(n as Number)
1.2 2.4 2.40 2.400
1.20 2.40 2.400 2.4000
1.200 2.400 2.4000 2.40000
1.2000 2.4000 2.40000 2.400000
1.20000 2.40000 2.400000 2.4000000
Try this code to see how VFP uses the DECIMALS setting in calculations. It sums 1/3 j times and compares that with j/3
PROCEDURE tryit(nDec as Integer)
SET DECIMALS TO (nDec)
FOR j = 46190 TO 46195 STEP 1
FOR i = 1 TO num
The propagation of precision reminds me of something closely related: probable error propagation.
When I was in physics class in the mid 70’s, I learned about probable error and how to manipulate it. The probable error (PE) is often expressed as a Plus or Minus (±) after a measurement.
For example, if you measure a square object, you might come up with a measurement of 12 ± 1.2 units (± 10 %). The perimeter can be calculated by multiplying by 4, resulting in 48. The probable error is sqrt(4*(1.2)^2) = 2.4
For more information about probable error and measurement uncertainty, and how they propagate through calculation, see Error Analysis