Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Numerical approximation error

I'm apparently having a very specific rounding/floating point error when using the numerical approximation function .n(). I reproduced it in the example below. The "1" would be a function whose numerical value amounts to 1.

Input:

print(int(-1/6*1.n() + 49/6))
print(int(-1/6 + 49/6))

Output:

7
8

The error does not happen if we specify a precision for approximation, such as: Input:

print(int(-1/6*1.n(1)+ 49/6))
print(int(-1/6 + 49/6))

Output:

8
8

Can this be corrected?