# Numerical approximation error

I'm apparently having a very specific rounding/floating point error when using the numerical approximation function .n(). I reproduced it in the example below. The "1" would be a function whose numerical value amounts to 1.

Input:

print(int(-1/6*1.n() + 49/6))
print(int(-1/6 + 49/6))


Output:

7
8


The error does not happen if we specify a precision for approximation, such as: Input:

print(int(-1/6*1.n(1)+ 49/6))
print(int(-1/6 + 49/6))


Output:

8
8


Can this be corrected?

edit retag close merge delete

Sort by » oldest newest most voted Once you use a numerical approximation, you probably shouldn't do further arithmetic and then another approximation (such as applying int) and expect reliable results. The number you're getting is very close to 8, just a little smaller, so when int truncates it, the result is 7.

sage: 8-(49/6-(1/6*1.n()))
8.88178419700125e-16


Perhaps it would be better to use round instead of int: int is a built in Python function, while round is better suited to work with Sage number types. In general, though, it's probably best not to use .n() until the end of the calculation.

more

One more comment: when you do -1/6 * 1.n(), the use of 1.n() forces all remaining computations to be done using approximate computer arithmetic, as opposed to -1/6 * 1 which Sage recognizes as being a fraction with exact value -1/6. That's why the rounding issues occur.