Ask Your Question

Numerical approximation error

asked 2021-03-02 15:44:22 +0200

danielfn gravatar image

I'm apparently having a very specific rounding/floating point error when using the numerical approximation function .n(). I reproduced it in the example below. The "1" would be a function whose numerical value amounts to 1.


print(int(-1/6*1.n() + 49/6))
print(int(-1/6 + 49/6))



The error does not happen if we specify a precision for approximation, such as: Input:

print(int(-1/6*1.n(1)+ 49/6))
print(int(-1/6 + 49/6))



Can this be corrected?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2021-03-03 06:02:54 +0200

Once you use a numerical approximation, you probably shouldn't do further arithmetic and then another approximation (such as applying int) and expect reliable results. The number you're getting is very close to 8, just a little smaller, so when int truncates it, the result is 7.

sage: 8-(49/6-(1/6*1.n()))

Perhaps it would be better to use round instead of int: int is a built in Python function, while round is better suited to work with Sage number types. In general, though, it's probably best not to use .n() until the end of the calculation.

edit flag offensive delete link more


One more comment: when you do -1/6 * 1.n(), the use of 1.n() forces all remaining computations to be done using approximate computer arithmetic, as opposed to -1/6 * 1 which Sage recognizes as being a fraction with exact value -1/6. That's why the rounding issues occur.

John Palmieri gravatar imageJohn Palmieri ( 2021-03-03 22:52:49 +0200 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools

1 follower


Asked: 2021-03-02 15:44:22 +0200

Seen: 69 times

Last updated: Mar 03