Ask Your Question

rounding error .n() and N()

asked 2013-04-08 04:53:32 -0500

kjhudson gravatar image

updated 2013-04-08 04:54:10 -0500

Why does 1.414 - sqrt(2).n(digits=4) not evaluate to zero?

sage: sqrt(2).n(digits=4)


sage: 1.414 - sqrt(2).n(digits=4)


edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted

answered 2013-04-08 07:01:42 -0500

sqrt(2).n(digits=4) returns a number which agrees with sort(2) to 4 digits of precision, but it need not be obtained by truncating sqrt(2) after 4 digits. I'm guessing that it's obtained by truncating sqrt(2) after the necessary number of binary digits:

sage: a = sqrt(2).n(digits=4)
sage: a.n(digits=30)

sage: b = sqrt(2).n(16)
sage: b.n(digits=30)
sage: a == b
edit flag offensive delete link more

answered 2013-04-08 07:03:04 -0500


This is because the two numbers do not live in the same ground field as you can see

sage: 1.414.parent()
Real Field with 53 bits of precision
sage: sqrt(2).n(digits=4).parent()
Real Field with 17 bits of precision

1.414 is not exactly represented in memory because it is not a diadic number. Two different diadic approximations with different precisions are different.


edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Question Tools


Asked: 2013-04-08 04:53:32 -0500

Seen: 80 times

Last updated: Apr 08 '13