Tuesday, November 27, 2018

Python: Floating Point vs. Decimal

In our Arduino projects (programmed in C/C++), we often use a Raspberry Pi for collecting data, and the Python programming language.

Math with floating point is inherently risky, because a decimal number does not store exactly in binary.

in python,
1.1+2.2 = 3.3000000000000003

not the 3.3 you might expect.

we can force the result we are looking for by importing the Decimal module, and setting the desired precision (2 in this example):

>>>from decimal import *
>>> getcontext().prec = 2
>>> Decimal(1.1)+Decimal(2.2)

Arduino doesn't have a Decimal function, it uses floating point, but you can work around this by using Fixed Point Arithmetic
Related Posts Plugin for WordPress, Blogger...