Have you performed simple arithmetic operations like 0.1 + 0.2? You might have gotten something strange: 0.1 + 0.2 = 0.30000000000000004.

  • germanatlas@lemmy.blahaj.zone
    link
    fedilink
    arrow-up
    0
    ·
    3 days ago

    Basically it’s a rounding error.

    It’s like 2/3, when you write it out, you usually write 0.666, 0.667 or something like that, because the decimal system simply does not allow us to write 2/3 fully as a floating point number. You could write out more sixes, but ultimately you have to cut off a rest and live with the rounding inaccuracy.

    The same thing happens with numbers in binary floating point representation, but with different numbers like 0.4 (0.0110011001100 and so on). They also have to be cut off at some points depending on the precision type, causing their “translation” to decimal to be very slightly off from a true result.