Skip to content

Precision-correct normalization factor may exceed representation range #115

@eyalroz

Description

@eyalroz

We "normalize" our floating-point values to get their power-of-10 to 1, so that we are left with the base-10 mantissa as a number. This works fine. However, if we also require a precision value above 0, we multiply the normalization factor to also account for the precision, so that instead of, say, 1.234, for precision 2 we would get 123.4 , and can return the integral part of that value.

Now, if the original number is close to the edge of the representation range ( ~10^308) - the extra factor for the precision can bring it past the range, resulting in infinity, and messing up the rest of the calculation.

Unfortunately, there's no unit test coverage for this situation; so, we should add one and address it.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingresolved-on-developA changeset fixing this issue has been commiutted to the development branchtestingOnly relevant to the testing programs rather than the main code itself

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions