MF resolution

[darknesspic]

I'm using V280 with Visilogic 9.8.64 build0

I have the simple formula A(ML)x1=B(MF), I try A=2147483647 but the output = 2.147484E+9 why it's not 2.147483647E+9 because in the manual said maximum MF range is 3.402E37

Please advise, thank you.

[Flex727]

I rarely use floating point operands, but my understanding is that it is stored in a 32-bit memory location, just like an ML, but it has to store both the significand and the exponent in that 32-bit location. The situation with MF is that it trades off between range and precision. You cannot get 10 digits of precision in an MF.

[Joe Tauser]

4 hours ago, Flex727 said:

The situation with MF is that it trades off between range and precision.

Unitronics uses single precision floating point-

https://en.wikipedia.org/wiki/Single-precision_floating-point_format

As described in the above link, once you get above 9 digits (as you show in your example) the magical exponent shift lever kicks in and knocks the number of significant digits down to 7 to allow for the exponent value.

Which is why cleverly manipulated long integers are more accurate and precise than floating point in many applications.

Joe T.

[darknesspic]

Thank you for your comments, I understood.