Floating Point Conversion Problem (OpenInsight 32-Bit)
At 17 JUN 2004 02:23:39AM Donald Bakke wrote:
I have casually perused the previous discussions regarding floating point conversion issues and Basic+ but I'm skeptical if this falls into that category. Consider the following:
Function Test(Void)
A=96667 / 100
B=966.67
C=A - B
Return C
The result returned is -1.13686837721616E-13 and not 0 as expected. If I Iconv/MD2 variables A and B first and then Oconv/MD2 the result then it works.
BTW, this problem occurs with OI16 and OI32 but not AREV and not in Universe
.
dbakke@srpcs.com
At 17 JUN 2004 04:49AM Steve Smith wrote:
You do a division which effectively gives a 32 bit float (scientific notation) as a result.
Floats are stored as abscissa and mantissa, I suspect.
The 32 bit float format is actually quite imprecise after the tenth significant figure. Blame Intel.
So given that the problem can be solved by iconv's - which effectively impose a "safer" type on the number, and constrain the decimal places, you've really answered your own question.
You should get 10 SF of accuracy, and you achieve this in your example, despite casting the data into a float / scientific notation.
OCONV (or field and substring) is your friend.
Steve
One equals two for sufficiently large values of one.
At 17 JUN 2004 09:36PM Pat McNerthney wrote:
Steve,
In this case it appears to be a bug in OI, at least in my opinion. The fact that both AREV and Universe convert this floating point value to 0 is a strong indication to me that OI is in error.
Pat
At 17 JUN 2004 11:06PM Steve Smith wrote:
There's always a way around it.
The slide rule comes to mind.