Maxiumum length of a numeric value (OpenInsight 32-Bit) [Revelation On-Line Wiki]

OpenInsight 32-Bit, Jeff Warvel, //www.sprezzatura.com]The Sprezzatura Group[/url], [email protected]

Join The Works program to have access to the most current content, and to be able to ask questions and get answers from Revelation staff and the Revelation community

At 29 NOV 2007 09:38:24AM Jeff Warvel wrote:

The problem is that numeric values with length ] 17 are not treated as a numeric.

Ex :

In OI 8.0

NUM('12345678901234567') evaluates to false
and therefore
'12345678901234567' + 2.4 results in a crash to debugger

but

NUM('1234567890') evaluates to true
and therefore
'1234567890' + 2.4 is ok

In AREV 2.12

NUM('12345678901234567') evaluates to true
and therefore
12345678901234567 + 2.4 work just fine

There is a small exception to this rule. If using a real number rather than integer the length limit is 16 (the decimal point adds one)

I've seen many of the post regarding precision and mathmetic comparisons, but I could not find any topics on this issue.

Why did this work in Arev, but doesn't work in OI (i've tested 4.x, 7.1, 8.0) ??

here is some sample code to demostrate this issue. run in OI and Arev to see the results.

NUM=0; NOTNUM=0

OUT.LIST="

I=1

MAXLEN=32

*CHECK.NUM=1.2 ;* check real numbers

*CHECK.NUM=.23 ;* check small real numbers

CHECK.NUM=1 ;* check integers

LOOP

stop when max len is reached.

WHILE ( LEN(CHECK.NUM) < MAXLEN) AND (I < 32)

report the len of CHECK.NUM and it's numeric status.

results stored to OUT.LIST
I += 1
OUT.LIST := "len= ": FMT(LEN(CHECK.NUM),"L#5")
IF NUM(CHECK.NUM) THEN
  NUM += 1
  OUT.LIST := "NUMERIC      ": CHECK.NUM: @FM
END ELSE
  NOTNUM += 1
  OUT.LIST := "NOT NUMERIC  ": CHECK.NUM: @FM
END
CHECK.NUM=CHECK.NUM : I-1,1     ;* append last digit of I.  does this make it a string?
REPEAT

OPEN "SYSLISTS" TO L.CHAN THEN
WRITE OUT.LIST TO L.CHAN, "JEFF.NUMERIC" ELSE NULL
END

At 29 NOV 2007 10:04AM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:

See bug 899

The Sprezzatura Group

World leaders in all things RevSoft

At 29 NOV 2007 12:01PM Jeff Warvel wrote:

Thanks Sprezz… I'll check the "Bug Tracker" next time before posting.

I'm surprised this was not discovered earlier in OI 4.x or 7.x

At 29 NOV 2007 06:28PM [email protected] wrote:

Hi to you and say hi to Jill! This is indeed weird that it hasn't been noticed before now. It is of course all to do with the limitations of 32 bit floating point numbers. Not an easy fix but a necessary one.

[email protected]

The Sprezzatura Group Web Site

World Leaders in all things RevSoft

View this thread on the Works forum...