Concatenation Speed (OpenInsight 32-Bit) [Revelation On-Line Wiki]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
{{tag>category:"OpenInsight 32-Bit" author:"Don Muskopf" author:"[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]" author:"Sean FitzSimons" author:"dsig@sigafoos.org" author:"Colin Rule" author:"support@sprezzatura.com" author:"Andrew McAuley" author:"dsigafoos" author:"Barry Stevens" author:"Mike Ruane" author:"[url=http://www.srpcs.com]SRP[/url]'s Kevin Fournier"}}

[[https://www.revelation.com/the-works|Join The Works program to have access to the most current content, and to be able to ask questions and get answers from Revelation staff and the Revelation community]]


==== Concatenation Speed (OpenInsight 32-Bit) ====
=== At 31 AUG 2006 02:14:33PM Don Muskopf wrote: ===
 
 
<QUOTE>
If you test run this simple program,



For i=1 to NumIterations

LIST := RECORD:@RM

Next i

* each RECORD is a 10 element array about 75 bytes in length.



the number of iterations increases linearly and so does the size of LIST, but the time it takes to complete the loop increases geometrically. For example on my computer (a stand alone P4 3.4Ghz with 2GB RAM):



10,000 iterations takes .062 sec.

25,000 iterations takes .437 sec.

50,000 iterations takes 1.844 sec.

100,000 iterations takes 7.391 sec.

250,000 iterations takes 46.453 sec.

and so on ..



Can anyone explain why concatenation operations take geometrically longer and longer?



Thanks.
</QUOTE>
----

=== At 31 AUG 2006 09:31PM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:  ===

<QUOTE>Don,



I'd say it was memory allocations.



As you're essentially building an 18MB string bit by bit the system is having to find another block of memory each time your request exceeds the currently allocated block - each memory alloc means copying and discarding and possibly even defragging RAM ... all very expensive operations. 



(I've also seen this in many other products - try building large JavaScript strings by concatenation in IE and you'll see what I mean)



The best way around it is to try and presize the string you are going to need before you start processing and move along it setting characters in the string as you go - that way you get far less allocations (maybe just the one if your presize is accurate) so this should speed you up...



e.g.





pre.code {

   background-color: #E5E5E5;

   border: 1px solid #000000;

   width: 640px;

   padding: 5px;

   font-family: courier, verdana, arial, serif;

   margin: 0px 10px auto;

}


<code>

   
   record=str( "X", 75 )
   xcount=250000
   lRec  =len( record ) + 1
   
   call dostime( sTime )

   list =str( \00\, lRec * xCount )
   pos  =1
   for x=1 to xCount
      listpos, lRec=@rm : record
      pos += lRec
   next
   call dostime( etime )
   
   timeTaken=( etime - stime )


</code>




[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]



</QUOTE>

----

=== At 01 SEP 2006 08:35AM Sean FitzSimons wrote:  ===

<QUOTE>Wow!



On my machine:

Concatenating with Don's code took 159 seconds;

Concatenating with Sprezz's code took 0.8 seconds.



As Grandpa Joe says:  Yippee!!!



Sean

</QUOTE>

----

=== At 01 SEP 2006 09:24AM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:  ===

<QUOTE>Cute isn't it :) Add that to the edittable speed improvements and your systems should fly! Remember you can always OVER allocate if you don't know the exact size and just 1, \00\ when the operation completes.



[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]



</QUOTE>

----

=== At 01 SEP 2006 10:05AM Don Muskopf wrote:  ===

<QUOTE>Holy cow! On my computer for 250,000 records, the time improved from 46.45 seconds to 0.266 seconds. This is great!!  No long waits any more.



Thanks Sprezz.

</QUOTE>

----

=== At 01 SEP 2006 10:27AM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:  ===

<QUOTE>Now if you combine that with recent edittable changes you'll find that you can load editables with 1/4 million rows in less than a second instead of the minutes it was taking a few releases back.



Lightspeed anyone ;)



[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]

</QUOTE>

----

=== At 01 SEP 2006 10:33AM Don Muskopf wrote:  ===

<QUOTE>It's actually better than going from minutes to seconds, because of the geometric increase in the time it takes. I had made the mistake of testing a million rows and, after an HOUR!, I broke out of it!

</QUOTE>

----

=== At 01 SEP 2006 11:18AM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:  ===

<QUOTE>Out of interest could you retry with the million?



[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]

</QUOTE>

----

=== At 01 SEP 2006 12:30PM Don Muskopf wrote:  ===

<QUOTE>Sorry, it's not one hour. I had forgotten that that particular test was done within my application and additional concatenation code was included in it.



For 1 million records, using the same code as I posted, it takes 12 minutes 22 seconds. Using your code with my records takes a mere 1.1 seconds. :)

</QUOTE>

----

=== At 01 SEP 2006 12:43PM dsig@sigafoos.org wrote:  ===

<QUOTE>so how many minutes for the user to scroll through a million rows to find the data they want?

</QUOTE>

----

=== At 01 SEP 2006 02:13PM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:  ===

<QUOTE>Hmmm let's see how we can shave a 100 msecs off that :)



[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]

</QUOTE>

----

=== At 01 SEP 2006 02:14PM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:  ===

<QUOTE>That's what the Search function is for ;)



[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]

</QUOTE>

----

=== At 05 SEP 2006 04:17AM Colin Rule wrote:  ===

<QUOTE>Just wondering, when using Table or Table to build up a table for inserting into an array, does OI use concatenation to create such a variable.

If so, is this susceptible to the same speed issues for building the string, and is it worth building up the string for table insert in the same manner as suggested.



Chances are there are other factors in building an table array such as file reads which may have a large impact on the time than the concatenation, but there may be improvements by doing such a generic function in this way.



Colin

</QUOTE>

----

=== At 05 SEP 2006 10:58AM dsig@sigafoos.org wrote:  ===

<QUOTE>Unless there has been some undocumented changes in the recent updates the angle brakets are the worst way to do LARGE concats.  If you are building small strings cool .. but large strings should always use square brackets.



Write a little test routine of varying size from 1K to 100meg and get timings along the way .. it will show the differences

</QUOTE>

----

=== At 10 SEP 2006 08:37AM support@sprezzatura.com wrote:  ===

<QUOTE>Just so you know, the compiler swaps out angle brackets with insert, extract and replace opcodes.







support@sprezzatura.com

The Sprezzatura Group Web Site

[i]World Leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]

</QUOTE>

----

=== At 13 SEP 2006 06:54AM Colin Rule wrote:  ===

<QUOTE>Interesting.  I am curious here.



Are we then saying that the INSERT statement does not to the same as TABLE within the compiler, ie 's does more to do the same function and is therefore slower?



If I were to want to build up a table to display, of say 20,000 rows of 10 columns, I would use a loop and increment my row counter building up the Table variable and then populate this into the ARRAY property of the control.  Normally I would use the angle brackets as it makes much better looking code, and saves space, but if this is taking a speed hit over using INSERT, then this would be worth changing.



My reasons for raising this here is that the topic indicates to me that using text replacement in square brackets would seem to be much faster, and I will do some tests to be sure, but is this really the best method of building up a string to populate an array.



Givin that is is 2006 and not 1976 when one might expect to have to write assembler style code to get speed, why is it that there is not an efficient means of creating fast variable structures to feed tables, or am I missing something here, and have been programming in Pick like systems for 30 years and everyone else knows some way of getting things to fly that I dont.



My problem is that now that the ARRAY update in OI 7.0 has resolved a signoficant speed problem (woohoo - fantastic), that there is still a possible bottleneck on the variable build side which could (or should) be within the OS compiler and not something that the developer should need to consider when coding.



Please advise.

Colin

</QUOTE>

----

=== At 13 SEP 2006 07:33AM Andrew McAuley wrote:  ===

<QUOTE>It's horses for courses and  has always been slower than .  is optimised for [i]extraction[i] so if you say  it will look for two field marks to know it is at the third field. This means a sequential scan of the string.



So when you say 







pre.code {

    background-color: #E5E5E5;

    border: 1px solid #000000;

    width: 450px;

    padding: 5px;

    font-family: courier, verdana, arial, serif;

    margin: 0px 10px auto;

}


<code>

   For X=1 To 5
     Array=Rnd(10)
   Next

</code>




What you are actually saying is put this in the first value, now starting at the beginning scan for the second value and replace it, now starting at the beginning scan for the third value and replace it, now starting at the beginning etc etc.



When you use  you are simply concatenating to a string so the engine just needs to grab the string length, move to there and add data. 



The engine doesn't know your intentions in creating a for/next loop so it can't use concatentation when you use . The only way that this could be optimised would be for the  operator to set an internal value for this array pointing to the end of the last operation and the next time a  was done check if it is one more than the last and if so concatentate. Whilst this is possible it probably won't happen anytime soon.



The preassignment of memory trick is a small price to pay for a language that stops us having to malloc ourselves. 



[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]

</QUOTE>

----

=== At 13 SEP 2006 07:52AM Andrew McAuley wrote:  ===

<QUOTE>Sorry about the italics...



R65 has some hints on optimisation and standards too. Download KBase.Zip.

</QUOTE>

----

=== At 13 SEP 2006 01:30PM dsigafoos wrote:  ===

<QUOTE>i malloc myself way too often as it is 

</QUOTE>

----

=== At 14 SEP 2006 03:43AM Colin Rule wrote:  ===

<QUOTE>Rubbish.

For a OS environment that bases its ideologies around multi-values to have to use string manipulation to improve speed and not the core fundamentals of the environment is bizarre.

</QUOTE>

----

=== At 14 SEP 2006 04:41AM Colin Rule wrote:  ===

<QUOTE>It certainly does make a difference... a big one.

This is one for future discussion and serious debate, but I am appalled that the core of OI (and Pick) being dynamic multivalued arrays is so poor when dealing with medium to large (say 20,000 rows) tables.



Sure, we can ignore all the MV stuff and do it all the hard way, but hey its 2006 for gods sake, time the application environment caught up with life. Not that I want to use VB or C, but Revelation wants to compete by aiming at the less 'malloc' aware programmer. Theres got to be a better way internally (OI side) to handle this. I now have to rewrite much of my core application structure to get this speed improvement.



I am just hoping that Revelation can do the same to this as they did with the ARRAY update speeds.  Just imagine if building a dynamic array using s was as quick as  s AND with the great ARRAY update speed.

Populating a database view would just take a miniscule amount of time and might even be as quick as SQLs Select statement... ooooohh I love a good debate.



Colin

</QUOTE>

----

=== At 14 SEP 2006 05:00AM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:  ===

<QUOTE>Colin 



Expecting one environment to work out your intentions and optimise for you, when it provides lots of ways for you to do it yourself is naive. If it were so easy to write compilers that "get it right" then you'd expect .Net to do it for you - given MS has spent millions reengineering it from the ground up. Now try typing "optimizing .Net applications" into Google. One of the first hits will give you this quote



[i]The technical book market today is flooded with a slew of titles on how to build applications with the .NET technology, and many are undoubtedly very good. But the majority of these titles simply take a "how-to" approach on programming with the .NET Framework. They often pay little more than cursory attention to the real-world issues and challenges that developers face. [b]Performance is one of those challenges[/b].[/i]



The concatenation vs angle brackets is a trick which was well known in the Rev community in the '80s. We'll send you a copy of a document from the '80s written by John Dunbar where this sort of stuff is documented. (Just as soon as we find it).



That doesn't mean that the manuals shouldn't address the various ways of squeezing extra performance from the system but understanding how an engine works will always help you to squeeze that extra bit of perfomance from it. The challenge for Rev is not to make OI the fastest possible development engine - that's why people still use assembler - it's to make OI the easiest to use so that people who don't want to delve into the arcane side of optimisation can still deliver working apps. 



[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]

</QUOTE>

----

=== At 14 SEP 2006 06:01AM Barry Stevens wrote:  ===

<QUOTE>]]ooooohh I love a good debate



Sounds like you are getting it.



Gee, that John Dunbar article sounds like it would be interestingif only sprezz can find it,.

How many more documents are floating around outside of this site re mv systems optimisation and\or tricks 



-B

</QUOTE>

----

=== At 14 SEP 2006 07:41AM Colin Rule wrote:  ===

<QUOTE>OK, Naive perhaps, in as much as when I was using Pick/Unidata there was not such a performance hit using angle brackets, hence such use of string structures in the 80s was now known to me as I was not into Revelation then.



Absolutely agree that it should be in the Online Help manual, as this is a key (in fact paramount) part of building fast applications, assuming one has preferences towards the user rather than the developer.



Sure, I agree that the compiler cant predict what to do, but seeing as MVs are core, one might be led to expect their internal use would be optimised to be highest efficiency.  If this cannot be done then fine, other ways always exists, and this is one, but it MUST be a generally available topic within the development manual, and not something that one has to stumble across before knowing how to make apps fly.



I am very keen to get my grubby hands on the article by John Dumbar, and even more so to find the best way of building an array of unknown number of rows (I could do two passes but defeats the object) and an unknown string length for each row.

If you have to predefine the variable to replace into using square brackets, you don't want to have to keep increasing its size by appending char(0)s  or do you.

I am sure I can figure it out, but if it has been around since the 80s, then someone must build database views to populate into tables in an efficient manner.



Seems to me that this is Revelations best kept secret, I wonder if there are more.



Seems to me also that it would be in Revelations interest to have developers produce fast applications, and by making such things (or even tools) to do so available it would make them more likely to gain additional developers.



The thing is, if on the old version of OI the array build took me 30 seconds, and the array update took another 30 seconds, ie one minute (or more in some cases) to populate a view, and now it takes under 1 second, then I am REALLY impressed.

Imagine what the users will think,



Thanks for your input.



Colin

</QUOTE>

----

=== At 14 SEP 2006 09:34AM [url=http://www.sprezzatura.com]The Sprezzatura Group[/url] wrote:  ===

<QUOTE>Sorry we can't send it immediately - we're all away on a programming retreat preparing some nice stuff for conference. Thank you incidentally - you've stirred an interesting internal debate as to what changes could be made to Extract and Replace to make existing applications perform with similar speeds on array creation using  vs . It's a complex subject so fun to knock around!



As for unknowns, just grab more memory than you think you're likely to need - it'll just slow down when it hits that wall.



[url=http://www.sprezzatura.com]The Sprezzatura Group[/url]

[i]World leaders in all things RevSoft[/i]

[img]http://www.sprezzatura.com/zz.gif[/img]

</QUOTE>

----

=== At 14 SEP 2006 09:42AM dsig@sigafoos.org wrote:  ===

<QUOTE>But it has always been that way in Rev products and as far as i know it is that way in the MV environments i have worked in.



Just like REMOVE is much faster than 



Even if you do assembler there are 'fast ways' to do something.

</QUOTE>

----

=== At 14 SEP 2006 09:45AM dsig@sigafoos.org wrote:  ===

<QUOTE>]] but I am appalled that the core of OI (and Pick) being dynamic multivalued arrays is so poor when dealing with medium to large (say 20,000 rows) tables



PICK is still much faster than OI .. and that being a multi user (real engine) environment.



But even there .. in all enviornments there are 'optimal' ways of doing things.

</QUOTE>

----

=== At 14 SEP 2006 09:50AM dsig@sigafoos.org wrote:  ===

<QUOTE>almost agree  except for the ' ..The challenge for Rev is not to make OI the fastest possible development engine..'



It does need to be easier and more consistent.  It does need to do virtually everything that other environments can do .. all the tools and features without wierd hack work arounds.



But optimization IS a big deal.  But of course like all of us we have our favorite picks of what should be optimized .. like the back end .. are REAL engine performing all data selection/reduction on the server.



It still staggers me how much faster PICK, Universe etc can be over OI.  Must be the shared resources?? 

</QUOTE>

----

=== At 14 SEP 2006 09:53AM dsig@sigafoos.org wrote:  ===

<QUOTE>how many more articles .. well probably not as many physically as the ones we remember 



This site started when .. 94?  There was a lot of info available on CompuServe before that.



By the way .. i wonder if anyone has put an ENHANCEMENT request for documentation covering this question?

</QUOTE>

----

=== At 14 SEP 2006 11:49AM Mike Ruane wrote:  ===

<QUOTE>There was a session at a conference a year or two ago that addressed this very issue, as well as an article in International Spectrum.

</QUOTE>

----

=== At 14 SEP 2006 12:12PM [url=http://www.srpcs.com]SRP[/url]'s Kevin Fournier wrote:  ===

<QUOTE>Colin,



If you find yourself unable to guess the size of your string, you could use a chunk by chunk method. I've included an example stored procedure. Copy this into OI and run it to see some benchmarks.



Essentially, I allocated a chunk at a time. When I reach the end of a chunk, I append a new chunk. This has more reallocate & copies than the precalculation method, but not nearly as many as using . And you can adjust the chunk based on your best guess of the data size.



In the demo proc, I create a 11,000,000 character size array. With a chunk size of 32000, it took 6 seconds. Chunk size of 64000 took 3 seconds. To compare, the demo also executes the precalculation method, which took 1 second on my machine.



Play around with it to see what you think.




<code>
Compile Function [/color]Demo_String_Builder_Unknown(VOID) 
[/color]**************************************************************************************** 
* Demonstrates a method for building large strings quickly when then total size          
* cannot be determined. This method uses a chunk by method. It will be only slightly     
* slower than a) predetermining the total size or b) allocated more than you need.       
*                                                                                        
* NOTE: Adjust the chunk size and loop count as desired.                                 
**************************************************************************************** 
[/color] 
[/color]Declare function [/color]GetTickCount 
[/color] 
[/color]// constants 
[/color]Equ [/color]CHUNK_SIZE$ [/color]to [/color]64000 
[/color]Equ [/color]NUM_LOOPS$  [/color]to [/color]1000000 
[/color] 
[/color]// Test data: let's make it 10 characters long 
[/color]Data=[/color]"0123456789" 
[/color]DataLen=[/color]10 
[/color] 
[/color]! This is the Chunk-by-Chunk algorithm 
[/color] 
[/color]// We're gonna time this 
[/color]StartTime=GetTickCount() 
[/color] 
[/color]// Initialize our target variable to the chunk 
[/color]Array=Str([/color]\00\[/color], CHUNK_SIZE$) 
[/color] 
[/color]// Track the available size and the current array length 
[/color]CurrAlloc=CHUNK_SIZE$ 
CurrLen=[/color]0 
[/color] 
[/color]// Track the position within the string 
[/color]CurrPos=[/color]1 
[/color] 
[/color]For [/color]i=[/color]1 [/color]to [/color]NUM_LOOPS$ 
     
    [/color]// first, check to see if there is enough room left 
    // (Add 1 to account for delimiter) 
    [/color]If [/color](CurrLen + DataLen + [/color]1[/color]) GT CurrAlloc [/color]then 
        [/color]// since there's not enough room, we have to allocate more 
        [/color]Array := Str([/color]\00\[/color], CHUNK_SIZE$) 
        CurrAlloc += CHUNK_SIZE$ 
    [/color]end 
     
    [/color]// append the data 
    [/color]ArrayCurrPos, DataLen=Data 
    CurrPos += DataLen 
    CurrLen += DataLen 
     
    [/color]// append the delimiter 
    [/color]ArrayCurrPos, [/color]1[/color]=[/color]@FM 
    [/color]CurrPos += [/color]1 
    [/color]CurrLen += [/color]1 
     
[/color]Next [/color]i 
[/color] 
[/color]// Get everything up to the first null character 
[/color]Array=Array[/color]1[/color], [/color]"F"[/color]:[/color]\00\[/color] 
[/color] 
[/color]// How long did it take? 
[/color]ChunkElapsedTime=GetTickCount() - StartTime; 
[/color] 
[/color]// for informational purposes, get the length of the array 
[/color]ArrayLen=Len(Array) 
[/color]debug 
[/color] 
[/color]! For comparison purposes, build the same array using precalculated size 
[/color] 
[/color]// We're gonna time this 
[/color]StartTime=GetTickCount() 
[/color] 
[/color]// Initialize our target variable to the total size 
[/color]Array=Str([/color]\00\, (DataLen + [/color]1[/color]) * NUM_LOOPS$) 
[/color] 
[/color]// Track the position within the string 
[/color]CurrPos=[/color]1 
[/color] 
[/color]For [/color]i=[/color]1 [/color]to [/color]NUM_LOOPS$ 
     
    [/color]// append the data 
    [/color]ArrayCurrPos, DataLen=Data 
    CurrPos += DataLen 
     
    [/color]// append the delimiter 
    [/color]ArrayCurrPos, [/color]1[/color]=[/color]@FM 
    [/color]CurrPos += [/color]1 
     
[/color]Next [/color]i 
[/color] 
[/color]// Remove the extra delimiter 
[/color]Array-[/color]1[/color], [/color]1[/color]=[/color]"" 
[/color] 
[/color]// How long did it take? 
[/color]PrecalcElapsedTime=GetTickCount() - StartTime; 
[/color]debug 
[/color] 
[/color]Return [/color]1[/color][/color][/size]         
               
</code>




kfournier@srpcs.com

[url=http://www.srpcs.com]SRP Computer Solutions, Inc.[/url]



[img]http://www.srpcs.com/srpicon1.gif[/img]

</QUOTE>

----

=== At 14 SEP 2006 10:45PM dsig@sigafoos.org wrote:  ===

<QUOTE>I am really surprised to hear that square brakets and concat is no faster in PICK and universe etc.  Not sure of D3 as it has been a while but they always appeared (simple timings) to work faster.  Same with remove etc

</QUOTE>


[[https://www.revelation.com/revweb/oecgi4p.php/O4W_HANDOFF?DESTN=O4W_RUN_FORM&INQID=WORKS_READ&SUMMARY=1&KEY=8DCA8B7F6FEC117A852571DB006435D7|View this thread on the Works forum...]]