Remove Statement speed (OpenInsight 32-Bit)
At 20 FEB 2011 11:40:30PM Colin Rule wrote:
I have a loop/remove, which is processing 50Mb of text, some 200,000 lines.
On my desktop (Vista 64bit, 2.5Ghz, 4Gb Ram), the processing takes around 1 second to run.
On my laptop (Vista 32bit, 1.86Ghz, 3Gb Ram), the processing takes ]60 minutes, as I killed it after 20 mins, and it was 1/10th the way through.
I tried using string extaction and COL2(), but this is a similar speed.
Is there a better way to do this, or any suggestions as to what might cause the laptop to perform so badly.
I have AVG running on both desktop and laptop.
Colin
At 21 FEB 2011 11:48AM Richard Hunt wrote:
I strongly believe that it is not the actual "REMOVE" statement speed that you are having issues with, it is probably the other statements you have within that loop of statements that are causing the slow down.
At 21 FEB 2011 02:22PM dsigafoos wrote:
Colin, i doubt that it is AVG. What else is running on the laptop. there must be something specific about the laptop .. memory etc
At 21 FEB 2011 03:39PM Barry Stevens wrote:
probably has memory errors and are being corrected.Run a 'PC tester' on it.
At 21 FEB 2011 04:29PM Colin Rule wrote:
I have removed all statements inside the loop, but no change in speed.
I do however also beleive it is PC related.
The problem might be due to memory paging.
I am going to reinstall Windows, in the hope it will resolve.
It is however running at an acceptable speed for all other functions.
Colin
At 21 FEB 2011 05:30PM Barry Stevens wrote:
I suppose OI is the same as the old AREV adage….if there is something wrong with a pc, AREV will find it.
At 22 FEB 2011 03:08AM Stefano Cavaglieri wrote:
Have you tried with BRemove ? Don't know about your environment, but at least in UTF8 mode this would make a huge difference.
At 22 FEB 2011 03:15AM Warren Auyong wrote:
Is the data on a network drive or copied on to local drives? If local what size hardware cache is on the drives? How do the drives compare with disk benchmarks? How do the drive specs compare - RPM, cache, random access times etc.? What are the L1 and L2 cache on the CPUs? What does the page fault, memory, disk usage look like in the system monitor?
At 22 FEB 2011 08:54AM Jared Bratu wrote:
These are all great questions and may be the source of the disparity between systems. If the data can be pre-loaded and then processed the network performance could be eliminated from the equation.
At 23 FEB 2011 06:13PM Colin Rule wrote:
Thanks Stefano, it was down to being in UTF8 mode.
Turning this off, caused the loop/remove to take about 1-2 seconds.
I have not seen the BREMOVE statement, there is nothing in the online help, but I will search the forum.
Colin
At 23 FEB 2011 07:33PM [email protected] wrote:
Hi Colin,
Here's the details:
There are some other UTF8 enhancements detailed on the blog as well that you might be interested in.
Battlestar Sprezzatura - BSG 77
Colonial leaders in all things RevSoft
