Setup

HP Netserver LH Pro dual 200mhz Intel Pentiums

256 megs ram

Hardware RAID5 controller with 3 x 9 gig drives

NT4 service pack 3

TCP/IP

10mbit network

Arev 3.12 application

NT Service and NPP

Workstations

P75 upwards HP Vectras running win95

Problems experienced with peformance degrading during the day. We have now isolated this to disk writes (averaging about 95% write activity which often 'flatlines' at 100% for extended periods)

When the disk writes are this high, performance is slow and workstation response to initial key press very slow.

I have managed to further narrow this down to indexing 'causing' the huge write activity (log in one workstation to an application with outstanding indexing and as soon as the indexing starts the disk writes shoot up - log out and the disk writes drop off)

I have also noticed the same sort of degredation with file re-sizes.

Revparam files set to serveronly=true

Any help with improving performance would be greatly appreciated.

Regards

Mike Horton

horton@mcmail.com

A couple of things. One is to set up a dedicated indexing machine. This will make sure indexes are as up to date as possible so that the load on a station will be reduced.

The others are to go through the network settings and make sure that the server is allocating it's time and engery properly. Make sure that process time is spent working on file and disk action and other background processing instead of screen savers, UI and other front end work.

apk@sprezzatura.com

Sprezzatura, Inc.

Aaron

Thanks for your response. We have had an index server running but have not disabled all the workstations from indexing as yet. We will be putting through a dedicated 100 mbit link to the index server and will then check performance (both with and without disabling indexing on the other workstations).

My one concern was that we are wacking +-13000 transactions into the system in a three day cycle, and processing will produce approx 10000 rows into two other tables during this processing (plus a few other arbitary rows in other tables) We have reduced our indexing to the bare minimum but find that with the dedicated indexer we still have a fair load of indexing being carried out by idle workstations.

The weekend procedure creates a history of the +- 13000 transactions and then clears down the transaction file for the next week

I still find that when any workstation starts indexing the disk writes immediately go through the roof (I though this may possibly connected to the RAID5 producing a huge amount of writes)

Transactions are imported a couple of hundred at a time so the option of turning the indexing off and rebuilding is not an option - I have considered using and index flush at the end of each import (I am running tests at the moment)

We are confused as we have the same system running at a different site (and although it carries a lighter workload) we have no performance issues (as we don't with the system running on Novell). I have requested a spec on the OK server to see what main differences there are and hopefully that will also give us an indication of the bottlenecks.

Sorry about rambling on but I am anxious to allay the clients fears that Arev is not their solution (after we have pushed the upgrade from RevG to Arev and they ask why???)

Once again, many thanks

Mike Horton

A dedicated indexer is your best bet. Actually, a custom indexing program would be even better. This way, it can give higher priority to the heavy transaction files, indexing as the transactions are added.

Both systems should be generating similar requests per transaction. Now, what could be happening, is if your keys are large, or non-sequential, and you have a large amount of identical index values, you could be jamming records in the middle of nodes forcing a full node re-write for each transaction.

I know of a site that used to take over 33 seconds to process an individual index transaction by the time the node information was updated. The key was 5 or 6 parts, with full @STATION (session number included) date, time, @username, @account and a random number. Think there was more there as well.

What happens if you turn off the RAID5?

apk@sprezzatura.com

Sprezzatura, Inc.

I have considered a custom indexing routine and will look at that. We have a most a 4 part key (12-15 digits)which are fairly random. Indexing station to become dedicated once the 10mb link is put in.

I now have our NT guru looking at the performance and he has asked why can we not run the indexing on the server (plenty of puch left over in terms of ram and processor) I have advised him that running on the server is unsupported and he has asked why?

As we have hardware RAID5 I have been advised it is not a case of just switching it off and testing - it requires a rebuild. We do have another site within the group that runs the same software but on a much lower spec server (with no raid and considerably less transactions) but do not have a performance issue there. Our own development system is on a lower spec server (again without raid) but here there are only test transactions going through and can not really compare things. One thing does come through in common is that if a work station sits idle for some time, response to a key press is slow, as is subsequent processing - this then picks up although a workstation that has been working consistently seems to carry on performing well.

I have also looked at the posibility of file resizes creating additional writes and though that as we know approximately how many transactions are coming in, that I should possibly size the files accordingly - again a bit in the dark as to how and when the files resize.

I have also discovered today that we end up with indexes a lot larger that the data file - a rebuild on the indexes in the table brings the index size down to expected sizes (have found this on files that have been cleared down) - it does seem to improve performance if we find this on one of the larger files and do the rebuild.

The NT guru is going to clear down any unused services and reboot the server and continue monitoring. Will post back any findings here.

Once again thanks for your response.

Regards

Mike Horton

See my response on the advanced revelation thread it may be of interest

Mike,

Your application is write intensive and your RAID 5 solution

involves multiple I/O 's to perform a single write successfully.

This happens because of the need for parity to be calculated

for each transaction and you are possibly writing over multiple

stripes across multiple hard disks.

All this I/O will slow your application down dramatically unless …..

You have a very high performance hardware RAID controller

AND you have a very high quality file server e.g. Digital or

Compaq.

No amount of tweeking of NT will make any difference!

Your solution may well be to remove RAID 5 for performance

and install a combination of RAID 0 and RAID 1 which will

provide faster I/O processing with disk mirroring for fault

tolerance.

Regards,

Kevin at KPW

email kpw1@compuserve.com

Mike, revisited your initial thread to check your server spec.

Unable to comment upon th HP server nor their RAID controllers.

Are you using NT Service V1.5? This definitely seems better

than previous versions for stability and performance.

Your disk I/O is the real issue so in your case we would definitely

recommend changing your RAID solution. Yes, it means a rebuild.

In a similar circumstance one of our clients experienced a massive

improvement after changing to pure RAID 0 but they lost fault

tolerance. The option to combine RAID 0 with RAID 1 is in fact

recommended as a safer option than RAID 5 because if a techo

removed the wrong hard disk from the set after a fault you lose it all!

Regards,

Kevin at KPW

View this thread on the forum...