Index Update Crashing OpenInsight - 10.08.1 (OpenInsight 64-bit)
At 27 JAN 2021 04:40:40PM Brad Bishop wrote:
I found that starting sometime yesterday when the RTI_INDEXER is invoked through REVRUN it just will hang. I found that today whenever the RTI_INDEXER is ran online though the db tools OpenInsight exits and creates an appcrash event in the Windows Event Log . As part of diagnosing the issue I removed the particular table the indexer stopped on and the issue moved to another table. The issue has "moved" to 4 different tables. I have ran LH_VERIFY on both the tables and indices and they report no GFE's. I have deleted indexes (had to manually finish the deletion) and rebuilt a new index file and the indexer will still crash on the table. I can rebuild the indices on all of the tables manually with no issues. When I try to view the LF Table info on any of the tables the app will also crash. I ran the RevLHSrvc.log and it seems to confirm there is a problem with table info. The rti_lh_info lines in the lof only appear on tables when the indexer is going to crash or when trying to view table info. Since this has occurred on multiple tables and appears to be a moving target I'm at a loss on how to proceed.
bb
[75165906] [75165984] 0 [75165984] ,40 [75165984] ,!PRODUCT_SPEC*ESSEX302 [75165984] :Read Only Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,0 [75165984] ,REVSOFT\SF_Trng\Database\essexdata\REV51357 [75165984] :Open File (resolved to E:\Revsoft\SF_Trng\Database\essexdata\REV51357)… [75165984] file 57.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,*INDEXES [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,*INDEXES [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,0 [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,0 [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,0 [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,14 [75165984] ,RTI_LH_INFO [75165984] :Read Only Record… [75165984] error:100,0,0.
[75165984] [75165984] 0 [75165984] ,22 [75165984] ,rti_lh_info getTableNumber*75165984 [75165984] :Read Record… [75165984] error:100,0,0.
[75165984] [75165984] 0 [75165984] ,40 [75165984] ,PRODUCT_SPEC*ESSEX302 [75165984] :Read Only Record… [75165984] done.
[75165984] [75166000] 0 [75166000] ,0 [75166000] ,REVSOFT\SF_Trng\Database\essexdata\REV50356 [75166000] :Open File (resolved to E:\Revsoft\SF_Trng\Database\essexdata\REV50356)… [75166000] file 58.
[75166000] [75166000] 0 [75166000] ,40 [75166000] ,DICT.PRODUCT_SPEC*ESSEX302 [75166000] :Read Only Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,0 [75166000] ,REVSOFT\SF_Trng\Database\essexdata\REV50355 [75166000] :Open File (resolved to E:\Revsoft\SF_Trng\Database\essexdata\REV50355)… [75166000] file 59.
[75166000] [75166000] 0 [75166000] ,59 [75166000] :Get File Header… [75166000] done.
[75166000] [75166000] 0 [75166000] ,59 [75166000] ,%FIELDS% [75166000] :Read Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,59 [75166000] ,%PROTECT.SPEC% [75166000] :Read Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,57 [75166000] ,! [75166000] :Read Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,40 [75166000] ,PRODUCT_SPEC*ESSEX302 [75166000] :Read Only Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,22 [75166000] ,rti_lh_info getTableInfo - Open revmedia*75166000 [75166000] :Read Record… [75166000] error:100,0,0.
[75166000] [75166000] 0 [75166000] ,40 [75166000] ,REVMEDIA [75166000] :Read Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,22 [75166000] ,rti_lh_info getFileStatistics \\FW-TR-OI19\REVSOFT\SF_TRNG\DATABASE\ESSEXDATA\REV50356*75166000 [75166000] :Read Record… [75166000] error:100,0,0.
[75166000] [75166000] 0 [75166000] ,0 [75166000] ,REVSOFT\SF_Trng\Database\essexdata\REV50356 [75166000] :Get Statistics… [75166000] done.
[75166000] [75167203] 0 [75167203] :Disconnect… [75167203] done.
At 27 JAN 2021 04:49PM bob carten wrote:
Hi Brad,
Those errors are from trying to look up the linear hash information, not from the indexer, correcf?
Is there any chance that the tables are over 2gb in size?
At 28 JAN 2021 08:09AM Brad Bishop wrote:
No tables over 2gb. The largest table in this system is 1.25GB which so far doesn't seem affected. As a side not I check my development system for this app and the View LH Statistics is also crashing by the indexer is unaffected. There is nothing directly shared between these two systems (they are in different servers).
Here is the log from the index build.
[75165906] [75165984] 0 [75165984] ,40 [75165984] ,!PRODUCT_SPEC*ESSEX302 [75165984] :Read Only Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,0 [75165984] ,REVSOFT\SF_Trng\Database\essexdata\REV51357 [75165984] :Open File (resolved to E:\Revsoft\SF_Trng\Database\essexdata\REV51357)… [75165984] file 57.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,*INDEXES [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,*INDEXES [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,0 [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,0 [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,57 [75165984] ,0 [75165984] :Read Record… [75165984] done.
[75165984] [75165984] 0 [75165984] ,14 [75165984] ,RTI_LH_INFO [75165984] :Read Only Record… [75165984] error:100,0,0.
[75165984] [75165984] 0 [75165984] ,22 [75165984] ,rti_lh_info getTableNumber*75165984 [75165984] :Read Record… [75165984] error:100,0,0.
[75165984] [75165984] 0 [75165984] ,40 [75165984] ,PRODUCT_SPEC*ESSEX302 [75165984] :Read Only Record… [75165984] done.
[75165984] [75166000] 0 [75166000] ,0 [75166000] ,REVSOFT\SF_Trng\Database\essexdata\REV50356 [75166000] :Open File (resolved to E:\Revsoft\SF_Trng\Database\essexdata\REV50356)… [75166000] file 58.
[75166000] [75166000] 0 [75166000] ,40 [75166000] ,DICT.PRODUCT_SPEC*ESSEX302 [75166000] :Read Only Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,0 [75166000] ,REVSOFT\SF_Trng\Database\essexdata\REV50355 [75166000] :Open File (resolved to E:\Revsoft\SF_Trng\Database\essexdata\REV50355)… [75166000] file 59.
[75166000] [75166000] 0 [75166000] ,59 [75166000] :Get File Header… [75166000] done.
[75166000] [75166000] 0 [75166000] ,59 [75166000] ,%FIELDS% [75166000] :Read Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,59 [75166000] ,%PROTECT.SPEC% [75166000] :Read Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,57 [75166000] ,! [75166000] :Read Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,40 [75166000] ,PRODUCT_SPEC*ESSEX302 [75166000] :Read Only Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,22 [75166000] ,rti_lh_info getTableInfo - Open revmedia*75166000 [75166000] :Read Record… [75166000] error:100,0,0.
[75166000] [75166000] 0 [75166000] ,40 [75166000] ,REVMEDIA [75166000] :Read Record… [75166000] done.
[75166000] [75166000] 0 [75166000] ,22 [75166000] ,rti_lh_info getFileStatistics \\FW-TR-OI19\REVSOFT\SF_TRNG\DATABASE\ESSEXDATA\REV50356*75166000 [75166000] :Read Record… [75166000] error:100,0,0.
[75166000] [75166000] 0 [75166000] ,0 [75166000] ,REVSOFT\SF_Trng\Database\essexdata\REV50356 [75166000] :Get Statistics… [75166000] done.
[75166000] [75167203] 0 [75167203] :Disconnect… [75167203] done.
[75167203] Stop pending…Stopped.
Terminate…done.
At 01 FEB 2021 11:33AM Brad Bishop wrote:
Do you need anything else from me? What's the next step ?
At 01 FEB 2021 03:59PM bob carten wrote:
Hi Brad,
I'm reading the code to see why rti_lh_info would be called when building an index. Do you have the lh_Statistics or lh_lock_info windows running at the same time?
At 01 FEB 2021 04:58PM Brad Bishop wrote:
No. It occurs when the indexer is the only thing running.
At 03 FEB 2021 03:07PM Brad Bishop wrote:
Several Updates to add to the confusion-
The initial issues of the indexer stopping on a table has been resolved (but not by the same actions). On the affected tables ddelete the indices, delete the bang file, manually removeing the bang file from revmedia, removing the table from the DBT. sign out of OI, sign back into OI, attach the table and then build new indices. Repeat this for the 3 other tables exhibiting the same behavior. This corrected my training system. This did not identify the original problem but is a work around to get my users back to testing.
On my Dev system where the LH statistics was blowing out the OI client. The statistics used to run through all the tables but lately it has been stopping on certain tables. Detaching the table that it stopped would allow the statistics to continue past where it was stopping. In an attempt to determine what was occurring I detached all application tables and re-attached them using local pathing (instead of network pathing) and the LH statistics completes. I have no idea what changed that would have started this behavior.
With out identifying the root causes of these issues I am reluctant to move forward with a production implementation. Given these apps are used in manufacturing plants that run 24*7*365 the systems have to be rock solid. I don't have the luxury of shutting down a product system to diagnose problems. The only down time I can get is a 2-4 hour window if it's pre-planned.