Data Warehousing Trials and Travails (OpenInsight Specific)
At 02 MAR 1999 02:27:13PM Matt Sorrell wrote:
I've run across a very irritating problem when trying to warehouse my LH data.
As long as I tell the Warehouse Manager to clear the tables by dropping and recreating them, everything runs perfectly.
However, if I tell the Warehouse Manager to NOT clear the tables, then nothing gets run in at all.
The error log is full of "Error processing ". I have actually verified, there is one entry for every record I'm trying to warehouse.
I have verified that the tables are empty before I run my process using the OI query tool and the ISQL query tool in SQL Enterprise Manager. Both return 0 rows.
I'm completely at a loss on this one. I really need to be able to run the records in in batches, and clearing the tables for each run is not an option.
Also, I have verified that I have access to all of the relevant network drives (data drive, SQL server, OI installation, etc.)
Any help would be greatly appreciated.
Thanks,
Matt
At 03 MAR 1999 11:34AM Nick Stevenson wrote:
I seem to remember we had something along those lines as well. I think we solved it by COMMITTING the dataset on each row processed. If you know what to do, try that. If not let me know and I'll look into our source code and show you how.
At 03 MAR 1999 01:23PM Matt Sorrell wrote:
Nick,
I'm rather new to OI, been doing AREV for almost 2 years now.
I know how to commit a transaction through VB, but don't know how to do it through OI.
Any help would be greatly appreciated.
Thanks,
Matt
At 03 MAR 1999 02:01PM Cameron Revelation wrote:
Matt,
If you don't clear the tables, the warehousing logic changes. Here is the explanation:
1) clear tables
2) select LH records
3) for each LH record
3.1) read LH record
3.2) insert data from LH record into DataSet
3.3) commit DataSet (only uses INSERT script since all data is "new")
Versus:
1) select LH records (only subset if Updates-Only option used, otherwise all)
2) for each LH record
2.1) read LH record
2.2) retrieve existing data from database (DataSet executes using SELECT script)
2.3) remove all data from the DataSet
2.4) insert all data from the LH record into the DataSet
2.5) commit DataSet (uses INSERT, UPDATE, and DELETE scripts since some data may be new, some may be modified, some may be removed)
So check your SELECT, UPDATE, and DELETE scripts – there is a problem in one of them.
Cameron Purdy
Revelation Software
At 08 MAR 1999 11:01AM Nick Stevenson wrote:
Matt
Sorry, I went to sleep a bit on that one. Try changing the "DELETE" part of your WP script to include the COMMIT:
* delete all rows from DataSets
if Ret thenfor i=1 to CountDSGetProperty(Handle[i], DS_ROWCOUNT$, RowCount)DSSetProperty(Handle[i], DS_ROWID$, 1)for iRow=1 to RowCountDSMethod(Handle[i], DS_DELETE$)DSMethod(Handle[i], DS_COMMIT$)next iRownext iendendI assume you are using the WP script as generated by the Warehouse Manager. If so, then just find the bit of code that matches the above snippet.
Worked for us…
Good luck,
Nick