Has anyone have any information on the reliability of the OSWRITE and other OS commands?

The reason I ask is that we have a couple of remote sites that are having some problems with our in-house transaction process that keeps our data in sync between our headquarters and the remote site.  We create a DOS file with the information that has changed in a row.  Then under a couple of different conditions the DOS file will be closed out and a new one started.  Tnen the old file is sent to headquarters to be loaded into the mirror database.  The manifsation of the problem we are having is that sometimes there will be a skip in the numbering sequence of the DOS files.  Sometimes no skips, sometimes little ones and other times big jumps without apparent cause or reason.  My consern is that in those numbering skips we are loosing little bits of information, not enough to be really noticable by themselfs but they will have an accumlative effect over time (something we are seeing and dealing with for quite a while now).  I've been looking at the problem off and on for a while and I still haven't found the

problem. In reading the code line by line (several times), it seems to be in good shape and covers all the bases (I can't find any whole in it). I've been doing some checking on different things and I'm left with apparently two possible causes for the problems we are experiencing. I'm looking into one of them by collecting data to see if I can find a pattern to our problems, thinking that maybe one of our routines is causing the problem. But at this time the other possability that has come to mind is that the OSWRITE and OSBWRITE and other OS commands may not be as roubust as I would think or like them to be.

So I was wondering if anyone out there has any information on the OS commands and under what conditions might they start to become unstable and not function as expected?

The systems that are having a problem is running under AREV 3.12 on a Novell 3.11 network with most if not all of the work stations running under Win95.  I don't think Win95 is the problem because we were having this problem before Win95 was installed.

Thank You,

Michael Slack

E-mail: [email protected]

As far as I know there are no problems with OS(B) read & write. We have used them for many years in transfer of data form courts etc and have had no data loss.

Are the numbers you are using sequential? Could you give a process explaination you are going through.

dsig

David Tod Sigafoos ~ SigSOlutions

[email protected] cis:70302,77 voice:503-639-8080

Mike,

Could you put in a check of the status() function after the OSwrite? You could have it append the full status of each write to a log, perhaps in the following format:

date (of OSwrite):sequential number range (first and last in the file):status as returned by the status() function after the oswrite.

You could also have it check the file size using DOS "dir" (??) after the oswrite, and append that to the log entry as well. Then you could at least know if the consolidating/sending part is working correctly. I've found in systems like that where you can't easily see much of the functioning that the creation of logs really helps you grasp what is happening, and if it is what you expect ot be happening.

Kenny

Yes, the DOS file numbers are supposed to be sequencial.

Here is the general process that we are using.

Certain AREV tables have an MFS attached to them.  So that if someone or a process changes anything within a row, it will write that information to a DOS file for later transmission to our main office.  Then at the main office it read in the file and update the data here with the changed information from our remote site.  The same process happens in reverse from the main office to the remote site.  That is a general overall view.

Here is a little more detail.

A piece of data is modified within a table that has our MFS on it.  The change takes place plus a transaction record is created within the current DOS trasacion file.  The file name is prefixed with alpha characters to disignate where it came from and a sequentential number is added to the name.  The information within the transaction DOS record contains a date/time stamp, the name of the table and the ID of the row, the position number, the type of change it is (change, delete, add, re-add and a few otheres), the original value and the new value.

The information for the maximum size of the trasaction file, current sequential number, the file name prefix and extention and location of the file, plus a few other peices of information are stored within an Arev table.  The next transaction file can be created one of two ways.  Normally when the file acceeds a preset file size, the current file is closed out and a new one is created.  The appropriate information within the transaction table within AREV is updated.  Or the adminstrator can close out a trasaction file and the system will treat it just like it had reached it max size by updateing the file number information.  Once a trasaction file is closed out we run it thru another process before sending it to the other location.

Some of the internals to the process is that the MFS is trigered.  It creates the trasaction record depending on the type of change that was made.  Then it comes to the section for writing it to the dos trasaction file.  It pulls in from the AREV table the file prefix the current file number and extentrion.  Builds the file name.  Checks to make sure it exists, if so it appends the record to the end of the file by way of an OSBWrite.  If it doesn't exist it creates the table by writing the record by a OSWRITE.  Then it checks to see if the table is maxed out or not.  If it is then it closes it out by coping the table to the same name but another extention.  Once that is done it deletes the original table.  Then it updates the file counter to the next sequential number.  Then the next transaction record sees that no file exists it creates the next one in line and writes the record.

In reading the code for this process, I can't find any programming problems.

Michael Slack

In reading the code line by line (several times), it seems to be in good shape and covers all the bases (I can't find any whole in it).

Perhaps that's the problem. You know what they say, the whole is greater than the sum of the parts. Therein lies the problem with computer code, although in some cases, the phrase should be, "the hole is greater than the sum of the parts."

Seriously though, I don't know of a problem with OSWRITE. There is a serious problem with OSBWRITE, however. See Steve Smith's Top 5 Arev Bugs page at http://www.ozemail.com.au/~stsm/top5.htm for details.

What is the nature of your sequential numbering scheme? Are you incrementing the name of the file each time there is an update or something? Do you have a way from the ACKs to go back when an update is successful?

I hope you are using locking logic. Otherwise you could get something like this:

Session 1 notices a change and logs the item (1) to the DOS file.

Session 2 finds a change, and gets the updated counter (2).

Session 1 notices another change and gets the updated counter (3).

Session 1 prepares to osbwrite to the file, first checking for the size of the file and noticing it has 1 record.

Session 2 prepares to osbwrite to the file, etc.

Next step: they both OSBWRITE to the same location, and one of the records is lost.

Other similar things can happen without appropriate locking.

Micheal,

To add to what victor mentioned ..

Would it be possible to change your MFS to write to another Arev Table. Then have a process which grabs those records and builds your export DOS records.

For me I would worry about any two 'processes' trying to write to the same DOS record .. or possible flushing problems(?)

The 'create dos record' could be attached to an index processor ..

Could you post the write section of your mfs .. maybe we can see something

dsig

David Tod Sigafoos ~ SigSOlutions

[email protected] cis:70302,77 voice:503-639-8080

As requested, I've included the section of code that I believe is where our problems occure.

Let me say that the process called MAPCON_REC_LOCK uses the AREV locking but does some checking on the resulting STATUS() code and displays messages as needed.  Plus the code is used to unlock rows as well.  ALll-in-all relatively straight forward code.

You asked if we could write the transactions to an AREV table and then create our DOS transaction file from there.  I see no major problem with that and that might be something we will have to consider doing if I can't find the cause for our DOS file numbers skipping.  I'm hesatant rework our transaction process at this point due to the large impact it would have if something went wrong, because our organization doesn't have any real testing procedures.  We seem to be more of a sink or swim organization so we do a good bit of flailing.

Victor Engel asked about locking, as you can see from the code we do locking.  We use the same locking subroutine for most of our record locking calls and we haven't had any trouble with it.  It has be quite stable.

Here is the header of our MFS and the subroutine that actually writes the DOS transaction record and maintains the current transaction file number within the TRANS_DATA table in the SHARE_DATA row.

Any insights or suggestions you may have will be greatfully accepted.  Thank you for your time and effort.

Michael Slack

E-mail: [email protected]

EXPENDABLE SUBROUTINE SHARE_DATA_MFS(CODE, BFS, HANDLE, NAME, FMC, RECORD, STATUS)

.

.

.

WRITE_TRANS_REC:

• * * PARAMETERS: (FILE_NAME,RECORD_ID,TYPE,TRANS_REC) * * WRITES TRANSACTION RECORDS TO DOS FILE, USING INFORMATION STORED * IN SHARE_DATA OF THE TRANSACTION_DATA FILE. SHARE_DATA IS LOCKED * TO SIMULATE LOCKING CONTROL OF THE DOS FILE. THE FILE NUMBER * AUTOMATICALLY ROLLS OVER TO 1 WHEN THE MAX FILE NUMBER IS EXCEEDED. **

IF PHYSICAL_SITE_STATUS%=T' ELSE

  END_DELIM=CHAR(13):CHAR(10)

  OPEN 'TRANS_DATA' TO TRANS_DATA THEN

    CALL MAPCON_REC_LOCK(VALID_LOCK,TRANS_DATA,'TRANS_DATA','SHARE_DATA','L')

    IF VALID_LOCK THEN

      TRANS_INFO=FILE_NAME:@VM:RECORD_ID:@VM:TRANS_INFO

      IF TRANS_REC=' THEN

        OUT_REC=TRANS_INFO

      END ELSE

        OUT_REC=TRANS_INFO:@FM:TRANS_REC

      END

      OUT_REC := END_DELIM ;* DELIM FOR END OF RECORD

      SUCCESS=0 ;*FALSE

      LOOP

        OSOPEN OUT_PATH%:FILE_PREFIX%:FILE_NUM%:OUT_EXTENSION% TO TRANS_FILE THEN

• ADD RECORD TO END OF FILE

          FILE_INFO=DIR(OUT_PATH%:FILE_PREFIX%:FILE_NUM%:OUT_EXTENSION%)

          OSBWRITE OUT_REC TO TRANS_FILE AT FILE_INFO

          OSCLOSE TRANS_FILE

          FILE_LEN=FILE_INFO + LEN(OUT_REC)

          SUCCESS=1 ;*TRUE

        END ELSE

• FILE DOES NOT EXIST. CHECK FOR NEW FILE OR CREATE NEW FILE.

          FILE_LEN=LEN(OUT_REC)

          READ SHARE_REC FROM TRANS_DATA, 'SHARE_DATA' THEN

            OUT_PATH%     =SHARE_REC 

            IF OUT_PATH%-1,1 # '\' THEN OUT_PATH% := '\'

            OUT_EXTENSION%=SHARE_REC 

            FILE_PREFIX%  =SHARE_REC 

            IF FILE_NUM%  =SHARE_REC  THEN

              FILE_NUM_LEN=8 - LEN(SHARE_REC )

              NEXT_KEY=SHARE_REC  + 1

              MAX_NUM=STR('9',FILE_NUM_LEN)

              IF NEXT_KEY ] MAX_NUM THEN

                ZERO_PAD=STR('0',FILE_NUM_LEN - 1)

                SHARE_REC =ZERO_PAD:'1'

              END ELSE

                KEY_LEN=LEN(NEXT_KEY)

                ZERO_PAD=STR('0',FILE_NUM_LEN - KEY_LEN)

                SHARE_REC =ZERO_PAD:NEXT_KEY

              END

              WRITE SHARE_REC TO TRANS_DATA, 'SHARE_DATA'

              FILE_NUM%=SHARE_REC 

              OSWRITE OUT_REC TO OUT_PATH%:FILE_PREFIX%:FILE_NUM%:OUT_EXTENSION%

              SUCCESS=1 ;*TRUE

            END ELSE

              FILE_NUM%=SHARE_REC 

            END

          END ELSE

• ERROR - READING SHARE_DATA

            MSG('ERROR READING SHARE_DATA. NO TRANSACTION RECORD CREATED.

REPORT THIS PROBLEM TO YOUR MAPCON ADMINISTRATOR!',,,) SUCCESS=1 FILE_LEN=0 END END UNTIL SUCCESS REPEAT IF MAX_FILE_SIZE%=' THEN MAX_FILE_SIZE%=XLATE('TRANS_DATA','SHARE_DATA',8,'X') END IF FILE_LEN ] MAX_FILE_SIZE% THEN SQUISH_FILTER=XLATE('TRANS_DATA','SQUISH_DATA',TD_IN_FILTER$,'X') SQUISH_EXTENSION=SQUISH_FILTER-4,4 CALL PUSH.SELECT(CURSOR, SAVE1, SAVE2, SAVE3) PCPERFORM "COPY ":OUT_PATH%:FILE_PREFIX%:FILE_NUM%:OUT_EXTENSION%:" ":OUT_PATH%:FILE_PREFIX%:FILE_NUM%:SQUISH_EXTENSION FILE_JUST_CREATED=OUT_PATH%:FILE_PREFIX%:FILE_NUM%:SQUISH_EXTENSION OSOPEN FILE_JUST_CREATED TO TEMP_HANDLE THEN PCPERFORM "DEL ":OUT_PATH%:FILE_PREFIX%:FILE_NUM%:OUT_EXTENSION% END ELSE ERROR=' BEGIN CASE CASE STATUS()=1 ERROR=BAD FILE NAME' CASE STATUS()=2 ERROR=ACCESS DENIED BY OS' CASE STATUS()=4 ERROR=FILE DOES NOT EXIST' * the *.mfs did not copy to the *.squ CASE STATUS()=5 ERROR=UNDEFINED ERROR' END CASE MSG("UNABLE TO DO DOS COPY FROM *.MFS TO *.SQU PLEASE CONTACT I.S. MAPCON SUPPORT WITH THIS INFORMATION! ERROR: ":ERROR,,,)

        END

        CALL POP.SELECT(CURSOR, SAVE1, SAVE2, SAVE3)

      END

      CALL MAPCON_REC_LOCK(VALID_LOCK,TRANS_DATA,'TRANS_DATA','SHARE_DATA','U')

    END ELSE

• ERROR - LOCKING SHARE_DATA

      MSG('ERROR LOCKING SHARE_DATA','','','')

    END

  END ELSE

• ERROR - OPENING TRANS_DATA

    MSG('ERROR OPENING TRANS_DATA','','','')

  END

END

RETURN

Michael,

Two points, for whatever they are worth.

In 15+ years of using Rev/Arev, I've never had an OS… function fail.

At several customer sites, I've noticed occasional flakiness with some sort of caching. The worst has been on W95 over an NT net. Is it possible that you are getting incorrect results from your DIR function? Or that the locking really isn't working?

Hello Curt:

Thanks for the input.  Anything is possible at this point.  The remote site that is having the problems on a Novell 3.11 (or something really close to that but not the 4.0), using AREV 3.12 with most of the workstations running under Win95.  So we don't have the NT in the mix there.

You have a good point about the caching being a possible problem.  I'll have to do some research but I seem to remember something about turning off backgroung writing from cach in Win95 and it should be turned off when using AREV.

Thanks,

Michael Slack

View this thread on the forum...