Background: we have a client serving web APIs via the OECGI. The OEngineServer and system tables are on one server (aka the web server) and the application tables are on another server (aka the database server). Occasionally the database server will drop its connection, but this is momentary. This is just enough for the web server to encounter failures when it needs to access a table on the database server. The situation is not recoverable. Restarting the OEngineServer, however, quickly resolves the issue because the database server is not longer disconnected.
Question 1: Is it possible for code running in an engine to know whether the OEngineServer was launched via the command prompt versus as a service?
Question 2: Is it possible for code running in an engine to restart the OEngineServer? (I think I can figure out this second question on my own, but I would appreciate receiving any known solutions. The more important question is #1 since I will want to restart the OEngineServer using the same technique that is currently being used.)
Hi Don,
Bryan may correct me on this but I don't think the OEServer exposes that sort of information to an engine. However, if the OEServer is running as a service you can use a few API calls in an engine to check if it's running in the context of an interactive workstation - if it's not then it's running from the service account.
Take a look at the GetProcessWindowStation and GetUserObjectInformation APi calls to see if the station name is "WinSta0" - if so you've been started from the user's desktop and not a service app. Here's a quick proc as an example (no error handling sorry)
It needs the following DLL prototypes in User32:
HANDLE STDCALL GetProcessWindowStation( VOID )
INT STDCALL GetUserObjectInformationA( HANDLE, INT, LPVOID, UINT, LPUINT ) as GetUserObjectInformation
compile function rti_IsInteractiveStation() declare function GetProcessWindowStation, GetUserObjectInformation $insert logical equ UOI_NAME$ to 2 equ MAX_PATH$ to 256 // First get a handle to the station hStation = GetProcessWindowStation() if ( hStation ) else // Unlikely ... but... return "" end // Lets get the name... cbBuffer = MAX_PATH$ buffer = str( \00\, MAX_PATH$ ) cbNeeded = 0 bOK = GetUserObjectInformation( hStation, | UOI_NAME$, | getPointer( buffer ), | cbBuffer, | cbNeeded ) if ( bOK ) then buffer = buffer[1,\00\] end else // Handle error... <whatever> return "" end return ( buffer == "WinSta0" )(The exception to this is if the service has the "allow desktop" flag set - then it will use the user's desktop, but this is unlikely in a production system)
Once you've got that you could probably use something like a simple batch file to restart the service - you need something outside of the engine as stopping the service will terminate it.
HTH
Not what you are asking, but related:
OECGI3 and OECGI4 allow you to specify a comma delimited list of serverUrl and ServerPort, then set it to round-robin or failover from one host to the next. You could run two engine server services on the same machine but different ports. Then you can restart one or the other on a regular basis without completely interrupting your website. See the OECGI quickstart guide in the documents folder for details.
You could run the engineserver as a service on multiple appserver machines rather than on the web server. In that case, in the webserver OECGI3 registry settings you would specify comma separated ip addresses of the appservers in the serverURL, with a matching list of comma separated port numbers. You could then reboot the appservers as desired with minimal disruption
Alternatively, you could run multiple oengineserver services on your existing server. The installer for the service is InstallAPP-NT.bat, located in the OESERVER folder under Oinsight. It installs the service using parameters in file named WRAPPER.CONF.TXT in the CONF folder under the OESERVER folder. To install multiple instances of the oengineserver service, you create a second eserver.cfg to specify different ports, a second conf file to specify the new eserver.cfg, then pass the path to the new conf file to InstallApp-Nt.bat
Below are examples of a modified wrapper2.conf.txt file and an modified eserver2.cfg to create an OengineServer2. Given these files, I was able to open a CMD prompt as administrator, change to the OESERVER folder under openinsight and run
InstallApp-NT conf\wrapper2.conf.txtIf you read through you see where I replaced OengineServer with OengineServer2 in multiple locations to install the second service with a different name. The important line is where I use the -L argument to make it it load a different eserver.cfg,
wrapper.app.parameter.1=com.revelation.oesocketserver.Main -l eserver2.cfgwrapper2.conf.txt
#******************************************************************** # Wrapper Properties #******************************************************************** # Java Application wrapper.java.command=c:\Program Files (x86)\Java\jre1.8.0_261\bin\java # Java Main class. This class must implement the WrapperListener interface # or guarantee that the WrapperManager class is initialized. Helper # classes are provided to do this for you. See the Integration section # of the documentation for details. wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp # Java Classpath (include wrapper.jar) Add class path elements as # needed starting from 1 wrapper.java.classpath.1=oeserver/lib/wrapper.jar wrapper.java.classpath.2=oesocketserver.jar # Java Library Path (location of Wrapper.DLL or libwrapper.so) wrapper.java.library.path.1=oeserver/lib wrapper.java.library.path.2=. # Java Additional Parameters #wrapper.java.additional.1= # Initial Java Heap Size (in MB) #wrapper.java.initmemory=3 # Maximum Java Heap Size (in MB) #wrapper.java.maxmemory=64 # Application parameters. Add parameters as needed starting from 1 wrapper.app.parameter.1=com.revelation.oesocketserver.Main -l eserver2.cfg #******************************************************************** # Wrapper Logging Properties #******************************************************************** # Format of output for the console. (See docs for formats) wrapper.console.format=PM # Log Level for console output. (See docs for log levels) wrapper.console.loglevel=INFO # Log file to use for wrapper output logging. wrapper.logfile=logs/wrapper2.log # Format of output for the log file. (See docs for formats) wrapper.logfile.format=LPTM # Log Level for log file output. (See docs for log levels) wrapper.logfile.loglevel=INFO # Maximum size that the log file will be allowed to grow to before # the log is rolled. Size is specified in bytes. The default value # of 0, disables log rolling. May abbreviate with the 'k' (kb) or # 'm' (mb) suffix. For example: 10m = 10 megabytes. wrapper.logfile.maxsize=10 # Maximum number of rolled log files which will be allowed before old # files are deleted. The default value of 0 implies no limit. wrapper.logfile.maxfiles=30 # Log Level for sys/event log output. (See docs for log levels) wrapper.syslog.loglevel=NONE #******************************************************************** # Wrapper Windows Properties #******************************************************************** # Title to use when running as a console wrapper.console.title=OEngineServer2 #******************************************************************** # Wrapper Windows NT/2000/XP Service Properties #******************************************************************** # WARNING - Do not modify any of these properties when an application # using this configuration file has been installed as a service. # Please uninstall the service before modifying this section. The # service can then be reinstalled. # Name of the service wrapper.ntservice.name=OEngineServer2 # Display name of the service wrapper.ntservice.displayname=OEngineServer2 # Description of the service wrapper.ntservice.description=Revelation OpenEngine Server Second copy # Service dependencies. Add dependencies as needed starting from 1 wrapper.ntservice.dependency.1= # Mode in which the service is installed. AUTO_START or DEMAND_START wrapper.ntservice.starttype=AUTO_START # Allow the service to interact with the desktop. wrapper.ntservice.interactive=falseeserver2.cfg
WebServerDefaultPage=oecgi3.exe/O4W_RUN_DASHBOARD?GRAPHID\=STATS_DASHBOARD_DAY;oecgi3.exe/O4W_RUN_DASHBOARD?GRAPHID\=STATS_DASHBOARD_DAY IdleCheck=600 Password= KeepAlive=0 Server= PortNumber=8289 MaxConnections=0 MaxEngines=10 TimerProc= StartupProc= StartupFlags=1 ShutdownFlags=65 Procedure= UserName= # built in web server WebServer_Disabled=0 WebServerPortNumber=8288 # triggers for web server to treat as cgi # if the url contains one of these then if will behave as if OECGI was called WebServerCGIProcedure=oecgi3.exe;online WebServerVirtualDirs=console;examples;sysprog;univbridge #Virtual paths is like the physical folder WebServerVirtualPaths=\\OI940\\o4w;\\OI940\\o4w;\\OI940\\o4w;\\OI940\\o4w #Webserverconnecton is like registry setting for OECGI #WebserverConnection_mypath = MYAPP,MYUSER,MYPW, startupflags,shutdownflags, cgiproc, , uploadmode, oifolder with escaped backslashes WebServerConnection_default=EXAMPLES,EXAMPLES,,1,1,RUN_OECGI_REQUEST,,,1 WebServerConnection_console=SYSPROG,OICONSOLE,OICONSOLEPW,1,1,RUN_OECGI_REQUEST,,,3,\\OI940Gold_Final_rjc\\o4w\\downloads\\, WebServerConnection_sysprog=SYSPROG,SYSPROG,SYSPROGPW,1,1,RUN_OECGI_REQUEST,,,3,\\OI940Gold_Final_rjc\\o4w\\downloads\\, SystemMonitor=RTI_OEMONITOR_CLEANUP,SYSPROG,OICONSOLE,OICONSOLE;RTI_OEMONITOR_INDEX,SYSPROG,OICONSOLE,OICONSOLE Procedure_1=REVCMD_LISTENER Statistics_Output_Disabled=1 IdleTimeout=150 Statistics_Input_Disabled=1 Mode= UTFPort_Disabled=1 SystemMonitorTime=60 ShutdownProc= Procedure_=JD3_LISTENER Application= //=Process to run for mode -1,0,2 at 'idle check'
Carl and Bob,
Thank you both for your responses. The ability to create failover connections is great (and we've already implemented this with some sites), but not entirely sure it will solve our particular problem. We also have batch files that will launch OEngineServer, which I was considering as part of the solution for this issue. I'll experiment with Carl's code to see how well that works.
However, upon further reflection I wonder if restarting the OEngineServer is necessary. Yes, it resolves the problem, but perhaps it is too much of a large mallet when I simply need a small hammer.
I think all I really need is a way to close the current engine. Then I can just let the OEngineServer spin up a new engine and it should be capable of connecting to the application server tables no differently than when I restart the OEngineServer. I believe RevCAPI uses the RevCloseEngine method to shutdown an engine. Perhaps there is a way to send a request back to the OEngineServer to issue this call?
Carl and Bob,
Thank you both for your responses. The ability to create failover connections is great (and we've already implemented this with some sites), but not entirely sure it will solve our particular problem. We also have batch files that will launch OEngineServer, which I was considering as part of the solution for this issue. I'll experiment with Carl's code to see how well that works.
However, upon further reflection I wonder if restarting the OEngineServer is necessary. Yes, it resolves the problem, but perhaps it is too much of a large mallet when I simply need a small hammer.
I think all I really need is a way to close the current engine. Then I can just let the OEngineServer spin up a new engine and it should be capable of connecting to the application server tables no differently than when I restart the OEngineServer. I believe RevCAPI uses the RevCloseEngine method to shutdown an engine. Perhaps there is a way to send a request back to the OEngineServer to issue this call?
Hi, Don. Are you hoping that a _different_ engine can close down a "badly attached" engine, or that the bad engine ITSELF can close itself down?
Also, when the "problem engine" fails, does it just misbehave with respect to your application routines, or are you getting an error being raised of some sort?
Thanks,
- Bryan Shumsky
Hi, Don. Are you hoping that a _different_ engine can close down a "badly attached" engine, or that the bad engine ITSELF can close itself down?
Also, when the "problem engine" fails, does it just misbehave with respect to your application routines, or are you getting an error being raised of some sort?
Thanks,
- Bryan Shumsky
Hi Bryan. I'm hoping the _current_ (or _problem_) engine can close itself down. In our own (non-HTTP aware) engine server farm utility, an engine can send a special keyword back to our engine server via Send_Info. Our engine server allows the engine to end its processing gracefully and then the engine server submits a request to RevCAPI to close that engine. This would be an ideal solution in our current situation with the OEngineServer.
When the _problem_ engine fails, it results in two types of FS errors. One is FS145 (table does not exist). The other is FS1105 (RCL error). Both make sense. FS145 would likely occur when the table had not ever been attached and when a program attempts to attach it, it fails. FS1105 would likely occur after a handle had alreaady been created but then an attempt to perform an operation (Read, Write, etc.) was attempted after the database server disconnected.
In the case of an FS145 error, we are considering building a solution that waits a few seconds and then attempts to attach the table again in hopes that the database server becomes available again. In the case of an FS1105 error, we can also try to re-attach the table, but I'm not sure if that would ever be successful. In either case, if it becomes impossible to re-connect to the database server I would simply like to have the the current engine get recycled. Ideally, the engine would be allowed to send a response back to the OECGI so the client can be notified that there is a temporary server issue (I would probably send back an HTTP 503 status code). This way the HTTP client can choose to attempt another request, which would then be routed to a new engine.
Hi, Don. Thanks for the response.
Is this all running on OI 10, or OI 9.x?
- Bryan Shumsky
Hi, Don. Thanks for the response.
Is this all running on OI 10, or OI 9.x?
- Bryan Shumsky
This is OI 9.x.
I am currently testing using the "debugger intercept" functionality with "phantom" processes (those regularly scheduled tasks that run via the engine server) to capture errors and resume operation at the "invoking" routine (rather than aborting entirely). This has required a modified version of the OEngine, which Carl has graciously put together for me in both OI 9.4 and OI 10 versions.
I wonder if similar functionality would work for you, as well? What I envision is that, on startup, your oengine would set itself to an alternate debugger routine, and as part of that routine you would log any errors and then abort back to the original invoking routine (which I guess would be RUN_OECGI_REQUEST or whatever you use in its place).
If you think this might work for you and are interested in testing it, let me know via email and we'll set something up.
Thanks!
- Bryan Shumsky
I am currently testing using the "debugger intercept" functionality with "phantom" processes (those regularly scheduled tasks that run via the engine server) to capture errors and resume operation at the "invoking" routine (rather than aborting entirely). This has required a modified version of the OEngine, which Carl has graciously put together for me in both OI 9.4 and OI 10 versions.
I wonder if similar functionality would work for you, as well? What I envision is that, on startup, your oengine would set itself to an alternate debugger routine, and as part of that routine you would log any errors and then abort back to the original invoking routine (which I guess would be RUN_OECGI_REQUEST or whatever you use in its place).
If you think this might work for you and are interested in testing it, let me know via email and we'll set something up.
Thanks!
- Bryan Shumsky
I'll follow-up via email, but I'm curious if this "alternate debugger routine" is something I would (or could) write or if this is something you would provide me. We already have an intercept debugger routine in place, so I just want to confirm that we would still have the ability to perform our own debugger intercept logic, whether in our routine or one that you would provide.
This post is to let everyone know of the wonderful solution Bryan was able to provide me. The following code (which will be shipped with OI 10, but this will work for OI 9.4) will allow the current engine to gracefully terminate (thus it can still respond with an appropriate HTTP status code, description, body, etc.) and a new engine will immediately launch:
Subroutine RTI_ESERVER_EXITENGINE(VOID) Equ COMMANDCMD$ To \08\ Equ DELIMITER$ To \01\ Declare Function revcmd_input CMDTYPE = "-4" ;* TELL THE CURRENT ENGINE TO REMOVE ITSELF PASSWORD = "REVSOFT" ;* the engine server “admin” password OUTSTR = commandcmd$ : "S" : delimiter$ : CMDTYPE : DELIMITER$ : PASSWORD : DELIMITER$ : "X" : DELIMITER$ dummy = REVCMD_INPUT(OUTSTR) Return 0Many thanks to Bryan for researching this and making this available.
Hi, Don. Glad that worked for you!
Please note that, since you can call this at any point, if you need to wait for your network to be "active" again (so that the oengine that gets created next doesn't face the same problem), you can do whatever you need to before invoking this and returning from your stproc - you might put in a delay, for example, or more proactively somehow "ping" your other machine to make sure it's alive before you terminate the current process.
- Bryan Shumsky