OEngine/Java locking up (OEngine Server)
At 25 SEP 2023 03:42:07PM John Bowen wrote:
We have an API that handles all .NetOI calls from Sql and .Net processes. Some of the eserver.cfg settings are below. OEngine runs as normal most of the day. Multiple engines will spin up as needed. Java.exe starts and can be seen in the Task Manager. At some point during the data a second Java.exe with a different PID is started. When this happens some of the engines start locking up.
I have a PowerShell script that I run to stop Oengine and Java. The problem that I'm having is that only one of the Java instances will stop. I cannot restart the Oengine until I manually stop the Java.exe through the Task Manager.
Has anyone else had similar issues with multiple Java's starting up and locking up Oengine? We are using an OpenSource version of Java on our server.
MaxEngines=30
MaxConnections=0
IdleTimeout=15
IdleCheck=60
MaxRunTime=0
MaxUpTime=30
At 25 SEP 2023 04:28PM bshumsky wrote:
Hi, John. Have you tried running this when the engine server is started in 'debug' mode (from a DOS cmd prompt), rather than as a service? You might be able to see if there is a 'crash' or other problem that way.
Which version of OI are you using this with? In the OESERVER\logs folder, you might find some log entries that could give you some more information as well.
Hope that helps,
- Bryan Shumsky
At 25 SEP 2023 04:40PM John Bowen wrote:
Bryan,
Thanks for the quick response.
We are running OI 10.2.
Should I try running the engine server in debug mode after I notice that there are two Java instances running? I don't think that I want to be running it in debug mode all day just to see if I can see when the issue happens.
I'll review the logs to see if anything pops.
John
At 25 SEP 2023 04:47PM John Bowen wrote:
Below is a snipit of the oeserver log. The oengine server was started yesterday morning. This morning around 8:00am it locked up. I restarted the Oengine when I got an error email around 9:30.
INFO|1348/0|Service OEngineServer (OpenJDK)|23-09-24 07:34:10|Started at 2023-09-24 07:34:10
INFO|1348/0|Service OEngineServer (OpenJDK)|23-09-25 08:05:53|Waiting…currently cannot create required engine without exceeding maximum allowed
INFO|1348/0|Service OEngineServer (OpenJDK)|23-09-25 08:06:19|Waiting…currently cannot create required engine without exceeding maximum allowed
INFO|1348/0|Service OEngineServer (OpenJDK)|23-09-25 08:06:50|Waiting…currently cannot create required engine without exceeding maximum allowed
INFO|1348/0|Service OEngineServer (OpenJDK)|23-09-25 08:07:15|Waiting…currently cannot create required engine without exceeding maximum allowed
INFO|1348/0|Service OEngineServer (OpenJDK)|23-09-25 08:09:15|Waiting…currently cannot create required engine without exceeding maximum allowed
INFO|1348/0|Service OEngineServer (OpenJDK)|23-09-25 08:09:15|Waiting…currently cannot create required engine without exceeding maximum allowed
INFO|1348/0|Service OEngineServer (OpenJDK)|23-09-25 08:16:54|Waiting…currently cannot create required engine without exceeding maximum allowed
At 25 SEP 2023 05:07PM bshumsky wrote:
Hi, John.
How often are you experiencing these lock-ups? I'd suggest you need to run the engine server in debug mode for at least as long as it's taking, on average, to see the problem. You should also set the startup and shutdown flags both to '1' in your NetOI calls so that your oengines are "visible".
From the log snippets, it looks like you've got the maximum number of engines active and new requests are queueing. While it's certainly possible that your system is so active that that's really the case, it's usually much more likely that you've got something 'crashing' in your oengines that are causing these bottlenecks, and perhaps having visual engines in the debug engine server will let you see what's going on.
Hope that helps,
- Bryan Shumsky
At 10 OCT 2023 04:00PM John Bowen wrote:
Bryan,
We're usually experiencing the lock-ups once daily. It usually happens late in the afternoon.
Based on your suggestions, I've been reviewing Stored Procedures that are called through the oengine to check for processes that are calling msg(), fsmsg() etc.
I haven't run the Oengine in debug mode yet. I don't want to affect performance on the server. If all else fails, I'll run in debug mode.
While reviewing eServer Configuration I noticed a possible issue. For the Background Processes, does the 'Application to Run In' need to be the same as the application that the Oengine is running in? We have one Application account that the Oengine is running in that interacts with a Sql database. The main Application account is where the Arev64 UI is running.
At 16 OCT 2023 11:24AM bshumsky wrote:
Hi, John. When the background process starts up, it will log in an addition engine (or engines) to the specified application, so it doesn't need to be the same as the application used by your other web requests.
Another option would be to set up a separate service to just run the background tasks. Also, this might be a good way to figure out whether the background tasks, or the main web requests, are causing the lockup…? You could do this by taking out the background tasks from your eserver.cfg and creating a new eserver_background.cfg file which contained just the relevant pieces. You could then run the background tasks either as a service, or just via the command line (while testing).
If you'd like some help setting that up, email info at revelation.com with your eserver.cfg and we can help configure that.
Regards,
- Bryan Shumsky