5 weeks in a row, when our CPS app server reboots, jboss fails to come up. We have one doctor who likes to catch up on saturday mornings and he pages IT emergency on call. GE's fix is to restart jboss until it comes up. Really? Sometimes we have to restart the server one more time. One GE support tech said to change the windows service to delayed start. Well that didn't work. Anyone encounter this? Any recommendations?
Make sure that your not rebooting JBoss at the same time SQL is rebooting.. We've seen if SQL is away when JBoss comes back up JBoss goes into an error state until it can be rebooted again. We stagger our SQL/JBoss restarts by about 30 minutes just in case.
Good call - I thought about checking that, but it was fine before the v12 upgrade. What's your reboot order? DB --> App server --> DTS --> Term servers?
because TS have no bearing on the first three, that's not part of our equation, but what you have there works fine, we give enough buffer time before each service does a restart to ensure the previous had enough time to finish.
delayed start should work if the delay is long enough, or don't rely on the service to start it, set it to manual start, then set a schedule task to start it 5 or 10 minutes after the server comes back up.
I have to giggle when ever I am listening to someone at GE and their way to tell if JBOSS is done starting or stopping is to "watch the processor activity".
You should check the logs for a successfully start or shutdown. or the jboss admin console.
If you cant get it to shutdown, kill the java process.
We have noticed that after the JBoss service starts, JBoss itself is not ready for all functions for quite a few minutes. It does thrash the multiple processors we have pretty hard during this startup interval. I'm curious why you are rebooting the server once a week and not just bouncing the service itself. We reboot once a month during our Microsoft Patch Night and only bounce JBoss if we need to reset it due to some CQR reason (which have been many). We are EMR 9.8.7.
It was set up this way in the past and I'm pretty sure I read that a weekly reboot is suggested? Can anyone confirm?
I can confirm based on conversations with GE engineers, the spec is a once a week reboot of SQL and Jboss (i've even had certain engineers indicate a more frequent reboot schedule for jboss..)
Just so I don't get those Sunday calls I moved my weekly reboots to Monday mornings just before people are scheduled to come in but after the backup. If the backup happens to be interrupted then it is no big deal as the Sunday backup would suffice in any DR situation
To answer the questions about why rebooting the server instead of the service, I will say that it is always better to reboot the server for Microsoft purposes. Before virtual servers a cold reboot would even be more preferable to a reboot.
I also use baretail to watch the jboss log file to see when it is actually loaded. It is similar to the tail linux command.
Mike Zavolas
Tallahassee Neurological Clinic
For the sake of simplicity, and eliminating it as a potential cause for issues, I have JBoss and SQL rebooting nightly after each is backed up (Acronis). I have noticed that JBoss itself can sometimes take 10 to 15 minutes to be fully restarted and ready for use. This has been the case from CPS 10 through 12. With JBoss rebooting at 4AM, it has never been an issue.
I typically restart our DB server on it's own. If jboss needs to be booted also that is done after the DB.
Each has it's own dedicated server so i go:
DB->JBoss-> DTS
Staggering each server 30 minutes worked for 2 weeks, but not last week. I wrote a vbscript that will check for the http response code of 500. If it encounters it, it will stop jboss, wait 3 minutes and then start it again. It will then wait 3 more minutes to check it again. I'll keep you updated.
It's sad that it's 2015 and we are still being told by a vendor that we pay six figures to for support/maintenance on a yearly basis to reboot daily or weekly as a "solution." I guess that means Centricity is a no go for a 24/7 clinic. Seriously - we don't do this with other SQL apps on Windows. They can run for months or more without any issues. We aren't dealing with Windows 3.1 anymore.
Where the heck is service pack 8 which was supposed to address even the top 15 of the top 40 listed bugs (of the hundreds that probably exist and have existed for years)?
Watch them release it a couple days before the next CHUG conference so they can tell everyone that asks this question that "everything is fixed in SP8 so go home and try it." I'm sure we'lll be disappointed in the quality of that release as well.
@mdoak - I feel the same exact way. Hire more programmers if you have all these problems. I'm tired of telling my colleagues there's nothing I can do when CPS crashed 6 times a day. I check everyday for SP8. Our hospital EMR needs daily reboots at 2am. Are you serious? WTF?
Hi:
We are experiencing major similar JBoss issues and have read your answers and can relate.
Typical response is "reboot" and call if you have any more problems. In meantime our practice goes down and doctors get upset.
Has any of the Service Paks corrected this problem? We are on 12.0.7 right now and have been told that 10 might?? help.
Any new feedback you all can provide would be great.
Thanks.
Michelle