Notifications
Clear all
Topic starter
We are having some extreme slowness issues that we have been working closely with GE with for 2 weeks.
We recently relocated our JBOSS, eprescribing and DTS to a location about 109 miles away from the EMR server. We have been told the connection speed is plenty fast for them to communicate, but I was wondering if anyone else has experienced this same set of circumstances and what your experience was? We can't find another source of the slowness and it's getting to the "I'll quit if you don't fix this" point.
Posted : February 3, 2017 1:13 am
Well, we've got 1 Gig bandwidth between ours, what have you got?
My expectation is that there is quite a lot of traffic between Jboss and the database (I am assuming that is what you mean by EMR Server) so I wouldn't want them remote from each other.
Steve
Posted : February 3, 2017 1:25 am
I agree with swestfisher I don't think I would have them separated, I would rather have them together, to many unforeseen variables.
Posted : February 3, 2017 2:49 am
Topic starter
10 gig, low latency...
but the emr at this point, is almost unusable for some!
Posted : February 3, 2017 3:03 am
I would first look at the bandwidth you are using, both between your JBoss server and your database server as well as clients to JBoss. Are they fat clients, terminal services, Citrix?
What was moved, database or JBoss?
Steve
Posted : February 3, 2017 3:10 am
Topic starter
DB stayed and some citrix stayed.
One loc: DB, some citrix (we have 15 total with that db)
Moved to New loc: JBOSS, App1, App2, DTS, ePrescribing
Posted : February 3, 2017 4:23 am
As I mentioned I would monitor bandwidth usage between JBoss and the database server, as that 10 Gig WAN would concern me. Any other changes/upgrades? If you built new servers when the change was made go through all the best practices and GE recommendations for the servers and make sure they are properly configured.
Steve
Posted : February 3, 2017 4:28 am
I agree with the general direction of the others as well. This smells like a latency issue.
The CPS client is super chatty and loves exchanging way too much data between itself, JBoss, and (the last time I did a packet capture) the database server itself. If the client queries JBoss, then waits on the results to query the database, then waits on the results to query JBoss again, your latency values can ramp up pretty quickly.
You need a network specialist that can capture packets, check interface stats on the server(s), switches, routers, etc and knows how to interpret that data.
Posted : February 3, 2017 6:10 am
We recently tried this, but the other way around. We moved our Citrix virtual machines that run the Centricity client about 300 miles away. This added about 10-20ms latency, and the decrease in performance was dramatic. After watching the packets going back and forth, we believe it's due to inefficient communication between the client and JBOSS. A lot of messages could be bundled together and sent as one big packet, but GE sends these all separately.
Posted : February 6, 2017 2:58 am
Even if the bandwidth is good, like 100/1000mbit, the distance of the circuit is the problem. Ethernet has a 100 meter (~328 feet) maximum distance for gigabit speed and that is what is supported by GE. Even if this circuit is 1000mbit fiber (not Ethernet) traveling at the speed of light the distance causes latency for each packet of data and that causes slower performance.
I have a 100 mbit circuit to a facility less than 100 miles from here. I have only ever seen about 40 mbit of actual throughput with 8ms of latency using iperf ( http://iperf.fr) to measure. The number of negotiations for each packet is the issue. For example I can transfer a single large file, like 500mbit faster than my ccc and clinicfoldername folders even though they are only 40mb of data because there are over 1600 tiny files in those directories, each has to be negotiated separately during transfer.
If you haven't already, I am guessing you'll be moving that server back real soon. The GE application was not designed for this setup.
Mike Zavolas
Tallahassee Neurological Clinic
Posted : February 6, 2017 9:52 am
I agree--the JBOSS and the SQL Server chat quite a bit back and forth and the distance between the two is a concern--you mention 10GB WAN, but I assume this is not dedicated meaning you have to share the traffic with others.
One potential item to check is the network adapter settings on both servers--is it possible they are operating at lower than capacity?
Posted : February 7, 2017 3:29 am
I agree that the distance could be part of the problem, but we have seen some serious latency issues with our JBoss and Database servers both being Virtual Servers in the exact same environment. One of the things I do after every upgrade (including Service Pack upgrades), is go through our Database looking for missing indexes. I have a script that identifies processes that could possibly benefit from creating a new index. I then review the index to see if I believe that it would be beneficial and then create it if I deem it necessary. After doing this our latency issues have drastically dropped.
Posted : February 10, 2017 9:47 am
Could you share the script?
Posted : February 10, 2017 10:08 am
Topic starter
Thanks everyone! It was DEFINITELY the issue. We moved the db server to the new location, and the remainder of the citrix servers and all is well...lesson learned!
Posted : February 13, 2017 4:48 am
Here is the script. Just keep in mind that the index creates a partial copy of the table, based on number of fields you use in the index, so large tables can considerably increase the size of your database. Also, not all queries will benefit from additional indexes. I usually just create ones identified as high percentage and I usually skip really small tables unless I expect that table to eventually grow large. Also, if you are creating indexes on large tables, you'll want to wait until after hours because it does lock the table while the index is building.
SELECT db.[name] AS [DatabaseName],
id.[statement] AS [FullyQualifiedObjectName],
gs.[avg_user_impact] AS [AvgUserImpact],
'CREATE NONCLUSTERED INDEX [<Name of Missing Index, CentricityPS>]'+' ON '+ id.[statement] +' ('+ISNULL(id.[equality_columns],'')
+CASE
WHEN id.[equality_columns] ISNOTNULL
AND id.[inequality_columns] ISNOTNULL
THEN','
ELSE''
END+ISNULL(id.[inequality_columns],'')+')'+ISNULL(' INCLUDE ('+ id.[included_columns] +')','')AS [ProposedIndex],
CAST(CURRENT_TIMESTAMPAS [smalldatetime])AS [CollectionDate]
FROM [sys].[dm_db_missing_index_group_stats] gs WITH (NOLOCK)
INNERJOIN [sys].[dm_db_missing_index_groups] ig WITH (NOLOCK)
ON gs.[group_handle] = ig.[index_group_handle]
INNERJOIN [sys].[dm_db_missing_index_details] id WITH (NOLOCK)
ON ig.[index_handle] = id.[index_handle]
INNERJOIN [sys].[databases] db WITH (NOLOCK)
ON db.[database_id] = id.[database_id]
ORDERBY db.name, id.statement, gs.[avg_user_impact] desc
Posted : February 14, 2017 10:15 am
