[BUG] Accessing Web Dashboard from cluster VIP or directly from server IP that has VIP creates CLOSE_WAIT connections that make mirth connect run slow and eventually crash.

[shanedewitt]

Describe the bug
Opening a browser to vip ip address on port 8080 or server that has the vip on port 8080 and using web dashboard leaves connection in CLOSE_WAIT status and makes mirth connect and appliance control panel run slow, eventually crashing.

To Reproduce

1. Use a web browser to connect to vip ip address on port 8080 or connect to server ip address that has vip on port 8080.
2. Open expert on server and search for CLOSE_WAIT. Note the number of CLOSE_WAITS.
3. Repeated process from another device. Notice the number of CLOSE_WAIT's increase.
4. Continue at you leisure, notice how control panel now takes ~2 minutes to finish loading. Adding users or changing their password will give you a 500 status error. Eventually mirth will crash.

Environment (please complete the following information):
Appliance Version 3.11.4
Mirth Connect Version | 3.9.1
Java 7 (64 bit) Version | 1.7.0_151-b15
Java 8 (64 bit) Version | 1.8.0_181-b13
OpenJDK 11 (64 bit) Version | 11+28
PostgreSQL Version | 9.4.5
Operating System | CentOS Linux release 7.2.1511 (Core)
Workaround(s)
Are there one or more workarounds for this issue currently?

Additional context
Accessing web dashboard from appliance that does not have vip seems to not create CLOSE_WAITS.

[mcanalld]

Had a similar issue running Centos 6 with 3.9.1, OpenJDK 8 275. CLOSE_WAITS were occurring and couldn't hit webpage or download JNLP. We had other systems with same setup but didn't have a problem. That's when we found that the two servers having issue were running kernel version - 2.6.32-754.35.1.el6.x86_64. The ones not having a problem were running - 2.6.32-358.el6.x86_64. Downgrading the kernel version on the problem servers fixed the problem.

FYI, we tried using different java versions (9, 11). Upgrading to latest mirth build 3.11. Nothing worked. Downgrading to Mirth 3.4.2 did work. Very odd issue, and maybe not be related to you since you are on Centos 7.

[shanedewitt]

This is actually a known issue cause by jetty library. Fix was to have mirth support downgrade jetty library.

KI Number | KI196763

Description: Due to a known issue with Jetty Server versions 9.4.21.v20190926 -> 9.4.23, HTTP Responses hang. As a result, Mirth Connect becomes inaccessible until the server is restarted. A thread dump points to a blocked HTTP Response from the ResponseWriter class.
Jetty Issue: jetty/jetty.project#4279

Actual Outcome: After some time, Mirth Connect will become inaccessible via the Administrator Client before ultimately timing out due to never receiving a response.
Expected Outcome: The Mirth Connect Administrator should not hang waiting for a response from the server.
Setup Steps (Pre-reqs): Install Mirth Connect 3.9.0 or later.
Steps to Reproduce: N/A - this behavior is sporadic.
Reported Site/Versions: 3.9.0, 3.9.1, 3.10.0, 3.10.1
Is this reproducible in the latest version? (If yes, include version): Yes, 3.10.1

Additional Info:
Browser(s) tested:
What is the workaround (if applicable)?: Rollback to Jetty Server version 9.4.9.20180320. Jetty version 9.4.23 could not be used in place of 9.4.9.v20190926 at this time, errors are thrown.
Regression issue (list details): N/A
Log output / stacktrace(s): N/A
Source Code Notes: N/A

Steps to Recreate
Setup Steps (Pre-reqs): Install Mirth Connect 3.9.0 or later.
Steps to Reproduce: N/A - this behavior is sporadic.

Work Around
What is the workaround (if applicable)?:
Rollback to Jetty Server version 9.4.9.20180320.
Jetty version 9.4.23 could not be used in place of 9.4.9.v20190926 at this time, errors are thrown.

[tonygermano]

@shanedewitt why did you close this? A workaround to manually downgrade libraries is not a fix.

The current version of Connect is 3.11.0, and it still includes Jetty version 9.4.21.v20190926. This still needs to be addressed in future versions.

[shanedewitt]

I agree, I re-opened issue.