99% CPU usage #5939
Replies: 5 comments 16 replies
-
|
Partial GC logs
|
Beta Was this translation helpful? Give feedback.
-
|
2023-09-27 13:36:03 "RMI TCP Connection(5)-192.168.12.53" #299 daemon prio=5 os_prio=0 tid=0x000000001e130000 nid=0x62c4 runnable [0x000000002846d000] "qtp286935302-298" #298 prio=5 os_prio=0 tid=0x000000001e127000 nid=0x5e1c runnable [0x000000002292e000] "qtp286935302-297" #297 prio=5 os_prio=0 tid=0x000000001e12f800 nid=0x72bc waiting on condition [0x00000000218bf000] "RMI TCP Connection(3)-192.168.12.53" #296 daemon prio=5 os_prio=0 tid=0x000000001e12e800 nid=0x762c runnable [0x000000002836e000] "RMI TCP Connection(4)-192.168.12.53" #294 daemon prio=5 os_prio=0 tid=0x000000001e12e000 nid=0x6898 runnable [0x000000002633e000] "JMX server connection timeout 293" #293 daemon prio=5 os_prio=0 tid=0x000000001e12d000 nid=0x6134 in Object.wait() [0x0000000025b3f000] "RMI Scheduler(0)" #292 daemon prio=5 os_prio=0 tid=0x000000001e125800 nid=0x5dd4 waiting on condition [0x000000002583e000] "RMI TCP Connection(1)-192.168.12.53" #291 daemon prio=5 os_prio=0 tid=0x000000001e126800 nid=0x7654 runnable [0x000000002573e000] "RMI TCP Accept-0" #290 daemon prio=5 os_prio=0 tid=0x000000001e12c800 nid=0x5354 runnable [0x000000002302e000] "qtp286935302-289" #289 prio=5 os_prio=0 tid=0x000000001e12a000 nid=0x2e68 waiting on condition [0x000000000118e000] "qtp286935302-182" #182 prio=5 os_prio=0 tid=0x000000001e128000 nid=0x72d0 runnable [0x000000002282e000] "qtp1425465485-180" #180 prio=5 os_prio=0 tid=0x000000001e12b800 nid=0x6228 waiting on condition [0x000000002826f000] "qtp1425465485-179" #179 prio=5 os_prio=0 tid=0x000000001e12b000 nid=0x6444 waiting on condition [0x000000002816e000] "qtp1425465485-178" #178 prio=5 os_prio=0 tid=0x000000001e451000 nid=0x6cac waiting on condition [0x000000002806e000] "qtp1425465485-177-acceptor-0@6137d474-ServerConnector@7cd9b2b{HTTP/1.1, (http/1.1)}{0.0.0.0:9996}" #177 prio=3 os_prio=-1 tid=0x00000000186a1000 nid=0x372c runnable [0x000000002743f000] "qtp1425465485-176" #176 prio=5 os_prio=0 tid=0x00000000186a6000 nid=0x44f0 runnable [0x000000002723e000] "qtp1425465485-175" #175 prio=5 os_prio=0 tid=0x00000000186a2800 nid=0x662c runnable [0x000000002713e000] "qtp1425465485-174" #174 prio=5 os_prio=0 tid=0x000000001f476800 nid=0x4f4 runnable [0x000000002703e000] "qtp1425465485-173" #173 prio=5 os_prio=0 tid=0x000000001f46f000 nid=0x6d74 runnable [0x0000000026f3e000] "qtp428645053-172-acceptor-0@5ddd21f8-ServerConnector@469bba7c{HTTP/1.1, (http/1.1)}{0.0.0.0:9999}" #172 prio=3 os_prio=-1 tid=0x000000001f475800 nid=0x50b0 runnable [0x0000000026e3e000] "qtp428645053-171" #171 prio=5 os_prio=0 tid=0x000000001f472000 nid=0x43e0 waiting on condition [0x0000000026d3e000] "qtp428645053-170" #170 prio=5 os_prio=0 tid=0x000000001f46e000 nid=0x2ad8 runnable [0x0000000026c3e000] "qtp428645053-169" #169 prio=5 os_prio=0 tid=0x000000001f46f800 nid=0x6390 runnable [0x0000000026b3e000] "qtp428645053-168" #168 prio=5 os_prio=0 tid=0x000000001f475000 nid=0x5a8 waiting on condition [0x0000000026a3e000] "qtp428645053-167" #167 prio=5 os_prio=0 tid=0x000000001f470800 nid=0x4ce4 runnable [0x000000002693e000] "qtp428645053-166" #166 prio=5 os_prio=0 tid=0x000000001f474000 nid=0x6c54 runnable [0x000000002683e000] "qtp428645053-165" #165 prio=5 os_prio=0 tid=0x000000001f473800 nid=0x61a4 waiting on condition [0x000000002673e000] "qtp286935302-162" #162 prio=5 os_prio=0 tid=0x0000000020d88000 nid=0x2344 runnable [0x000000002643e000] "qtp286935302-92" #92 prio=5 os_prio=0 tid=0x0000000020d85000 nid=0x3990 runnable [0x000000000138e000] "Session-Scheduler-1e2c855a-1" #87 prio=5 os_prio=0 tid=0x0000000020d84800 nid=0x3678 waiting on condition [0x0000000025a3f000] "Connector-Scheduler-5e280c60-1" #82 prio=5 os_prio=0 tid=0x0000000020d81800 nid=0x1f54 waiting on condition [0x000000000128e000] "Java2D Disposer" #78 daemon prio=10 os_prio=2 tid=0x0000000020d83800 nid=0x56f4 in Object.wait() [0x0000000027f6e000] "Timer-2" #77 prio=5 os_prio=0 tid=0x000000001f47c800 nid=0x4b38 in Object.wait() [0x0000000027c6f000] "qtp1937643607-76" #76 prio=5 os_prio=0 tid=0x000000001f47a000 nid=0x6598 waiting on condition [0x0000000027b6f000] "qtp1937643607-75" #75 prio=5 os_prio=0 tid=0x000000001f478800 nid=0x4a4 runnable [0x0000000027a6f000] "qtp1937643607-74" #74 prio=5 os_prio=0 tid=0x000000001f479800 nid=0x6988 waiting on condition [0x000000002796e000] "qtp1937643607-73-acceptor-0@3755dfe8-ServerConnector@533e2b8e{HTTP/1.1, (http/1.1)}{0.0.0.0:18020}" #73 prio=3 os_prio=-1 tid=0x000000001f47b800 nid=0x1ff8 runnable [0x000000002786e000] "qtp1937643607-72" #72 prio=5 os_prio=0 tid=0x000000001f477000 nid=0x2454 waiting on condition [0x000000002776e000] "qtp1937643607-71" #71 prio=5 os_prio=0 tid=0x000000001f47d000 nid=0x1208 runnable [0x000000002766e000] "qtp1937643607-70" #70 prio=5 os_prio=0 tid=0x000000001f47b000 nid=0x4e5c runnable [0x000000002756e000] "qtp1937643607-69" #69 prio=5 os_prio=0 tid=0x000000001f478000 nid=0x3c7c runnable [0x000000002733e000] "DashboardConnectorEventListener Consumer Thread" #39 prio=5 os_prio=0 tid=0x0000000020d7c000 nid=0x2e04 waiting on condition [0x0000000023bee000] "Statistics Updater Thread" #38 prio=5 os_prio=0 tid=0x0000000020d7d000 nid=0x6b18 waiting on condition [0x000000002384f000] "qtp286935302-37-acceptor-0@7c555c7a-sslconnector@5e280c60{SSL, (ssl, http/1.1)}{0.0.0.0:18443}" #37 prio=3 os_prio=-1 tid=0x00000000186a7000 nid=0x7634 runnable [0x000000002374e000] "Session-HouseKeeper-743e6c25-1" #35 prio=5 os_prio=0 tid=0x00000000186a4800 nid=0x7734 waiting on condition [0x0000000022f2e000] "qtp286935302-34" #34 prio=5 os_prio=0 tid=0x00000000186a4000 nid=0x774c runnable [0x0000000022c2f000] "qtp286935302-33" #33 prio=5 os_prio=0 tid=0x00000000186a0000 nid=0x7588 runnable [0x0000000022b2f000] "qtp286935302-32-acceptor-0@73eccfac-connector@2ef3315{HTTP/1.1, (http/1.1)}{0.0.0.0:18080}" #32 prio=3 os_prio=-1 tid=0x00000000186a3000 nid=0x70a8 runnable [0x0000000022a2f000] "qtp286935302-29" #29 prio=5 os_prio=0 tid=0x00000000186a1800 nid=0x3d80 runnable [0x0000000021c4e000] "qtp286935302-27" #27 prio=5 os_prio=0 tid=0x000000001e450000 nid=0x54c0 runnable [0x00000000215be000] "Timer-1" #26 prio=5 os_prio=0 tid=0x000000001e44f800 nid=0x201c in Object.wait() [0x00000000212bf000] "DataPruner_QuartzSchedulerThread" #25 prio=5 os_prio=0 tid=0x000000001e44e800 nid=0x7384 in Object.wait() [0x00000000217bf000] "DataPruner_Worker-1" #24 prio=5 os_prio=0 tid=0x000000001e44e000 nid=0x10ac in Object.wait() [0x00000000216bf000] "HikariPool-2 housekeeper" #22 daemon prio=5 os_prio=0 tid=0x0000000018fec000 nid=0x5604 waiting on condition [0x00000000214be000] "HikariPool-1 housekeeper" #20 daemon prio=5 os_prio=0 tid=0x000000001f29e800 nid=0x5740 waiting on condition [0x00000000211be000] "derby.rawStoreDaemon" #19 daemon prio=5 os_prio=0 tid=0x0000000019004800 nid=0x68d0 in Object.wait() [0x0000000020ebe000] "Timer-0" #18 daemon prio=5 os_prio=0 tid=0x0000000018d2d800 nid=0x77c in Object.wait() [0x000000001d78e000] "mysql-cj-abandoned-connection-cleanup" #17 daemon prio=5 os_prio=0 tid=0x0000000018c05000 nid=0x75e4 in Object.wait() [0x000000001d68e000] "Main Server Thread" #13 prio=5 os_prio=0 tid=0x0000000018516800 nid=0x3410 waiting on condition [0x000000001c3bf000] "DefaultAlertWorker Consumer Thread" #15 prio=5 os_prio=0 tid=0x0000000019cce800 nid=0x5720 waiting on condition [0x000000001c2bf000] "AuditableEventListener Consumer Thread" #14 prio=5 os_prio=0 tid=0x000000001841f800 nid=0xfc0 waiting on condition [0x000000001c1be000] "Service Thread" #10 daemon prio=9 os_prio=0 tid=0x0000000018460000 nid=0x6a44 runnable [0x0000000000000000] "C1 CompilerThread3" #9 daemon prio=9 os_prio=2 tid=0x0000000016851000 nid=0x2770 waiting on condition [0x0000000000000000] "C2 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x0000000016850000 nid=0x6dc0 waiting on condition [0x0000000000000000] "C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x0000000016849000 nid=0x2dd8 waiting on condition [0x0000000000000000] "C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x0000000016845000 nid=0x24b4 waiting on condition [0x0000000000000000] "Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x0000000016833000 nid=0x463c waiting on condition [0x0000000000000000] "Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000016830000 nid=0x6474 runnable [0x0000000000000000] "Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000167c2800 nid=0x60c4 in Object.wait() [0x0000000017d8f000] "Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000167bb800 nid=0x6ef0 in Object.wait() [0x0000000017c8f000] "main" #1 prio=5 os_prio=0 tid=0x0000000000fc6800 nid=0x74bc waiting on condition [0x0000000000000000] "VM Thread" os_prio=2 tid=0x0000000015b9d000 nid=0x6268 runnable "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000000fdf800 nid=0x434 runnable "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000000fe0800 nid=0x6570 runnable "GC task thread#2 (ParallelGC)" os_prio=0 tid=0x0000000000fe2000 nid=0x68e0 runnable "GC task thread#3 (ParallelGC)" os_prio=0 tid=0x0000000000fe3000 nid=0x281c runnable "GC task thread#4 (ParallelGC)" os_prio=0 tid=0x0000000000fe5000 nid=0x1138 runnable "GC task thread#5 (ParallelGC)" os_prio=0 tid=0x0000000000fe6000 nid=0x4c5c runnable "GC task thread#6 (ParallelGC)" os_prio=0 tid=0x0000000000fe9000 nid=0x2f20 runnable "GC task thread#7 (ParallelGC)" os_prio=0 tid=0x0000000000fea000 nid=0x3e58 runnable "VM Periodic Task Thread" os_prio=2 tid=0x0000000018474000 nid=0x2e48 waiting on condition JNI global references: 704 |
Beta Was this translation helpful? Give feedback.
-
|
I ran your thread dump through an analyser - https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjMvMDkvMjcvb3VybGFuZy50aHJlYWQuZHVtcC50eHQtLTE2LTM2LTIw& I dont see any common problems. No deadlocks. No blocked threads. How much memory are you giving Mirth? What is your XMX setting? How many DB connections is Mirth configured to use to SQL server? A search for the DLL you showed has a lot of results. |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
My service has a total of 2 channels and 6 destinations |
Beta Was this translation helpful? Give feedback.
-
|
Beta Was this translation helpful? Give feedback.
-
|
20 DB connections for 2 channels is a lot. Reduce it to 10. Your memory settings are good. Task Manager shows ~225MB used out of 2GB allowed. That is good. What source connector types do your channels use? Pollers? TCP listeners? |
Beta Was this translation helpful? Give feedback.
-
|
source connector is http listeners。 2 channels have been modified to10 DB ，No effect after modification。 CPU usage or 95% |
Beta Was this translation helpful? Give feedback.
-
|
Based on the thread dump it looks like you have multiple HTTP listeners. On my test machine each HTTP listener creates about 8 threads. So doing 4 HTTP Listener channels will cause my CPU to spike around 80-90% at Mirth service startup. but settles down after a little bit. |
Beta Was this translation helpful? Give feedback.
-
|
Have you replicated this problem on a clean install of Windows and Mirth? And on Linux do you see the same problem? I'd expect Mirth to settle down unless your channels are busy. |
Beta Was this translation helpful? Give feedback.
-
|
@ourlang Mirth doesn't support configuring the size of the thread pool used by HTTP Listeners. That means even just an empty Mirth instance will create 8 threads at startup for the embedded Jetty server that provides the Mirth API and WebStart endpoints. What I see from the source code, each destination creates a thread as well. It would be interesting to see a fork that runs on JDK 21, Jetty 12, and uses virtual threads to see if there's a CPU spike at startup. My test machine only had 2 vCPUs so seeing the CPU spike makes sense for 4 HTTP Listener channels. |
Beta Was this translation helpful? Give feedback.
-
|
@pacmano1 This issue has been replicated on the computer on the new Windows, and Linux has not been tested yet |
Beta Was this translation helpful? Give feedback.
-
|
@jonbartels @pacmano1 @ab-mg-23 @gbortis After each restart, the CPU usage is almost 1%, and after approximately 4 hours of use, the CPU usage is 99%. The channel only handles data processing and forwarding, without any database operations |
Beta Was this translation helpful? Give feedback.
-
|
Show the output of Show the MC "connection log" panel when the CPU usage is high. |
Beta Was this translation helpful? Give feedback.
-
|
We've been experiencing exactly same issue for months now. Ever since we updated to 4.3. We are running Live production server with 130 channels and TONS of destination. It is a busy one - but has plenty of resources. We run it on Linux - so it's not system related. Clearly something changed with 4.3. It doesn't happen every 4 hours - more like once every 12 hours but at quite random - and mostly during hours when traffic is the lowest. It's not resource issue - we have 8GB allocated to mirth and don't seem to ever hit even close to 50% - network has been analysed many times and no issues there. It's causing quite a few headaches - specially because NextGen support answer is to update to 4.4 - then to 4.4.1 - otherwise they can't help any further. We have provided them with all the possible logs and even offered them to dig through our server themselves - but after talking with their support - feels like they are clueless to what's going on. Every time this happens - service needs to be force killed - and after restarting it - it's back to normal for another 12 or so hours. No other process runs at that time, and this server is purely for Mirth only. Would love to get to the bottom of this. |
Beta Was this translation helpful? Give feedback.
-
|
@marilaki take a thread dump when the CPU usage gets high. |
Beta Was this translation helpful? Give feedback.
-
|
Also helpful to post the complete enviroment, OS, Java vendor and version, DB type/version, when DB backups occur, Mirth premium extensions installed, mirth.properties tuning if any, vmoptions set, etc. |
Beta Was this translation helpful? Give feedback.
-
1、Middle connect version 4.4, after the Win10 system boots up, the CPU temporarily uses over 80%, only five channels are set incorrectly? Such a high CPU utilization rate causes other software and computers to be too laggy. What can we do to reduce the CPU to below 50%?
2、computer info
When the Mirth service is active the CPU consumption level reaches 100%.
Environment (please complete the following information):
OS: Windows 10
Java Distribution/Version
jdk version "1.8"
Connect Version 4.4
database:SqlServer
3、The screenshot of using CPU detection tool is as follows
Beta Was this translation helpful? Give feedback.
All reactions