Long locked thread issues

[rsignell-usgs]

I'm also getting Plumbr reports of "long locked thredds".

Are these useful or useless?

On 5 occasions threads were locked due to the same underlying problem. Total time for those locks was 40s:

The thread was waiting in synchronized block in uk.ac.rdg.resc.edal.cdm.LookUpTableGrid.generate() method line 88 for the java.util.HashMap lock to be released.

Full call stack for the waiting thread:

uk.ac.rdg.resc.edal.cdm.LookUpTableGrid.generate():88
uk.ac.rdg.resc.edal.cdm.CdmUtils.createHorizontalGrid():278
uk.ac.rdg.resc.edal.cdm.CdmUtils.readCoverageMetadata():173
uk.ac.rdg.resc.edal.cdm.CdmUtils.readCoverageMetadata():145
thredds.server.wms.ThreddsDataset.<init>():136
thredds.server.wms.ThreddsDataset.getThreddsDatasetForRequest():280
thredds.server.wms.ThreddsWmsController.dispatchWmsRequest():165
uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.handleRequestInternal():200
org.springframework.web.servlet.mvc.AbstractController.handleRequest():153
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle():48
org.springframework.web.servlet.DispatcherServlet.doDispatch():919
org.springframework.web.servlet.DispatcherServlet.doService():851
org.springframework.web.servlet.FrameworkServlet.processRequest():953
org.springframework.web.servlet.FrameworkServlet.doGet():844
javax.servlet.http.HttpServlet.service():621
org.springframework.web.servlet.FrameworkServlet.service():829
javax.servlet.http.HttpServlet.service():722
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():305
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
thredds.servlet.filter.RequestQueryFilter.doFilter():118
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():243
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
thredds.servlet.filter.RequestCORSFilter.doFilterInternal():49
org.springframework.web.filter.OncePerRequestFilter.doFilter():106
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate():343
org.springframework.web.filter.DelegatingFilterProxy.doFilter():260
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():243
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
thredds.servlet.filter.RequestPathFilter.doFilter():94
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():243
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
thredds.server.RequestBracketingLogMessageFilter.doFilter():81
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():243
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
org.apache.catalina.core.StandardWrapperValve.invoke():225
org.apache.catalina.core.StandardContextValve.invoke():123
org.apache.catalina.authenticator.AuthenticatorBase.invoke():472
org.apache.catalina.core.StandardHostValve.invoke():168
org.apache.catalina.valves.ErrorReportValve.invoke():98
org.apache.catalina.valves.AccessLogValve.invoke():927
org.apache.catalina.core.StandardEngineValve.invoke():118
org.apache.catalina.connector.CoyoteAdapter.service():407
org.apache.coyote.http11.AbstractHttp11Processor.process():1001
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process():585
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run():310
java.util.concurrent.ThreadPoolExecutor.runWorker():1110
java.util.concurrent.ThreadPoolExecutor$Worker.run():603
java.lang.Thread.run():722


Call stack 1 (2 threads)

uk.ac.rdg.resc.edal.cdm.LookUpTableGrid.generate():88
uk.ac.rdg.resc.edal.cdm.CdmUtils.createHorizontalGrid():278
uk.ac.rdg.resc.edal.cdm.CdmUtils.readCoverageMetadata():173
uk.ac.rdg.resc.edal.cdm.CdmUtils.readCoverageMetadata():126
thredds.server.wms.ThreddsDataset.<init>():95
thredds.server.wms.ThreddsDataset.getThreddsDatasetForRequest():270
thredds.server.wms.ThreddsWmsController.dispatchWmsRequest():165
uk.ac.rdg.resc.ncwms.controller.AbstractWmsController.handleRequestInternal():200
org.springframework.web.servlet.mvc.AbstractController.handleRequest():153
org.springframework.web.servlet.mvc.SimpleControllerHandlerAdapter.handle():48
org.springframework.web.servlet.DispatcherServlet.doDispatch():919
org.springframework.web.servlet.DispatcherServlet.doService():851
org.springframework.web.servlet.FrameworkServlet.processRequest():953
org.springframework.web.servlet.FrameworkServlet.doGet():844
javax.servlet.http.HttpServlet.service():621
org.springframework.web.servlet.FrameworkServlet.service():829
javax.servlet.http.HttpServlet.service():722
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():305
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
thredds.servlet.filter.RequestQueryFilter.doFilter():118
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():243
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
thredds.servlet.filter.RequestCORSFilter.doFilterInternal():49
org.springframework.web.filter.OncePerRequestFilter.doFilter():106
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate():343
org.springframework.web.filter.DelegatingFilterProxy.doFilter():260
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():243
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
thredds.servlet.filter.RequestPathFilter.doFilter():94
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():243
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
thredds.server.RequestBracketingLogMessageFilter.doFilter():81
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter():243
org.apache.catalina.core.ApplicationFilterChain.doFilter():210
org.apache.catalina.core.StandardWrapperValve.invoke():225
org.apache.catalina.core.StandardContextValve.invoke():123
org.apache.catalina.authenticator.AuthenticatorBase.invoke():472
org.apache.catalina.core.StandardHostValve.invoke():168
org.apache.catalina.valves.ErrorReportValve.invoke():98
org.apache.catalina.valves.AccessLogValve.invoke():927
org.apache.catalina.core.StandardEngineValve.invoke():118
org.apache.catalina.connector.CoyoteAdapter.service():407
org.apache.coyote.http11.AbstractHttp11Processor.process():1001
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process():585
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run():310
java.util.concurrent.ThreadPoolExecutor.runWorker():1110
java.util.concurrent.ThreadPoolExecutor$Worker.run():603
java.lang.Thread.run():722