IndexRecordMultiThreaded does not handle or log RuntimeException's thrown during execution

[ansell]

RuntimeExeceptions are not logged or handled by IndexRecordMultiThreaded during execution. They instead escape and cause the thread to stop and corrupt the resulting index.

https://github.com/AtlasOfLivingAustralia/biocache-store/blob/master/src/main/scala/au/org/ala/biocache/index/IndexRecordMultiThreaded.scala#L481

An example of an exception that escapes but does not cause the entire process to fail is the OOM exception which should be caught and the program should fully halt after it occurs as there is no way that a consistent index can be obtained after that point:

Exception in thread "Thread-7" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
at java.lang.StringCoding.encode(StringCoding.java:344)
at java.lang.String.getBytes(String.java:916)
at org.apache.solr.common.util.ContentStreamBase$StringStream.getStream(ContentStreamBase.java:170)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:162)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:99)
at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
at org.apache.solr.client.solrj.embedded.EmbeddedSolrServer.request(EmbeddedSolrServer.java:150)
at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:68)
at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:54)
at au.org.ala.biocache.index.SolrIndexDAO.indexFromMap(SolrIndexDAO.scala:639)
at au.org.ala.biocache.index.IndexRunner$$anonfun$run$7.apply(IndexRecordMultiThreaded.scala:478)
at au.org.ala.biocache.index.IndexRunner$$anonfun$run$7.apply(IndexRecordMultiThreaded.scala:466)
at au.org.ala.biocache.persistence.CassandraPersistenceManager$$anonfun$pageOver$1.apply(CassandraPersistenceManager.scala:341)
at au.org.ala.biocache.persistence.CassandraPersistenceManager$$anonfun$pageOver$1.apply(CassandraPersistenceManager.scala:333)
at scala.collection.immutable.List.foreach(List.scala:318)
at au.org.ala.biocache.persistence.CassandraPersistenceManager.pageOver(CassandraPersistenceManager.scala:333)
at au.org.ala.biocache.persistence.CassandraPersistenceManager.pageOverAll(CassandraPersistenceManager.scala:470)
at au.org.ala.biocache.index.IndexRunner.run(IndexRecordMultiThreaded.scala:466)
at java.lang.Thread.run(Thread.java:745)

[ansell]

The exception stack trace above does not seem to have been propagated out of the foreach method at:

biocache-store/src/main/scala/au/org/ala/biocache/index/BulkProcessor.scala

Line 160 in cbabd9a

threads.foreach(thread => thread.join)