Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MongoDB and Solr Service sinks may retain some records under some conditions #409

Open
mathieu-rossignol opened this issue Aug 29, 2018 · 1 comment
Milestone

Comments

@mathieu-rossignol
Copy link
Member

In MongoDB as well as Solr Services, we use the take() method on the BlockingQueue queue,
waiting for incoming records. This may lead to some records not flushed into the sinks.
Consider the following scenario:
In a short period of time lower than the flush interval (and at the very startup of the service or just after a batch has been flushed) a bulkput processor sends few records but less than the batch size to the service.
If there are no more immediate incoming records, and as we use the blocking method take(), those few records are blocked (in the prepared query waiting for processing) until some new records arrive.

Proposed fix: use the poll() method which has timeout support (as done for instance in the Chronix service)

MongoDB:
https://github.com/Hurence/logisland/blob/develop/logisland-services/logisland-mongodb/logisland-mongodb-client-service/src/main/java/com/hurence/logisland/service/mongodb/MongoDBUpdater.java#L81
Solr:
https://github.com/Hurence/logisland/blob/develop/logisland-services/logisland-solr-client-service/logisland-solr-client-service-api/src/main/java/com/hurence/logisland/service/solr/api/SolrUpdater.java#L52
Chronix:
https://github.com/Hurence/logisland/blob/develop/logisland-services/logisland-solr-client-service/logisland-solr_6_4_2-chronix-client-service/src/main/java/com/hurence/logisland/service/solr/ChronixUpdater.java#L82

@amarziali
Copy link
Contributor

MongoDB: #430 fixes the issue

@oalam oalam added the processor label Sep 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants