-
Notifications
You must be signed in to change notification settings - Fork 24.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Transform] latest
transform skipping some source documents
#106363
Labels
Comments
sync.time.delay
field has no effect for the latest
transformlatest
transform skipping some source documents
Pinging @elastic/ml-core (Team:ML) |
If there currently any workarounds or setting that could be adjusted? |
Any news or version ETA on this issue? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Elasticsearch Version
8.13
Installed Plugins
No response
Java Version
bundled
OS Version
MacOS
Problem Description
Latest transform was reported to skip some source documents.
I identified 2 potential issues:
@timestamp
value, thelatest
transform only picks one of them.sync.time.delay
field does not seem to influence the filterrange
queries issued by thelatest
transform.Ad 1.:
This is how we build the range query in the code:
So I think it can be that because of this
lt
the documents that have the same timestamp as the document that was already involved in the checkpoint will not get processed.This should be taken care of by the
time.sync.delay
but apparently it doesn't work in this case (Ad 2.)Steps to Reproduce
This has been reproduced by the Kibana team (https://github.com/elastic/security-team/issues/8893).
Now I'm working on reproducing it locally.
Logs (if relevant)
No response
The text was updated successfully, but these errors were encountered: