Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Transform] Updating frequency should apply after next checkpoint #86898

Open
sophiec20 opened this issue May 18, 2022 · 1 comment
Open

[Transform] Updating frequency should apply after next checkpoint #86898

sophiec20 opened this issue May 18, 2022 · 1 comment
Labels
>bug :ml/Transform Transform Team:ML Meta label for the ML team

Comments

@sophiec20
Copy link
Contributor

sophiec20 commented May 18, 2022

Found in 8.2.0-SNAPSHOT

Documentation says
All updated properties except description do not take effect until after the transform starts the next checkpoint. This is so there is data consistency in each checkpoint.

However in practice, updating frequency does not "seem" to apply until the transform is stopped and started.

image

To reproduce:

  1. Created and started a continuous transform on a dataset (timestamps ranging between now-10d to now+10d), with frequency: 30s
  2. Closed the source index POST source-index/_close
  3. Observed timestamps of warning notifications, 30s apart
  4. Opened source index POST source-index/_open
  5. Updated transform to set frequency: 10s
  6. Inserted current timestamp docs into source index (to force the checkpoint)
  7. Observed notification message confirming checkpoint occurred
  8. Closed the source index
  9. Observed timestamps of warning notifications -- 30s apart, instead of expected 10s apart
  10. Open the source index
  11. Stop and restart the transform
  12. Close the source index
  13. Observed timestamps of warning notifications -- now at 10s apart

Despite updating frequency: 10s beforehand, the notification messages still show that the action is occurring every 30s after a checkpoint. After stopping and restarting the transform, then the notification messages occur every 10s.

Either
1/ the docs require a clarification, or
2/ the test isn't valid, or
3/ there is a bug in the way frequency is being applied

@elasticmachine elasticmachine added the Team:ML Meta label for the ML team label May 18, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@przemekwitek przemekwitek self-assigned this Jun 27, 2022
@przemekwitek przemekwitek removed their assignment Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :ml/Transform Transform Team:ML Meta label for the ML team
Projects
None yet
Development

No branches or pull requests

3 participants