-
Notifications
You must be signed in to change notification settings - Fork 24.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lazily create the failure store #109289
Lazily create the failure store #109289
Conversation
Rather than initializing the failure store right away when a new data stream is created, we leave it empty and mark it for lazy rollover. This results in the failure store only being initialized (i.e. an index created) when a failure has actually occurred. The exception to the rule is when a failure occurs while the data stream is being auto-created. In that case we do want to initialize the failure store right away.
Pinging @elastic/es-data-management (Team:Data Management) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this @nielsbauman ! It looks good, I have started a couple of discussions and I am still running through potential scenarios. I haven't finished this yet, but I thought we do not have to delay the existing feedback while I do this :)
...reams/src/internalClusterTest/java/org/elasticsearch/datastreams/DataStreamsSnapshotsIT.java
Show resolved
Hide resolved
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/MetadataRolloverService.java
Outdated
Show resolved
Hide resolved
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/MetadataRolloverService.java
Outdated
Show resolved
Hide resolved
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java
Outdated
Show resolved
Hide resolved
...ck/plugin/ccr/src/main/java/org/elasticsearch/xpack/ccr/action/TransportPutFollowAction.java
Show resolved
Hide resolved
...t/java/org/elasticsearch/action/bulk/TransportBulkActionIndicesThatCannotBeCreatedTests.java
Outdated
Show resolved
Hide resolved
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java
Show resolved
Hide resolved
# Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java
server/src/main/java/org/elasticsearch/action/admin/indices/create/CreateIndexRequest.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/action/admin/indices/create/CreateIndexRequest.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm so excited for this PR, everything is looking really good. I left a couple of comments, mostly validation related.
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/MetadataRolloverService.java
Show resolved
Hide resolved
server/src/main/java/org/elasticsearch/cluster/metadata/MetadataCreateDataStreamService.java
Outdated
Show resolved
Hide resolved
...streams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/170_modify_data_stream.yml
Show resolved
Hide resolved
# Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java
This reverts commit 5f2e5c6.
...ams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/200_rollover_failure_store.yml
Show resolved
Hide resolved
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/MetadataRolloverService.java
Show resolved
Hide resolved
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/TransportRolloverAction.java
Show resolved
Hide resolved
...r/src/main/java/org/elasticsearch/action/admin/indices/rollover/MetadataRolloverService.java
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm just about ready to ✅ this PR, but one last test case comes to mind.
...ams/src/yamlRestTest/resources/rest-api-spec/test/data_stream/200_rollover_failure_store.yml
Show resolved
Hide resolved
# Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Thanks for all the hard work! 🚀
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! 😃
Rather than initializing the failure store right away when a new data stream is created, we leave it empty and mark it for lazy rollover. This results in the failure store only being initialized (i.e. an index created) when a failure has actually occurred.
The exception to the rule is when a failure occurs while the data stream is being auto-created. In that case, we do want to initialize the failure store right away.