-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(elasticsearch): Add rest.li endpoint that does truncation cleanup of a timeseries index #8277
Conversation
...data-io/src/main/java/com/linkedin/metadata/search/elasticsearch/update/ESBulkProcessor.java
Outdated
Show resolved
Hide resolved
...data-io/src/main/java/com/linkedin/metadata/search/elasticsearch/update/ESBulkProcessor.java
Outdated
Show resolved
Hide resolved
...restli-servlet-impl/src/main/java/com/linkedin/metadata/resources/entity/EntityResource.java
Show resolved
Hide resolved
...data-io/src/main/java/com/linkedin/metadata/search/elasticsearch/update/ESBulkProcessor.java
Outdated
Show resolved
Hide resolved
end to end test on quickstart: Use the following script to ingest timeseries aspects into the dataset usage statistics aspect:
Use get index sizes endpoint to check the size of the index:
Get the number of entries in the index using the ElasticSearch API:
Pick a timestamp within the range of those inserted and get the number of records less than that timestamp using the ElasticSearch API:
Dry run the deletion: using the new endpoint
Run the deletion for real using the new endpoint:
Query number of rows less than the timestamp after the deletion using ElasticSearch API:
Get the total number of entries after the deletion using ElasticSearch API:
Get the index size using the getIndexSizes endpoint:
Get task status endpoint:
|
return client.tasks().get(taskRequest, RequestOptions.DEFAULT); | ||
} catch (IOException e) { | ||
log.error(String.format("ERROR: Failed to get task status for %s:%d. See stacktrace for a more detailed error:", nodeId, taskId)); | ||
e.printStackTrace(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
e.printStackTrace(); |
Remove printStackTrace
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can add the e
to the log.error("message", e) to get this in the log
.../java/com/linkedin/metadata/timeseries/elastic/ElasticSearchTimeseriesAspectServiceTest.java
Show resolved
Hide resolved
public class OperationsResourceTest extends TestCase { | ||
private static final String TASK_ID = "taskId123"; | ||
|
||
TimeseriesAspectService mockTimeseriesAspectService = new TimeseriesAspectService() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's extract this preemptively as a mock implementation external class for possible reuse, people probably won't check to see if there are mock classes already inside tests before trying to write their own.
Overall this look good! Just look at the failing test and make sure that the test is deleting just what we expect and nothing else per Ryan's comment. Thank you! |
Example usage:
Checklist