Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Could DataStreamTimestampFieldMapper not iterate over all fields? #92297

Closed
jpountz opened this issue Dec 12, 2022 · 3 comments · Fixed by #110603
Closed

Could DataStreamTimestampFieldMapper not iterate over all fields? #92297

jpountz opened this issue Dec 12, 2022 · 3 comments · Fixed by #110603
Assignees
@lkts
Labels
>enhancement :StorageEngine/Mapping The storage related side of mappings Team:StorageEngine

Comments

@jpountz
Copy link
Contributor

jpountz commented Dec 12, 2022 •
edited by lkts
Loading

Description

DataStreamTimestampFieldMapper#postParse scans over all fields that have been created during the parsing process to verify that there is exactly one @timestamp field. This even happens twice with the TSDB mode because TsidExtractingFieldMapper#createField does another pass to extract the timestamp that it needs to generate the ID.

Could we instead hook into the parsing logic of DateFieldMapper#parse to record the timestamp value of the document somewhere on the LuceneDocument so that it's ready to use for DataStreamTimestampFieldMapper#postParse and TsidExtractingFieldMapper#createField and we don't have to scan over fields again?
@jpountz jpountz added >enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types labels Dec 12, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@elasticsearchmachine elasticsearchmachine added the Team:Search Meta label for search team label Dec 12, 2022
@romseygeek
Copy link
Contributor

+1. I tried a while ago to make @timestamp a metadata field in #84294 but ended up getting bogged down in trying to make it magically re-write mappings. But it could probably be revived without the magic bits.

@javanna javanna added :StorageEngine/Mapping The storage related side of mappings Team:StorageEngine and removed :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Meta label for search team labels Jun 25, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@lkts lkts self-assigned this Jul 5, 2024
@lkts lkts mentioned this issue Jul 8, 2024
Merged
@lkts lkts changed the title Could DataStreamTimestampFieldMapperTest not iterate over all fields? Could DataStreamTimestampFieldMapper not iterate over all fields? Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lkts lkts

Labels
>enhancement :StorageEngine/Mapping The storage related side of mappings Team:StorageEngine
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Stop iterating over all fields to extract @timestamp value lkts/elasticsearch
5 participants
@jpountz @javanna @romseygeek @lkts @elasticsearchmachine