TSDB: terms query with order by top_metrics on a counter field reports null values when applied on mixed index (non-TSDB rolled over to TSDB) #96192

dej611 · 2023-05-17T10:28:35Z

Elasticsearch Version

8.8.0-SNAPSHOT

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin -- 21.6.0 Darwin Kernel Version 21.6.0: ; root:xnu-8020.240.18.700.8~1/RELEASE_ARM64_T6000 arm64

Problem Description

This issue can be reproduced on a index which was originally non-TSDB then migrated to TSDB.
This happens when using the k8s integration package in kibana.

When sending a terms agg using a top_metrics with a counter field to order results, depending on the selected time range results vary from correct to null:

if the time range filter starts and end before the rollover timing then the correct data is reported
if the time range filter starts before the rollover and ends after it, then null values are returned
if the time range filter starts and ends after the rollover timing, then the correct data is reported.

No shards failures reported.

Steps to Reproduce

More details with query can be found in this Kibana issue: elastic/kibana#157839 (comment)

Logs (if relevant)

No response

The text was updated successfully, but these errors were encountered:

elasticsearchmachine · 2023-05-17T10:28:59Z

Pinging @elastic/es-analytics-geo (Team:Analytics)

martijnvg · 2023-05-17T11:11:41Z

@dej611 Is it possible to add an ES reproduction? I'm wondering whether what documents exist in the two indices that match with the range filter. This could explain why the pie chart renders differently.

dej611 · 2023-05-17T12:07:09Z

@tetianakravchenko created the environment where I managed to reproduce the issue.

salvatore-campagna · 2023-05-18T13:24:29Z

I tried to reproduce the issue unsuccessfully using the following YAML test which does the following:

create a test1 index with mode: time_series
create a test2 index (no time series)
index data in both indices including the same fields and the same mappings and excluding time series settings for the non-time series index
query the two indices using 3 queries with time ranges set as explained above (before rollover, cross rollover, after rollover)
create a data stream
add the two existing indices to the data stream
query the two indices as before but using the data stream as the query target

I see the same results. For this reason I think the issue might be rollover-related.

---
"mixed time series mode":
  - skip:
      features: allowed_warnings
      version: " - 8.1.99"
      reason: tsdb indexing changed in 8.2.0

  - do:
      indices.create:
        index: test1
        body:
          settings:
            index:
              number_of_shards: 3
          mappings:
            properties:
              "@timestamp":
                type: date
              metricset:
                type: keyword
              dataset:
                type: keyword
              resource:
                type: keyword
              count:
                type: long

  - do:
      indices.create:
        index: test2
        body:
          settings:
            index:
              mode: time_series
              routing_path: [ metricset ]
              time_series:
                start_time: 2021-04-28T00:00:00Z
                end_time: 2021-04-29T00:00:00Z
              number_of_shards: 3
          mappings:
            properties:
              "@timestamp":
                type: date
              metricset:
                type: keyword
                time_series_dimension: true
              dataset:
                type: keyword
              resource:
                type: keyword
                time_series_dimension: true
              count:
                type: long
                time_series_metric: counter

  - do:
      bulk:
        refresh: true
        index: test1
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 4 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 4 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 5 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 6 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 13 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 10 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:55:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 11 }'

  - do:
      bulk:
        refresh: true
        index: test2
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 13 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 14 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 21 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 16 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 25 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:55:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 29 }'

  - do:
      search:
        index: test1,test2
        body:
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-27T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:53:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 13 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:55:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 11 }

  - do:
      search:
        index: test1,test2
        body:
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

  - do:
      search:
        index: test1,test2
        body:
          fields:
            - field: "@timestamp"
              format: date
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    format: strict_date_optional_time
                    gte: "2021-04-28T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
                shard_size: 25
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                          minimum_should_match: 1
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

  - do:
      allowed_warnings:
        - "index template [data-stream-template] has index patterns [data-*] matching patterns from existing older templates [global] with patterns (global => [*]); this template [my-template] will take precedence during new index creation"
      indices.put_index_template:
        name: data-stream-template
        body:
          index_patterns: [ data-* ]
          data_stream: { }

  - do:
      indices.create_data_stream:
        name: data-stream-mixed
  - is_true: acknowledged

  - do:
      indices.modify_data_stream:
        body:
          actions:
            - add_backing_index:
                data_stream: "data-stream-mixed"
                index: "test1"
  - is_true: acknowledged

  - do:
      indices.modify_data_stream:
        body:
          actions:
            - add_backing_index:
                data_stream: "data-stream-mixed"
                index: "test2"
  - is_true: acknowledged

  - do:
      indices.get_data_stream:
        name: data-stream-mixed

  - length: { data_streams: 1 }
  - length: { data_streams.0.indices: 3 }
  - match: { data_streams.0.indices.0.index_name: "test2" }
  - match: { data_streams.0.indices.1.index_name: "test1" }

  - do:
      indices.get_mapping:
        index: "test2"

  - match: { [email protected]: "date" }
  - match: { test2.mappings.properties.count.type: "long" }
  - match: { test2.mappings.properties.count.time_series_metric: "counter" }

  - do:
      indices.get_mapping:
        index: "test1"

  - match: { [email protected]: "date" }
  - match: { test1.mappings.properties.count.type: "long" }
  - match: { test1.mappings.properties.count.time_series_metric: null }

  - do:
      search:
        index: data-stream-mixed
        body:
          size: 20
          query:
            match_all: { }

  - length: { hits.hits: 16 }

  - do:
      search:
        index: data-stream-mixed
        body:
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-27T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:53:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 13 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:55:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 11 }

  - do:
      search:
        index: data-stream-mixed
        body:
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

  - do:
      search:
        index: data-stream-mixed
        body:
          fields:
            - field: "@timestamp"
              format: date
          size: 0
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    format: strict_date_optional_time
                    gte: "2021-04-28T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
                shard_size: 25
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                          minimum_should_match: 1
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

salvatore-campagna · 2023-05-24T12:18:19Z

I did another test introducing a rollover operation for one of the non-time series indices but still I am not able to reproduce the issue.

---
"mixed time series mode":
  - skip:
      features: allowed_warnings
      version: " - 8.1.99"
      reason: tsdb indexing changed in 8.2.0

  - do:
      allowed_warnings:
        - "index template [blue] has index patterns [purple-index] matching patterns from existing older templates [global] with patterns (global => [*]); this template [blue] will take precedence during new index creation"
      indices.put_index_template:
        name: blue
        body:
          index_patterns: [ "test-00000*" ]
          template:
            mappings:
              properties:
                "@timestamp":
                  type: date
                metricset:
                  type: keyword
                dataset:
                  type: keyword
                resource:
                  type: keyword
                count:
                  type: long

  - do:
      indices.create:
        index: test-000001
        body:
          settings:
            index:
              number_of_shards: 1
              number_of_replicas: 1

  - do:
      indices.create:
        index: test-time-series-000001
        body:
          settings:
            index:
              mode: time_series
              routing_path: [ metricset ]
              time_series:
                start_time: 2021-04-28T00:00:00Z
                end_time: 2021-04-29T00:00:00Z
              number_of_shards: 1
              number_of_replicas: 1
          mappings:
            properties:
              "@timestamp":
                type: date
              metricset:
                type: keyword
                time_series_dimension: true
              dataset:
                type: keyword
              resource:
                type: keyword
                time_series_dimension: true
              count:
                type: long
                time_series_metric: counter

  - do:
      indices.put_alias:
        index: test-000001
        name: alias_test-000001

  - do:
      indices.put_alias:
        index: test-time-series-000001
        name: alias_test-time-series-000001

  - do:
      bulk:
        refresh: true
        index: test-000001
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 4 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 4 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 5 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 6 }'

  - do:
      indices.rollover:
        alias: "alias_test-000001"
        wait_for_active_shards: 1
  - match: { old_index: "test-000001" }
  - match: { new_index: "test-000002" }
  - match: { rolled_over: true }
  - match: { dry_run: false }

  - do:
      bulk:
        refresh: true
        index: test-000002
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 13 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 10 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-27T18:55:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 11 }'

  - do:
      bulk:
        refresh: true
        index: test-time-series-000001
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:51:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 13 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:52:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 12 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 14 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:53:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 21 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "foo", "count": 16 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:54:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 25 }'
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:55:00.000Z", "dataset": "kubernetes", "metricset": "pod", "resource": "bar", "count": 29 }'

  - do:
      search:
        index: test-000001,test-000002,test-time-series-000001
        body:
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-27T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 3 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:53:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 13 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 5 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-27T18:55:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 11 }

  - do:
      search:
        index: test-000001,test-000002,test-time-series-000001
        body:
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    gte: "2021-04-27T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 7 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 9 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

  - do:
      search:
        index: test-000001,test-000002,test-time-series-000001
        body:
          fields:
            - field: "@timestamp"
              format: date
          query:
            bool:
              filter:
                range:
                  "@timestamp":
                    format: strict_date_optional_time
                    gte: "2021-04-28T18:00:00.000Z"
                    lte: "2021-04-28T19:00:00.000Z"
          aggs:
            0:
              terms:
                field: resource
                order:
                  1-bucket>1-metric[count]: desc
                size: 10
                shard_size: 25
              aggs:
                1-bucket:
                  filter:
                    bool:
                      filter:
                        bool:
                          should:
                            exists:
                              field: count
                          minimum_should_match: 1
                  aggs:
                    1-metric:
                      top_metrics:
                        metrics:
                          field: count
                        size: 1
                        sort:
                          "@timestamp": desc

  - length: { aggregations.0.buckets: 2 }

  - match: { aggregations.0.buckets.0.key: "bar" }
  - match: { aggregations.0.buckets.0.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:55:00.000Z" }
  - match: { aggregations.0.buckets.0.1-bucket.1-metric.top.0.metrics.count: 29 }

  - match: { aggregations.0.buckets.1.key: "foo" }
  - match: { aggregations.0.buckets.1.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.doc_count: 4 }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.sort.0: "2021-04-28T18:54:00.000Z" }
  - match: { aggregations.0.buckets.1.1-bucket.1-metric.top.0.metrics.count: 16 }

elasticsearchmachine · 2024-03-20T16:33:30Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

dej611 added >bug needs:triage Requires assignment of a team area label :StorageEngine/TSDB You know, for Metrics labels May 17, 2023

elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) and removed needs:triage Requires assignment of a team area label labels May 17, 2023

martijnvg assigned salvatore-campagna May 23, 2023

salvatore-campagna mentioned this issue May 24, 2023

[Lens] [Pie visualisation] incorrect visualisation for non-tsdb and tsdb data elastic/kibana#157839

Closed

wchaparro removed the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 20, 2024

elasticsearchmachine added the Team:StorageEngine label Mar 20, 2024

salvatore-campagna removed their assignment Jun 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TSDB: terms query with order by top_metrics on a counter field reports null values when applied on mixed index (non-TSDB rolled over to TSDB) #96192

TSDB: terms query with order by top_metrics on a counter field reports null values when applied on mixed index (non-TSDB rolled over to TSDB) #96192

dej611 commented May 17, 2023

elasticsearchmachine commented May 17, 2023

martijnvg commented May 17, 2023

dej611 commented May 17, 2023

salvatore-campagna commented May 18, 2023 •

edited

Loading

salvatore-campagna commented May 24, 2023

elasticsearchmachine commented Mar 20, 2024

TSDB: terms query with order by top_metrics on a counter field reports null values when applied on mixed index (non-TSDB rolled over to TSDB) #96192

TSDB: terms query with order by top_metrics on a counter field reports null values when applied on mixed index (non-TSDB rolled over to TSDB) #96192

Comments

dej611 commented May 17, 2023

Elasticsearch Version

Installed Plugins

Java Version

OS Version

Problem Description

Steps to Reproduce

Logs (if relevant)

elasticsearchmachine commented May 17, 2023

martijnvg commented May 17, 2023

dej611 commented May 17, 2023

salvatore-campagna commented May 18, 2023 • edited Loading

salvatore-campagna commented May 24, 2023

elasticsearchmachine commented Mar 20, 2024

salvatore-campagna commented May 18, 2023 •

edited

Loading