Access aggregate_metric_double fields from runtime fields and scripts

[salvatore-campagna]

Description

Customers need the ability to process indices and/or data streams including both raw data and downsampled data using ingest pipelines (configured, for instance with index.default_pipeline and index.final_pipeline).

When an index is downsampled, downsampling target indices have slightly different mappings and settings which might prevent pipeline scripts or runtime fields to work correctly. Ideally they would like to have a mechanism that is as easy as possible to use and that allows them to seamlessly process raw data and downsampled data using runtime fields and/or scripts without having to care much about whether the index includes raw data or downsampled data and such that they don't have to maintain multiple scripts (one for raw data and one for downsampled data).

Right now the main obstacle is about having the ability to access fields of type aggregate_metric_double that the downsampling operation creates when downsampling some metric fields. We need support for accessing such fields from pipeline scripts and runtime fields to start with.

Related to #96478.

[elasticsearchmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[salvatore-campagna]

Related to #88534

[salvatore-campagna]

Right now the ability to access these fields is available through the Fields API like demonstrated by the following YAML test:

setup:
  - skip:
      version: " - 8.0.99"
      reason: introduced in 8.1.0

  - do:
      indices.create:
        index: test
        body:
          settings:
            index:
              mode: time_series
              routing_path: [ metricset ]
              time_series:
                start_time: 2021-04-28T00:00:00Z
                end_time: 2021-04-29T00:00:00Z
          mappings:
            properties:
              "@timestamp":
                type: date
              metricset:
                type: keyword
                time_series_dimension: true
              aggregate:
                type: aggregate_metric_double
                metrics: [ min, max, sum, value_count ]
                default_metric: sum

  - do:
      bulk:
        refresh: true
        index: test
        body:
          - '{"index": {}}'
          - '{"@timestamp": "2021-04-28T18:50:00.000Z", "metricset": "pod", "counter": 1, "gauge": 100, "aggregate": { "min": 1, "max": 10, "sum": 23, "value_count": 5 }}'

---
"painless on aggregate_metric_double field":
  - do:
      search:
        index: test
        body:
          _source: false
          fields: [ "aggregate_sum" ]
          runtime_mappings:
            aggregate_sum:
              type: double
              script:
                source: "emit(doc['aggregate'].value)"
                lang: painless
          query:
            match_all: {}

  - match: { hits.total.value: 1 }
  - match: { hits.hits.0.fields.aggregate_sum.0: 23.0 }

Access to one of the aggregate values is provided by taking advantage of default_metric into the mapping.
Note that all metrics are of type double.
Note also that if there is a single metric, that is used without having to specify a default one.

[salvatore-campagna]

My understanding is that, at the moment, we are not able to expose all metrics under the aggregate metric field because accessing the value returns an array and we have no way to index into that array by means of a keyword like min or max.

[wchaparro]

@giladgal FYI

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)