Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/elasticsearch] OTel mode serialization #33290

Merged

Conversation

aleksmaus
Copy link
Contributor

@aleksmaus aleksmaus commented May 29, 2024

Description:

Implements OTel (OpenTelemetry-native) mode serialization for elasticsearch exporter.
This is an initial cut in order to get the discussion going.
This is approach was tested as internal POC.

It leverages Elasticsearch "passthrough" fields mapping initially introduced in Elasticsearch 8.13 allowing users to query the document/scope/resources attributes as top level fields, making the ECS queries compatible with OTel sematic convention schema. Another benefit is the simplicity of conversion of stored document from Elasticsearch back to Otel data model format.

The document/scope/resources attributes are dynamically mapped and stored as flattened keys.

Here is an example of index template mappings with "passthrough" fields:

PUT _index_template/logs_otel
{
  "priority": 250,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "logs"
        },
        "codec": "best_compression",
        "mapping": {
          "ignore_malformed": "true"
        }
      }
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "date_detection": false,
      "dynamic": "strict",
      "dynamic_templates": [
        {
          "all_strings_to_keywords": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        },
        {
          "complex_attributes": {
            "path_match": [
              "resource.attributes.*",
              "scope.attributes.*",
              "attributes.*"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date_nanos",
          "ignore_malformed": false
        },
        "data_stream": {
          "type": "object",
          "properties": {
            "type": {
              "type": "constant_keyword"
            },
            "dataset": {
              "type": "constant_keyword"
            },
            "namespace": {
              "type": "constant_keyword"
            }
          }
        },
        "observed_timestamp": {
          "type": "date_nanos",
          "ignore_malformed": true
        },
        "severity_number": {
          "type": "long"
        },
        "severity_text": {
          "type": "keyword"
        },
        "body_text": {
          "type": "match_only_text"
        },
        "body_structured": {
          "type": "flattened"
        },
        "attributes": {
          "type": "passthrough",
          "dynamic": true,
          "priority": 2
        },
        "dropped_attributes_count": {
          "type": "long"
        },
        "trace_flags": {
          "type": "byte"
        },
        "trace_id": {
          "type": "keyword"
        },
        "span_id": {
          "type": "keyword"
        },
        "scope": {
          "properties": {
            "name": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            },
            "attributes": {
              "type": "passthrough",
              "dynamic": true,
              "priority": 1
            },
            "dropped_attributes_count": {
              "type": "long"
            },
            "schema_url": {
              "type": "keyword"
            }
          }
        },
        "resource": {
          "properties": {
            "dropped_attributes_count": {
              "type": "long"
            },
            "schema_url": {
              "type": "keyword"
            },
            "attributes": {
              "type": "passthrough",
              "dynamic": true,
              "priority": 0
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "logs-*.otel-*"
  ],
  "data_stream": {}
}

Here is an example of the auditd document in Elasticsearch abbreviated:

{
    "@timestamp": "2024-05-29T13:30:25.085926000Z",
    "attributes": {
        "foo": "bar",
        "some.bool": true
    },
    "body_structured": {
        "MESSAGE": "AVC apparmor=\"STATUS\" operation=\"profile_replace\" info=\"same as current profile, skipping\" profile=\"unconfined\" name=\"/usr/bin/evince-previewer\" pid=2702 comm=\"apparmor_parser\"",
        "SYSLOG_FACILITY": "4",
        "SYSLOG_IDENTIFIER": "audit",
        "_SOURCE_REALTIME_TIMESTAMP": "1716989425080000",
        "_TRANSPORT": "audit",
    },
    "dropped_attributes_count": 0,
    "observed_timestamp": "2024-05-29T14:49:26.534908898Z",
    "resource": {
        "attributes": {
            "data_stream.dataset": "auditd.otel",
            "data_stream.namespace": "default",
            "data_stream.type": "logs",
            "host.arch": "arm64",
            "host.cpu.cache.l2.size": 0,
            "host.cpu.family": "",
            "host.cpu.model.id": "0x000",
            "host.cpu.model.name": "",
            "host.cpu.stepping": "0",
            "host.cpu.vendor.id": "Apple",
            "host.id": "cae0e0147d454a80971b0b747c8b62b9",
            "host.ip": [
                "172.16.3.131",
                "fe80::20c:29ff:fe66:3012",
            "host.name": "lebuntu",
            "host.os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic #117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
            "host.os.type": "linux",
            "os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic #117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
            "os.type": "linux"
        },
        "dropped_attributes_count": 0,
        "schema_url": "https://opentelemetry.io/schemas/1.6.1"
    },
    "severity_number": 0,
    "trace_flags": 0
}

Here is an example of ECS compatible query that works on this Otel native schema:

GET logs-auditd.otel-default/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "host.name": "lebuntu"
          }
        }
      ]
    }
  }
}

Link to tracking Issue:
No tracking issue yet.

Testing:
Added unit test for OTel transformation.
Tested with journald OTel receiver.

Documentation:
No documentation is added yet.

@felixbarny
Copy link
Contributor

Functionally, the PR looks good to me. The only missing piece is the data stream routing. Maybe a test case with complex (map, list) attributes could be useful. I'll defer the actual code review to folks more familiar with go and the code base.

Copy link
Contributor

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added the Stale label Jun 21, 2024
@aleksmaus
Copy link
Contributor Author

Updated PR based on the some code review feedback. Added the data_stream routing support, that was initially implemented in Josh's POC per reviewer request.

@aleksmaus
Copy link
Contributor Author

Don't have permissions to remove the Stale label. Please advise.

@github-actions github-actions bot removed the Stale label Jun 25, 2024
@felixbarny
Copy link
Contributor

#33794 is doing some changes wrt. routing. After that is merged, please merge in the changes and make sure the same routing mechanism is used. Let us know if you think that would work.

@aleksmaus
Copy link
Contributor Author

#33794 is doing some changes wrt. routing. After that is merged, please merge in the changes and make sure the same routing mechanism is used. Let us know if you think that would work.

Yeah, that why I mentioned in DM before that I didn't want to add the routing since the work was happening in that area. It's easy to rollback.

@@ -245,19 +245,19 @@ func (doc *Document) Dedup() {
// Serialize writes the document to the given writer. The serializer will create nested objects if dedot is true.
//
// NOTE: The documented MUST be sorted if dedot is true.
func (doc *Document) Serialize(w io.Writer, dedot bool) error {
func (doc *Document) Serialize(w io.Writer, dedot bool, otel bool) error {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have to change objmodel for otel mode? Is there a reason why it isn't done like https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/model.go#L124 where we just add the attributes under a key?

Copy link
Contributor Author

@aleksmaus aleksmaus Jul 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I remember, at the time of implementing this feature, I stumbled on the issue where the document could only be serialized as completely flat from the root or "dedotted" (by default https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/elasticsearchexporter/factory.go#L80), all the comma separated properties were unwrapped into the nested object, which was not stored correctly "flattened" with the passthrough attributes in ES.

With OTel-native serialization the serialized document structure is a mix, needed to keep only the ".attributes*" flattened, while the rest of the document from the root is not flattened.

Let me know if anything changed in that area recently, this PR was open for some time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the context. I'm just learning about how passthrough field type works, and what you describe here makes sense. I wonder if it could be structured better in the code, but I don't have any good ideas at the moment and it is not a blocker.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it makes sense to revisit this once we remove the dedot option: #33772.

Copy link
Contributor

@felixbarny felixbarny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than a small suggestion for the docs, LGTM.
If Carson is happy with this, I think we can merge it.

exporter/elasticsearchexporter/README.md Outdated Show resolved Hide resolved
@aleksmaus
Copy link
Contributor Author

Accepted the README suggestion, merged the latest main

Copy link
Contributor

@carsonip carsonip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks! changelog nit, otherwise lgtm

.chloggen/feature_elasticsearch_otel_model.yaml Outdated Show resolved Hide resolved
.chloggen/feature_elasticsearch_otel_model.yaml Outdated Show resolved Hide resolved
@aleksmaus
Copy link
Contributor Author

Accepted the suggestions, resolved the latest set of conflicts with main.

@djaglowski
Copy link
Member

Please resolve CI failures and merge conflict

@aleksmaus
Copy link
Contributor Author

Resolved conflicts

Copy link
Contributor

@carsonip carsonip left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

=== Failed
=== FAIL: internal/objmodel TestDocument_Serialize_Otel/otel (0.00s)
    objmodel_test.go:387: 
                Error Trace:    /home/carson/projects/opentelemetry-collector-contrib/exporter/elasticsearchexporter/internal/objmodel/objmodel_test.go:387
                Error:          Not equal: 
                                expected: "{\"@timestamp\":\"2024-03-18T21:09:53.645578000Z\",\"attributes\":{\"auditd.log.op\":\"PAM:session_open\",\"auditd.log.record_type\":\"USER_START\",\"auditd.log.sequence\":6082,\"auditd.log.subj\":\"unconfined\",\"auditd.log.uid\":\"1000\"},\"resource\":{\"attributes\":{\"blah.num\":234,\"blah.str\":\"something\"}},\"scope\":{\"attributes\":{\"bar.one\":\"boo\",\"foo.two\":\"bar\"}}}"
                                actual  : "{\"@timestamp\":\"2024-03-18T21:09:53.645578000Z\",\"attributes\":{\"auditd.log.op\":\"PAM:session_open\",\"auditd.log.record_type\":\"USER_START\",\"auditd.log.sequence\":6082,\"auditd.log.subj\":\"unconfined\",\"auditd.log.uid\":\"1000\"},\"resource\":{\"attributes\":{\"blah\":{\"num\":234,\"str\":\"something\"}}},\"scope\":{\"attributes\":{\"bar\":{\"one\":\"boo\"},\"foo\":{\"two\":\"bar\"}}}}"
                                
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -{"@timestamp":"2024-03-18T21:09:53.645578000Z","attributes":{"auditd.log.op":"PAM:session_open","auditd.log.record_type":"USER_START","auditd.log.sequence":6082,"auditd.log.subj":"unconfined","auditd.log.uid":"1000"},"resource":{"attributes":{"blah.num":234,"blah.str":"something"}},"scope":{"attributes":{"bar.one":"boo","foo.two":"bar"}}}
                                +{"@timestamp":"2024-03-18T21:09:53.645578000Z","attributes":{"auditd.log.op":"PAM:session_open","auditd.log.record_type":"USER_START","auditd.log.sequence":6082,"auditd.log.subj":"unconfined","auditd.log.uid":"1000"},"resource":{"attributes":{"blah":{"num":234,"str":"something"}}},"scope":{"attributes":{"bar":{"one":"boo"},"foo":{"two":"bar"}}}}
                Test:           TestDocument_Serialize_Otel/otel

=== FAIL: internal/objmodel TestDocument_Serialize_Otel (0.00s)

=== FAIL: internal/objmodel TestDocument_Serialize_Otel/otel (re-run 1) (0.00s)
    objmodel_test.go:387: 
                Error Trace:    /home/carson/projects/opentelemetry-collector-contrib/exporter/elasticsearchexporter/internal/objmodel/objmodel_test.go:387
                Error:          Not equal: 
                                expected: "{\"@timestamp\":\"2024-03-18T21:09:53.645578000Z\",\"attributes\":{\"auditd.log.op\":\"PAM:session_open\",\"auditd.log.record_type\":\"USER_START\",\"auditd.log.sequence\":6082,\"auditd.log.subj\":\"unconfined\",\"auditd.log.uid\":\"1000\"},\"resource\":{\"attributes\":{\"blah.num\":234,\"blah.str\":\"something\"}},\"scope\":{\"attributes\":{\"bar.one\":\"boo\",\"foo.two\":\"bar\"}}}"
                                actual  : "{\"@timestamp\":\"2024-03-18T21:09:53.645578000Z\",\"attributes\":{\"auditd.log.op\":\"PAM:session_open\",\"auditd.log.record_type\":\"USER_START\",\"auditd.log.sequence\":6082,\"auditd.log.subj\":\"unconfined\",\"auditd.log.uid\":\"1000\"},\"resource\":{\"attributes\":{\"blah\":{\"num\":234,\"str\":\"something\"}}},\"scope\":{\"attributes\":{\"bar\":{\"one\":\"boo\"},\"foo\":{\"two\":\"bar\"}}}}"
                                
                                Diff:
                                --- Expected
                                +++ Actual
                                @@ -1 +1 @@
                                -{"@timestamp":"2024-03-18T21:09:53.645578000Z","attributes":{"auditd.log.op":"PAM:session_open","auditd.log.record_type":"USER_START","auditd.log.sequence":6082,"auditd.log.subj":"unconfined","auditd.log.uid":"1000"},"resource":{"attributes":{"blah.num":234,"blah.str":"something"}},"scope":{"attributes":{"bar.one":"boo","foo.two":"bar"}}}
                                +{"@timestamp":"2024-03-18T21:09:53.645578000Z","attributes":{"auditd.log.op":"PAM:session_open","auditd.log.record_type":"USER_START","auditd.log.sequence":6082,"auditd.log.subj":"unconfined","auditd.log.uid":"1000"},"resource":{"attributes":{"blah":{"num":234,"str":"something"}}},"scope":{"attributes":{"bar":{"one":"boo"},"foo":{"two":"bar"}}}}
                Test:           TestDocument_Serialize_Otel/otel

@djaglowski djaglowski merged commit 13366cc into open-telemetry:main Jul 23, 2024
154 checks passed
@github-actions github-actions bot added this to the next release milestone Jul 23, 2024
abhishek-at-cloudwerx pushed a commit to Internal-Cloudwerx/opentelemetry-collector-contrib that referenced this pull request Aug 9, 2024
**Description:**

Implements OTel (OpenTelemetry-native) mode serialization for
elasticsearch exporter.
This is an initial cut in order to get the discussion going.
This is approach was tested as internal POC.

It leverages Elasticsearch ```"passthrough"``` fields mapping initially
introduced in Elasticsearch 8.13 allowing users to query the
document/scope/resources attributes as top level fields, making the ECS
queries compatible with OTel sematic convention schema. Another benefit
is the simplicity of conversion of stored document from Elasticsearch
back to Otel data model format.

The document/scope/resources attributes are dynamically mapped and
stored as flattened keys.

Here is an example of index template mappings with ```"passthrough"```
fields:
```
PUT _index_template/logs_otel
{
  "priority": 250,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "logs"
        },
        "codec": "best_compression",
        "mapping": {
          "ignore_malformed": "true"
        }
      }
    },
    "mappings": {
      "_source": {
        "enabled": true
      },
      "date_detection": false,
      "dynamic": "strict",
      "dynamic_templates": [
        {
          "all_strings_to_keywords": {
            "mapping": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "match_mapping_type": "string"
          }
        },
        {
          "complex_attributes": {
            "path_match": [
              "resource.attributes.*",
              "scope.attributes.*",
              "attributes.*"
            ],
            "match_mapping_type": "object",
            "mapping": {
              "type": "flattened"
            }
          }
        }
      ],
      "properties": {
        "@timestamp": {
          "type": "date_nanos",
          "ignore_malformed": false
        },
        "data_stream": {
          "type": "object",
          "properties": {
            "type": {
              "type": "constant_keyword"
            },
            "dataset": {
              "type": "constant_keyword"
            },
            "namespace": {
              "type": "constant_keyword"
            }
          }
        },
        "observed_timestamp": {
          "type": "date_nanos",
          "ignore_malformed": true
        },
        "severity_number": {
          "type": "long"
        },
        "severity_text": {
          "type": "keyword"
        },
        "body_text": {
          "type": "match_only_text"
        },
        "body_structured": {
          "type": "flattened"
        },
        "attributes": {
          "type": "passthrough",
          "dynamic": true,
          "priority": 2
        },
        "dropped_attributes_count": {
          "type": "long"
        },
        "trace_flags": {
          "type": "byte"
        },
        "trace_id": {
          "type": "keyword"
        },
        "span_id": {
          "type": "keyword"
        },
        "scope": {
          "properties": {
            "name": {
              "type": "keyword"
            },
            "version": {
              "type": "keyword"
            },
            "attributes": {
              "type": "passthrough",
              "dynamic": true,
              "priority": 1
            },
            "dropped_attributes_count": {
              "type": "long"
            },
            "schema_url": {
              "type": "keyword"
            }
          }
        },
        "resource": {
          "properties": {
            "dropped_attributes_count": {
              "type": "long"
            },
            "schema_url": {
              "type": "keyword"
            },
            "attributes": {
              "type": "passthrough",
              "dynamic": true,
              "priority": 0
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "logs-*.otel-*"
  ],
  "data_stream": {}
}
```

Here is an example of the auditd document in Elasticsearch abbreviated:
```
{
    "@timestamp": "2024-05-29T13:30:25.085926000Z",
    "attributes": {
        "foo": "bar",
        "some.bool": true
    },
    "body_structured": {
        "MESSAGE": "AVC apparmor=\"STATUS\" operation=\"profile_replace\" info=\"same as current profile, skipping\" profile=\"unconfined\" name=\"/usr/bin/evince-previewer\" pid=2702 comm=\"apparmor_parser\"",
        "SYSLOG_FACILITY": "4",
        "SYSLOG_IDENTIFIER": "audit",
        "_SOURCE_REALTIME_TIMESTAMP": "1716989425080000",
        "_TRANSPORT": "audit",
    },
    "dropped_attributes_count": 0,
    "observed_timestamp": "2024-05-29T14:49:26.534908898Z",
    "resource": {
        "attributes": {
            "data_stream.dataset": "auditd.otel",
            "data_stream.namespace": "default",
            "data_stream.type": "logs",
            "host.arch": "arm64",
            "host.cpu.cache.l2.size": 0,
            "host.cpu.family": "",
            "host.cpu.model.id": "0x000",
            "host.cpu.model.name": "",
            "host.cpu.stepping": "0",
            "host.cpu.vendor.id": "Apple",
            "host.id": "cae0e0147d454a80971b0b747c8b62b9",
            "host.ip": [
                "172.16.3.131",
                "fe80::20c:29ff:fe66:3012",
            "host.name": "lebuntu",
            "host.os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic open-telemetry#117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
            "host.os.type": "linux",
            "os.description": "Ubuntu 22.04.4 LTS (Jammy Jellyfish) (Linux lebuntu 5.15.0-107-generic open-telemetry#117-Ubuntu SMP Mon Apr 29 14:37:09 UTC 2024 aarch64)",
            "os.type": "linux"
        },
        "dropped_attributes_count": 0,
        "schema_url": "https://opentelemetry.io/schemas/1.6.1"
    },
    "severity_number": 0,
    "trace_flags": 0
}
```

Here is an example of ECS compatible query that works on this Otel
native schema:
```
GET logs-auditd.otel-default/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "host.name": "lebuntu"
          }
        }
      ]
    }
  }
}
```


**Link to tracking Issue:**
No tracking issue yet.

**Testing:**
Added unit test for OTel transformation.
Tested with journald OTel receiver. 

**Documentation:**
No documentation is added yet.

---------

Co-authored-by: Felix Barnsteiner <[email protected]>
Co-authored-by: Carson Ip <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants