out_loki: add support for structured metadata #9048
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for structured metadata in the loki output. As of Loki 3.0, structured metadata has been made GA.
Implementation wise, labels and structured metadata are very similar: both are key/value pairs, and both support record accessors, consequently this PR does some light refactoring to avoid nontrivial code duplication between labels and structured metadata.
This PR only implements the analog of the
labels
config parameter, which is calledstructured_metadata
. Analogs forlabel_keys
andlabel_map_path
were not implemented to keep the change both simple and immediately useful, though they could be added later.Fixes #8080.
Testing
Because the code paths are slightly different when labels and/or structured metadata employ record accessors, I have two simple configurations: one that uses structured metadata with record accessors, and one that only uses static values.
Our Loki deployment uses mTLS for auth so I've kept that in the config to represent testing, although the Loki endpoint is obviously redacted.
All tests include a log containing combined output with Fluent Bit run under Valgrind.
Test 0: baseline without structured metadata
0-baseline-without-structured-metadata.log
Test 1: structured metadata with record accessors
1-with-record-accessor.log
Test 2: structured metadata (and labels) without record accessors
2-without-record-accessor.log
Documentation
Will submit a docs PR after this PR gets some initial feedback or indication it's likely to be accepted.
Backporting
N/A. It would be nice to make this in time for Fluent Bit 3.1 though. :)
Licensing
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.