[pkg/stanza] Add regexp based tests & examples for filter operator (o…

…pen-telemetry#31698)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
This PR adds some additional unit tests and examples to cover the regexp
based filter-in/filter-out functionalities of the [filter
operator](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/filter.md).

This is an important functionality spotted during a comparison of the
`filelogreceiver`'s capabilities with those of
[Filebeat](https://github.com/elastic/beats/tree/main/filebeat#filebeat).

[Filebeat](https://github.com/elastic/beats/tree/main/filebeat#filebeat)'s
[filestream
input](https://github.com/elastic/beats/tree/main/filebeat/input/filestream)
supports this filtering with its
[include_lines](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-include-lines)
and
[exclude_lines](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-exclude-lines)
options.
The added tests and examples are to ensure the parity of this
capability.

**Link to tracking Issue:** <Issue number if applicable>

**Testing:** <Describe what testing was performed and which tests were
added.>
-
[FilterOutRegexp](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/31698/files#diff-01d783f62fa2a61d9fbfe9342f0ea20629604563c550cee56b0fa0ba179556a1R50)
-
[FilterInRegexp](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/31698/files#diff-01d783f62fa2a61d9fbfe9342f0ea20629604563c550cee56b0fa0ba179556a1R60)

**Documentation:** <Describe the documentation added.>
Added the [Filtering log messages based on
content](https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/31698/files#diff-7156c0db24790afae5276c0908dd0e02ecc7c23947607996d50008b8e4ab574dR1)
example.

PS: I'm not sure if that change technically require a changelog entry.
If so, please let me know.

Signed-off-by: ChrsMark <[email protected]>

examples/logline-filtering/README.md

@@ -0,0 +1,13 @@
+## Filtering log messages based on content
+
+Filelog receiver provides support for filtering logs based on their content. This can be achieved by using
+the [filter operator](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/filter.md),
+configured with matching regular expressions.
+
+With this happening at the collection point, a lot of resources at the destination backend
+can be saved since no additional processing would need to take place.
+
+A full configuration example on how to filter out logs that start with the `INFO:` pattern is
+provided in the [example config](./otel-col-config-filter-out-logs.yaml).
+A full configuration example on how to only collect logs that start with the `WARN:` pattern is provided in
+the [example config](./otel-col-config-filter-in-logs.yaml)

examples/logline-filtering/otel-col-config-filter-in-logs.yaml

@@ -0,0 +1,17 @@
+receivers:
+ filelog:
+ include: [/var/log/busybox/simple.log]
+ operators:
+ - type: filter
+ expr: 'body not matches "^WARN:"'
+
+service:
+ pipelines:
+ logs:
+ receivers: [filelog]
+ exporters: [otlp/custom]
+ processors: []
+
+exporters:
+ otlp/custom:
+ endpoint: http:https://0.0.0.0:4242

examples/logline-filtering/otel-col-config-filter-out-logs.yaml

@@ -0,0 +1,17 @@
+receivers:
+ filelog:
+ include: [/var/log/busybox/simple.log]
+ operators:
+ - type: filter
+ expr: 'body matches "^INFO:"'
+
+service:
+ pipelines:
+ logs:
+ receivers: [filelog]
+ exporters: [otlp/custom]
+ processors: []
+
+exporters:
+ otlp/custom:
+ endpoint: http:https://0.0.0.0:4242

pkg/stanza/operator/transformer/filter/filter_test.go

@@ -46,6 +46,26 @@ func TestTransformer(t *testing.T) {
  `body.message == "test_message"`,
  false,
  },
+ {
+ "FilterOutRegexp",
+ &entry.Entry{
+ Body: map[string]any{
+ "message": "INFO: this is an info message",
+ },
+ },
+ `body.message matches "^INFO:"`,
+ true,
+ },
+ {
+ "FilterInRegexp",
+ &entry.Entry{
+ Body: map[string]any{
+ "message": "WARN: this is a warning message",
+ },
+ },
+ `body.message not matches "^WARN:"`,
+ false,
+ },
  {
  "MatchAttribute",
  &entry.Entry{