OTTL 'log' function needed for debugging complex transform pipelines

[ringerc]

Component(s)

processor/transform

Is your feature request related to a problem? Please describe.

It's difficult to inspect and diagnose the behaviour of complex transform processor pipelines because of the lack of a means of inspecting the state and execution partway through the pipeline.

Answering questions like "why does this 'where' clause not match" is considerably more difficult than might be expected.

Describe the solution you'd like

An OTTL debug_log function that writes its argument(s) out in json form to the otel collector log stream would be very helpful. If passed multiple arguments it would log each in turn. If passed a path, it would serialize and log all children of the path, following the usual access rules.

It would take a key/value pairing of log field label to log field value, and emit them through the otel collector's logger, so if json logging is turned on the results would be distinct json logging keys.

E.g.

  transform/afterdiscovery-vendor:
    metric_statements:
      - context: resource
        statements:
          - |
            set(attributes["foo"], true)
            where attributes["bar"] == "baz"
          - debug_log("attribute foo", attributes["foo"], "attribute bar", attributes["bar"])

For use in the filter processor, logging converter variants may make sense. These would take their first argument and return it unchanged, then take all additional parameters as data to emit as logs.

Describe alternatives you've considered

Presently debugging complex pipelines often involves commenting out various parts, or splitting them up into two separate processors linked by a connector with a debug exporter in the middle. It's wholly impractical.

Ideally some form of attachable debugger could be added, where it could establish a socket to the running collector and place probes dynamically. This is far beyond what's practical to add to the OTTL stack at present however.

Sometimes it's possible to get the info needed by using set to apply various attributes to carry the info required, then inspect with a debug exporter. But this quickly fall down for more complex transforms.

Additional context

No response

[github-actions]

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

@ringerc maybe instead of parameters a function like debug could print the current TransformContext as json (exluding the metric slice from the metric context)?

Ultimately what I'd really like to do some time is add tracing to OTTL/transformprocessor. It would be great to create a span for each statement and record in the trace how the telemetry was change with each statement. Since the collector can export its spans now this is doable.

[github-actions]

Pinging code owners for pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[ringerc]

@TylerHelmuth Sounds viable, though very noisy in the logs if the context is large.

This was prompted by wrestling a quite complex transform pipeline that adds different attributes for different target monitoring platform vendors.

I'm hoping to get some time to actually work on the collector itself soon.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/transform: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley
• pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[TylerHelmuth]

It seems I made a duplicate with #31912, lets keep the discussion here.

A highlight from that issue is that I think it would be nice to be able to print the TransformContext after every statement without having to manually add debug statements via a dedicated function.

[TylerHelmuth]

We should consider if a solution can be added specifically to OTTL so that any component that uses OTTL can benefit. It might be enough to add some debug logs to Execute and Eval.

[TylerHelmuth]

Since we're tracking trace work via #33433, I consider this closed via #33274