[processor/transform] Add ContextStatements config

[TylerHelmuth]

Description:
Adds the ContextStatement config and its supporting API. This config will allow users to specify a list of statements associated to a context. The processor will then be updated to use this API to processing all the statements associated to a context in the defined order.

As new Contexts are added the switch statement and ParserCollection will grow. Traces, Metrics, and Logs processors will use the corresponding New[Signal]ParserCollection to gain access to the parsers needed for their possible contexts.

Link to tracking Issue:
#14578

Related to #15381

[kentquirk]

I have to admit, this is a bit wordier than I had originally thought we were talking about -- but from an efficiency point of view this is exactly the right way to do it -- if you don't specify a deep context you don't have to pay for it.

This is solid work. Thanks.

[TylerHelmuth]

@bogdandrutu @evan-bradley please review.

[kovrus]

lgtm!

[evan-bradley]

Looks good to me. I have some minor nitpicks/questions, but nothing blocking.

[TylerHelmuth]

@kovrus would the routing processor want to take advantage of this type of configuration/API?

[kovrus]

@kovrus would the routing processor want to take advantage of this type of configuration/API?

As of now, routing statements use the resource context. I am now aware of any request to extend the context scope, but I guess, once we get such request we can make use of the ContextStatement configuration.

[TylerHelmuth]

@bogdandrutu @evan-bradley @kentquirk @kovrus I push some commits that I think gives the best of all worlds.

The API now has Trace/Metric/Log specific Contexts so that the Trace/Metric/Log specific processors can use their specific types. At the same time, ParserCollection is now a base that allows handling ResourceStatements and ScopeStatements are parsed in exactly 1 place and reused for the [Trace/Metric/Log]ParserCollection that encapsulates everything that a user would need to parse statements for the specific signal. We end up with an API that has no extraneous functions/interfaces.

When the logs processor uses this API it looks like

type Processor struct {
	contexts []common.LogsContext
}

func NewProcessor(statements []common.ContextStatements, settings component.TelemetrySettings) (*Processor, error) {
	ottlp := common.NewLogParserCollection(Functions(), settings)
	parsedStatements, err := ottlp.ParseContextStatements(statements)
	if err != nil {
		return nil, err
	}
	return &Processor{
		contexts: parsedStatements,
	}, nil
}

func (p *Processor) ProcessLogs(ctx context.Context, td plog.Logs) (plog.Logs, error) {
	for _, context := range p.contexts {
		err := context.ProcessLogs(ctx, td)
		if err != nil {
			return td, err
		}
	}
	return td, nil
}

Please re-review with fresh eyes.

[bogdandrutu]

Why do you need these specialized calls?

[TylerHelmuth]

@bogdandrutu I made specialized calls to represent the possibility of Resource-only functions or Scope-only functions.

[bogdandrutu]

Not sure we need a public API for that, just add a comment to the "Functions" that can be initialized with multiple contexts.

[TylerHelmuth]

@bogdandrutu This isn't a public API, it would never be moved to the OTTL. This is a way for the transform processor to enforce its definition of what functions are allowed to be used for Scope and Resource. Similar to how each signal in the transform processor has their own functions.go.

[bogdandrutu]

What is the reason to return a slice? I think we can return just one and call the statements sequentially.

[TylerHelmuth]

We could, but it will mean implementing loops in config.Validate and each signal's NewProcessor function

[bogdandrutu]

I think otherwise you will duplicate the loops in the user code (consumer of the package).

[TylerHelmuth]

@bogdandrutu If we remove this function's ability to handle a list of ContextStatements then the NewProcessor and Validate functions all look like:

type Processor struct {
	contexts []consumer.Logs
}

func NewProcessor(contextStatements []common.ContextStatements, settings component.TelemetrySettings) (*Processor, error) {
	pc, err := common.NewLogParserCollection(Functions(), settings)
	if err != nil {
		return nil, err
	}
	contexts := make([]consumer.Logs, len(contextStatements))
	for i, c := range contextStatements {
		context, err := pc.ParseContextStatements(c)
		if err != nil {
			return nil, err
		}
		contexts[i] = context
	}
	return &Processor{
		contexts: contexts,
	}, nil
}

I would definitely prefer to hide this shared logic in the ParseContextStatement functions. In the future, if this all moves into the OTTL, we can also expose a function for handling single ContextStatements if necessary

[bogdandrutu]

Same question I asked, why not replace this with consumer.Traces

[TylerHelmuth]

@bogdandrutu I don't see the value of type traceStatements []*ottl.Statement[ottltraces.TransformContext] implementing a Capabilities() Capabilities function. Also, I don't like how the API becomes ConsumeTraces instead of ProcessTraces as the function does not consume traces in the sense that the consumer.Traces API means.

[TylerHelmuth]

@bogdandrutu switched to consumer so you can see what it looks like.

[bogdandrutu]

Overall I think the direction is good.

[bogdandrutu]

Do we need this type, or we can replace it with something like this:

func ParseLogStatements(functions map[string]interface{}, settings component.TelemetrySettings, contextStatements ContextStatements) (consumer.Logs, error) {
	logParser := ottllogs.NewParser(functions, settings)
	switch contextStatements.Context {
	case Log:
		lStatements, err := logParser.ParseStatements(contextStatements.Statements)
		if err != nil {
			return nil, err
		}
		return logStatements(lStatements), nil
	default:
		statements, err := pc.parseCommonContextStatements(contextStatements)
		if err != nil {
			return nil, err
		}
		return statements, nil
	}
}

[TylerHelmuth]

@bogdandrutu I started with this approach but added the extra type to simplify the ParseSignalStatements functions parameters. If there isn't enough value, initializing in the function isn't a big deal.

[TylerHelmuth]

But if ParseSignalStatements only takes in a single ContextStatement, then we will be recreating each parser every time we call the function. Seems wasteful.

[bogdandrutu]

Lets iterate after we see the usage in the processor.