[pkg/telemetryquerylanguage] Add Telemetry Query Language package (op…

…en-telemetry#11771)

* Add Telemetry Query Language package

* Update changelog

* Rename package

* Fix lint

* Add changelog entry

* revert changelog

* revert changelog

* Make telemetryquerylanguage a module

* Fix checks

* Update codeowners

* move replace statement

* Updated readme

.github/CODEOWNERS

@@ -98,6 +98,7 @@ internal/splunk/ @open-telemetry/collector-c
 pkg/batchpersignal/ @open-telemetry/collector-contrib-approvers @jpkrohling
 pkg/resourcetotelemetry/ @open-telemetry/collector-contrib-approvers @mx-psi
 pkg/stanza/ @open-telemetry/collector-contrib-approvers @djaglowski
+pkg/telemetryquerylanguage/ @open-telemetry/collector-contrib-approvers @TylerHelmuth @anuraaga
 pkg/translator/jaeger/ @open-telemetry/collector-contrib-approvers @open-telemetry/collector-approvers
 pkg/translator/opencensus/ @open-telemetry/collector-contrib-approvers @open-telemetry/collector-approvers
 pkg/translator/prometheus/ @open-telemetry/collector-contrib-approvers @dashpole @bertysentry

go.mod

@@ -732,6 +732,8 @@ replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/resourceto
 
 replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza => ./pkg/stanza
 
+replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/telemetryquerylanguage => ./pkg/telemetryquerylanguage
+
 replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/jaeger => ./pkg/translator/jaeger
 
 replace github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/opencensus => ./pkg/translator/opencensus

pkg/telemetryquerylanguage/Makefile

@@ -0,0 +1 @@
+include ../../Makefile.Common

pkg/telemetryquerylanguage/go.mod

@@ -0,0 +1,25 @@
+module github.com/open-telemetry/opentelemetry-collector-contrib/pkg/telemetryquerylanguage
+
+go 1.17
+
+require (
+ github.com/alecthomas/participle/v2 v2.0.0-alpha9
+ github.com/stretchr/testify v1.8.0
+ go.opentelemetry.io/collector/pdata v0.55.0
+ go.uber.org/multierr v1.8.0
+)
+
+require (
+ github.com/davecgh/go-spew v1.1.1 // indirect
+ github.com/gogo/protobuf v1.3.2 // indirect
+ github.com/golang/protobuf v1.5.2 // indirect
+ github.com/pmezard/go-difflib v1.0.0 // indirect
+ go.uber.org/atomic v1.7.0 // indirect
+ golang.org/x/net v0.0.0-20201021035429-f5854403a974 // indirect
+ golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4 // indirect
+ golang.org/x/text v0.3.3 // indirect
+ google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 // indirect
+ google.golang.org/grpc v1.47.0 // indirect
+ google.golang.org/protobuf v1.28.0 // indirect
+ gopkg.in/yaml.v3 v3.0.1 // indirect
+)

pkg/telemetryquerylanguage/tql/CONTRIBUTING.md

@@ -0,0 +1,12 @@
+# Contributing
+
+This guide is specific to the Telemetry Query Language. All guidelines in [Collector Contrib's CONTRIBUTING.MD](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md) must also be followed.
+
+## New Values
+
+When adding new values to the grammar you must:
+
+1. Update the `Value` struct with the new value. This may also mean adding new token(s) to the lexer.
+2. Update `NewFunctionCall` to be able to handle calling functions with this new value.
+3. Update `NewGetter` to be able to handle the new value.
+4. Add new unit tests.

pkg/telemetryquerylanguage/tql/README.md

@@ -0,0 +1,169 @@
+# Telemetry Query Language
+
+The Telemetry Query Language is a query language for transforming open telemetry data based on the [OpenTelemetry Collector Processing Exploration](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/processing.md).
+
+This package reads in TQL queries and converts them to invokable conditions and functions based on the TQL's grammar.
+
+The TQL is signal agnostic; it is not aware of the type of telemetry on which it will operate. Instead, the conditions and functions returned by the package must be passed a TransformContext, which provide access to the signal's telemetry. 
+
+## Grammar
+
+The TQL grammar includes Invocations, Values and Conditions.
+
+### Invocations
+
+Invocations represent a function call. Invocations are made up of 2 parts
+
+- a string identifier. The string identifier must start with a letter or an underscore (`_`).
+- zero or more Values (comma separated) surrounded by parentheses (`()`).
+
+**The TQL does not define any functions implementations.** Users must supply a map between string identifiers and the actual function implementation. The TQL will use this map and reflection to generate Invocations, that can then be invoked by the user.
+
+Example Invocations
+- `drop()`
+- `set(field, 1)`
+
+### Values
+
+Values are the things that get passed to an Invocation or used in a Condition. Values can be either a Path, a Literal, or an Invocation. 
+
+Invocations as Values allows calling functions as parameters to other functions. See [Invocations](#invocations) for details on Invocation syntax.
+
+#### Paths
+
+A Path Value is a reference to a telemetry field. Paths are made up of string identifiers, dots (`.`), and square brackets combined with a string key (`["key"]`). **The interpretation of a Path is NOT implemented by the TQL.** Instead, the user must provide a `PathExpressionParser` that the TQL can use to interpret paths. As a result, how the Path parts are used is up to the user. However, it is recommended, that the parts be used like so:
+
+- Identifiers are used to map to a telemetry field. 
+- Dots (`.`) are used to separate nested fields.
+- Square brackets and keys (`["key"]`) are used to access maps or slices.
+
+Example Paths
+- `name`
+- `resource.name`
+- `resource.attributes["key"]`
+
+#### Literals
+
+Literals are literal interpretations of the Value into a Go value. Accepted literals are:
+
+- Strings. Strings are represented as literals by surrounding the string in double quotes (`""`).
+- Ints. Ints are represented by any digit, optionally prepended by plus (`+`) or minus (`-`). Internally the TQL represents all ints as `int64`
+- Floats. Floats are represented by digits separated by a dot (`.`), optionally prepended by plus (`+`) or minus (`-`). The leading digit is optional. Internally the TQL represents all Floats as `float64.
+- Bools. Bools are represented by the exact strings `true` and `false`.
+- Nil. Nil is represented by the exact string `nil`. 
+- Byte slices. Byte slices are represented via a hex string prefaced with `0x`
+
+Example Literals
+- `"a string"`
+- `1`, `-1`
+- `1.5`, `-.5`
+- `true`, `false`
+- `nil`,
+- `0x0001`
+
+### Conditions
+
+Conditions allow a decision to be made about whether an Invocation should be called. The TQL does not force a condition to be used, it only allows the opportunity for the condition to be invoked before invoking the associated Invocation. Conditions allways return true or false.
+
+Conditions are made up of a left Value, an operator, and a right Value. See [Values](#values) for details on what a Value can be.
+
+Operators determine how the two Values are compared. The valid operators are:
+
+- Equal (`==`). Equal (`==`) checks if the left and right Values are equal, using Go's `==` operator.
+- Not Equal (`!=`). Not Equal (`!=`) checks if the left and right Values are not equal, using Go's `!=` operator.
+
+## Examples
+
+These examples contain a SQL-like declarative language. Applied statements interact with only one signal, but statements can be declared across multiple signals. Functions used in examples are indicative of what could be useful, but are not implemented by the TQL itself.
+
+### Remove a forbidden attribute
+
+```
+traces:
+ delete(attributes["http.request.header.authorization"])
+metrics:
+ delete(attributes["http.request.header.authorization"])
+logs:
+ delete(attributes["http.request.header.authorization"])
+```
+
+### Remove all attributes except for some
+
+```
+traces:
+ keep_keys(attributes, "http.method", "http.status_code")
+metrics:
+ keep_keys(attributes, "http.method", "http.status_code")
+logs:
+ keep_keys(attributes, "http.method", "http.status_code")
+```
+
+### Reduce cardinality of an attribute
+
+```
+traces:
+ replace_match(attributes["http.target"], "/user/*/list/*", "/user/{userId}/list/{listId}")
+```
+
+### Reduce cardinality of a span name
+
+```
+traces:
+ replace_match(name, "GET /user/*/list/*", "GET /user/{userId}/list/{listId}")
+``` 
+
+### Reduce cardinality of any matching attribute
+
+```
+traces:
+ replace_all_matches(attributes, "/user/*/list/*", "/user/{userId}/list/{listId}")
+``` 
+
+### Decrease the size of the telemetry payload
+
+```
+traces:
+ delete(resource.attributes["process.command_line"])
+metrics:
+ delete(resource.attributes["process.command_line"])
+logs:
+ delete(resource.attributes["process.command_line"])
+```
+
+### Drop specific telemetry
+
+```
+metrics:
+ drop() where attributes["http.target"] = "/health"
+```
+
+### Attach information from resource into telemetry
+
+```
+metrics:
+ set(attributes["k8s_pod"], resource.attributes["k8s.pod.name"])
+```
+
+### Group spans by trace ID
+
+```
+traces:
+ group_by(trace_id, 2m)
+```
+
+
+### Update a spans ID
+
+```
+logs:
+ set(span_id, SpanID(0x0000000000000000))
+traces:
+ set(span_id, SpanID(0x0000000000000000))
+```
+
+### Create utilization metric from base metrics.
+
+```
+metrics:
+ create_gauge("pod.cpu.utilized", read_gauge("pod.cpu.usage") / read_gauge("node.cpu.limit")
+```

pkg/telemetryquerylanguage/tql/condition.go

@@ -0,0 +1,57 @@
+// Copyright The OpenTelemetry Authors
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+// http:https://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package tql // import "github.com/open-telemetry/opentelemetry-collector-contrib/pkg/telemetryquerylanguage/tql"
+
+import (
+ "fmt"
+)
+
+type CondFunc = func(ctx TransformContext) bool
+
+var alwaysTrue = func(ctx TransformContext) bool {
+ return true
+}
+
+func newConditionEvaluator(cond *Condition, functions map[string]interface{}, pathParser PathExpressionParser) (CondFunc, error) {
+ if cond == nil {
+ return alwaysTrue, nil
+ }
+ left, err := NewGetter(cond.Left, functions, pathParser)
+ if err != nil {
+ return nil, err
+ }
+ right, err := NewGetter(cond.Right, functions, pathParser)
+ // TODO(anuraaga): Check if both left and right are literals and const-evaluate
+ if err != nil {
+ return nil, err
+ }
+
+ switch cond.Op {
+ case "==":
+ return func(ctx TransformContext) bool {
+ a := left.Get(ctx)
+ b := right.Get(ctx)
+ return a == b
+ }, nil
+ case "!=":
+ return func(ctx TransformContext) bool {
+ a := left.Get(ctx)
+ b := right.Get(ctx)
+ return a != b
+ }, nil
+ }
+
+ return nil, fmt.Errorf("unrecognized boolean operation %v", cond.Op)
+}