Skip to content

Commit

Permalink
[receiver/postgresql] Convert database and table metric attributes to…
Browse files Browse the repository at this point in the history
… resource attributes (#12967)

- Remove database and table metric attributes
- Add corresponding resource attributes
- Control change with feature flag
  • Loading branch information
schmikei committed Aug 10, 2022
1 parent f5a44be commit 29a3e33
Show file tree
Hide file tree
Showing 12 changed files with 3,881 additions and 45 deletions.
31 changes: 30 additions & 1 deletion receiver/postgresqlreceiver/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,4 +64,33 @@ The full list of settings exposed for this receiver are documented [here](./conf
Details about the metrics produced by this receiver can be found in [metadata.yaml](./metadata.yaml)

[beta]: https://github.com/open-telemetry/opentelemetry-collector#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib

### Feature gate configurations

#### Transition from metrics without "resource_attributes"

All metrics are being transitioning to moving the metric attributes `table` and `database` to resource attributes `postgresql.table` and `postgresql.database` respectively. This effort is motivated via the resource specification found [in the metrics data model](https://github.com/open-telemetry/opentelemetry-specification/blob/141a3ef0bf1eba0b6d260335bbe0ce7af9387cfc/specification/metrics/data-model.md#resource-attributes-1).

Eventually the move will be finalized, but there will be a transitional period where metrics will emit with resource attributes behind a feature gate.

##### Transition Schedule

1. v0.58.0, August 2022:

- The version of the metrics receiver with resource attributes will be available via feature gates.
- The old metrics with `table` and `database` metric attributes are deprecated with a warning.
- `receiver.postgresql.emitMetricsWithResourceAttributes` is *disabled* by default.
- `receiver.postgresql.emitMetricsWithoutResourceAttributes` is *enabled* by default.

2. v0.60.0, September 2022:

- The new collection method with resource attributes is enabled by default. The old metrics with the `table` and `database` metric attributes is disabled by default.
- `receiver.postgresql.emitMetricsWithResourceAttributes` is *enabled* by default.
- `receiver.postgresql.emitMetricsWithoutResourceAttributes` is *disabled* by default.

3. v0.62.0, October 2022:

- The feature gates are removed.
- Metrics collection using resource attributes are always emitted
- Metrics collection using the `database` and `table` metric attributes are no longer available.
7 changes: 7 additions & 0 deletions receiver/postgresqlreceiver/documentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,13 @@ metrics:
enabled: <true|false>
```

## Resource attributes

| Name | Description | Type |
| ---- | ----------- | ---- |
| postgresql.database.name | The name of the database. | String |
| postgresql.table.name | The schema name followed by the table name. | String |

## Metric attributes

| Name | Description | Values |
Expand Down
31 changes: 31 additions & 0 deletions receiver/postgresqlreceiver/integration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,16 +29,22 @@ import (
"github.com/testcontainers/testcontainers-go/wait"
"go.opentelemetry.io/collector/component/componenttest"
"go.opentelemetry.io/collector/consumer/consumertest"
"go.opentelemetry.io/collector/service/featuregate"

"github.com/open-telemetry/opentelemetry-collector-contrib/internal/scrapertest"
"github.com/open-telemetry/opentelemetry-collector-contrib/internal/scrapertest/golden"
)

type configFunc func(hostname string) *Config

// cleanupFunc exists to allow integration test cases to clean any registries it had
// to modify in order to change behavior of the integration test. i.e. featuregates
type cleanupFunc func()

type testCase struct {
name string
cfg configFunc
cleanup cleanupFunc
expectedFile string
}

Expand Down Expand Up @@ -86,6 +92,28 @@ func TestPostgreSQLIntegration(t *testing.T) {
},
expectedFile: filepath.Join("testdata", "integration", "expected_all_db.json"),
},
{
name: "with_resource_attributes",
cfg: func(hostname string) *Config {
featuregate.GetRegistry().MustApply(map[string]bool{
emitMetricsWithResourceAttributesFeatureGateID: true,
})
f := NewFactory()
cfg := f.CreateDefaultConfig().(*Config)
cfg.Endpoint = net.JoinHostPort(hostname, "15432")
cfg.Databases = []string{}
cfg.Username = "otel"
cfg.Password = "otel"
cfg.Insecure = true
return cfg
},
cleanup: func() {
featuregate.GetRegistry().MustApply(map[string]bool{
emitMetricsWithResourceAttributesFeatureGateID: false,
})
},
expectedFile: filepath.Join("testdata", "integration", "expected_all_with_resource_attributes.json"),
},
}

container := getContainer(t, testcontainers.ContainerRequest{
Expand All @@ -105,6 +133,9 @@ func TestPostgreSQLIntegration(t *testing.T) {

for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
if tc.cleanup != nil {
defer tc.cleanup()
}
expectedMetrics, err := golden.ReadMetrics(tc.expectedFile)
require.NoError(t, err)

Expand Down
125 changes: 125 additions & 0 deletions receiver/postgresqlreceiver/internal/metadata/custom.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http:https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package metadata // import "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/postgresqlreceiver/internal/metadata"

import "go.opentelemetry.io/collector/pdata/pcommon"

// RecordPostgresqlDbSizeDataPointWithoutDatabase adds a data point to postgresql.db_size metric without a database metric attribute
func (mb *MetricsBuilder) RecordPostgresqlDbSizeDataPointWithoutDatabase(ts pcommon.Timestamp, val int64) {
mb.metricPostgresqlDbSize.recordDatapointWithoutDatabase(mb.startTime, ts, val)
}

func (m *metricPostgresqlDbSize) recordDatapointWithoutDatabase(start pcommon.Timestamp, ts pcommon.Timestamp, val int64) {
if !m.settings.Enabled {
return
}
dp := m.data.Sum().DataPoints().AppendEmpty()
dp.SetStartTimestamp(start)
dp.SetTimestamp(ts)
dp.SetIntVal(val)
}

// RecordPostgresqlBackendsDataPointWithoutDatabase adds a data point to postgresql.backends metric.
func (mb *MetricsBuilder) RecordPostgresqlBackendsDataPointWithoutDatabase(ts pcommon.Timestamp, val int64) {
mb.metricPostgresqlBackends.recordDatapointWithoutDatabase(mb.startTime, ts, val)
}

func (m *metricPostgresqlBackends) recordDatapointWithoutDatabase(start pcommon.Timestamp, ts pcommon.Timestamp, val int64) {
if !m.settings.Enabled {
return
}
dp := m.data.Sum().DataPoints().AppendEmpty()
dp.SetStartTimestamp(start)
dp.SetTimestamp(ts)
dp.SetIntVal(val)
}

// RecordPostgresqlBlocksReadDataPointWithoutDatabaseAndTable adds a data point to postgresql.blocks_read metric.
func (mb *MetricsBuilder) RecordPostgresqlBlocksReadDataPointWithoutDatabaseAndTable(ts pcommon.Timestamp, val int64, sourceAttributeValue AttributeSource) {
mb.metricPostgresqlBlocksRead.recordDatapointWithoutDatabaseAndTable(mb.startTime, ts, val, sourceAttributeValue.String())
}

func (m *metricPostgresqlBlocksRead) recordDatapointWithoutDatabaseAndTable(start pcommon.Timestamp, ts pcommon.Timestamp, val int64, sourceAttributeValue string) {
if !m.settings.Enabled {
return
}
dp := m.data.Sum().DataPoints().AppendEmpty()
dp.SetStartTimestamp(start)
dp.SetTimestamp(ts)
dp.SetIntVal(val)
dp.Attributes().InsertString("source", sourceAttributeValue)
}

// RecordPostgresqlCommitsDataPointWithoutDatabase adds a data point to postgresql.commits metric without the database metric attribute
func (mb *MetricsBuilder) RecordPostgresqlCommitsDataPointWithoutDatabase(ts pcommon.Timestamp, val int64) {
mb.metricPostgresqlCommits.recordDatapointWithoutDatabase(mb.startTime, ts, val)
}

func (m *metricPostgresqlCommits) recordDatapointWithoutDatabase(start pcommon.Timestamp, ts pcommon.Timestamp, val int64) {
if !m.settings.Enabled {
return
}
dp := m.data.Sum().DataPoints().AppendEmpty()
dp.SetStartTimestamp(start)
dp.SetTimestamp(ts)
dp.SetIntVal(val)
}

// RecordPostgresqlRollbacksDataPointWithoutDatabase adds a data point to postgresql.commits metric without the database metric attribute
func (mb *MetricsBuilder) RecordPostgresqlRollbacksDataPointWithoutDatabase(ts pcommon.Timestamp, val int64) {
mb.metricPostgresqlRollbacks.recordDatapointWithoutDatabase(mb.startTime, ts, val)
}

func (m *metricPostgresqlRollbacks) recordDatapointWithoutDatabase(start pcommon.Timestamp, ts pcommon.Timestamp, val int64) {
if !m.settings.Enabled {
return
}
dp := m.data.Sum().DataPoints().AppendEmpty()
dp.SetStartTimestamp(start)
dp.SetTimestamp(ts)
dp.SetIntVal(val)
}

// RecordPostgresqlRowsDataPointWithoutDatabaseAndTable adds a data point to postgresql.rows metric without the database or table metric attribute.
func (mb *MetricsBuilder) RecordPostgresqlRowsDataPointWithoutDatabaseAndTable(ts pcommon.Timestamp, val int64, stateAttributeValue AttributeState) {
mb.metricPostgresqlRows.recordDatapointWithoutDatabaseAndTable(mb.startTime, ts, val, stateAttributeValue.String())
}

func (m *metricPostgresqlRows) recordDatapointWithoutDatabaseAndTable(start pcommon.Timestamp, ts pcommon.Timestamp, val int64, stateAttributeValue string) {
if !m.settings.Enabled {
return
}
dp := m.data.Sum().DataPoints().AppendEmpty()
dp.SetStartTimestamp(start)
dp.SetTimestamp(ts)
dp.SetIntVal(val)
dp.Attributes().InsertString("state", stateAttributeValue)
}

// RecordPostgresqlOperationsDataPointWithoutDatabaseAndTable adds a data point to postgresql.operations metric without the database or table metric attribute
func (mb *MetricsBuilder) RecordPostgresqlOperationsDataPointWithoutDatabaseAndTable(ts pcommon.Timestamp, val int64, operationAttributeValue AttributeOperation) {
mb.metricPostgresqlOperations.recordDatapointWithoutDatabaseAndTable(mb.startTime, ts, val, operationAttributeValue.String())
}

func (m *metricPostgresqlOperations) recordDatapointWithoutDatabaseAndTable(start pcommon.Timestamp, ts pcommon.Timestamp, val int64, operationAttributeValue string) {
if !m.settings.Enabled {
return
}
dp := m.data.Sum().DataPoints().AppendEmpty()
dp.SetStartTimestamp(start)
dp.SetTimestamp(ts)
dp.SetIntVal(val)
dp.Attributes().InsertString("operation", operationAttributeValue)
}

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

39 changes: 28 additions & 11 deletions receiver/postgresqlreceiver/metadata.yaml
Original file line number Diff line number Diff line change
@@ -1,20 +1,37 @@
name: postgresqlreceiver

resource_attributes:
postgresql.database.name:
description: The name of the database.
type: string
postgresql.table.name:
description: The schema name followed by the table name.
type: string

attributes:
database:
description: The name of the database.
type: string
table:
description: The schema name followed by the table name.
type: string
source:
description: The block read source type.
enum: [ heap_read, heap_hit, idx_read, idx_hit, toast_read, toast_hit, tidx_read, tidx_hit ]
enum:
- heap_read
- heap_hit
- idx_read
- idx_hit
- toast_read
- toast_hit
- tidx_read
- tidx_hit
operation:
description: The database operation.
enum: [ ins, upd, del, hot_upd ]
enum: [ins, upd, del, hot_upd]
state:
description: The tuple (row) state.
enum: [ dead, live ]

enum: [dead, live]

metrics:
postgresql.blocks_read:
Expand All @@ -25,7 +42,7 @@ metrics:
value_type: int
monotonic: true
aggregation: cumulative
attributes: [ database, table, source ]
attributes: [database, table, source]
postgresql.commits:
enabled: true
description: The number of commits.
Expand All @@ -34,7 +51,7 @@ metrics:
value_type: int
monotonic: true
aggregation: cumulative
attributes: [ database ]
attributes: [database]
postgresql.db_size:
enabled: true
description: The database disk usage.
Expand All @@ -43,7 +60,7 @@ metrics:
value_type: int
monotonic: false
aggregation: cumulative
attributes: [ database ]
attributes: [database]
postgresql.backends:
enabled: true
description: The number of backends.
Expand All @@ -52,7 +69,7 @@ metrics:
value_type: int
monotonic: false
aggregation: cumulative
attributes: [ database ]
attributes: [database]
postgresql.rows:
enabled: true
description: The number of rows in the database.
Expand All @@ -61,7 +78,7 @@ metrics:
value_type: int
monotonic: false
aggregation: cumulative
attributes: [ database, table, state ]
attributes: [database, table, state]
postgresql.operations:
enabled: true
description: The number of db row operations.
Expand All @@ -70,7 +87,7 @@ metrics:
value_type: int
monotonic: true
aggregation: cumulative
attributes: [ database, table, operation ]
attributes: [database, table, operation]
postgresql.rollbacks:
enabled: true
description: The number of rollbacks.
Expand All @@ -79,4 +96,4 @@ metrics:
value_type: int
monotonic: true
aggregation: cumulative
attributes: [ database ]
attributes: [database]
Loading

0 comments on commit 29a3e33

Please sign in to comment.