Skip to content

Latest commit

 

History

History
 
 

googlecloudpubsubreceiver

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
internal
internal
 
 
testdata
testdata
 
 
Makefile
Makefile
 
 
README.md
README.md
 
 
config.go
config.go
 
 
config_test.go
config_test.go
 
 
factory.go
factory.go
 
 
factory_test.go
factory_test.go
 
 
go.mod
go.mod
 
 
go.sum
go.sum
 
 
receiver.go
receiver.go
 
 
receiver_test.go
receiver_test.go
 
 

README.md

Google Pubsub Receiver

Status
Stability beta
Supported pipeline types traces, logs, metrics
Distributions none

⚠️ This is a community-provided module. It has been developed and extensively tested at Collibra, but it is not officially supported by GCP.

This receiver gets OTLP messages from a Google Cloud Pubsub subscription.

The following configuration options are supported:

  • project (Optional): The Google Cloud Project of the client connects to.
  • subscription (Required): The subscription name to receive OTLP data from. The subscription name should be a fully qualified resource name (eg: projects/otel-project/subscriptions/otlp).
  • encoding (Optional): The encoding that will be used to received data from the subscription. This can either be otlp_proto_trace, otlp_proto_metric, otlp_proto_log, or raw_text (see encoding). This will only be used as a fallback, when no content-type attribute is present.
  • compression (Optional): The compression that will be used on received data from the subscription. When set it can only be gzip. This will only be used as a fallback, when no content-encoding attribute is present.
  • endpoint (Optional): Override the default Pubsub Endpoint, useful when connecting to the PubSub emulator instance or switching between global and regional service endpoints.
  • insecure (Optional): allows performing “insecure” SSL connections and transfers, useful when connecting to a local emulator instance. Only has effect if Endpoint is not ""
receivers:
  googlecloudpubsub:
    project: otel-project
    subscription: projects/otel-project/subscriptions/otlp-logs
    encoding: raw_json

Encoding

You should not need to set the encoding of the subscription as the receiver will try to discover the type of the data by looking at the ce-type and content-type attributes of the message. Only when those attributes are not set must the encoding field in the configuration be set.

ce-type] ce-datacontenttype encoding description
org.opentelemetry.otlp.traces.v1 application/protobuf Decode OTLP trace message
org.opentelemetry.otlp.metrics.v1 application/protobuf Decode OTLP metric message
org.opentelemetry.otlp.logs.v1 application/json Decode OTLP log message
- - otlp_proto_trace Decode OTLP trace message
- - otlp_proto_metric Decode OTLP trace message
- - otlp_proto_log Decode OTLP trace message
- - raw_text Wrap in an OTLP log message

When the encoding configuration is set, the attributes on the message are ignored.

The receiver can be used for ingesting arbitrary text message on a Pubsub subscription and wrap them in OTLP Log message, making it a convenient way to ingest log lines from Pubsub.

Pubsub subscription

The Google Cloud Pubsub receiver doesn't automatically create subscriptions, it expects the subscription to be created upfront. Security wise it's best to give the collector its own service account and give the subscription Pub/Sub Subscriber permission.

The subscription should also be of delivery type Pull.

Filtering

When the messages on the subscription are accompanied by the correct attributes and you only need a specific type in your pipeline, the messages can be filtered on the subscription saving on egress fees.

An example of filtering on trace message only:

attributes.ce-type = "org.opentelemetry.otlp.traces.v1"
AND
attributes.content-type = "application/protobuf"