Status | |
---|---|
Stability | development: traces, metrics, logs |
Distributions | contrib |
Issues | |
Code Owners | @jmacd, @moh-osman3 |
Receives telemetry data using OpenTelemetry Protocol with Apache Arrow and standard OTLP protocol via gRPC.
The OpenTelemetry Protocol with Apache Arrow receiver is an extension of the core OpenTelemetry Collector OTLP receiver component with additional support for OpenTelemetry Protocol with Apache Arrow.
OpenTelemetry Protocol with Apache Arrow supports column-oriented data transport using the Apache Arrow data format. The OpenTelemetry Protocol with Apache Arrow exporter converts OTLP data into an optimized representation and then sends batches of data using Apache Arrow to encode the stream. This component contains logic to reverse the process used in the OpenTelemetry Protocol with Apache Arrow exporter.
The use of an OpenTelemetry Protocol with Apache Arrow exporter-receiver pair is recommended when the network is expensive. Typically, expect to see a 50% reduction in bandwidth compared with the same data being sent using standard OTLP/gRPC and gzip compression.
This component includes all the features and configuration of the core OTLP receiver, making it possible to upgrade from the core component simply by replacing "otlp" with "otelarrow" as the component name in the collector configuration.
To enable the OpenTelemetry Protocol with Apache Arrow receiver, include it in the list of receivers for a pipeline. No further configuration is needed. This receiver listens on the standard OTLP/gRPC port 4317 and serves standard OTLP over gRPC out of the box.
receivers:
otelarrow:
Users may wish to configure gRPC settings, for example:
receivers:
otelarrow:
protocols:
grpc:
...
endpoint
(default = 0.0.0.0:4317 for grpc protocol): host:port to which the receiver is going to receive data. The valid syntax is described at https://github.com/grpc/grpc/blob/master/doc/naming.md.
Several common configuration structures provide additional capabilities automatically:
In the arrow
configuration block, the following settings are available:
memory_limit_mib
(default: 128): limits the amount of concurrent memory used by Arrow data buffers.
When the limit is reached, the receiver will return RESOURCE_EXHAUSTED error codes to the receiver, which are conditionally retryable, see exporter retry configuration.
In the arrow
configuration block, zstd
sub-section applies to all
compression levels used by exporters:
memory_limit_mib
limits memory dedicated to Zstd decompression, per stream (default 128)max_window_size_mib
: maximum size of the Zstd window in MiB, 0 indicates to determine based on level (default 32)concurrency
: controls background CPU used for decompression, 0 indicates to letzstd
library decide (default 1)
As a gRPC streaming service, the OTel Arrow receiver is able to limit stream lifetime through configuration of the underlying http/2 connection via keepalive settings.
Keepalive settings are vital to the operation of OTel Arrow, because longer-lived streams use more memory and streams are fixed to a single host. Since every stream of data is different, we recommend experimenting to find a good balance between memory usage, stream lifetime, and load balance.
gRPC libraries do not build-in a facility for long-lived RPCs to learn about impending http/2 connection state changes, including the event that initiates connection reset. While the receiver knows its own keepalive settings, a shorter maximum connection lifetime can be imposed by intermediate http/2 proxies, and therefore the receiver and exporter are expected to independently configure these limits.
receivers:
otelarrow:
protocols:
grpc:
keepalive:
server_parameters:
max_connection_age: 1m
max_connection_age_grace: 10m
In the example configuration above, OpenTelemetry Protocol with Apache
Arrow streams will have reset initiated after 10 minutes. Note that
max_connection_age
is set to a small value and we recommend tuning
max_connection_age_grace
.
OTel Arrow exporters are expected to configure their
max_stream_lifetime
property to a value that is slightly smaller
than the receiver's max_connection_age_grace
setting, which causes
the exporter to cleanly shut down streams, allowing requests to
complete before the http/2 connection is forcibly closed. While the
exporter will retry data that was in-flight during an unexpected
stream shutdown, instrumentation about the telemety pipeline will show
RPC errors when the exporter's max_stream_lifetime
is not configured
correctly.
See the exporter README for more
guidance. For the
example where max_connection_age_grace
is set to 10 minutes, the
exporter's max_stream_lifetime
should be set to the same number
minus a reasonable timeout to allow in-flight requests to complete.
For example, an exporter with 9m30s
stream lifetime:
exporters:
otelarrow:
timeout: 30s
arrow:
max_stream_lifetime: 9m30s
endpoint: ...
tls: ...
In addition to the the standard
obsreport
metrics, this component provides network-level measurement instruments
which we anticipate will become part of obsreport
in the future. At
the normal
level of metrics detail:
receiver_recv
: uncompressed bytes received, prior to compressionreceiver_recv_wire
: compressed bytes received, on the wire.
Arrow's compression performance can be derived by dividing the average
receiver_recv
value by the average receiver_recv_wire
value.
At the detailed
metrics detail level, information about the stream
of data being returned from the receiver will be instrumented:
receiver_sent
: uncompressed bytes sent, prior to compressionreceiver_sent_wire
: compressed bytes sent, on the wire.
There several OpenTelemetry Protocol with Apache Arrow-consumer related metrics available to help diagnose internal performance. These are disabled at the basic level of detail. At the normal level, these metrics are introduced:
arrow_batch_records
: Counter of Arrow-IPC records processedarrow_memory_inuse
: UpDownCounter of memory in use by current streamsarrow_schema_resets
: Counter of times the schema was adjusted, by data type.
service
...
telemetry:
...
metrics:
...
level: detailed