Support for table creation on cluster

[dimei-BT]

Component(s)

exporter/clickhouse

Is your feature request related to a problem? Please describe.

Hi, it seems the current implementation will create the tables on the current server. Could we add the support for ON CLUSTER clause? Thanks!

Describe the solution you'd like

Potentially, we can add a config flag to control whether the query will include the ON CLUSTER clause.

Describe alternatives you've considered

Or we can have the query with ON CLUSTER clause by default.

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Frapschen]

we can have a section to config cluster relate param and I think we also need to consider support config Data Replication relate params:

cluster_model:
  enabled: true
  zoo_path_prefix: /clickhouse/tables/{shard}
  replica_name: {replica}

zoo_path_prefix is the prefix of /clickhouse/tables/{shard}/table_name, for trace table, it will be /clickhouse/tables/{shard}/otel_traces.

if zoo_path and replica_name are not empty we need to create tables with Replicated prefix.

@dimei-BT @hanjm any other suggestion?

[hanjm]

I think no need to config zoo_path_prefix and replica_name, it is a common pattern.
in cluster mode, should config collector write local table or distributed table. write local table by default allowing for better scalability.

[oliverdding]

Keeping an eye on this, I think it's a necessary feature that should be included.

It's better to make zoo_path_prefix and replica_name configurable, just like @Frapschen posted, since there may be possible conflict. And, cluster_name should be exported too.

Summary:

cluster_model:
  enabled: true
  cluster_name: default_cluster
  zoo_path_prefix: /clickhouse/tables/{database}/{table}/{shard}
  replica_name: {replica}

[oliverdding]

Hi @hanjm , I would vote for inserting local table for better performance. And the exporter should connect to the endpoint every time the data in ready.

In cluster mode (for example, 2 shard 2 replicas), users may use reverse-proxy, so he/she may set endpoint in two ways:

exporter:
  endpoint: reverse-proxy:9000

or

exporter:
  endpoint: clickhouse0:9000,clickhouse1:9000,clickhouse2:9000,clickhouse3:9000

• In the first circumstance, exporter should build connection on every inserting.
• In the latter one, exporter can hold multiple long-time connection and choose one on every inserting.

So, for simplicity, it's a good idea to connect the endpoint on every inserting.

[hanjm]

So, for simplicity, it's a good idea to connect the endpoint on every inserting

Agree, we can set short connection simply via db.SetMaxIdleConns(-1) .

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[rauanmayemir]

I manually created tables with ReplicatedMergeTree and it works, but what would be a proper DDL for otel_traces_trace_id_ts_mv?

[Frapschen]

I manually created tables with ReplicatedMergeTree and it works, but what would be a proper DDL for otel_traces_trace_id_ts_mv?

you can refer: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/clickhouseexporter/exporter_traces.go#L251

assume you had created a table manually for trace named otel.otel_trace , then create the id_ts table and mv table:

create table IF NOT EXISTS otel.otel_trace_id_ts (
     TraceId String CODEC(ZSTD(1)),
     Start DateTime64(9) CODEC(Delta, ZSTD(1)),
     End DateTime64(9) CODEC(Delta, ZSTD(1)),
     INDEX idx_trace_id TraceId TYPE bloom_filter(0.01) GRANULARITY 1
) ENGINE MergeTree()
TTL toDateTime(Timestamp) + toIntervalDay(1)
ORDER BY (TraceId, toUnixTimestamp(Start))
SETTINGS index_granularity=8192;

CREATE MATERIALIZED VIEW IF NOT EXISTS %s_trace_id_ts_mv
TO otel.otel_trace_id_ts
AS SELECT
TraceId,
min(Timestamp) as Start,
max(Timestamp) as End
FROM
otel.otel_trace
WHERE TraceId!=''
GROUP BY TraceId;

[rauanmayemir]

@Frapschen I've already looked them up and tweaked the engine.

But for Materialized View, it's not as trivial as just adding something like ReplicatedSummingMergeTree, there are also order and partition keys to consider.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.