[exporter/clickhouse] Why are batch inserts not used in clickhouse exporter?

[takyoni]

Component(s)

exporter/clickhouse

Describe the issue you're reporting

It seems that using batch insert is much more performant than just a large number of records.
Example: https://github.com/SigNoz/signoz-otel-collector/blob/main/exporter/clickhouselogsexporter/exporter.go#L129

[github-actions]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[cwegener]

Can you explain why you think that the clickhouse exporter is not using batched inserts? As far as I can tell it is batching the insert statements, but I might be wrong.

[takyoni]

In the code you can see that the insertion is performed by adding a large number of ordinary inserts to the transaction.
Batch insert example https://github.com/ClickHouse/clickhouse-go/blob/main/examples/clickhouse_api/batch.go#L52

[cwegener]

I see. I think maybe your question then is why is the Clickhouse exporter using the standard database/sql interface and not the native clickhouse API interface of the clickhouse-go driver?

I'm not sure why why standard database/sql interface was chosen.

[takyoni]

Yes you are right. This is not clear to me either.

[hanjm]

#18090 (comment)

#18090 (comment)