Add event-listener plugin with Kafka sink

[marton-bod]

Currently open source Trino has only MySQL and http event listener plugins. These are not always scalable, especially when several larger Trino clusters are sending hundreds of thousands of events to the same centralized store. Kafka is equipped to handle that sort of load and therefore is used by several companies to sink their QueryCreatedEvents and QueryCompletedEvents. I would propose to create a new Trino plugin called trino-kafka-event-listener.

[mosabua]

That would be awesome .. I heard a mention at Trino Fest 2024 that you might know where such a plugin already exists ..

[YuriyGavrilov]

Hi @marton-bod try to look on https://quickwit.io/docs/ingest-data/ingest-api this should be scale enough for the hundred s of terabytes if you need.

I work to try to realize such logs on Quickwit and our Trino cluster. There are also ElasticSearch compatible api to add those log back to Trino as external connector.

For this moment I need to have full Event of Trino to make correct index in Quickwit: https://quickwit.io/docs/get-started/quickstart

it seams to be completely simply to add Trino events in Quickwit and no need to have connector. just need to have proper index yaml and event metadata from Trino.

Could you please guys @marton-bod @mosabua share some complete events metadata just an example?

so.... making some test and it's working out of the box:

with such config:

event-listener.name=http
http-event-listener.log-created=true
http-event-listener.connect-ingest-uri=http:https://quick:7280/api/v1/trino/ingest?commit=force
http-event-listener.log-created=true
http-event-listener.log-completed=true