Question: Is there a good way to split the collector?

[frzifus]

Is your feature request related to a problem? Please describe.
I'm not sure if the issue belongs here or more in contrib, but...
Currently core and contrib apk's, deb's and rpm's are published for the linux package managers. Distributions with a specific set of components can only be created by a custom build (e.g. with the help of ocb). This package must then be installed manual or made available via extra package sources.

I am looking for a way to divide the collector into different modules and make them installable via a package manager.

Example

graph TD;
    A[otelcol-core-receiver-otlp-v0.80.0.rpm]-->C[otlpcol-core-v0.80.0.rpm];
    B[otelcol-core-exporter-logging-v0.80.0.rpm]-->C[otlpcol-core-v0.80.0.rpm];
    D[otelcol-contrib-exporter-jaeger-v0.80.0.rpm]-->C[otlpcol-core-v0.80.0.rpm];

Loading

Describe the solution you'd like
Some way to dynamically load extensions, receivers, connectors and exporters into the collector.

Describe alternatives you've considered
I haven't really looked into it yet, but I have seen that go might support WASI in the future. Maybe it would be a possibility to load modules into a WASI runtime.

Additional context
With a few modifications, I got a prototype up and running. I replaced the components.go file and added a main file in a plugin folder in the otlpreceiver and loggingexporter components.
See: diff to v0.80.0

[mx-psi]

I am not sure using plugin is a good idea; see golang/go#19282 (comment) where the fourth most active contributor to the Go compiler says that it is half-baked and has significant issues on supported platforms. The comment is not very specific about what those issues are, so it's hard to check if they are fixed, but it is very concerning. It would also not be available on Windows as can be seen on golang/go/issues/19282.

For a WebAssembly based solution, see open-telemetry/opentelemetry-collector-contrib/issues/11772, which was abandoned but could be retaken. This could be done in pure Go.

I also wonder if we could leverage OpAMP to do this. I tried doing this but got delayed by other priorities, but I think, while OpAMP package installation is designed for version updates, using it for installing different Collector versions that have different component sets should work.

[frzifus]

I am not sure using plugin is a good idea; ...

oh me neither. It was the first practical solution that worked well for me, at least locally (linux/amd64). There is also a big warning list regarding go plugins, which I have come across.

But since the same source tree should always be used in the Linux packaging scenario and it refers to packaging exclusively under Linux, I didn't see any problems so far. Furthermore, because it did not change anything in the current distributions. It would legendarily create another "otel-plugin" distro.

Just as an aside, go plugins are not exactly ideal because of the file size. An empty collector + otlpreceiver + loggingexporter would consume ~100MB disk space. 😅

$ ls -la
.rwxr-xr-x@  38M frzifus 23 Jun 23:10 otelplugcol
.rw-r--r--@  28M frzifus 23 Jun 23:08 loggingexporter.so
.rw-r--r--@  33M frzifus 23 Jun 23:06 otlpreceiver.so

For a WebAssembly based solution...

It looks like collector-contrib#11772 is a bit different. It focuses on adding a (e.g. processor) to a collector distro, which can then load other wasm processors. I am looking for a seamless integration where the default collector is delivered without a single processor, receiver or exporter, but those can be added by once needed.

But yes, basically you need a wasi runtime in pure go, a way to hide loaded and created vm's behind the factory interfaces, and the ability to make requests of any kind from within these vm's. (e.g. to read env variables and use them inside a processor or establish a tcp connection to act like an exporter)

I also wonder if we could leverage OpAMP to do this..

In that sense that your OpAMP server would build a new distro, based on your needs and tell the client to exchange the existing one? In that case, its hard to support it. Building distros on demand was recently discussed in the operator SIG and had turned out to be tricky and seems to end up in a new k8s distro.

---

In principle, I think such a plugin system could also positively influence the operator. The proposed set of components for a Kubernetes distro could be included in the bundle and, if necessary, additional modules could be made available to the operator via configmap. Removing the need of building specific distros.

[mx-psi]

Furthermore, because it did not change anything in the current distributions. It would legendarily create another "otel-plugin" distro.

It seems like the current code in opentelemetry-collector used as a library can support having this as a downstream distro. I think it's good if we try this out, but as I said above, I am skeptical that the plugin approach specifically would be a good approach and that we should make any kind of 'official' releases based on it if the Go team does not support this.

It focuses on adding a (e.g. processor) to a collector distro, which can then load other wasm processors. I am looking for a seamless integration where the default collector is delivered without a single processor, receiver or exporter, but those can be added by once needed.

What I was trying to say is that open-telemetry/opentelemetry-collector-contrib/issues/11772 would bring us one step closer to make Collector components installable separately. It is not a complete solution (it seems directly applicable only to processors and connectors), but it does bring a couple of good ideas that can be reused for working towards a full solution.

In that sense that your OpAMP server would build a new distro, based on your needs and tell the client to exchange the existing one?

Yes, that's basically what I had in mind

In that case, its hard to support it. open-telemetry/opentelemetry-operator#1817 was recently discussed in the operator SIG and had turned out to be tricky and seems to end up in a open-telemetry/opentelemetry-collector-releases#357.

Thanks for the link! Reading through the meeting notes I don't quite understand what parts seem hard. Could you elaborate on that? I think any solution is going to be hard, so I wouldn't want to discard this approach right off the start.

[frzifus]

cc @chilleregeravi @ridwanmsharif @henrikrexed