-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Configure second-stage exporters #1609
Comments
Closes open-telemetry#1609. Signed-off-by: Juraci Paixão Kröhling <[email protected]>
cc @tigrannajaryan, @bogdandrutu, @pjanotti , would you have a few minutes to share your opinions? |
What about other aspects of pipelines? They support processors and multiple exporters, for example. If you just need exporters then declaring a pipeline seems redundant.
Your use case seems specific enough to have explicit dedicated section in the config. Also, the "second stage" term needs to be documented IMO. |
Good point about the processors. Multiple exporters are indeed part of the design.
To be honest, this was the least intrusive way I found to get the exporter instances ready and shared across the components. If we are going with a dedicated construct, it might have a bigger impact in the current code base, but it could look like this: receivers:
examplereceiver:
processors:
exampleprocessor:
exporters:
exampleexporter:
exampleexporter/second-stage:
exampleexporter/unused:
service:
pipelines:
traces:
receivers: [examplereceiver]
processors: [exampleprocessor]
exporters: [exampleexporter]
extra: # what could be a better name here?
traces:
exporters: [exampleexporter/second-stage] |
Unfortunately I am not sure I like either of the suggested approaches.
I think it will be confusing to have a pipelines that consist of just exporters.
I am not sure how you can do this. Exporters are not created at all unless they are part of a pipeline. The Collector will not know how to create an exporter because the pipeline type dictates how the exporter is created. Ideally the Collector should be able to figure out which exporters need to be created by looking at the |
How about a new interface, like: type WithDependentExporters interface {
DependentExportersNames() []string // Like: {traces/exampleexporter/second-stage}
} The config loader, once it loads the exporter, would check whether the exporter implements the interface and would add the returned dependencies to the list of exporters to instantiate. This would allow us to have a config without the receivers:
examplereceiver:
processors:
exampleprocessor:
exporters:
exampleexporter:
exampleexporter/second-stage:
exampleexporter/unused:
routing:
default_exporters:
traces: [exampleexporter/second-stage]
service:
pipelines:
traces:
receivers: [routing]
processors: [exampleprocessor]
exporters: [exampleexporter] |
Alternatively, you may just embed "second stage" exporters into the routing exporter: exporters:
routing:
embedded_exporters:
# duplicates the top-level exporters config
logging:
otlp:
jaeger/globex:
endpoint: "https://jaeger.globex.example.com:14250"
jaeger/ecorp:
endpoint: "https://jaeger.ecorp.example.com:14250"
saas/ecorp:
endpoint: "https://saas.globex.example.com:8443"
license: xyz
# references to exporters from the embedded_exporters, not top-level exporters
default_exporters: [otlp]
table:
- value: acme
exporters: [logging]
- value: globex
exporters: [jaeger/globex, saas/globex]
- value: ecorp
exporters: [jaeger/ecorp] This will require reusing code from the exporters builder opentelemetry-collector/service/builder/exporters_builder.go Lines 153 to 154 in eb0896e
but will give you a complete freedom on the preferred configuration format / flexibility. In that case you don't need any changes in the core collector code at all. |
One of the previous concerns was about reusing instances from the main collector. In your example, the routing exporter would have second-stage instances of its own, right? I don't think it would be a problem, but perhaps @tigrannajaryan has a specific concern about it? |
This is an option but makes the "second stage" exporters special. For example they will be no longer visible from GetExporters() functions to other components. I don't think this is desirable.
This is also a possibility but requires changes to the core which I am not sure we want to do yet. I would try something else for now. Instead of making this an exporter make it a routing processor, like this: processors:
routing:
from_attribute: X-Tenant
default_exporters: jaeger/default
table:
- value: acme
exporters: [jaeger/acme]
service:
pipelines:
traces/routing:
receivers: [otlp]
processors: [routing]
exporters:
- jaeger/default
- jaeger/acme This way no changes are need in the core. This requires that exporters are listed twice: once in the "service.exporters" and again in the routing table of "processors.routing", which is a downside, but limits the awkwardness too routing component only. The routing processor will obviously have to not propagate anything to its default consumer in the pipeline (which would be all of the exporters) and will only need to send the data to the exporter matching the routing table. If the routing exporter see a lot of usage we can look into adding more support to the core to remove config redundancy. Long term we may want to have more flexible ways of chaining pipelines which may also handle the routing use case. What do you think? |
We can catch bugs related to this in the routing processor and log accordingly.
This could be confusing to users at first, as it would basically mean that this processor has exporter semantics. But sounds good. Can we determine whether the next consumer is a processor? If we detect that this is the case, we might want to log a warning as well (like: "the next consumer is a processor, but it won't ever receive any data"). That said, I'm +1 to this idea, especially because it allows us to start small and iterate based on feedback. |
Yes, we can by checking that the next consumer implements the Processor interface (which is distinct from Exporter interface).
Good idea. |
Closing this since we decided to go with a processor instead of an exporter for now. |
Closes open-telemetry/opentelemetry-collector#1609 Signed-off-by: Juraci Paixão Kröhling <[email protected]>
Closes open-telemetry/opentelemetry-collector#1609 Signed-off-by: Juraci Paixão Kröhling <[email protected]>
Closes open-telemetry/opentelemetry-collector#1609 Signed-off-by: Juraci Paixão Kröhling <[email protected]>
Closes open-telemetry/opentelemetry-collector#1609 Signed-off-by: Juraci Paixão Kröhling <[email protected]>
As part of #1260, we need a way to add second-stage exporters to the
GetExporters()
. I can see two possible solutions for it:GetExporters()
so that it returns all exporters from the configuration, even if they aren't declared as part of any pipelines.I think the first suggestion is cleaner, as it requires users to declare any second-stage exporters explicitly, differentiating them from unused exporters.
If we decide to go with the second suggestion, one additional aspect to solve is that exporters are tied to their types, which is determined based on the exporter's declaration in the pipeline:
In the case above,
Exporters#ToMapByDataType
will not returnexampleexporter/2
even when it's part of the list of exporters.The text was updated successfully, but these errors were encountered: