[receiver/jmx] Ability to discovery jmx endpoints dynamically

[neo-waiariki-koia]

Component(s)

receiver/jmx

Is your feature request related to a problem? Please describe.

At the moment, the endpoint for jmx receiver has to be hardcoded. Unfortunately, for my use case, this isn't feasible. I was hoping some sort of service discovery mechanism could be implemented to find endpoints dynamically? The prometheus receiver has this and, although different, Datadog agents can facilitate similar functionality.

Describe the solution you'd like

I'd like for the ability to discover jmx endpoints dynamically within a Kubernetes cluster. This could be done by filtering pods based on fields, labels or annotations.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/jmx: @rmfitzpatrick

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[JaredTan95]

From your description, I think it might be more appropriate to put this requirement into otel-operator?

[joerg]

+1 for that feature. Currently, fetching other metrics than the built in ones requires building a custom jar file (see #17072) which is an unreasonable amount of overhead for just a list of metrics to gather.
In my case, I need to gather metrics from the Atlassian tools (e.g. Jira https://confluence.atlassian.com/adminjiraserver/live-monitoring-using-the-jmx-interface-939707304.html) which would require me to:

• write the custom groovy script
• build the jar file
• and then build a custom otel-collector supporting my custom target system (https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/jmxreceiver/config.go#L200).

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/jmx: @rmfitzpatrick

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[hughesjj]

Hey, so there are observer that you can use with the receiver creator, but it doesn't introspect whether it's a jmx endpoint or not. We could try figuring out if there's any indicators to make it more likely to determine whether an endpoint is jmx, or add some special observer for such.

[hughesjj]

However, this doesn't resolve the problem with the jmx-metrics-gatherer not supporting arbitrary queries. There's some history on that topic, will update as I find the artifacts. TL;DR inclusion of jmx receiver in this distro was predicated on not allowing arbitrary code to be run

Updates:

• Issue 17072 -- prior request for custom groovy scripts
• Background summary of what was done in response to security concerns

We can talk about this more in a SIG meeting or on the CNCF slack (invite) if you prefer