kibana metrics endpoint reporting as down wth "server returned HTTP status 404 Not Found"

[davejhahn]

Which chart:
kibana-5.2.6

Describe the bug
Installed bitnami Kibana chart with metrics.enabled=true,metrics.serviceMonitor.enabled=true,metrics.serviceMonitor.namespace=monitoring. The ServiceMonitor is created, but there are no pods or sidecars created for metrics. Target endpoints appears in Prometheus but as DOWN with "server returned HTTP status 404 Not Found".

To Reproduce
Steps to reproduce the behavior:

1. Install Prometheus Operator
2. Install Kibana helm chart with values metrics.enabled=true,metrics.serviceMonitor.enabled=true,metrics.serviceMonitor.namespace=
3. Visit Prometheus http:https://127.0.0.1:9090 and select Status->Targets
4. Reports http:https://10.244.2.150:5601/_prometheus/metrics as DOWN with error "server returned HTTP status 404 Not Found". Endpoint being used in my case is http:https://10.244.2.150:5601/_prometheus/metrics

Expected behavior
Endpoint is accessible and returning UP

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13.8"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"d224476cd0730baca2b6e357d144171ed74192d6", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.9", GitCommit:"a2cfecc33f6911261fb8a670d2df2714094002a6", GitTreeState:"clean", BuildDate:"2020-05-29T21:50:25Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Additional context
Based on the URL, not sure if metrics are built into Kibana (same port as Kibana, 5601) or if this URL being reported is wrong. Mentioned above, no other pods are created or sidecars added so not sure where metrics are supposed to come from.

[pbn4]

Did you solve your issue?

[davejhahn]

@pbn4 no I didn't actually solve it.

I had closed it because I read in the helm chart a note that it requires kibana-prometheus-exporter plugin which I didn't realize and assumed that was the issue.

However, I added the 7.8.0 version of the kibana-prometheus-exporter plugin (to align with the Kibana version for the latest chart) using the plugins[0] value and although it added it (I can see it in the in /opt/bitnami/kibana/plugins/kibana-prometheus-exporter) I still get a 404 error when hitting the endpoint (e.g. http:https://127.0.0.1:5601/_prometheus/metrics from the pod itself or from the Prometheus endpoints, it shows DOWN).

On the pod, I ran: ./kibana-plugin list which shows:
[email protected]

So it appears to have installed.

I opened an issue with pjhamton pjhampton/kibana-prometheus-exporter#169 but he said it is working / there have been lots of downloads without any reports, so at this point I'm not sure why although the plugin looks to install through the Bitnami helm chart but it isn't running.

If you have any thoughts, that would be great, kind of at a dead end right now.

[davejhahn]

Not able to get the plugin to return metrics on the Bitnami helm chart version. Looking for some thoughts.

[marcosbc]

Hi @davejhahn, you're right. Kibana requires the kibana-prometheus-exporter plugin to be installed, it is vaguely mentioned in values.yaml and NOTES (which appears when installing the chart) of the Kibana subchart, and not the Elasticsearch.

The following options allow the Kibana chart to be installed with the Prometheus plugin:

--set global.kibanaEnabled=true --set 'kibana.metrics.enabled=true' --set 'kibana.metrics.serviceMonitor.enabled=true' --set 'kibana.metrics.serviceMonitor.namespace=' --set 'kibana.plugins[0]=https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/7.8.0/kibana-prometheus-exporter-7.8.0.zip'

diff --git a/bitnami/kibana/values.yaml b/bitnami/kibana/values.yaml
index a09a3a606..defc31864 100644
--- a/bitnami/kibana/values.yaml
+++ b/bitnami/kibana/values.yaml
@@ -58,6 +58,7 @@ updateStrategy:
 ## List of plugins to install
 ##
 plugins:
+- https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/6.8.10/kibana-prometheus-exporter-6.8.10.zip
 # - https://github.com/fbaligand/kibana-enhanced-table/releases/download/v1.5.0/enhanced-table-1.5.0_7.3.2.zip

 ## Saved objects to import (NDJSON format)

However, it looks like the plugin gets installed after starting Kibana, meaning that it still does not work. In the meantime there are several workarounds:

• Init scripts should stop Kibana after starting to execute init scripts. It should be started again afterwards.

  In this case, the container image should persist plugins (which are currently not being persisted). This can be workaround with the KIBANA_FORCE_INITSCRIPTS environment variable (--set kibana.forceInitScripts=true in the chart), unless the "cmd" is replaced (see below).

• The chart should not install plugins via init scripts, but instead run a new cmd script for installing plugins. I.e.:

  #!/bin/bash
  /path/to/install-plugins.sh
  exec /run.sh
  

  This would also require to remove the logic that mounts the install-plugins.sh script to the init scripts directory, as it would not be executed as such.

Also, the chart should make it a lot simpler to enable monitoring, by documenting the required steps to enable monitoring.

As you can see, this requires some major changes to the initialization of the container/chart (or at least a rethink), so I'm afraid I cannot give an ETA for when this will be fixed. If you feel confident to try and make the changes for your self, we'd be glad to review a PR that fixes this issue!

[davejhahn]

@marcosbc I created my own image from the bitnami kibana image, simply with the difference being installation of the plugin. That doesn't seem to work either, it returns a 500 error when accessing the endpoint. So unlike the helm package, that installs it after it is started, this is installed when the container starts up so the above problem you described shouldn't be happening.

The author if the widely used plugin asked if bitnami has a fork of Kibana--that it works fine for him (and apparently everyone else using it). I'm assuming no, but wasn't sure.

Is there any reason this wouldn't work either? I'm almost at the point of giving up getting metrics to work with the bitnami instance. My alternative at this point is to use a different helm package and create my own service monitor, since everything is working nicely (aside from this) I am hesitant to do that.

Any insight would be great.

[andresbono]

Hi @davejhahn, we have been investigating the issue. We saw this pjhampton/kibana-prometheus-exporter#186 where you are showing the Dockerfile that you are using. We tried the following:

1. Build a custom image extending the Bitnami one:

FROM bitnami/kibana:7.9.0-debian-10-r0
RUN /opt/bitnami/kibana/bin/kibana-plugin install https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/7.9.0/kibana-prometheus-exporter-7.9.0.zip

docker build -t bitnami/kibana:custom .

2. Make the image available in your k8s cluster.

3. Install kibana chart with custom values just for testing purposes:

image:
  tag: custom

elasticsearch:
  hosts:
    - localhost
  port: 9200
sidecars:
  - name: elasticsearch
    image: bitnami/elasticsearch:latest
    ports:
      - name: http
        containerPort: 9200

helm install kibana bitnami/kibana -f custom-values.yaml

4. Check Kibana logs show the following message to be sure Kibana is up and running:

kubectl logs kibana-695bbc5c9-xrnlq kibana | grep "http server running"
{"type":"log","@timestamp":"2020-09-08T09:04:57Z","tags":["info","http","server","Kibana"],"pid":26,"message":"http server running at http:https://:::5601"}

5. Ensure container is running the metrics plugin:

kubectl exec kibana-695bbc5c9-xrnlq -c kibana -- kibana-plugin list
[email protected]

6. Query the metrics endpoint:

kubectl exec kibana-695bbc5c9-xrnlq -c kibana -- curl -s http:https://localhost:5601/_prometheus/metrics
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

kubectl logs kibana-695bbc5c9-xrnlq kibana
{"type":"response","@timestamp":"2020-09-08T10:07:27Z","tags":[],"pid":27,"method":"get","statusCode":404,"req":{"url":"/::/api/status?extended=","method":"get","headers":{"host":":5601","accept":"application/json","connection":"close"},"remoteAddress":"::ffff:127.0.0.1","userAgent":"::ffff:127.0.0.1"},"res":{"statusCode":404,"responseTime":1,"contentLength":9},"message":"GET /::/api/status?extended= 404 1ms - 9.0B"}
{"type":"error","@timestamp":"2020-09-08T10:07:27Z","tags":[],"pid":27,"level":"error","error":{"message":"404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}","name":"StatusCodeError","stack":"StatusCodeError: 404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}\n    at new StatusCodeError (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/errors.js:32:15)\n    at Request.plumbing.callback (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/plumbing.js:104:33)\n    at Request.RP$callback [as _callback] (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/plumbing.js:46:31)\n    at Request.self.callback (/opt/bitnami/kibana/node_modules/request/request.js:185:22)\n    at Request.emit (events.js:198:13)\n    at Request.<anonymous> (/opt/bitnami/kibana/node_modules/request/request.js:1161:10)\n    at Request.emit (events.js:198:13)\n    at IncomingMessage.<anonymous> (/opt/bitnami/kibana/node_modules/request/request.js:1083:12)\n    at Object.onceWrapper (events.js:286:20)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/_prometheus/metrics","path":"/_prometheus/metrics","href":"/_prometheus/metrics"},"message":"404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}"}
{"type":"response","@timestamp":"2020-09-08T10:07:27Z","tags":[],"pid":27,"method":"get","statusCode":500,"req":{"url":"/_prometheus/metrics","method":"get","headers":{"host":"localhost:5601","user-agent":"curl/7.64.0","accept":"*/*"},"remoteAddress":"::1","userAgent":"::1"},"res":{"statusCode":500,"responseTime":7,"contentLength":9},"message":"GET /_prometheus/metrics 500 7ms - 9.0B"}

We the steps above, we could reproduce your issue.

---

It seems the issue is caused due to some kind of incompatibility with IPv6 or at least due to the configuration key server.host: "::" (Change introduced in #2401). If we create a custom ConfigMap updating that setting to IPv4, everything works as expected:

1. Create ConfigMap:

kibana-conf.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-kibana-conf
data:
  kibana.yml: |
    pid.file: /opt/bitnami/kibana/tmp/kibana.pid
    server.host: "0.0.0.0"
    server.port: 5601
    elasticsearch.hosts: [http:https://localhost:9200]
    server.rewriteBasePath: false

kubectl create -f kibana-conf.yaml

2. Upgrade the Kibana deployment:

helm upgrade kibana bitnami/kibana -f custom-values.yaml --set configurationCM=custom-kibana-conf

3. Verify the configuration file is updated:

kubectl exec kibana-5f644c846f-l6xkf -c kibana -- cat /opt/bitnami/kibana/conf/kibana.yml
pid.file: /opt/bitnami/kibana/tmp/kibana.pid
server.host: "0.0.0.0"
server.port: 5601
elasticsearch.hosts: [http:https://localhost:9200]
server.rewriteBasePath: false

4. Query the metrics endpoint again:

kubectl exec kibana-5f644c846f-l6xkf -c kibana -- curl -s http:https://localhost:5601/_prometheus/metrics | head -n4
# HELP kibana_status kibana_status
# TYPE kibana_status gauge
kibana_status 1
# HELP kibana_millis_uptime kibana_millis_uptime

Could you please try on your side?

[davejhahn]

I will give it a try and report back, thanks!

[davejhahn]

@andresbono I created the config map and upgraded specifying the configmap name in configurationCM, but the readiness probe is failing.

Readiness probe failed: HTTP probe failed with statuscode: 503
Readiness probe failed: Get http:https://10.240.0.140:5601/app/kibana: dial tcp 10.240.0.140:5601: connect: connection refused

Any ideas?

[davejhahn]

Got a chance to look at this again, thinking it was the elasticsearch.hosts setting--I changed it to point to the bitnami install I am using (verified the URL with curl from working Kibana before upgrading) with http:https://elasticsearch-coordinating-only:9200.

It still can't get passed the readiness probe because Kibana doesn't start:

Kibana pod log:

�[0m�[38;5;2mINFO �[0m ==> Starting gosu...
�[0m�[38;5;2mINFO �[0m ==> Kibana started with PID 30. Waiting for it to be started
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 30 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 29 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 28 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 27 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 26 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 25 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 24 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 23 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 22 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 21 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 20 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 19 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 18 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 17 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 16 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 15 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 14 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 13 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 12 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 11 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 10 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 9 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 8 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 7 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 6 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 5 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 4 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 3 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 2 remaining attempts...
�[0m�[38;5;2mINFO �[0m ==> Waiting for Kibana server: 1 remaining attempts...
Error: Kibana is not available after 30 retries

kubectl describe pod:

  Warning  Unhealthy               7s (x9 over 2m47s)     kubelet, secretnode  Readiness probe failed: HTTP probe failed with statuscode: 503

[andresbono]

Hi @davejhahn, you can try to add NAMI_LOG_LEVEL=trace to see if you get more information from the log. You will need to redeploy Kibana with this additional values:

extraEnvVars:
  - name: NAMI_LOG_LEVEL
    value: trace

In any case, I would recommend you to use a sidecar elasticsearch container as explained in #3058 (comment) (only for testing purposes), just to confirm that at least the problem with the metrics plugin is resolved.

[davejhahn]

@andresbono ah, after looking at the logging and trying the URL that it was complaining about, I found it was just wrong:

was http:https://elasticsearch-coordinating-only:9200 instead of http:https://elasticsearch-elasticsearch-coordinating-only:9200

Once I updated the config map, everything was good. Seeing the endpoint as up in Prometheus and data reporting.

Thanks for your help!

[davejhahn]

Leaving this open for the original issue....

[andresbono]

Great, @davejhahn! Thanks for confirming it.

[Prakash-droid]

I too faced the issue but your suggested workaround (custom image and configmap) helped me to fix the issue

[andresbono]

Nice, @Prakash-droid! Thanks for confirming it.

The original issue is still pending to be resolved.

[fmubaidien]

Hello, I am facing a similar issue however for me the probes fail due to the path being /login for the probes, when i server side apply it to check /status it works fine, can I add the ability to change healthcheck route as a PR or is there another solution?

[miguelaeh]

Hi @fmubaidien ,
Of course, we are always open and happy to review PRs.

Regarding the feature you need, currently it supports a custom path, but the /login will be concatenated. Check this. I guess that is because checking other endpoints without being logged in will produce an 403 error.