Imported AKS cluster does not show any workload level metrics

[jls-tschanzc]

What kind of request is this (question/bug/enhancement/feature request):

bug

Steps to reproduce (least amount of steps as possible):

1. Import an AKS Cluster
2. Enable Monitoring (0.0.7 or 0.1.0)
3. Wait a while
4. Go to Project -> Namespace -> Workload (Or browse to any workload/pod metrics in Grafana)
5. Open the "Workload Metrics"

Result:

It always states "Not enough data for graph". In Grafana it only shows requests/limits and no CPU/Memory data.

Other details that may be helpful:

It has already been this way on Rancher 2.3.5 / Monitoring 0.0.7 and remained the same after upgrading to Rancher 2.4.4 / Monitoring 0.1.0.

It does show cluster level metrics.

Environment information

• Rancher version (rancher/rancher/rancher/server image tag or shown bottom left in the UI): 2.4.4
• Installation option (single install/HA): HA

Cluster information

• Cluster type (Hosted/Infrastructure Provider/Custom/Imported): AKS Imported
• Machine type (cloud/VM/metal) and specifications (CPU/memory): Cloud, 1.9 vCPU / 4.5 GB
• Kubernetes version (use kubectl version): v1.15.10

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T23:34:25Z", GoVersion:"go1.14.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.10", GitCommit:"059c666b8d0cce7219d2958e6ecc3198072de9bc", GitTreeState:"clean", BuildDate:"2020-04-03T15:17:29Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

gz#12942

[Rancheroo]

I also have the same issue on
Rancher 2.3.3
AKS 1.14.8
Monitoring 0.0.6

Please advise if more info is required.

[Ileriayo]

Try opening port 8472 on the cluster nodes.

[bentastic27]

I'm seeing the same behavior

Rancher 2.4.8
AKS 1.17.11 (Rancher created even)
Monitoring 0.1.2

Some logs from the metrics-server include the following, so perhaps related.

E1110 14:30:11.289027       1 reststorage.go:160] unable to fetch pod metrics for pod default/nginx-649c7cb7f9-xqmxg: no metrics known for pod

[bentastic27]

This doesn't seem to work with monitoring 0.1.4 either.

[GGGitBoy]

I tried to reproduce these steps and found this error.
If your case is the same as mine. Then the following may be helpful to you.

View the kubelet process of the aks node is the default port 10250, but this is different from the port we scraped

Why is it like this:

After enabling monitoring for some imported clusters, ui will set exporter-kubelets.https to false

So our scraping port will be 10255 instead of 10250, for aks cluster this is not what we expected

Workaround:

1. Add monitoring answer exporter-kubelets.https: true, for example:

2. Save and wait for the monitoring deployment to complete

now it works fine

[bentastic27]

@GGGitBoy that is a solid workaround for the time being. The downside is that answer disappears from the monitoring config page after you hit save so you have to remember to re-add it on every change.

[GGGitBoy]

yep, this is because UI sends the incorrect exporter-kubelets.https parameter according to the cluster type (aks). it will be fixed later

[catherineluse]

@gaktive I realized that this issue is for the deprecated monitoring V1. Therefore I suggest closing it because we are no longer updating the V1 apps. In addition to that, the UI is moving away from injecting certain values into Helm charts because it could potentially introduce a version incompatibility issue between the Helm chart version and the Rancher version.

I think it would be better to address this issue through documentation in these places:

• Rancher documentation for imported AKS clusters
• Rancher documentation for the monitoring application
• The monitoring Helm chart README

[gaktive]

Closing as won't fix. For docs people who see this, we should document as @catherineluse suggests.

[catherineluse]

I have opened a docs ticket to track the work of figuring out if this kind of requirement needs to be documented for Monitoring V2. rancher/rancher-docs#377