Adding github repo - Request entity too large: limit is 3145728

[naisanzaa]

SURE-6732

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

Unable to add a public github repository to Continuous Delivery.

Error:

Time="2023-07-14T01:45:17Z" level=fatal msg="Request entity too large: limit is 3145728"

Expected Behavior

Add github repo.

Steps To Reproduce

1. go to Continuous Delivery
2. add github https link
3. error

Environment

Rancher	v2.7.5
Dashboard	v2.7.5
Helm	v2.16.8-rancher2
Machine	v0.15.0-rancher100

Logs

No response

Anything else?

related: #205

[kkaempf]

Please add a link to the public github repo here so we can reproduce your problem.

[naisanzaa]

Please add a link to the public github repo here so we can reproduce your problem.

https://github.com/TheShellLand/automonisaur

[kkaempf]

Thanks.

This repo has huge docs (1.8 MB) and a huge Python stack (automon 2.2 MB) - not really suitable for Fleet 🤔

You probably want to move the files relevant for Fleet to a sub-directory.

[Martin-Weiss]

I have the same issue after upgrade of rancher 2.6.8 to 2.7.5 and fleet just uninstalled the app from all downstream and from the local cluster! This is really critical...

There is not much / just yaml files in our git repo!

Any idea for a workaround and ETA for a fix?

[kkaempf]

Hmm, it's probably not Git or Fleet but Kubernetes (resp. etcd).

Googling around reveals e.g.
https://stackoverflow.com/questions/60468110/kubernetes-object-size-limitations/60492986#60492986
And this from Helm 🤔

[Martin-Weiss]

Hm - but why do we see this in Rancher 2.7.5 and did not see this in Rancher 2.6.8? What has been changed in fleet between these two versions that would bring us to this problem?

[kkaempf]

I don't think Fleet's the culprit here 😉

[Martin-Weiss]

Fleet does an uninstall after the upgrade of Rancher from 2.6.8 to 2.7.5. So something must have changed in fleet and maybe it is a design issue in fleet.. we really need to find a way to get more debug info.

[olblak]

Hm - but why do we see this in Rancher 2.7.5 and did not see this in Rancher 2.6.8? What has been changed in fleet between these two versions that would bring us to this problem?

🤔 Well that's a good question we went from Fleet 0.3.11 to [0.7.0]

[Martin-Weiss]

In my case it really seems that the root cause of the problem is that targetCustomizations did not work for repo and version.. and now that works. In combination with the large rancher-monitoring helm chart (4MB extracted) this seems to cause some limits to be hit..

So this causes a problem (rancher-monitoring-crd chart - but same problem exists with rancher-monitoring chart even worse):

defaultNamespace: cattle-monitoring-system
helm:
  repo: https://registry-2.di.customer.de/chartrepo/prdr
  version: "100.1.3+up19.0.3"
  chart: rancher-monitoring-crd
  releaseName: rancher-monitoring-crd
  values:
targetCustomizations:
- name: sbxr
  helm:
    repo: https://registry-2.di.customer.de/chartrepo/sbxr
    version: "100.1.2+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: sbxr
- name: devr
  helm:
    repo: https://registry-2.di.customer.de/chartrepo/devr
    version: "100.1.2+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: devr
- name: tstr
  helm:
    repo: https://registry-2.di.customer.de/chartrepo/tstr
    version: "100.1.2+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: tstr
      stage: tstr
- name: prdr
  helm:
    repo: https://registry-2.di.customer.de/chartrepo/prdr
    version: "100.1.2+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: prdr
- name: local
  helm:
    repo: https://registry-2.di.customer.de/chartrepo/prdr
    version: "100.1.3+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: local

This does not cause a problem:

defaultNamespace: cattle-monitoring-system
helm:
  repo: https://registry-2.di.customer.de/chartrepo/prdr
  version: "100.1.3+up19.0.3"
  chart: rancher-monitoring-crd
  releaseName: rancher-monitoring-crd
  values:
targetCustomizations:
- name: sbxr
  helm:
    #repo: https://registry-2.di.customer.de/chartrepo/sbxr
    #version: "100.1.2+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: sbxr
- name: devr
  helm:
    #repo: https://registry-2.di.customer.de/chartrepo/devr
    #version: "100.1.2+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: devr
- name: tstr
  helm:
    #repo: https://registry-2.di.customer.de/chartrepo/tstr
    #version: "100.1.2+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: tstr
      stage: tstr
- name: prdr
  helm:
    #repo: https://registry-2.di.customer.de/chartrepo/prdr
    #version: "100.1.2+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: prdr
- name: local
  helm:
    #repo: https://registry-2.di.customer.de/chartrepo/prdr
    #version: "100.1.3+up19.0.3"
    values:
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: local

[manno]

• document that the fix from 0.6 now puts multiple helm chart versions into the bundle
• make sure that the fatal error doesn't cause deletion of an existing deployment when upgrading

[raulcabello]

In my case it really seems that the root cause of the problem is that targetCustomizations did not work for repo and version.. and now that works. In combination with the large rancher-monitoring helm chart (4MB extracted) this seems to cause some limits to be hit..

That's correct. targetCustomizations for helm repo and versions was fixed in v0.6.0 more info. That means the Bundle now contains both versions of the chart. That Bundle is now too big to be stored in etcd, that's why you see the Request entity too large: limit is 3145728 error.

Fleet does an uninstall after the upgrade of Rancher from 2.6.8 to 2.7.5. So something must have changed in fleet and maybe it is a design issue in fleet.. we really need to find a way to get more debug info.

That looks like a separate issue. I found out some resources were redeployed when upgrading from 2.6.8 to 2.7.5. The problem with the monitoring chart is that old resources are removed, but the new Bundle can't be created because of the limit issue. I'm still investigating why resources are redeployed

[raulcabello]

For each BundleDeployment (or at least for some BundleDeployments) a helm upgrade with empty resources happens after the migration from 2.6.8 to 2.7.5. This is removing all resources. Then another upgrade with all the resources happens with the execution of fleet apply in the Job created by fleet. This last upgrade doesn't happen if the Job fails , this is the case for the rancher-monitoring with multiple versions.

The empty resources upgrade might be related to something that has changed in the way fleet stores the helm manifest, but this needs more investigation.

[raulcabello]

Resources are disappearing because a helm upgrade with empty resources is done. This is happening on Rancher upgrades for Bundles that contains a repo or version overridden in the helm options, which was fixed in fleet v0.6.0 more info.

The problem is that the checksum is different as it is now using the right helm version from the targetCustomization override. This is causing that the fleet-agent doesn't find the Chart.yaml or any resources as the checksum is used for the files prefix, therefore it performs an upgrade with no resources. The BundleDeployment is modified when the fleet job modifies the Bundle, then the right checksum is used and resources are deployed successfully.

The problem with the rancher-monitoring chart is that it is too big to be stored in a Bundle when using multiple versions. Therefore, the upgrade with the empty resources is done, but the Bundle is not modified as the job fails

Examples:

• https://github.com/raulcabello/fleet-test/blob/main/limit-test-two/fleet.yaml resources are recreated when upgrading Rancher from v2.6.8 to v2.7.5. helm ls shows two upgrades: one with empty resource, and another with all resources.
• https://github.com/raulcabello/fleet-test/blob/main/limit-test/fleet.yaml resources are removed from the downstream cluster. A helm upgrade with empty resources is done, and the the fleet job is failing with Request entity too large: limit is 3145728

[manno]

• Document etcd blob size depends on cluster
• Review existing size warning in docs
• Document multiple version via target customization increase bundle size

We might add an OCI repo as an alternative storage in the future, or extend the bundle/contents to use multiple k8s resources to work around the limits.

[weyfonk]

Closing this (see previous comment for a rationale).

[jhoblitt]

I ran into this problem as having two versions of the kube-prometheus-stack chart are now enough to trigger this. It seems like this is likely to be a frequent issue and the only solution is to split the bundle into multiple objects, one resource per chart version.

[shkarface]

I have the same issue while trying to install the groundcover chart. this is because Bundle contains non templated helm chart, so it's too big to store in etcd. is there any way to store only the templated resources instead of the whole chart (including comments, documentations and everything)

[strowi]

Hi, it seems we just started seeing this problem with a bundle exceeding 3M with ~100 kustomize overlays. Is there any documentation/way we can test this (or a timeline for release)?