[Feature] Find a solution to reduce size of the chart

[eddycharly]

Problem Statement

Helm chart is growing beyond the size helm can manage.
We need to find a solution to the release size under control.

Xref #8130 (comment)

Solution Description

Let's throw ideas on how we can do that.

Alternatives

No response

Additional Context

No response

Slack discussion

https://kubernetes.slack.com/archives/C032MM2CH7X/p1693306174401429

Research

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[MarcelMue]

From the discussion in the Weekly:

• We want to initially see if there are some quick wins for 1.11
• We want to also understand which long term solutions are possible (maybe inclusion of CRD apply in the controller)

[chipzoller]

Also need an analysis of how close we actually are here and how much headroom we have.

[JimBugwadia]

As a short term solution, can we strip comments / help on all non-default versions?

[vishal-chdhry]

Some references I came across:

1. Kubernetes CRD too big issue: Large CRDs go over size limits (e.g. those with embedded podspecs) kubernetes/kubernetes#82292
2. $ref in CRD validation schema: Allow use of $ref in in CRD validation schema kubernetes/kubernetes#62872
3. Crossplane stripped down CRDs: remove descriptions from stacks crds to reduce the filesize crossplane/crossplane#702
4. Prometheus CRD too big issue: Fix prometheus CRD being too big prometheus-operator/prometheus-operator#4439 (comment)
5. Jaeger Operator removed descriptions: Removed descriptions from CRD jaegertracing/jaeger-operator#932

[MarcelMue]

We should have enough headroom in the near future with the removal of descriptions in #8185

[eddycharly]

IMO we should try to reduce more:

Helm secret size (1015880 bytes) is below the max allowed (1030000 bytes)

[eddycharly]

Side note prometheus-community/helm-charts#3548

[MarcelMue]

Side note prometheus-community/helm-charts#3548

This is also the approach I mentioned in the call initially, but we need to remove/deal (possibly with globals) with any rendering of names / labels we have in the CRDs ideally. Subcharts are supposed to be able to be standalone in helm, so you can't just "import" values from the parent chart. If we only had plain-text CRDs, then this would be the trivial solution.

-> helm docs: https://helm.sh/docs/chart_template_guide/subcharts_and_globals/

[eddycharly]

Isn't the solution related to using the special crds folder ?
Last time i checked subcharts contributed the same data size to the secret as direct templates.

[MarcelMue]

Isn't the solution related to using the special crds folder ? Last time i checked subcharts contributed the same data size to the secret as direct templates.

No - if you check the PR, it is moved from the crds subfolder to a subchart called crds and then there into the crds folder: https://github.com/prometheus-community/helm-charts/pull/3547/files

You can also find the chart.yaml of the subchart here: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/charts/crds/Chart.yaml

[eddycharly]

But when you install the kube-prometheus-stack chart, the manifests from grafana for example are part of the secret, it's a subchart though.

[tmsdce]

Hi,

Maybe it can help others. On our side we use ArgoCD and the CRD size issue went away thanks to SSA with the use of a specific annotation applied on CRD objects argocd.argoproj.io/sync-options: ServerSideApply=true (I think SSA is also supported on Flux).

Also, as ArgoCD uses helm template, this "trick" is only possible because the CRDs are part of the chart templates through the .Values.crds.annotations (it would also work in a dedicated chart or subchart). If they were moved to the special crds folder however, this wouldn't work as this folder is not templated by helm.

The key was to use SSA and it is quite straightforward to leverage it when using gitops operators, but may be more cumbersome to handle without these tools.

[MariamFahmy98]

I believe we can create a separate helm chart for the CRDs.

[chipzoller]

I really think the most sustainable thing going forward is to handle CRD creates/updates via an initContainer. It's more initial work but we can do whatever we want with total freedom and it makes this issue go away.

[tobru]

Handling CRDs directly with the controller or in the Pod has one big drawback: You'd need elevated privileges to manage CRDs, it makes RBAC painful.

[chipzoller]

Yes, but you can use specific resources names because they're your CRDs.

[MariamFahmy98]

This is how prometheus handles the CRDs through helm charts:

1. kube-prometheus-stack has a subchart for crds (can be found here).
   I installed it to check if the CRDs are a part of the helm secret and it is not.
2. A separate helm chart for CRDs prometheus-operator-crds.

@eddycharly - I amn't sure why grafana manifests exist in the helm secret whereas CRDs don't.

This is the dependencies of the kube-prometheus-stack. You will see that there's a separate helm chart for grafana and they include it in the dependencies of the chart, however for CRDs, they use the crds subchart they have created instead of using the prometheus-operator-crds i.e. the separate chart for CRDs. I wonder if this is why crds don't exist in secret templates whereas the grafana does exist.

By default, the chart installs the CRDs.

[MariamFahmy98]

This is how I checked if CRDs templates exist in the helm secret:

kubectl get secrets <helm-secret-name> -o jsonpath="{ .data.release }" | base64 -d | base64 -d | gunzip -c | jq '.chart.templates[].data' | tr -d '"' | base64 -d >> templates.o

Then search for CustomResourceDefinition in templates.o, and there is none.

Let me know if it isn't the correct approach.

[treydock]

CRDs in a subchart is how Kyverno used to ship the CRDs and it caused some issues for the community since you'd have to install the CRD chart before the Kyverno chart, so Kyverno Helm install was at minimum a 2 step process rather than 1 step. Some ways to reduce size would move many/all of the helpers to a common chart and then maybe move some of the various Kyverno components in the current chart into subcharts.