Allow users to add extra objects and resources when installing a Helm chart

[wallrj]

In cert-manager/cert-manager#5900 @pierluigilenoci and I have been discussing whether it is sustainable for each Helm chart to implement its own values.extraObjects field.
Judging by the number of popular charts that have added extraObjects, this is clearly a needed feature.

Some use cases are:

1. Platform team deploying cert-manager using Helm want to bundle some CNI manifests (e.g. Calico variant of NetworkPolicy) so that those cert-manager specific policies can be maintained alongside the cert-manager settings and uninstalled along with cert-manager.
2. Platform team deploying cert-manager using Helm, want to bundle ConfigMaps containing Grafana dashboards for monitoring cert-manager. "It makes sense that all Grafana dashboards related to cert-manager are installed with the chart of the software itself without having to resort to external means.".

I think this should be implemented in Helm, probably as a helm install command line option, for the following reasons:

1. It should be a more maintainable to implement helm install --user-supplied-manifests extra-objects/, than to add Values.extraObjects to each Helm chart.
2. It should be more consistent to implement helm install --user-supplied-manifests extra-objects/, than for each Helm chart to implement the feature slightly differently
3. It should be more user friendly to document helm install --user-supplied-manifests extra-objects/ thoroughly at helm.sh, than to document it (or not document it) chart-by-chart in the chart README file, for example.
4. It should be more time efficient to discuss and plan this feature at https://github.com/helm/helm/, than to repeat the discussion with the maintainers of every Helm chart.

Links of interest, in no particular order:

• https://www.reddit.com/r/kubernetes/comments/yuw27c/deploying_with_helm_extra_manifests/
• https://artifacthub.io/packages/helm/argo/argo-cd#general-parameters
• https://tech.aabouzaid.com/2020/09/3-ways-to-customize-off-the-shelf-helm-charts-with-kustomize-kubernetes.html
• Add extraObjects value that allows creating supportive / adhoc resources traefik/traefik-helm-chart#595
• [prometheus] Accept template string as server.extraObjects prometheus-community/helm-charts#2139
• BREAKING CHANGE: Remove .Values.extraObjects aws/karpenter-provider-aws#3803
• extraObjects provisioner warning aws/karpenter-provider-aws#3705
• Add extraObjects to values cert-manager/cert-manager#5997
• No matches for kind "ClusterTriggerAuthentication" if this CRD is referenced in extraObjects kedacore/charts#513
• Add extraObjects value that allows creating supportive / adhoc resources apache/couchdb-helm#72
• Add a new chart template that uses the conventions I have grown accustomed to colearendt/helm#38
• Included extraObjects example in flux2/values.yaml doesn't appear to work. fluxcd-community/helm-charts#183
• [BUG][Opensearch] ExtraManifests do not allow structured data opensearch-project/helm-charts#463
• generic: add extraObjects morremeyer/charts#320
• Add Extra Manifest kubecost/cost-analyzer-helm-chart#2325
• extraObjects does work with ApplicationSet in ArgoCd argoproj/argo-helm#1882
• [FR] Allow the Chart to create extra manifest grafana/helm-charts#1001
• [agent-operator] Add extraObjects functionality grafana/helm-charts#2192
• [tempo-distributed] Add the ability to add extraObjects grafana/helm-charts#2722
• feat(helm): add extra object option crossplane/crossplane#4664

Originally posted by @wallrj in #7077 (comment)

[mattfarina]

Before I dig in further, why wouldn't the extra objects be a second chart someone installs that add those?

Consider package managers for other platforms (like Linux) installing something like PHP or ffmpeg. For the "additional" parts there are secondary packages one would install. Why wouldn't that same style experience be followed?

[wallrj]

Before I dig in further, why wouldn't the extra objects be a second chart someone installs that add those?

My understanding is that users find it laborious to create and maintain a second chart.
They prefer the extraObjects approach because it is less effort and because it has become a familiar pattern in other charts.
All the Bitnami charts have this option, for example.

There is an example of this in traefik/traefik-helm-chart#595 (comment) and apache/couchdb-helm#72 (comment) where @colearendt wrote:

In many cases, it can avoid the need for additional (fake helm chart w/ a few resources), extended (chart that extends the traefik chart), or adhoc deployments (kubectl apply and friends).

[pierluigilenoci]

@wallrj, you forgot to mention this: fluent/helm-charts#331 😉

[stevehipwell]

I think that Helm is currently lacking documentation & conventions (and maybe some tooling) to correctly separate the various roles who interact with it; this has led to charts trending towards having fewer constraints and being less opinionated. The addition of the extraObjects pattern in public Helm charts is a prime example of this direction of travel. I think this is a case of convenience over correctness as the documented pattern for this functionality is using public charts as sub-charts with additional resources strongly typed.

This said I can see value in adding native Helm support for rendering external templates as part of a chart install. I think the wrapper chart pattern is a better solution but anything to help move away from the extraObjects pattern would be a step in the right direction.

---

IMHO a public Helm chart (for a component) should aim to be as opinionated as possible while following the open-closed principal.

[pierluigilenoci]

@wallrj Also, this is related kubernetes/autoscaler#5633

[wallrj]

Before I dig in further, why wouldn't the extra objects be a second chart someone installs that add those?

@mattfarina are you persuaded by the reasons given in the comments above?

[pierluigilenoci]

New use case: kubernetes/ingress-nginx#11351 (Always me 😛)