Support patching a JSON/YAML value inside a ConfigMap

[yanniszark]

Hi all! I am reopening an old issue with a lot of likes, which I think is really needed in a lot of scenarios:
#680

Problem Statement

Kustomize has the ability to perform powerful patches (JSONPatch, SMP) on Kubernetes resources (yaml objects with apiVersion, kind, metadata.name keys). This enables a powerful workflow where manifests consumers are decoupled from manifests consumers and keep their changes in separate files.

It is very common for applications to declare their configuration as a JSON or YAML structure inside a configmap key. However, all the powerful kustomize abilities do not apply to that the JSON inside that key, meaning that consumers must include all the configuration again, instead of just patching the values they want.

This is especially frustrating, as it hinders upgrades. For every minor change in the producer's config, the consumer must now copy the config again and make their changes ontop. Because of that, it violates kustomize's principle of keeping consumer changes separate from the producer's files, via patches.

Proposed Solution

I am opening this issue to discuss how we could apply kustomize's existing powerful patching mechanisms to configmap JSON / YAML values. This would greatly enhance many peoples' workflows, as is evident from the activity in #680.

@monopole @Shell32-Natsu @pwittrock what do you think? This is a long-standing problem, so maybe you already have some thoughts on this. Since we are invested in using kustomize, both as Arrikto and in the Kubeflow community as well, we'd be happy to put in the work required for such a feature.

[pwittrock]

cc @KnVerey

[zhech2]

Would it be possible to break a JSON document down into keys and values.

// base configuration
{ 
  "Configuration": {
     "Property1": "Value1",
     "Property2": "Value2"
   },
   list: [ "item0", "item1"]
}

// overlay
{ 
  "Configuration": {
     "Property1": "OverrideValue"
   }
}

Base Configuration

Configuration:Property1=Value1
Configuration:Property2=Value2
Configuration:list:0=item0
Configuration:list:1=item1

Overlay

Configuration:Property1=OverrideValue

Final Result

Configuration:Property1=OverrideValue
Configuration:Property2=Value2

Then generate a JSON ConfigMap for the configuration based on this?

[Benly-walter]

Is there an update to this?

[marchmallow]

Interested as well

[goto100]

It's very useful, and kubectl edit has already made an example for this usage. It shows a structured value rather than a string if the value is a legal yaml string.

[liger1978]

Patching data within ConfigMaps would remove the last reason for templating.

[jonaskello]

This would be perfect to use with kustomize as a helm postRenderers directive. Many helm charts create a configmap with a yaml file in it. It is today not possible to patch that yaml file without replacing the whole file and thereby overwriting the file created by the helm chart. This suggestion would enable that if I understand it correctly.

[KnVerey]

As @monopole mentioned here, the maintainer team is open to accepting this feature as long as it is explicitly triggered and the edit it performs is fully structured. However, we do not have the bandwidth to implement it ourselves.

@monopole previously requested a full KEP, but we now have a brand new, lighter weight process for aligning on Kustomize features that are large and/or particularly contentious before time is spent on implementation: a mini in-repo KEP. If someone following this issue still feels strongly about it and is interested in contributing an implementation, please create an in-repo KEP with your proposal.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[onedr0p]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[natasha41575]

/remove-lifecycle stale

A proposal for this feature has been submitted as a KEP: #4558

[ringerc]

This is a common need whenever someone has operators that don't have full CRDs. For example, the OpenTelemetry Collector has a resource

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector

but the bulk of its configuration is in a

spec:
  config: |
     ...inline yaml here ...

which can not be easily modified by Kustomize. It does support environment variable substitution, but that's quite limited.

Ideally everyone should just use proper CRDs, but in reality...

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[satrox28]

Hi,

Is there any update on this feature request - inline yaml support for configmap?.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[onedr0p]

/remove-lifecycle stale

[amold1]

Is there any update on this? This is a very useful feature to have to avoid repeating configuration

[satrox28]

I am still looking for this feature to be available.

[kpanic9]

Could be helpful for Secrets too.

[LH8PPL]

Can we make this happen?

[seanmorton]

This would also be very helpful for overriding specific fields in the spec.source.helm.values of ArgoCD's Application CRD, for example when using a shared reusable base of Application manifests.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[webwurst]

Is this now covered by #4517?

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.