Deprecate vars

[natasha41575]

The new replacements field is intended to be the replacement for the vars feature. Now that its implementation is complete, we should look into deprecating vars.

Some outstanding migration issues:

#3978
#4012
#4337
#4359

These issues have the same underlying problem that users are adding vars in strings where the var is not delimited, e.g. Host(www-$(NAMESPACE).example.com).

We should also take a look at use cases such as #4080 where the user wants to replace a string in an array without having to rely on the order of items in the array.

Generators for service accounts have been proposed as the solution for the first issue, but does not easily resolve the second. options.prefix and options.suffix have been proposed for the third (more details in the issue).

Additionally, #4120 requests a way to replace a template variable.

[k8s-ci-robot]

@natasha41575: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aodinokov]

@natasha41575
JFYI Maybe it will give some inspiration :) .. When I experimented with krm-based replacement transformer last year I came to the solution with
multiref source
This solution was inspired by kpt substitutions. where it was possible to concat several values using some template/pattern.

To make something similar to Host(www-$(NAMESPACE).example.com) it's possible to do the following

- source:
    multiref:
      refs:
      - objref:
          kind: Secret
          name: node1-bmc-secret
        fieldref: stringData.somefield # let's say it has value 'docker-ce . grep 19.03.12 ...' and we want to substitute 19.03.12 with value from another yaml ephemeral-catalogue and field versions.docker-ce.. 
      - objref:
          name: ephemeral-catalogue
        fieldref: versions.docker-ce
      template: |
        {{ regexReplaceAll "docker-ce . grep 19.03.12" (index .Values 0) (printf "docker-ce | grep %s" (index .Values 1)) }}
  target:
    objref:
      kind: Secret
      name: node1-bmc-secret
    fieldrefs: 
    - stringData.somefield

Basically what this solution does - in case muliref object is set all values extracted there are passed to go-template as values. go template engine uses that as .Values and it has to produce resulting string that will go to target. Target in this example is the same field as source[0]. Go-template engine had Sprig-functions included.
this example would work for $(VERSION) substring as well... - it was necessary to replace $(VERSION) instead of docker-ce . grep 19.03.12 and to user index .Values 1' instead of printf`, e.g.:

- source:
    multiref:
      refs:
      - objref:
          kind: Secret
          name: node1-bmc-secret
        fieldref: stringData.somefield # let's say it contains Host(www-$(NAMESPACE).example.com)
      - objref:
          name: ephemeral-catalogue
        fieldref: versions.docker-ce # and this field contains the value to put instead of $(NAMESPACE)
      template: |
        {{ regexReplaceAll "$(NAMESPACE)" (index .Values 0) (index .Values 1) }}
  target:
    objref:
      kind: Secret
      name: node1-bmc-secret
    fieldrefs: 
    - stringData.somefield

This approach in practice demonstrated a lot of flexibility - it's possible to concat string, build string with patterns, substitute part of it and many other things, because go template is basically a language and that allows alot.. I guess it's much more flexible than approach with delimiter.

From implementation standpoint it was really easy to implement it:
https://github.com/aodinokov/noctl-airship-poc/blob/0e2e126c0601bdc820ade673d47ff506b328edf0/kpt-functions/replacement/function.go#L78 - definition of multiref source.
https://github.com/aodinokov/noctl-airship-poc/blob/0e2e126c0601bdc820ade673d47ff506b328edf0/kpt-functions/replacement/function.go#L175 - is function that collects all sources and runs go-template with their values
plus some checks that multiref and other options are mutually exclusive...

[yuwenma]

Per offline discussion, we keep var in kustomize for one year according to the kubernetes one year deprecating policy. When var is detected in kustomization.yaml, kustomize warns users about the deprecation date, and request them to migrate to the replacements field.

[natasha41575]

@KnVerey @yuwenma @monopole Are any of the outstanding issues blockers for vars deprecation? I am tempted to mark them deprecated now, and address these users' concerns in between deprecation and removal.

[KnVerey]

The question that comes to mind for me is how exactly we want to do this deprecation. Since this is a field in the Kustomization API (as opposed to a CLI cmd/flag), if we follow the Kubernetes API backwards compatibility policy, we shouldn't ever remove it from v1beta1 Kustomization; instead, we should cut v1 with vars etc. already removed, and have the CLI support both v1beta1 and v1 for the year deprecation period. This would also mean a single migration guide for all the deprecated features, and support in the kustomize edit fix command for updating the API version (and all associated conversions).

So overall the process would be:

1. Create v1 Kustomization and support both in the CLI
2. Ensure kustomize edit fix can perform upgrade to the extent possible, and docs are fully up to date.
3. Emit deprecation warnings on use of v1beta1 (or unspecified, assumed to be v1beta1)*
4. Wait a year from the above and address any feature parity gaps during that time.
5. Make CLI stop supporting v1beta1 with a major version bump

* A huge downside to that is that it requires all users to take some action, not just the ones affected by a deprecation. Perhaps before emitting deprecation warnings, we could also implement a silent conversion feature to enable v1beta1 support indefinitely if and only if the instance is using fields that exist in both versions or have exact equivalents (e.g. bases->resources but not vars->replacements). In other words, if and only if carrying that support will not incur technical debt / inability to remove unsupported features from the codebase.

WDYT?

If we go with that plan, we can start it any time IMO--the first step is mostly a matter of confidence in the feature set we want v1 to retain, and of figuring out how to do this cleanly in the codebase. Experience adding Composition (which I'm starting now) may be informative for the latter.

Regardless, you're right that the long deprecation period leaves us plenty of time to implement more enhancements to replacements as required. Incidentally, several of the asks seem to be for unstructured edits / variable substitution, which we will definitely not support. For the cases where the problem is essentially problematically structured CRD fields, the plugin graduation story may also play a big role in preparedness for feature removal.

[natasha41575]

The overall process you've proposed LGTM. We can start looking at some of the issues in https://github.com/kubernetes-sigs/kustomize/projects/12 to prepare for a v1 Kustomization.

Perhaps before emitting deprecation warnings, we could also implement a silent conversion feature to enable v1beta1 support indefinitely if and only if the instance is using fields that exist in both versions or have exact equivalents (e.g. bases->resources but not vars->replacements).

SGTM. So long as all the fields are supported, I see no issue with converting v1beta1 to v1 silently.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[KnVerey]

/remove-lifecycle stale

[natasha41575]

/lifecycle frozen

[tis-rpage]

@natasha41575 JFYI Maybe it will give some inspiration :) .. When I experimented with krm-based replacement transformer last year I came to the solution with multiref source This solution was inspired by kpt substitutions. where it was possible to concat several values using some template/pattern.

To make something similar to Host(www-$(NAMESPACE).example.com) it's possible to do the following

- source:
    multiref:
      refs:
      - objref:
          kind: Secret
          name: node1-bmc-secret
        fieldref: stringData.somefield # let's say it has value 'docker-ce . grep 19.03.12 ...' and we want to substitute 19.03.12 with value from another yaml ephemeral-catalogue and field versions.docker-ce.. 
      - objref:
          name: ephemeral-catalogue
        fieldref: versions.docker-ce
      template: |
        {{ regexReplaceAll "docker-ce . grep 19.03.12" (index .Values 0) (printf "docker-ce | grep %s" (index .Values 1)) }}
  target:
    objref:
      kind: Secret
      name: node1-bmc-secret
    fieldrefs: 
    - stringData.somefield

Basically what this solution does - in case muliref object is set all values extracted there are passed to go-template as values. go template engine uses that as .Values and it has to produce resulting string that will go to target. Target in this example is the same field as source[0]. Go-template engine had Sprig-functions included. this example would work for $(VERSION) substring as well... - it was necessary to replace $(VERSION) instead of docker-ce . grep 19.03.12 and to user index .Values 1' instead of printf`, e.g.:

- source:
    multiref:
      refs:
      - objref:
          kind: Secret
          name: node1-bmc-secret
        fieldref: stringData.somefield # let's say it contains Host(www-$(NAMESPACE).example.com)
      - objref:
          name: ephemeral-catalogue
        fieldref: versions.docker-ce # and this field contains the value to put instead of $(NAMESPACE)
      template: |
        {{ regexReplaceAll "$(NAMESPACE)" (index .Values 0) (index .Values 1) }}
  target:
    objref:
      kind: Secret
      name: node1-bmc-secret
    fieldrefs: 
    - stringData.somefield

This approach in practice demonstrated a lot of flexibility - it's possible to concat string, build string with patterns, substitute part of it and many other things, because go template is basically a language and that allows alot.. I guess it's much more flexible than approach with delimiter.

From implementation standpoint it was really easy to implement it: https://github.com/aodinokov/noctl-airship-poc/blob/0e2e126c0601bdc820ade673d47ff506b328edf0/kpt-functions/replacement/function.go#L78 - definition of multiref source. https://github.com/aodinokov/noctl-airship-poc/blob/0e2e126c0601bdc820ade673d47ff506b328edf0/kpt-functions/replacement/function.go#L175 - is function that collects all sources and runs go-template with their values plus some checks that multiref and other options are mutually exclusive...

Any chance of multiref becoming a core API of kustomize/replacements? Without this capability, I'm afraid deprecating vars breaks the templating capabilities of native kustomize and requires using helmCharts in kustomization abuse to introduce the same capability in a non-native kustomize approach.

[natasha41575]

Without this capability, I'm afraid deprecating vars breaks the templating capabilities of native kustomize

That's the point. Kustomize was built around the idea of template-free YAML - this is the first line of the README. There are lots of reasons that we believe that templating is not great, you can read about them here under "unstructured edits": https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/ and many more details about the pitfalls of parameterization in Brian Grant's Declarative Management Application proposal.

Introducing vars was a mistake and a gateway to users building kustomization files full of templates that ends up with all the same problems that we've been trying to avoid. To meet users where they are, we are trying to offer support in kustomize to build on top of helm (#4401), but kustomize was born out of explicitly not wanting to natively support templating, so keeping vars as a feature doesn't make sense.

That being said, we understand that users are already using vars, and we are trying to come up with reasonable migration solutions for those users.

[tis-rpage]

That being said, we understand that users are already using vars, and we are trying to come up with reasonable migration solutions for those users.

Using replacements with multiref enables structured parsing of arbitrary fields and joining them together, this would largely resolve issues with the deprecation of vars, and was entirely ignored by your response.

[natasha41575]

Using replacements with multiref enables structured parsing of arbitrary fields and joining them together, this would largely resolve issues with the deprecation of vars, and was entirely ignored by your response.

Apologies for not addressing it in my response, I did read it and we are still thinking about it. The original proposal above has some syntax with templating/regex that we would most likely avoid but allowing the source to come from multiple places and concatenate them together is a very common use case that I myself have dealt with as well.

In my own work, what I've done for this is write a KRM function (I use the kpt starlark function) to generate the desired string, pulling it from multiple sources and concatenating it, and then using replacements to propagate the value to the correct places. Though this by no means is the official recommended workflow for kustomize, in my own experience it has worked well and I believe that it is a reasonable option once we get kustomize into a place where we can recommend KRM functions to users.

Replacements is already quite verbose and has complex syntax, and I have concerns that introducing multiref natively in replacements will exacerbate its complexity.

[tis-rpage]

Apologies for not addressing it in my response

Thanks for addressing it. It is a complex topic, but it's a very common use-case and a "standard" approach to resolving the issue would benefit the community. Unfortunately, leveraging KRM/kpt expands the trusted deployment code base, and for auditable environments that becomes impractical due to review processes involved. That overhead can be so significant that it is preferable to wait for a new kubectl release rather than pull in a dedicated kustomize build.

Replacements is already quite verbose and has complex syntax, and I have concerns that introducing multiref natively in replacements will exacerbate its complexity.

Yes, the implementation is complex/verbose, however I think replacements is a great swiss-army knife of kustomize.

===========================

My most common use cases where I need these capabilities and kustomize flounders badly, is with generating strings of formats such as:

• $(pathing)/$(program)/$(environment)/$(cluster)/$(namespace)/$(name)
• jdbc:https://$(name).$(namespace).$(cluster).$(environment).$(program).$(domain):$(port)/$(dbname)
• s3:https://$(program)-$(environment)-$(cluster)-$(namespace)-$(name)

Below is how I use configMaps to generate "structured variables", however schema prefixed strings like bucket names and connection strings with their non-uniform delimiters can't work with this approach.

kind: Component



configMapGenerator:
  - name: aws
#   behavior: merge
#   arn:partition:service:region:account:resource
    options:
      disableNameSuffixHash: true
    literals:
      - partition=aws
      - region=us-east-1
      - account=1234567890
      - iam.resource_path=$(program)/k8s
      - iam.permitted_arn=arn:[^:]*:iam::[^:]*:role
      - ecr_url=1234567890.dkr.ecr.us-east-1.amazonaws.com
      - REF_AWS_REGION=region #literal source in replacements would remove the need for this reference to grab the region


#   valuesInline:
#     serviceAccount:
#       annotations:
#         eks.amazonaws.com/role-arn: arn:$(aws.partition):iam::$(aws.account):role/$(aws.iam.resource_path)/$(cluster)/$(namespace)/role
#     deployment:
#       annotations:
#         iam.amazonaws.com/role: $(aws.iam.resource_path)/$(cluster)/$(namespace)/role
replacements:
  - source:
      kind: ConfigMap
      name: aws
      fieldPath: "data.[iam.resource_path]"
    targets:
      - options: { delimiter: "/", index: 1 }
        select: { kind: Namespace }
        fieldPaths: [ "metadata.annotations.[iam.amazonaws.com/permitted]" ]
      - options: { delimiter: "/", index: 1 }
        select: { kind: ServiceAccount }
        fieldPaths: [ "metadata.annotations.[eks.amazonaws.com/role-arn]" ]
      - options: { delimiter: "/", index: 0 }
        select: { kind: Pod }
        fieldPaths: [ "metadata.annotations.[iam.amazonaws.com/role]" ]
      - options: { delimiter: "/", index: 0 }
        select: { kind: Deployment }
        fieldPaths: [ "spec.template.spec.containers.*.metadata.annotations.[iam.amazonaws.com/role]" ]
      - options: { delimiter: "/", index: 0 }
        select: { kind: StatefulSet }
        fieldPaths: [ "spec.template.spec.containers.*.metadata.annotations.[iam.amazonaws.com/role]" ]
  - source:
      kind: ConfigMap
      name: aws
      fieldPath: "data.[iam.permitted_arn]"
    targets:
      - options: { delimiter: "/", index: 0 }
        select: { kind: Namespace }
        fieldPaths: [ "metadata.annotations.[iam.amazonaws.com/permitted]" ]
  - source:
      kind: ConfigMap
      name: aws
      fieldPath: data.partition
    targets:
      - options: { delimiter: ":", index: 1 }
        select: { kind: ServiceAccount }
        fieldPaths: [ "metadata.annotations.[eks.amazonaws.com/role-arn]" ]
  - source:
      kind: ConfigMap
      name: aws
      fieldPath: data.account
    targets:
      - options: { delimiter: ":", index: 4 }
        select: { kind: ServiceAccount }
        fieldPaths: [ "metadata.annotations.[eks.amazonaws.com/role-arn]" ]
  - source:
      kind: ConfigMap
      name: aws
    targets:
      - options: { create: true }
        select:
          kind: Deployment
        fieldPaths:
          - spec.template.spec.containers.[name=controller].env.[name=AWS_REGION].valueFrom.configMapKeyRef.name
  - source: # literal here would be super helpful to jam in `AWS_REGION` into the key, but currently requires a dummy configmap value to implement
      kind: ConfigMap
      name: aws
      fieldPath: data.REF_AWS_REGION
    targets:
      - options: { create: true }
        select:
          kind: Deployment
        fieldPaths:
          - spec.template.spec.containers.[name=controller].env.[name=AWS_REGION].valueFrom.configMapKeyRef.key

[joseph-skupniewicz-sp]

I'd like to transition from vars to replacements and get rid of the deprecation warning..and it works in all but one case where I am unable to substitute a var that is part of an env var name that is set in my Deployment conf:

kustomization.yaml

..
..
replacements:
  - source:
      kind: ConfigMap
      name: my-service-region-env
      fieldPath: data.AWS_REGION_SHORTHAND
    targets:
      - select:
          kind: Deployment
        fieldPaths:
          - spec.template.spec.containers[*].env[*].valueFrom.secretKeyRef.name

I am using this patch file:

patch-env-vars.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-service
spec:
  template:
    spec:
      containers:
        - name: my-service
          env:
            - name: SERVICE_ACCOUNT
              valueFrom:
                secretKeyRef:
                  name: my-service-credentials-$(AWS_REGION_SHORTHAND)
                  key: account
            - name: SERVICE_USERNAME
              valueFrom:
                secretKeyRef:
                  name: my-service-credentials-$(AWS_REGION_SHORTHAND)
                  key: username
            - name: SERVICE_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: my-service-credentials-$(AWS_REGION_SHORTHAND)
                  key: password

But I get this error on an apply:

The Deployment "my-service-use1" is invalid: 
* spec.template.spec.containers[0].env[0].valueFrom.secretKeyRef.name: Invalid value: "$(AWS_REGION_SHORTHAND)": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

I am trying to make this replacement in my patch file from a common folder that is referenced in my resources: block for each downstream folder in my directory structure.

I'd like to avoid having to add this patch-env-vars.yaml in each of my downstream folders and hardcode the region in the name and would prefer to use variable substitution but there seems to be something off with doing this against this specific fieldPath?

Using replacements for variable substitution in other strings is working for other value-type paths, but not this for some reason. Is it because it is a key name?

[farioas]

Hi @joseph-skupniewicz-sp,

I'm trying to do the same as you. As far as I understood, we have to use delimiter: -, in that case your line will be splitted into an array:

- my
- service
- credentials
- $(AWS_REGION_SHORTHAND)

Then you have to replace the object under index: 3.

New replacements requires to write 3 times more code than vars.

[joebowbeer]

More discussion here: #5046