Unable to replace variables in deployment's env definition #4337

itsnagaraj · 2021-12-09T03:06:32Z

I have the following deployment manifest

cat <<EOF >$DEMO_HOME/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myapp
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: myapp-container
        image: busybox:1.29.0
EOF

I have a patch file with following contents

cat <<EOF >$DEMO_HOME/env-java-opts.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  template:
    spec:
      containers:
        - name: myapp-container
          env:
          - name: APP_SECRET
            valueFrom:
              secretKeyRef:
                name: $(ENVIRONMENT_SECRET_PREFIX)-app-external-secret
                key: app_secret

EOF

external secret with following contents

cat <<EOF >$DEMO_HOME/app-secrets.yaml
apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: $(ENVIRONMENT_SECRET_PREFIX)-app-external-secret
secretDescriptor:
  backendType: secretsManager
  data:
    - key: $(ENVIRONMENT_SECRET_PREFIX)/app
      name: app_secret
      property: app_secret
EOF

kustomization file with following contents

cat <<EOF >$DEMO_HOME/kustomization.yaml
configMapGenerator:
  - name: app-vars
    literals:
      - ENVIRONMENT_SECRET_PREFIX="nonprod"

vars:
  - name: ENVIRONMENT_SECRET_PREFIX
    objref:
      kind: ConfigMap
      name: platform-services-address-vars
      apiVersion: v1
    fieldref:
      fieldpath: data.ENVIRONMENT_SECRET_PREFIX

resources:
  - deployment.yaml
  - app-secrets.yaml

patchesStrategicMerge:
  - env-java-opts.yaml

EOF

On running kustomize build the variable are not replaced.

apiVersion: v1
data:
  ENVIRONMENT_SECRET_PREFIX: nonprod
kind: ConfigMap
metadata:
  name: app-vars-ff7mhhf9cg
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myapp
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - env:
        - name: APP_SECRET
          valueFrom:
            secretKeyRef:
              key: app_secret
              name: $(ENVIRONMENT_SECRET_PREFIX)-app-external-secret
        image: busybox:1.29.0
        name: myapp-container
---
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: $(ENVIRONMENT_SECRET_PREFIX)-app-external-secret
secretDescriptor:
  backendType: secretsManager
  data:
  - key: $(ENVIRONMENT_SECRET_PREFIX)/app
    name: app_secret
    property: app_secret

Is this expected? Or are there other ways to solve this?

The text was updated successfully, but these errors were encountered:

natasha41575 · 2021-12-30T21:54:15Z

We are expecting to deprecate and remove vars, I would encourage you to use replacements instead. Please see the documentation: https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/

For your case, to replace the variable in your secret, you would need something like:

app-secrets.yaml

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: ENVIRONMENT_SECRET_PREFIX-app-external-secret
secretDescriptor:
  backendType: secretsManager
  data:
    - key: ENVIRONMENT_SECRET_PREFIX/app
      name: app_secret
      property: app_secret

kustomization.yaml

configMapGenerator:
  - name: app-vars
    literals:
      - ENVIRONMENT_SECRET_PREFIX="nonprod"

resources:
  - app-secrets.yaml

replacements:
- source: 
    kind: ConfigMap
    name: app-vars
    fieldPath: data.ENVIRONMENT_SECRET_PREFIX
  targets:
  - select:
      name: ENVIRONMENT_SECRET_PREFIX-app-external-secret
      kind: ExternalSecret
    fieldPaths: 
    - secretDescriptor.data.[name=app_secret].key
    options:
      delimiter: '/'
      index: 0
  - select:
      name: ENVIRONMENT_SECRET_PREFIX-app-external-secret
      kind: ExternalSecret
    options:
      delimiter: '-'
      index: 0

Running kustomize build with the above files will give you the following output, with your variables replaced:

apiVersion: v1
data:
  ENVIRONMENT_SECRET_PREFIX: nonprod
kind: ConfigMap
metadata:
  name: app-vars-ff7mhhf9cg
---
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: nonprod-app-external-secret
secretDescriptor:
  backendType: secretsManager
  data:
  - key: nonprod/app
    name: app_secret
    property: app_secret

natasha41575 · 2021-12-30T21:54:36Z

/kind support
/triage accepted

itsnagaraj · 2021-12-30T23:34:31Z

We are expecting to deprecate and remove vars, I would encourage you to use replacements instead. Please see the documentation: https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/replacements/

For your case, to replace the variable in your secret, you would need something like:

app-secrets.yaml

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: ENVIRONMENT_SECRET_PREFIX-app-external-secret
secretDescriptor:
  backendType: secretsManager
  data:
    - key: ENVIRONMENT_SECRET_PREFIX/app
      name: app_secret
      property: app_secret

kustomization.yaml

configMapGenerator:
  - name: app-vars
    literals:
      - ENVIRONMENT_SECRET_PREFIX="nonprod"

resources:
  - app-secrets.yaml

replacements:
- source: 
    kind: ConfigMap
    name: app-vars
    fieldPath: data.ENVIRONMENT_SECRET_PREFIX
  targets:
  - select:
      name: ENVIRONMENT_SECRET_PREFIX-app-external-secret
      kind: ExternalSecret
    fieldPaths: 
    - secretDescriptor.data.[name=app_secret].key
    options:
      delimiter: '/'
      index: 0
  - select:
      name: ENVIRONMENT_SECRET_PREFIX-app-external-secret
      kind: ExternalSecret
    options:
      delimiter: '-'
      index: 0

Running kustomize build with the above files will give you the following output, with your variables replaced:

apiVersion: v1
data:
  ENVIRONMENT_SECRET_PREFIX: nonprod
kind: ConfigMap
metadata:
  name: app-vars-ff7mhhf9cg
---
apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: nonprod-app-external-secret
secretDescriptor:
  backendType: secretsManager
  data:
  - key: nonprod/app
    name: app_secret
    property: app_secret

If there multiple keys would we have to repeat the replacements multiple times for each key. Could we do it in one go?

natasha41575 · 2021-12-31T00:00:22Z

If there multiple keys would we have to repeat the replacements multiple times for each key. Could we do it in one go?

You can provide multiple fieldpaths for a single target, e.g.

replacements:
- source: 
    kind: ConfigMap
    name: app-vars
    fieldPath: data.ENVIRONMENT_SECRET_PREFIX
  targets:
  - select:
      name: ENVIRONMENT_SECRET_PREFIX-app-external-secret
      kind: ExternalSecret
    fieldPaths: 
    - secretDescriptor.data.[name=app_secret_1].key
    - secretDescriptor.data.[name=app_secret_2].key
    options:
      delimiter: '/'
      index: 0

Does that answer your question?

itsnagaraj · 2021-12-31T04:00:46Z

If there multiple keys would we have to repeat the replacements multiple times for each key. Could we do it in one go?

You can provide multiple fieldpaths for a single target, e.g.
replacements:
- source: 
    kind: ConfigMap
    name: app-vars
    fieldPath: data.ENVIRONMENT_SECRET_PREFIX
  targets:
  - select:
      name: ENVIRONMENT_SECRET_PREFIX-app-external-secret
      kind: ExternalSecret
    fieldPaths: 
    - secretDescriptor.data.[name=app_secret_1].key
    - secretDescriptor.data.[name=app_secret_2].key
    options:
      delimiter: '/'
      index: 0
Does that answer your question?

Yes. I will give it a go thanks

itsnagaraj · 2022-01-05T23:00:37Z

All the above suggestions and examples worked @natasha41575 and thank you so much for the assistance.

Is it possible to replace multiple variables in a line using replacements or should I continue using variables for the scenario below?

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myapp
  name: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - env:
            - name: JAVA_OPTS
              value: -Dspring.profiles.active=$(SPRING_PROFILE) -Dnewrelic.environment=$(NEWRELIC_ENVIRONMENT_NAME) -Dnewrelic.config.app_name=$(NEWRELIC_ENVIRONMENT_NAME)-myapp

natasha41575 · 2022-01-05T23:31:16Z

Is it possible to replace multiple variables in a line using replacements or should I continue using variables for the scenario below?

It is possible to replace multiple variables in a line using replacements, but what is not possible currently is to use replacements with values that are not delimited. Your above variable $(SPRING_PROFILE) has an = on one side and a on the other, which is a use case that replacements does not yet support.

itsnagaraj · 2022-01-06T04:21:36Z

Ok. Will continue to use vars in this case

gbharadwaj · 2023-05-02T17:15:07Z

Definition of Done - Consul template as part of base repo will have ENV as a placeholder so this task is to replace ENV placeholder with deployment target environment . Example for dev , it should replace ENV with DEV .

Suggestion : Please use kustomize configmap which hold key value pair and replace dynamically in consul template container before actual application read it

How would I achieve this by having a single ENV file as a template and change the environmental variables by DEV / QA /UAT

gbharadwaj · 2023-05-02T17:17:46Z

kustomize yaml file

namespace: oasis-dev
resources:

../../base
patchesStrategicMerge:
oasisapi.yaml
patches:
target:
kind: Ingress
patch: |-
- op: add
  path: /metadata/annotations/kubernetes.io~1ingress.class
  value: nginx-oasis-dev
target:
kind: Ingress
name: oasis-sdcweb-ui
patch: "- op: replace\n path: /spec/rules/0/host\n value: pro-dev.suppressiondesigncenter.com "
configMapGenerator:
name: oasis-oasis-api
files:
- start.sh
- config.hcl
- template.ctmpl=oasisapi.ctmpl
  generatorOptions:
  disableNameSuffixHash: true
  images:
name: oasisapi
newName: oasis-dev-docker-virtual.jfrog-prod.debsreplatform.cloud/fs-oasis-oasis
newTag: "1.0.0-eb9ccc49"

k8s-ci-robot added needs-kind Indicates a PR lacks a `kind/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 9, 2021

natasha41575 mentioned this issue Jan 5, 2022

Deprecate vars #4049

Open

itsnagaraj closed this as completed Jan 6, 2022

randmonkey mentioned this issue Nov 8, 2022

test: add testing on kong router flavor and recover from invalid config Kong/kubernetes-ingress-controller#3127

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to replace variables in deployment's env definition #4337

Unable to replace variables in deployment's env definition #4337

itsnagaraj commented Dec 9, 2021

natasha41575 commented Dec 30, 2021 •

edited

Loading

natasha41575 commented Dec 30, 2021

itsnagaraj commented Dec 30, 2021

natasha41575 commented Dec 31, 2021

itsnagaraj commented Dec 31, 2021

itsnagaraj commented Jan 5, 2022

natasha41575 commented Jan 5, 2022

itsnagaraj commented Jan 6, 2022

gbharadwaj commented May 2, 2023

gbharadwaj commented May 2, 2023

Unable to replace variables in deployment's env definition #4337

Unable to replace variables in deployment's env definition #4337

Comments

itsnagaraj commented Dec 9, 2021

natasha41575 commented Dec 30, 2021 • edited Loading

natasha41575 commented Dec 30, 2021

itsnagaraj commented Dec 30, 2021

natasha41575 commented Dec 31, 2021

itsnagaraj commented Dec 31, 2021

itsnagaraj commented Jan 5, 2022

natasha41575 commented Jan 5, 2022

itsnagaraj commented Jan 6, 2022

gbharadwaj commented May 2, 2023

gbharadwaj commented May 2, 2023

natasha41575 commented Dec 30, 2021 •

edited

Loading