[Question] How to use ReplacementTransformer for fields that need a text placeholder?

[rkr-kununu]

Since it's been announced that vars will be deprecated in favor of ReplacementTransformer, I'd like to replace our use of vars in our test environment.

However, we're using Traefik's CRDs which is running within a dedicated namespace (with a wildcard ssl cert). This means each test environment exists within their respective namespace and share a single (public) Traefik instance to mock our public facing services.

Traefik's CRD introduces IngressRoute which has a Match section which contains an arbitrary number of rules. Because our instance of Traefik is shared, we use the Host in the request to have Traefik route the request to the proper namespace.

We've been using vars to extract the equivalent of the namespace. So our IngressRoute looks something like:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: app-cms
spec:
  entryPoints:
    - websecure
  routes:
  - match:
      Host(`www-$(NAMESPACE).example.com`) && Path(`/test$}`)
    kind: Rule
    services:
    - name: app-cms
      namespace: $(NAMESPACE)
      port: 80
  - match:
      Host(`www-$(NAMESPACE).example.com`) && (PathPrefix(`/example`)
      || Path(
          `/`, `/a`, `/b`, `/c`, `/d`)
      )
    kind: Rule
    services:
    - name: app-cms
      namespace: $(NAMESPACE)
      port: 80
  tls: {}

However, from the documentation/experiments I've made it appears that ReplacementTransformer is pretty limited when it comes to supporting how we're using Host().

Is there some undocumented feature I'm missing?

[k8s-ci-robot]

@rkr-kununu: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[natasha41575]

This is a problem we are aware of - a similar use case is described in #3978. For now we are thinking about using generators for such use cases, but it will be difficult for us to provide generators for all possible CRDs.

We've written a builtin generator plugin for generating service account resources. Something you can do is write your own generator plugin (a guide is here) for your CRD.

[rkr-kununu]

Thank you for the feedback - I'm relieved to know it's on the radar. However, I'm not too thrilled with the prospect of needing to write/install a custom generators for the various CRDs we want to use or expecting the kubernetes community to provide them. Especially, when great tools like crossplane exist, which create an easy way to make custom opinionated CRDs.

Could we instead unify the configuration of the ReplacementTransformer with the ValueAddTransformer and extend (potentially both of them)?

An example of the current ValueAddTransformer:

apiVersion: builtin
kind: ValueAddTransformer
metadata:
  name: MyNamespaceFolder

targets:
  - selector:
      kind: Namespace
    fieldPath: metadata/annotations/my-namespace
    # Replace any existing value
    filePathPosition: 0

An example of the current ReplacementTransformer:

apiVersion: builtin
kind: ReplacementTransformer
metadata:
  name: notImportantHere
replacements:
- source: 
    kind: Deployment
    fieldPath: spec.template.spec.containers.0.image
  targets:
  - select:
      kind: Deployment
    fieldPaths: 
    - spec.template.spec.containers.1.image

...by "unifying" I specifically mean take the targets yaml structure from the ValueAddTransformer and use it in ReplacementTransformer (thus deprecating the current targets yaml structure).

apiVersion: builtin
kind: ReplacementTransformer
metadata:
  name: notImportantHere
replacements:
- source: 
    kind: Deployment
    fieldPath: spec.template.spec.containers.0.image
  targets:
  - selector:
      kind: Deployment
    fieldPath: spec.template.spec.containers.1.image
    # Replace any existing value
    filePathPosition: 0

Then we can modify the ReplacementTransformer (and optionally the ValudAddTransformer) to "extend" it's features like filePathPosition and introduce a generic one like placeholderName.

So, how this would relate to my use-case... Given:
the replacement.yaml

apiVersion: builtin
kind: ReplacementTransformer
metadata:
  name: notImportantHere
replacements:
- source: 
    kind: Namespace
    fieldPath: metadata/name
  targets:
  - selector:
      kind: IngressRoute
    fieldPath: spec.routes.*.match
    # This will replace all-occurrences of NAMESPACE with the value referred in replacements/source/fieldPath (ie: test)
    placeholderName: NAMESPACE
  - selector:
      kind: IngressRoute
    fieldPath: spec.routes.*.services.*.namespace
    # Replace any existing value
    filePathPosition: 0

...and the routes.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: app-cms
  annotations:
    my-namespace: test
spec:
  entryPoints:
    - websecure
  routes:
  - match:
      Host(`www-NAMESPACE.example.com`) && Path(`/test$}`)
    kind: Rule
    services:
    - name: app-cms
      namespace: IGNORED
      port: 80
  - match:
      Host(`www-NAMESPACE.example.com`) && (PathPrefix(`/example`)
      || Path(
          `/`, `/a`, `/b`, `/c`, `/d`)
      )
    kind: Rule
    services:
    - name: app-cms
      namespace: IGNORED
      port: 80
  tls: {}

Which would ultimately yield:

apiVersion: v1
kind: Namespace
metadata:
  name: test
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: app-cms
spec:
  entryPoints:
    - websecure
  routes:
  - match:
      Host(`www-test.example.com`) && Path(`/test$}`)
    kind: Rule
    services:
    - name: app-cms
      namespace: test
      port: 80
  - match:
      Host(`www-test.example.com`) && (PathPrefix(`/example`)
      || Path(
          `/`, `/a`, `/b`, `/c`, `/d`)
      )
    kind: Rule
    services:
    - name: app-cms
      namespace: test
      port: 80
  tls: {}

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rkr-kununu]

/remove-lifecycle stale
For me, this is really a critical defect in the ReplacementTransformer that needs to be resolved before vars can be retired.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[rkr-kununu]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[jonashackt]

We had the same problem and also didn't want to write/maintain our own custom generator plugin. We created a full-blown stackoverflow Q&A - but here are the brief steps:

As a workaround we can create yaml files inline in our console using the syntax cat > ./myyamlfile.yml <<EOF ... EOF and also use the inline variable substitution. So first define the branch name as variable:

    NAMESPACE=foobar

And then use the described syntax to create a ingressroute-patch.yml file inline:

    cat > ./ingressroute-patch.yml <<EOF
    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRoute
    metadata:
      name: app-cms
    spec:
      entryPoints:
        - websecure
      routes:
      - match:
          Host(`www-$NAMESPACE.example.com`) && Path(`/test$}`)
        kind: Rule
        services:
        - name: app-cms
          namespace: $NAMESPACE
          port: 80
    
    EOF

The last step is to use the ingressroute-patch.yml file as patchesStrategicMerge inside our kustomization.yaml like this:

    patchesStrategicMerge:
      - ingressroute-patch.yml

Now running kustomize build . should output the desired IngressRoute:

    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRoute
    metadata:
      name: app-cms
    spec:
      entryPoints:
        - websecure
      routes:
      - match:
          Host(`www-foobar.example.com`) && Path(`/test$}`)
        kind: Rule
        services:
        - name: app-cms
          namespace: foobar
          port: 80

It's only a workaround, but might help to bridge the time until @rkr-kununu 's great idea might get implemented into Kustomize.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.