add genericGenerator

[sethpollack]

for example:

genericGenerator:
  resourcePath: cronjob.yaml
  fieldPathArgs:
  - path: metadata.name
    value: mycronjob
  - path: spec.schedule
    value: "* * * * *"

[Liujingfang1]

@sethpollack Can you add some explanation of meaning of each field? I also opened #122 for supporting extensible transformation. It seems to have some overlap with the motivation of genericGenerator you proposed.

[sethpollack]

Sure! This would be a transformation where you can take a base resource resourcePath and modify its fields by passing fieldPathArgs with the path to the field and the replacement value.

A good example would be cronjobs where I may want to create a bunch of them from the same base - just with different names, commands, and schedules.

The transformation would just loop through the provided fieldPathArgs rather than hardcoded PathConfigs and just call mutateField for each.

[Liujingfang1]

Is resourcePath a file path to a resource file in the base folder? Usually the overlays are not supposed to know the exact files or structs of the base.

[sethpollack]

yes

[Liujingfang1]

This feature looks promising to me. I am also thinking about some other use cases that it might support. Say I want to replace a container image name no matter it is in a deployment object or in a pod. How could I define the resource and field path so that it can work?

Can you write a brief design doc about this genericGenerator(maybe some other name genericTransformer)? Let have others' opinion on it.

[sethpollack]

I think the genericGenerator solves a different problem than the genericTransformer.

For example If I have 25 cronjob variations I would need 25 subdirectories with a kustomization and a patch file in each and then do something like your suggestion in #38 to generate them all from the same base resource:

kustomizations:
- ./cronjobs/cronjob1
- ./cronjobs/cronjob2
...

The idea for the genericGenerator would be to use 25 genericGenerator's in the base kustomization file instead.

genericGenerator:
  resourcePath: cronjob.yaml
  fieldPathArgs:
  - path: metadata.name
    value: cronjob1
  - path: spec.schedule
    value: "* * * * *"

genericGenerator:
  resourcePath: cronjob.yaml
  fieldPathArgs:
  - path: metadata.name
    value: cronjob2
  - path: spec.schedule
    value: "* * * * *"
...

As for the genericTransformer we can do something with PathConfigs like this:

genericTransformer:
  pathConfigs:
  - path: spec.template.spec.containers.name
    createIfNotPresent: true
    groupVersionKind:
      kind: Deployment
    matches: docker.io/nginx:latest
    value: docker.io/nginx:1.15.0
  - path: spec.jobTemplate.spec.template.spec.containers
    createIfNotPresent: true
    groupVersionKind:
      kind: CronJob
    matches: docker.io/nginx:latest
    value: docker.io/nginx:1.15.0    

[Liujingfang1]

Regardless of the different formats, my understanding based on your examples is that genericGenerator is used to generate new object, while genericTransformer doesn't generate new object, it only updates existing ones.

[sethpollack]

correct.

[sethpollack]

I can work on a PR for the genericTransformer - once Im done working on the secrets PR.

[Liujingfang1]

No rush on that. I need to think more thoroughly on genericTransformer so that it can serve different usage.

[sethpollack]

Ok

[droot]

@sethpollack

For example If I have 25 cronjob variations I would need 25 subdirectories with a kustomization and a patch file in each and then do something like your suggestion in #38 to generate them all from the same base resource:

What is your major concern about the above (current way of doing it) ?

[sethpollack]

@droot I don't think there is a current way - other than duplicating the entire cronjob for each variant of it.

Also I don't think that patches would work anyway since you cant change the name with a patch.

[Liujingfang1]

Let's revisit this request. Recently we added support for multibases with a common base. https://github.com/kubernetes-sigs/kustomize/blob/master/examples/multibases/README.md
So for you use case, you can create multiple cronjobs(as in different variants) from a common base. Then compose those variants together in another overlay.
With this available, I think we don't need genericGenerator. @sethpollack What do you think?

[Liujingfang1]

This genericGenerator idea can be done by JSON patch support #169

[Liujingfang1]

The original request should be able to achieved by multibases support and json patch. Basically, you create a common base for your cronjob. Then create multiple overlays with different prefixes. In those overlays, apply a proper json patch to the cronjob object.

[yubessy]

I've created an example of #126 (comment) for future issue visitors like me.
https://github.com/yubessy/example-kustomize-cronjob-multiple-schedule

[abdulsalama]

hmm I actually think that having genericGenerator is still a great feature to add and is really needed. I totally get the point that this can be done with json patch, but it comes with so much unnecessary overhead. Something like genericGenerator would be much cleaner and more intuitive for this kind of need which is very common.

[sl805]

@Liujingfang1 Actually genericGenerator is a crucial feature from operations standpoint. Its key difference with anything existing before it is a capability to create new object, instead of modifying existing one. As of now in order to create one modified object, one needs to create a base for specific resource kind, then refer it, and only then patch it. So to create 3 services, you'll have to create one base and then refer it 3 times and patch it 3 times, but it's getting worse if you have other resources have to be managed this way. Anyone who deals with huge fleet of similar applications will run into this issue when there's a need create a bunch of nearly identical (but not identical) resources like Jobs, Services, Deployments, networkPolicies, serviceAccounts just to name a few.

Since we have replacement transformer, can we ask to add newObject: false|true to replacement config, so it could create new target object instead of modifying existing one, if this option set to true ?