[v1alpha2] Generate CRD validation in Kubernetes 1.11

[gaocegege]

Ref #561 (comment)

I wrote a tool to generate the validation from the OpenAPI specification: https://github.com/gaocegege/crd-validation. Generated CRD for tfjob v1alpha2 is https://github.com/gaocegege/crd-validation/blob/master/generated/tfjob-crd-v1alpha2.yaml

While we meet an issue from Kubernetes side: kubernetes/kubernetes#59485 (comment). Kubernetes does not support addtionalproperties while it is needed for map type. Unfortunately, TFRelicaSpec is a map.

The upstream said it will be implemented in 1.11. Maybe we should wait for it. After 1.11 I think we are able to use the tool above to generate validations for all crds in kubeflow community.

[gaocegege]

This will limit the version of Kubernetes to 1.11.

[gaocegege]

/assign @codeflitting

[ddysher]

what's the plan for crd validation, @codeflitting @gaocegege can anyone of you summarize here? are we limiting user to only use kubernetes 1.11, or we follow @jessesuen 's suggestion?

[gaocegege]

@ddysher

We now follow jessesuen@'s suggestion to use the unstructured informer. While this is a WIP issue to use crd validation feature in Kubernetes to validate our CRD. If we do, we need to use k8s 1.11 because of kubernetes/kubernetes#59485 (comment)

[jlewi]

Would it be possible to only enable CRD validation on K8s 1.11 and newer? So that we could continue to support older K8s? Would that be worth doing?

[gaocegege]

I think it is not in high priority. We definitely could enable it since it just needs some config in crd yaml file.

[jlewi]

@johnugeorge @richardsliu should we try to do this in 0.5?

[richardsliu]

We should add this in 0.5.

[richardsliu]

@gaocegege I am not sure if we can do this actually. We are adding support for multiple CRD versions (#932), and version-specific schema validation is not yet supported. ETA is k8s 1.16.

[richardsliu]

Moving this out of 0.5.0.

[gaocegege]

@richardsliu Now Kubebuilder could generate openAPISchema for CRD, maybe we could take a look at the implementation of kubebuilder.

[richardsliu]

We have not decided if we are using kubebuilder yet. See discussion here: kubeflow/common#7 (comment)

[k82cn]

maybe we could take a look at the implementation of kubebuilder.

It's hard to import it which is binding with kubebuilder; in kube-batch/volcano, I build a shadow project to generate CRD openAPISchema and copy back to my project; and then modify it manually if necessary :)

[gaocegege]

@richardsliu Yeah I saw the comments. We can use the function of validation generation to generate openAPISchema for all our CRDs. As for the other functions in Kubebuilder which requires re-implementation, we should discuss it carefully.

[gaocegege]

We can run

go run vendor/sigs.k8s.io/controller-tools/cmd/controller-gen/main.go crd --output-dir ./examples/crd/

to use https://github.com/kubernetes-sigs/controller-tools generate CRD with validation for tf-operator. But the group will be tensorflow.kubeflow.org, not kubeflow.org. I think I could maintain a fork or file an PR to controller-tool to avoid tensorflow prefix then use the fork to generate the CRD from our API.

[richardsliu]

@johnugeorge Should we fix this for v1.0?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.