Consider $ref in CRDs or recommending authors add a GVK hint in replacements

[anthonyrisinger]

Merged with #62872

Original type information is lost when CRD authors expand their openapi definitions. openapi-gen-based operator code is similarly affected. CRDs forbid $ref (implicit GVK link) and x-* properties (GVK hints).

This information enables early, developer-friendly policy checks and useful manifest transforms. Our CICD automation extends the base k8s schema with custom policy and relies on the schema's $ref-erential knowledge to inspect and operate on target resource types, anywhere they appear, eg. deeply nested or embedded within custom resources. Prior to validation, a simple callback might first:

ObjectMeta

• Require teams set app.kubernetes.io/* labels.
• Require teams set org-specific cost/usage/alert aggregation labels.
• Back and forward compat handling, eg. app, app.kubernetes.io/name, team, ...

Container

• Inject dynamic ENV and volumes.
• Resolve string properties with templates.
• Require several resources.limits.* are set.

PodSpec ... PodSecurityContext ... Volume ...
Deployment ... CronJobSpec ... JobSpec ...
ServiceSpec ... SecurityContext ... (many more) ...

$ref expansion makes CRDs opaque to our automation. A pre-processing step is required to repair the type information and restore visibility into CRDs. Options considered are:

• Import all golang-based openapi-gen'ed CRDs directly (wherever they write openapi_generated.go) and call each GetOpenAPIDefinitions function. Merge all results.
• Run openapi-gen on k8s and CRD sources ourselves. Avoids multiple GetOpenAPIDefinitions calls and potential linking problems.
• Use kube-apiserver. Run hack/update-openapi-spec.sh with extra CRDs "builtin" and enabled so GET /openapi/v2 simply returns what we need.

I'd love $ref support, but I understand why that could be difficult or undesirable. Alas, if CRDs permitted a known property, eg. x-kubernetes-group-version-kind (or similar), CRD libraries can set it accordingly for each unique $ref expansion (no need to nest GVK hints), allowing consumers (or even k8s itself) to restore this relationship if and when useful.

Are there options we can move on?

[anthonyrisinger]

@kubernetes/sig-api-machinery-feature-requests

[k8s-ci-robot]

@anthonyrisinger: Reiterating the mentions to trigger a notification:
@kubernetes/sig-api-machinery-feature-requests

In response to this:

@kubernetes/sig-api-machinery-feature-requests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liggitt]

/cc @sttts

[caesarxuchao]

/cc @jpbetz

[roycaihw]

discussion related to $ref idea: #62872

[liggitt]

closing as a duplicate of #62872

/close

[k8s-ci-robot]

@liggitt: Closing this issue.

In response to this:

closing as a duplicate of #62872

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[anthonyrisinger]

I'm not sure this is a duplicate. The other issue specifically states:

This request is only to refer CRD internal types, not to refer k8s core types or external types.

This ticket is the exact inverse -- I don't care about the internal CRD types -- we need proper references to kubernetes objects within CRD definitions.

They are definitely related asks, but if we're going to leave the other one open can we add the requirements from this one so they do not get lost? I did go through some effort in preparing options and a succinct report likely to produce a solution.

(Perhaps there's still time for a simple, zero-overhead hint like x-kubernetes-whatever to make the aforementioned KEP? There wasn't strong pushback from @sttts, it could be optional, and could be removed any time in the future when a proper solution drops)

In lieu of a permanent option, is there any guidance on how to remedy the situation? I tried to build extension objects directly into kube-apiserver; it seems to work, but when I enable they can't be found so they're not registering properly.

[liggitt]

if we're going to leave the other one open can we add the requirements from this one so they do not get lost?

yes, let's collect the $ref requests in a single issue

[liggitt]

Perhaps there's still time for a simple, zero-overhead hint like x-kubernetes-whatever to make the aforementioned KEP? There wasn't strong pushback from @sttts, it could be optional, and could be removed any time in the future when a proper solution drops

fields can't be dropped without a version rev, so adding things to the API is a long-term commitment. when we were working through the v1 requirements, this was deferred (xref kubernetes/enhancements#1002 (comment))

[liggitt]

merged this with #62872

[anthonyrisinger]

Fair enough; being represented in the other ticket is enough for me :-) I'm happy to wait and see where `$ref` support lands and I'm confident the result will be useful to us.

…

On Fri, May 3, 2019, 9:49 AM Jordan Liggitt ***@***.***> wrote: Perhaps there's still time for a simple, zero-overhead hint like x-kubernetes-whatever to make the aforementioned KEP? There wasn't strong pushback from @sttts <https://github.com/sttts>, it could be optional, and could be removed any time in the future when a proper solution drops fields can't be dropped without a version rev, so adding things to the API is a long-term commitment. when we were working through the v1 requirements, this was deferred (xref kubernetes/enhancements#1002 (comment) <kubernetes/enhancements#1002 (comment)> ) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#76965 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABRLHN2RDAOQZBH6GNM4ATPTRGGNANCNFSM4HH6V4HQ> .