Allow use of $ref in in CRD validation schema

[Liujingfang1]

(combined with #76965)

$ref scenario 1:

In go types, we can define a type and use it in several other types. When generating CRD from this go types, we want to use reference for that type in the validation so that the schema has better readability.

package v1beta1

import (
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
type DataType struct {
  Value int `json:"value,omitempty"`
}
type MyKindSpec struct {
  Data []DataType `json:"data,omitempty"`
}

type MyKindStatus struct {
  Data []DataType `json:"data,omitempty"`
}

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// MyKind
// +k8s:openapi-gen=true
// +resource:path=mykinds
type MyKind struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   MyKindSpec   `json:"spec,omitempty"`
    Status MyKindStatus `json:"status,omitempty"`
}

What you expected to happen:
In the CRD for this type, we expect to see the validation as

     spec:
          properties:
            data:
              items:
                $ref: '#/definitions/<...>.DataType'

$ref scenario 2:

Original type information is lost when CRD authors expand their openapi definitions. openapi-gen-based operator code is similarly affected. CRDs forbid $ref (implicit GVK link) and x-* properties (GVK hints).

This information enables early, developer-friendly policy checks and useful manifest transforms. Our CICD automation extends the base k8s schema with custom policy and relies on the schema's $ref-erential knowledge to inspect and operate on target resource types, anywhere they appear, eg. deeply nested or embedded within custom resources. Prior to validation, a simple callback might first:

ObjectMeta

• Require teams set app.kubernetes.io/* labels.
• Require teams set org-specific cost/usage/alert aggregation labels.
• Back and forward compat handling, eg. app, app.kubernetes.io/name, team, ...

Container

• Inject dynamic ENV and volumes.
• Resolve string properties with templates.
• Require several resources.limits.* are set.

PodSpec ... PodSecurityContext ... Volume ...
Deployment ... CronJobSpec ... JobSpec ...
ServiceSpec ... SecurityContext ... (many more) ...

$ref expansion makes CRDs opaque to our automation. A pre-processing step is required to repair the type information and restore visibility into CRDs. Options considered are:

• Import all golang-based openapi-gen'ed CRDs directly (wherever they write openapi_generated.go) and call each GetOpenAPIDefinitions function. Merge all results.
• Run openapi-gen on k8s and CRD sources ourselves. Avoids multiple GetOpenAPIDefinitions calls and potential linking problems.
• Use kube-apiserver. Run hack/update-openapi-spec.sh with extra CRDs "builtin" and enabled so GET /openapi/v2 simply returns what we need.

I'd love $ref support, but I understand why that could be difficult or undesirable. Alas, if CRDs permitted a known property, eg. x-kubernetes-group-version-kind (or similar), CRD libraries can set it accordingly for each unique $ref expansion (no need to nest GVK hints), allowing consumers (or even k8s itself) to restore this relationship if and when useful.

Are there options we can move on?

[Liujingfang1]

/sig api-machinery

[Liujingfang1]

/assign mbohlool

[lavalamp]

/assign @mbohlool

I agree it seems good to allow references within a single CRD (as opposed to across resources, which I think we're not ready for).

[liggitt]

The reference-expanding code in the schema lib we're using for CRD validation is scary… if we do this, we shouldn't use an expander that has any possibility of making network or filesystem checks in the process

[nikhita]

The reference-expanding code in the schema lib we're using for CRD validation is scary… if we do this, we shouldn't use an expander that has any possibility of making network or filesystem checks in the process

👍

/area custom-resources
/cc @sttts

[nikhita]

/area custom-resources

[sttts]

reference-expanding code in the schema lib we're using for CRD validation is scary

Sounds like we have to abstract the reference resolver in kube-apiserver and instantiate it with something less scary.

[sttts]

/cc @ayushpateria

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

Will need to investigate if/how we want to do this.

[alegacy]

Has there been any progress on this investigation? We have a usecase where using references from within a single CRD would be useful.

[the-voidl]

Is there still any chance you let the users use $ref in CRDs?
I'd estimate a recursive CRD as relatively common - even the definitions of io.k8s.api resources highly rely on $ref.

I see you points @liggitt, still would love any feedbeek if anyone will be implementing this.

[liggitt]

This doesn't seem likely to make progress in an issue (no one is working on a design that addresses the significant additional complexity and failure modes introduced). I'd suggest closing this issue, and bringing up the topic in a sig-api-machinery meeting to gauge whether addition of this feature is likely to be accepted and whether there is priority / bandwidth to shepherd a design and implementation for it now

[lavalamp]

I think this issue has value as the place everyone is apparently watching.

I agree it's not making progress right now and to make progress needs an owner to drive it.

The first hurdle here is to gather requirements and demonstrate that some (possibly limited) version of this would be very helpful to a sufficient number of users. This could either start with a SIG meeting agenda item, or end with a SIG meeting agenda item once evidence has been collected, or both.

Schema recursion seems actually necessary to me to express some schemas? So I expect some utility to be forthcoming. However it is (obviously) extremely dangerous, too.

[zenbones]

I would call any schema meant to describe an object hierarchy, that can't handle self-referential recursion... impoverished. Would it really be that hard to handle a $ref that just recursively imported the schema pointed to by $ref? Inline the entire thing and then run the validation, which should be the same as having to write the whole thing out, which is currently allowed. I'm just looking for a way to avoid massive cut and paste that I need to find/replace on whenever I make a change. It can not be that hard to allow $ref in the same file as a starting point. I would do it but I find Go impoverished and hate grinding my teeth while I code. I know, that last comment is both unfair and unnecessary, and I apologize in advance, but I'm leaving it because it's true.

[lavalamp]

Some such references expand infinitely and can't be expanded, that's a large part of what makes it dangerous.

[rjeberhard]

I don't want to exclude others' use cases, but I would be satisfied with a mechanism to reference types from built-in schemas. For instance, we have a CRD where our controller will create Pods based on the contents of the custom resource. It would be very convenient to be able to reference existing types for volume and volume mounts, etc. These types (e.g. volumes) if properly expressed in a CRD's schema are very large. Note: there are other solutions such as having the custom resource reference a PodTemplate, but this isn't as convenient for our users.

[liggitt]

I don't want to exclude others' use cases, but I would be satisfied with a mechanism to reference types from built-in schemas

github helpfully hid my comment at #62872 (comment), but even referencing types from built-in schemas has a lot of potential problems

[zenbones]

Some such references expand infinitely and can't be expanded, that's a large part of what makes it dangerous.

How about a recursion limit as a tried and tested solution? Expand, look for expansions, expand... oh we've done that 10 times, throw an error. I mean, it's my machine and my compute time so I would appreciate a simple argument to set the limit, but I'd happily take a reasonable static limit.

[jessefeinman]

I'm facing a similar situation where $ref support would be helpful.

Specifically $ref support for accessing other locations in the same file would be nice but more importantly would be $ref support for accessing k8s types (or other loaded CRDs though thats not necessary).

e.g. we have rules to validate ObjectMeta (among other types) in our CICD pipeline, we currently have to do a bunch of tedious code generation in order to have the type references in the schemas we validate against. These references let us walk through the yaml and run custom rules against the specific types when they appear. We are unable to migrate to validating against the schemas defined by the CRDs directly because we wouldn't be able to correctly detect that a type defined in a CRD is actually a k8s type that we have a rule for. However, if there was a $ref with a path to a k8s type, then a) the crd is simpler to understand and b) we can determine if that path matches a type we have a custom rule for already and can then apply it.

[lavalamp]

a $ref with a path to a k8s type

This gets dicey around upgrades when a k8s type does some (legal) modification like add a field.

This is not an easy change to make and it's not currently prioritized (but we might do it eventually).

[jpbetz]

We've also seen some general problems with type sharing in the API. This came up recently with pod containers and initContainers sharing the same type, which caused problems when wanting to add a field needed only for initContainers (xref #115362).

That said, I can see some value to doing a $ref to things like podTemplate, which seems reasonable if the goal is to just envelope the podTemplate without needing to care about the contents.

[rjeberhard]

That said, I can see some value to doing a $ref to things like podTemplate, which seems reasonable if the goal is to just envelope the podTemplate without needing to care about the contents.

This use case would really help. We have a variety of operators that create pods and deployments and where our customers would like to customize the generated pods by specifying sidecar containers, volumes, etc. We've experimented with letting the customer specify a reference to a pod template, but they found this confusing. So, instead, we've reproduced the rather large schemas of these types in our CRD.