Umbrella issue for CRDs moving to GA

[nikhita]

This is a continuation of the CustomResourceDefinition feature.

As discussed earlier, the plan is to go GA. This is the list of tasks that need to be addressed for CRDs.

Please categorize (do not remove, please) whichever does not seem appropriate for moving to GA and put your name behind it. If you want to add any notes specific to a task, please list it as a bullet point and again, put your name behind it.

Features

GA-blocking

• Generate OpenAPI spec for registered CRDs (Publish CRD openapi #71192, "kubectl explain" should be able to explain "apiservices" and "customresourcedefinition" #49465)
  • @sttts: features in kubectl rely on this
• CustomResourceValidation feature to stable CRD validation umbrella issue #62323 (will be promoted along with v1 API)
• CustomResourceSubresources feature to stable (will be promoted along with v1 API)
• CustomResourceWebhookConversion feature to stable (will be promoted along with v1 API)
  • Follow-ups for CRD versioning #64136
  • ConversionReview API to v1 (will be promoted along with v1 API)
  • scale test, ensure adequate metrics - tracked in sig-api-machinery: Add scale targets to CRDs to GA KEP enhancements#1015
  • resolve how webhooks indicate they want v1 or v1beta1 ConversionReview objects sent to them
• rename validation -> schema in v1 (CRD: rename validation -> schema #64483)
• Support quota for custom resources (quota support for custom resources #53777, Limits around the number of CRs community#798, Support object count quota for custom resources #64201)
  • @sttts: critical for enterprise workloads
• Prevent metadata update on subresource update (validate no changes were made)
  • @nikhita: done for CRs, but not CRDs
  • @liggitt: the current behavior for CRs was the important one for GA
• Defaulting and pruning for custom resources (Strip unknown fields in custom resources #59151, apiextensions: [alpha] implement defaulting #63604)
  • @nikhita: design proposals and WIP PRs open (KEP: Pruning for CustomResources community#2443, apiextensions: [alpha] implement defaulting #63604)

Post-GA

• Human readable status from conditions for a CRD using additionalPrinterColumns (Human readable status from conditions for a CRD using additionalPrinterColumns #67268)
  • @liggitt: this sounds like a good candidate for future enhancement, but not GA-blocking
• Consider changing the schema language in CRDs (CRD: Consider changing schema language #67840)
  • @liggitt: resolved by https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190425-structural-openapi.md
• Should support graceful deletion for storage (Should support graceful deletion for CR storage  #68508)
• Enable arbitrary CRD field selectors by supporting a whitelist of fields in CRD spec (Enable arbitrary CRD field selectors #53459)
  • @sttts: optional, can be added later
• Support graceful deletion for custom resources (Grace period on DeleteOptions has no effect on CRs #56567)
  • @sttts: optional, can be added later
• CRD validation webhooks (Add design details for Custom resource definition webhook validation. community#1418)
  • @sttts: admission webhooks work today. More can be added later
  • @liggitt: validating webhooks cover this sufficiently
• Allow OpenAPI references in the CRD validation schema (Feature Request: Allow openapi references in CRD validation schema #54579, Allow use of $ref in in CRD validation schema #62872)
  • @sttts: optional, convenience, can be added later
• Generate json-schema for use in the CRDs from the go types (Generator for CRD Validation Schema #59154, Create json-schema for use in the CRDs from the go types. sample-controller#2, Generate json-schema from go types code-generator#28)
  • @sttts: only developer convenience
• Support for namespace-local CRD (Add support for namespace-local CRD #65551)
  • @sttts: optional feature, can be added later
  • @nikhita: we will probably not do this any time soon
• Support proto encoding for custom resources (CRDs: support binary encoding  #63677)
  • @nikhita: can be added later
• labelSelectorPath should be allowed not be under .status (CRD /scale labelSelectors shouldn't need to be under status #66688)
  • @liggitt: this can be expanded if there is a need for it, and it is not clear this is desired
• Support arbitrary non-standard subresources for CRDs (Support arbitrary subresources for custom resources #72637)
  • @liggitt: it is not clear that this is a pattern we want to encourage, and should not block graduating the standard REST spec/status feature set

Completed features

• Support multiple CRD APIVersions and migration (Multiple CRD APIVersions and migration #57316, Custom Resource Definition Versioning enhancements#544, [WIP] Crd versioning #60113, CRD Versioning with no-op Conversion #63518)
  • Support for no-op conversion versioning (Crd versioning with nop Conversion #63830)
  • Webhook conversion for custom resources (Webhook Conversion for Custom Resources enhancements#598, KEP for CRD conversion webhook community#2420)
• Allow retries on patch conflicts (CRDs merge-patch failing with "unable to find api field in struct Unstructured for the json field "metadata"" #58017, Support strategic merge patch for custom resources #58414)
• Subresources for Custom Resources (Sub-resources for Custom Resources #38113)
• Better kubectl printing for custom resources (Proposal: better output when getting 3prs #31343, Generate OpenAPI spec for registered CRDs #58603 60991)
• Allow to specify categories for custom resources (Allow to specify categories for custom resources #59561)
• CustomResourceDefinitions should support maps in OpenAPI schema (CustomResourceDefinitions should support maps in OpenAPI schema #59485)
• Allow more fields at root of CRD schema when status is enabled (Re-evaluate fields at root of the CRD schema when status subresource is enabled  #65293)
• Support API Paging for CRDs and custom resources (Set paging feature correctly for aggregator and crd storage #67861)

Rejected or out-of-scope feature requests

• Allow modifying CR metadata on status update via modifying webhook (Allow modifying CR metadata on status update via modifying webhook #70084)
  • @liggitt: allowing admission to make changes that would not be allowed via the API is not desired
• Support CRDs in dashboard (Support Custom Resource Definitions dashboard#2493)
  • @sttts: the dashboard is only one of many consumers. Not critical
• Fix kubectl get --export for custom resources (Unstructured represents nil Object fields inconsistently #48211)
  • @sttts: optional feature
  • @liggitt: export support for all objects is still under consideration - Implement export functionality server-side for all resources #24855

Bugs

GA-blocking

• kubectl get with a short name/category will error on the first try (Getting new CRD with short name/category will occur error #65517)
• Invalid regex in OpenAPI validation crashes API server (Invalid regex in OpenAPI validation crashes API server #65470)
• Rename CustomResourceColumnDefinition JSONPath as jsonPath for v1 (v1: CustomResourceColumnDefinition JSONPath should be serialized as "jsonPath" #66531, tracked in https://github.com/orgs/kubernetes/projects/28)
• CRD Established condition has null for lastTransitionTime (CRD Established condition has null for lastTransitionTime #69654, tracked in https://github.com/orgs/kubernetes/projects/28)
• CRDs created in old versions don't have .metadata.generation (Custom resources created in old versions don't have .metadata.generation #68998, tracked in https://github.com/orgs/kubernetes/projects/28)
• CRD validation error from 1.11 server was not correct, lacks details of errors (CR validation error from 1.11 server was not correct, lacks details of errors #68787, tracked in https://github.com/orgs/kubernetes/projects/28)
• CRD cannot have a field named 'items' (CRD cannot have a field named 'items' #68466, tracked in https://github.com/orgs/kubernetes/projects/28)

Non-GA blocking

• Watcher falls into infinite loop on bad Custom Resource manifests (Watcher error loop "Unable to decode an event from the watch stream" from bad CR manifests #57705)
  • @sttts: should be fixed, but is no deep problem
• CRD age does not change from <invalid> as soon as it becomes Established (CRD age does not change from <invalid> as soon as it becomes Established #57042)
  • @sttts: should be fixed, but is no deep problem
  • @liggitt: this is actually a client clock skew issue
• Apiextensions discovery is broken without kube-aggragator (Apiextensions discovery is broken without kube-aggragator #68971)
  • @liggitt: still under investigation, working as intended in kube-apiserver

completed bugs

• kubectl is inconsistent when the CRD has the same names as that of a native resource (kubectl should use TPR full name instead of subname #42521)
  • @sttts: consistency issue, must be fixed
• Collect prometheus metrics for custom resources (Collect prometheus metrics for custom resources #55146)
• Validate metadata and persist label/annotation values properly (Namespaced CRD cannot used label selector #59451)
• Allow required and description, along with properties in the root schema when the status subresource is enabled (apiextensions: allow "required" at root with status subresource #63533, apiextensions: allow Description at root with status subresource #64766)
• Creating a custom resource with non-traditional plurals and subresources doesn't work with kubectl for 1.11 (When subresource is given, non-conventional CRD resource pluralization not working in kubectl #67235)
• resourceVersion is incremented for custom resources on no-op writes (k8s 1.11 custom object resource version changes on no-op status patch #67541)

Tech debt

• Add CRD garbage collector tests to cover aggregated API server (Add garbage collector tests to cover aggregated API servers #49801)
• Revisit Creating CustomResourceDefinitions from apiextensions-apiserver integration tests (Revisit Creating CustomResourceDefinitions from apiextensions-apiserver integration tests #63656)
• Add e2e test for CRD watch (No e2e test for CRD watch (?) #55538)
• apimachinery passes through Unstructured in inconsistent ways (apimachinery passes through Unstructured in inconsistent ways #60312)

/cc @sttts @deads2k @enisoc @liggitt @ironcladlou @mbohlool @lavalamp @caesarxuchao @jennybuckley @ash2k @munnerz

[nikhita]

/sig api-machinery
/area custom-resources

[hzxuzhonghu]

/cc

[nikhita]

Added #59151.

[nikhita]

Added #59154.

/cc @frankgreco

[munnerz]

/cc

[nikhita]

Quick update for folks on this thread: have updated the main issue body with more features and bugs we need to get in.

[fedebongio]

/cc

[lavalamp]

Should we support export for CRDs somehow? It's mentioned above but the link doesn't seem to relate in an obvious way?

[rmohr]

Enable arbitrary CRD field selectors by supporting a whitelist of fields in CRD spec (#53459)
@sttts: optional, can be added later

How important is it for others that CRs support custom field selectors? In combination with the status subresource it becomes pretty ugly to set labels. We have for instance a status.nodeName on our object and also set a corresponding label, so that the node daemons can see the objects. With the status subresource this is not an atomic operation anymore (#70084).

Right now we first set a label and then do the status update and have some extra logic in the daemons to detect the "glitch" and wait until the hand-over is completely done.

[liggitt]

How important is it for others that CRs support custom field selectors? In combination with the status subresource it becomes pretty ugly to set labels

that is on the list of features to consider post-GA

[warmchang]

/cc

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[nikhita]

/remove-lifecycle stale

[marun]

@liggitt An update via the status endpoint for a core type (e.g. Namespace) supports changes to metadata. It appears that an update via the status endpoint for a CR does not allow changes to metadata. Is my testing in error, or is this an intentional deviation in API behavior for CRDs?

[liggitt]

Some built-in resources support updating metadata via status subresources, and some do not. Allowing it was an oversight in early built-in resources. See #20978 and #45539

[liggitt]

closing this issue. GA features are covered by https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20180415-crds-to-ga.md and work in progress is tracked/categorized in https://github.com/orgs/kubernetes/projects/28