📢 Kubernetes API removals on 1.25/1.26 and Openshift 4.12/4.13 might impact your Operator. How to deal with it? #1182
Replies: 1 comment
-
|
Hi All, If you are trying to check if your solution is or not using APIs that will be removed in the next release using the OCP Alerts then, we would like just to let you aware that: The alerts are fired only one hour later, the deprecated API be used on the cluster, and they do not provide any info to let us know who made the request. (before the alert is fired, we can check them in a Pending state). Then, we must use command-line commands with Therefore, we recommend you use the command line and not the UI, which seems the best approach to avoid any misleading when trying to ensure/QA your solution to verify if it is or not using APIs that will be removed in the next release. |
Beta Was this translation helpful? Give feedback.
-
Some APIs versions are deprecated and will no longer be served on the Kubernetes version
1.25/1.26and consequently on Openshift4.12/4.13. For further information, check the guide.4.11+(at least) that don’t use the Kubernetes API versions removed in Openshift4.12/k8s 1.25. Remember that cluster admins MUST have installed a version of your Operator that can work successfully in Openshift4.12+before upgrading the OCP cluster from4.11to4.12. Otherwise, they might face workflow issues1.22and Openshift4.9. Resources using these APIs are not provided via the bundle manifests, so we cannot check them with SDK commands such asoperator-sdk bundle validate ./bundle --select-optional suite=operatorframework --optional-values=k8s-version=1.254.11( K8s1.23+) to check if your versions are or not using these APIs. In this case, you need to perform a full regression test and use the APIRequestCount( such as it is suggested to cluster admins verify if they can safely upgrade their cluster to Openshift4.9with the current solutions installed on it ). You can also check warnings in the Operator logs. (Please, ensure that you check the section below:How can I find the usage of the deprecated APIsto know more about it.)🙆 When the errors will appear
It will be unlikely to find problems by only installing the Operator bundle on the cluster. These APIs are commonly used at runtime (e.g. reconciliations made when the user creates CRs to provision the Operand).
Example:
An Operator reconciles a Kind/API to create a CronJob using
batch/v1beta1or raises an Event usingevents.k8s.io/v1beta1. Then, these operations will not work from Kubernetes1.25+and Openshift4.12+. The API versions will not be found on the cluster, and you will probably check out errors in Operator logs (api not found).💁♀️ What does it mean for you?
You must ensure that you no longer use these API versions. Otherwise, any operation of your project which requires these versions will fail on Kubernetes versions
1.25/1.26+and Openshift4.12/4.13+.Ideally, to avoid workflow issues and ensure the best user experience for OCP/OLM users you should:
For all Operator versions published on OCP 4.11 (at least):
OCP 4.12/k8s 1.25)OR
olm.maxOpenShiftVersionset with the value4.11to block cluster upgrades (from OCP4.11to4.12) when versions of our Operators that you are knowledgeable that will not work on OCP4.12+are installed. In this way, avoid workflow issues. (You will find a better explanation of it in the below sectionSuppose I found the usage of these APIs for my next publication. What should I do?)Therefore, for OCP 4.12 release: you must ensure that the latest version ( head of channel(s) ) published on OCP
4.11(at least) works onOpenshift 4.12+.Otherwise, with your Operator installed, cluster admins cannot upgrade their cluster from OCP4.11to4.12.🙋 What are the APIs? How can I upgrade my Operator?
You can find all apis/versions removed on
1.25/1.26on the guide guide as the required steps to upgrade them. However, the following is a summary:APIs removed from k8s
1.25/ Openshift4.12:v1instead ofv1beta1.v1has been available since k8s v1.21/Openshift 4.8. No notable changesv1instead ofv1beta1.v1has been available since k8s v1.21/Openshift 4.8. (See the guide to check the notable changes)v1instead ofv1beta1.v1has been available since k8s v1.19/Openshift 4.6. (See the guide to check the notable changes)v2instead ofv2beta1.v2has been available since k8s v1.23/Openshift 4.10. (No notable changes)v1instead ofv1beta1.v1has been available since k8s v1.21/Openshift 4.8. (See the guide to check the notable changes.)v1instead ofv1beta1.v1has been available since k8s v1.20/Openshift 4.7. (No notable changes)APis removed from k8s
1.26Openshift4.13:v2instead ofv2beta2.v2has been available since k8s v1.23/Openshift 4.10(No notable changes)v1beta2instead ofv1beta1.v1beta2has been available since k8s v1.23/Openshift 4.10(No notable changes)v1beta2instead ofv1beta1.v1beta2has been available since k8s v1.23/Openshift 4.10(No notable changes)🙋♀️ How can I find the usage of the deprecated APIs:
1. Inspect your Operator source code. Look for manifests and imports using the removed versions, e.g.:
2. By running your Operator on the cluster, you can check that the Kubernetes API will also try to warn you when a workflow that uses these API versions is executed, e.g.:
0.12.0and k8s1.23+. Please, do the tests against Kubernetes API1.23+. Ideally, Openshift4.11+to ensure that the warnings will be fired.3. Test all Operator workflows on the cluster and check if it's calling these deprecated APIs (which will be removed).
Be aware that you need to test all workflows of your Operator ( apply all custom resources so that you can try to trigger all possible reconciliations and code paths ). The Openshift API APIRequestCount or Kubernetes metrics can only count the APIs which were effectively called and hit on the cluster.
💡 Note that it is not easy to identify if your Operator is doing the calls. Other components running on the cluster can also have called them. In this way, we recommend that you; initialise a new cluster for these tests. Then, before beginning to test your Operator verify the results so that you can compare the calls to these APIs before your Operator and workflows are executed with its result at the end.
Testing the Operator on Openshift 4.11
You will need to use the command
$ oc get apirequestcounts. To check all steps and options with examples, see the 4.2. Evaluating your cluster for removed APIs provided to cluster admins know how the could check if the solutions installed were using the removed APIs onk8s 1.22/4.9before they upgrade the cluster. Also, you will see in the 4.2.3. Using APIRequestCount to identify which workloads are using the removed APIs, that is possible to identify who made the call by looking at theUSERNAMEandUSERAGENT.Example
v1beta1:Run
oc get apirequestcounts -o jsonpath='{range .items[?(@.status.removedInRelease!="")]}{.status.removedInRelease}{"\t"}{.metadata.name}{"\n"}{end}'Then, you will check:
v1beta1:v1beta1:Above, you can see that the username is
kube:adminbecause the test was made by running the Operator from an local development environment ( using$ make runmakefile target provided in the projects built using Operator-SDK/Kubebuilder )IMPORTANT NOTE: You can safely ignore the entries that appear in the results from system:serviceaccount:openshift and from the following usernames if you check them:
Testing the Operator on a Kubernetes cluster ( i.e. Kind )
The APIRequestCount is an Openshift API, so you will not find this option on Kubernetes clusters. Howsoever, you can use Kubernetes metrics to check if your Operator calls the deprecated APIs (More info).
Example
1.25:Then, we checked as a result:
{ "labels": { "code": "200", "component": "apiserver", "dry_run": "", "group": "policy", "resource": "podsecuritypolicies", "scope": "cluster", "subresource": "", "verb": "LIST", "version": "v1beta1" }, "value": "1" }{ "labels": { "code": "200", "component": "apiserver", "dry_run": "", "group": "batch", "resource": "cronjobs", "scope": "cluster", "subresource": "", "verb": "LIST", "version": "v1beta1" }, "value": "1" } { "labels": { "code": "200", "component": "apiserver", "dry_run": "", "group": "policy", "resource": "podsecuritypolicies", "scope": "cluster", "subresource": "", "verb": "LIST", "version": "v1beta1" }, "value": "1" } { "labels": { "code": "201", "component": "apiserver", "dry_run": "", "group": "batch", "resource": "cronjobs", "scope": "resource", "subresource": "", "verb": "POST", "version": "v1beta1" }, "value": "1" }💁 Suppose I found the usage of these APIs for my next publication. What should I do?
If you see that your Operator version uses these Kubernetes API versions that will be removed on Openshift
4.12/4.13(K8s1.25/1.26), then the recommendation would be to upgrade the APIs. However, suppose you cannot fix it for the next publication, OR this publication must be done for Openshift versions where the new API version is not supported.In this case, the recommendation is to use the
olm.maxOpenShiftVersionannotation so that OCP/OLM users with these versions installed on4.11cannot upgrade their cluster before upgrading the Operator version installed. Cluster admins will need to upgrade their Operator versions installed to versions that can work on OCP4.12+before upgrading their cluster. Also, it would be recommended to ensure that your solution is not published on the OCP4.12+by defining the valid OCP range via the annotationcom.redhat.openshift.versionsinto themetadata/annotations.yaml.Example:
I verified that my new Operator bundle version will be released and is using CronJob
v1beta1. Also, I will not fix that for this release. So, I should use theolm.maxOpenShiftVersionannotation not to let OCP/OLM users upgrade their clusters to Openshift4.12if this version is installed:Then, in this case, I also want to ensure that this solution is added to the Openshift catalog from
4.6. Therefore, I will have the OCP label with the rangev4.6-v4.11in themetadata/annotations.yaml:ℹ️ For further information see the documentation.
💁 Suppose I checked that what is published so far can NOT successfully work on Openshift
4.12+. What should I do?If you have Operator solutions published on this repo it means that these Operators are distributed via its index catalog source (
registry.redhat.io/redhat/community-operator-index) then, you can update what was merged yet. ( To understand how the Openshift catalog is built and its sources, see here )In this case, you can update the previous versions by pushing a new pull request to update the CSV with the
olm.maxOpenShiftVersionannotation to not let OCP/OLM users upgrade their clusters from previous OCP versions to Openshift4.12as described in the above example. Also, if your solutions are distributed using the bundle format, you can use the labelcom.redhat.openshift.versionsin themanifest/annotations.yamlto provide a valid range to not allow these Operator(s) versions end up OCP4.12+. ( i.e.com.redhat.openshift.versions: v4.6-v4.11).com.redhat.openshift.versions, ' your Operator versions distributed via this repository will be added to all available tags for the index catalogregistry.redhat.io/redhat/community-operator-index. The image tags represent the Openshift versions. (i.e. _tagv4.10means that this image is the source available on Openshift 4.10_ ). In this way, if you are still using the previous format (PackageManifest`), you should upgrade your solutions to bundle format to use this feature. See here how can you achieve it.💡 You can use OPM to check the content available in the index catalogues:
You can use the OPM tool and the command
$ opm alpha list bundles registry.redhat.io/redhat/community-operator-index:v<OCP release> <packageName>to check all bundles available for your package on OCP4.11, e.g.:Beta Was this translation helpful? Give feedback.
All reactions