-
Notifications
You must be signed in to change notification settings - Fork 7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CRDs not being applied correctly #2994
Comments
Look at prometheus-operator, helm is not waiting for third party resource to become fully registered, therefore one off install hook is necessary |
In this chart the services aren't creating the CRDs. The chart itself creates the CRDs. I've used hooks to order the creation but Helm is having trouble applying them to Kubernetes |
Is this a cascading failure from #2995? Let's try the proposed fix in that issue first and see if that fixes things. |
This didn't help. I believe there's some funniness when using CRDs/TPRs with hooks. #2680 |
Related: #2925 |
As a note, we are going to pull the ordering part of this into 2.7 |
Pulling this out of 2.7 in favor of #2925 |
Issues go stale after 90d of inactivity. Prevent issues from auto-closing with an If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or |
/lifecycle frozen |
Whats the status of this? Is ordered support for CRDs in the roadmap for a future release or being worked on? |
This hasn't been slated into a release and nobody's working on this AFAIK. The root issue here is that we need some way to figure out how to install CRDs before beginning to validate the rest of the chart. It's a substantial change to the existing workflow, but it is possible to accomplish this in Helm 2 if someone wants to tackle the issue. |
@bacongobbler thanks for the reply! I'm not sure I'm necessarily the right person to spearhead the effort, but I love helm and would love to help if possible. I think given the direction of the community right now and all the talk at Kubecon, this seems like something that could be a huge win for the tool. Do you guys have community/SIG meetings on zoom? I can try and drop by the next one and learn more and see how/where I can be of help. |
TPR/CRDs are a particular issue that often necessitates some ordering control. But there are other ordering issues for umbrella charts with many sub-charts. It would be nice to have a simple and general ordering mechanism, rather than a special behavior for TPR/CRD. As identified here, it is critical to consider when validation occurs. One possible approach would to allow weights to be applied to subcharts in Deletions would follow the same grouping in reverse, deleting resources for equally weighted groups of parent and/or sub-charts, from highest weight to lowest weight. With this optional feature the processing of all existing charts (with no weights) would be unaffected - for both deployment order, validation behavior, and deletion. Only people who really need ordering would specify any weights for This mechanism would enable TPR/CRDs to be applied first, simply by including them in a lower weight chart (parent or sub-chart) than the charts (parent and/or sub-charts) that rely on those TPR/CRDs. Considerations:
|
I think issue is just with helm validation. Instead of introduce weighs for requirements, is possible just to validate on installation level with communication with tiller instead of a pre installation step? At least just for CRD. With the introduction of operator in kubernetes, CRD are installed by the operator itself and not by the helm chart, which make still no possible to use the requirements weighs you suggest. Is something like a registration step possible? That you can defined a job or something that validates the CRD installation using helm hooks. In my opinion the approach should be as stated in the helm documentation
Which can be done by installing the CRD as pre-install hook. With his the ordered is correct, and you can use hook weights for a more controlled ordering. Issue is still with helm validation, which will fail if it detects a custom resource that depends on a CRD that is being installed in the same install (even if the order is right, so CRD will be installed before the customer resource). |
and certmanager |
I get same thing as @fiunchinho but for heptio-ark when trying to upgrade/install |
I am using helm in an OpenShift setup, and trying to manage everything using helm hitting ordering issues very hard... In OpenShift there is a CRD called Project, which is a complete replacement of Namespace (it then creates a "virtual" Namespace object Read-Only), thus when I try creating a Project with Quotas, and other official objects from k8s the ordering is messed up. I think a good way to let users control ordering with CRDs, would be to, instead of having a static ordering like defined here https://github.com/helm/helm/blob/release-2.13/pkg/tiller/kind_sorter.go#L29 use an integer priority to each object and allow users to set a priority for Kinds in metadata of the chart or something. |
@dabelenda I've suggested elsewhere that subcharts in
|
I am "only" creating the Project, Quotas, and RoleBindings here. I think of Prometheus-Operator Prometheus ServiceMonitoring etc |
I'm using helm3 and experience the same problem sporadically, this time with istio resources:
It works occasionally, but most often it just throws me that error. It doesn't happen always. |
Error
Versions
|
Hey @Ciantic and @JulienBreux so this is because your local types/discovery cache is stale after you install a CRD. We have handled this with the new |
@thomastaylor312 it doesn't work. I guess because the main helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.3.0/charts/
helm repo update
kubectl create namespace istio-system
# Install crds
helm install istio-init istio.io/istio-init --namespace istio-system
# See https://github.com/helm/helm/issues/2994 for the deleting cache
rm -rf ~/.kube/cache/discovery/
sleep 3s # This really is installation step!
# Install istio
helm install istio istio.io/istio \
--namespace istio-system \
--set gateways.istio-ingressgateway.type=NodePort Throws error:
Only thing that works 100% time is normal I want to note that above script works sometimes but usually not. So something funny and non-deterministic there is. |
Probably because support for the crd-install hook was dropped in Helm 3 in favour of a |
You are right. Apparently istio can't be installed with helm3, and probably never will be, because they are working on a different installer. See istio/istio#17167 |
closing as resolved with the |
As hinted in [1], it seems to be not possible (after multiple trials) to manage CRDs using Helm 3's crd-install hook using the crds/ folder [2]. It seems like the only way would be to bundle crds inside the templates/ like cert-manager has been doing for a while. About bundling CRDs inside templates/, Rob Percival mentions in [1] that Helm has a problem with the CRD ordering [3] and that the issue has not been fixed yet, which means installing operators like google-cas-issuer breaks when the CRDs are inside templates/. [1]: GoogleCloudPlatform/marketplace-k8s-app-tools#303 [2]: https://helm.sh/docs/developing_charts/#defining-a-crd-with-the-crd-install-hook [3]: helm/helm#2994 Signed-off-by: Maël Valais <[email protected]>
As detailed in [1], CRDs that are applied through the CRD pre-install hook will not ever be updated or upgraded. The Helm documentation reads [4]: > The resources that a hook creates are not tracked or managed as part of > the release. Once Tiller verifies that the hook has reached its ready > state, it will leave the hook resource alone. > > Practically speaking, this means that if you create resources in a hook, > you cannot rely upon helm delete to remove the resources. To destroy such > resources, you need to either write code to perform this operation in a > pre-delete or post-delete hook or add "helm.sh/hook-delete-policy" > annotation to the hook template file. On top of that, it seems to be not possible (after multiple trials) to manage CRDs using Helm 3's crd-install hook using the crds/ folder [2]. It seems like the only way would be to bundle crds inside the templates/ like cert-manager has been doing for a while. Note that installing CRDs using the templates/ way also causes trouble. Rob Percival mentions in [1] that Helm has a problem with the CRD ordering [3] and that the issue has not been fixed yet, which means installing operators like google-cas-issuer breaks when the CRDs are inside templates/. [1]: GoogleCloudPlatform/marketplace-k8s-app-tools#303 [2]: https://helm.sh/docs/developing_charts/#defining-a-crd-with-the-crd-install-hook [3]: helm/helm#2994 [4]: https://v2.helm.sh/docs/developing_charts/#hook-resources-are-not-managed-with-corresponding-releases Signed-off-by: Maël Valais <[email protected]>
I'm seeing the following when installing the chart in this PR helm/charts#2369
If I render the files with CRDs manually using
helm template
and install usingkubectl create -f
they work just fine.The text was updated successfully, but these errors were encountered: