unable to drain k8s node running Istio components with PDB

[jansmets]

Deployed istio 1.1.0 RC with helm chart (not template)

I would like to drain out a specific node because it needs a reboot.
The PODs

Annotations:        scheduler.alpha.kubernetes.io/critical-pod:

prevents that.

error when evicting pod "istio-policy-d48655744-s8ggt" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

(also note that annotation scheduler.alpha.kubernetes.io/critical-pod is deprecated and will be removed in k8s 1.14!)

[dwradcliffe]

I don't think this is related to the critical-pod annotation, but instead the PodDisruptionBudget.

You can view with kubectl describe pdb istio-policy -n istio-system.
I'm guessing you only have 1 pilot pod running, and the PDB is preventing the eviction because that would violate the budget.

Try running 2 replicas of pilot.

[linsun]

agreed, try scale out istio-policy first before drain the node as PDB requires 1 replica to be alive.

[linsun]

@jansmets is this still an issue?

[Lysholm]

The default helm installs are set up with 1 replica and a PDB for several deployments / HPAs. This seems like a case of a bad default config. PDBs are a feature for HA deployments, and HA deployments imply 2+ replicas.

The reason this is bad is that it breaks kubectl drain. In itself not good but automatic node-pool upgrades may also fail or take unacceptably long to apply. Node-pools will automatically upscale but then may fail to downscale as well.

Don't know if this is a feature or a bug but it's not a great default config for istio installs. I would either remove the PDBs or increase default replicas to 2.

Affected deployments (spec.replicas = 1): certmanager, istio-galley
Affected HPA (spec.minReplicas = 1): istio-egressgateway, istio-ingressgateway, istio-policy, istio-telemetry, istio-pilot

[kinihun]

Is there any plan to resolve this issue. It's still a blocker when using such a tool like kured. It results in a node being effectively locked out of action almost indefinitely.

[johscheuer]

We also ran into this issue and worked around it by setting the replicaCount for all components greater than 1.

Actually resolving this issue has some challenges from my which i'm not sure how to solve best:

We want to create a PodDisruptionBudget only in the following cases:

• replicaCount is set to 1 or greater
• When PodDisruptionBudget is activated obviously

but what if you don't set the replicaCount because you use the autoscaling feature with minReplicas: 1? Should the pdb created since there can be cases where more than 1 replica is available but if there is no load on your cluster you will have the same challenges as before.

Probably a quick win would be to allow setting the pdb behaviour on components e.g. you can globally activate pdb but deactivate if for specific components.

[linsun]

Think this is still a valid issue for the new installer/operator.

The resulting is we produce a production high availability profile that have multiple replicas and specifically targeted to resolve upgrade without down time prob. We don't want this to be a solution just for istio-policy as istio-policy is going away soon.

How does that sound @johscheuer @jansmets @dwradcliffe?

cc @howardjohn @sdake @ostromart

[Arnold1]

@dwradcliffe @linsun running in the same issue:

$ kops rolling-update cluster --yes
...
error when evicting pod "istio-policy-578bcb878f-2dmlw" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pod "istio-galley-84749d54b7-24qrp" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pod "istio-policy-578bcb878f-2dmlw" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pod "istio-galley-84749d54b7-24qrp" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

$ kubectl describe pdb istio-policy -n istio-system
Name:           istio-policy
Namespace:      istio-system
Min available:  1
Selector:       app=policy,istio=mixer,istio-mixer-type=policy,release=istio
Status:
    Allowed disruptions:  0
    Current:              1
    Desired:              1
    Total:                1
Events:                   <none>

what to do to fix it?

[vigohe]

@Arnold1 you need to add one more replica for istio-policy & istio-galley.

If you did install istio with helm you need to add the following lines to values.yaml

galley:
  enabled: true
  replicaCount: 2

mixer:
  policy:
    enabled: true
    replicaCount: 2

Then apply it:

helm upgrade --install istio istio.io/istio --namespace istio-system -f values.yaml

[colincoghill]

citadel has the same problem in istio 1.4, except you don't appear to be able to specify its replicaCount, that I can see.

[vigohe]

@colincoghill for citadel it's the security section.

values.yaml

security:
  enabled: true
  replicaCount: 2

[colincoghill]

aha, brilliant, that worked for me, thank you @vigohe

[Arnold1]

@colincoghill @vigohe can I use this values.yaml file for istio 1.4?

values.yaml:

galley:
    enabled: true
    replicaCount: 2

mixer:
  policy:
    enabled: true
    replicaCount: 2

security:
    enabled: true
    replicaCount: 2

@vigohe I get this error:

$ helm upgrade --install istio istio.io/istio --namespace istio-system -f values.yaml
Error: failed to download "istio.io/istio" (hint: running `helm repo update` may help)

[Arnold1]

what worked is:
cd istio-1.4.0
and than edit install/kubernetes/helm/istio/values.yaml:

  defaultPodDisruptionBudget:
    enabled: false

and than:

helm upgrade --install istio install/kubernetes/helm/istio --namespace istio-system -f install/kubernetes/helm/istio/values.yaml

is that the recomended way for istio 1.4?

[kovalyukm]

So, what is Istio recommended solution to correctly drain node with any istio component?

Istio 1.4.2 installed with istioctl has these values for prod (default) profile:

k get pdb -n istio-system
NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE
certmanager 1 N/A 0 12d
ingressgateway 1 N/A 0 12d
istio-citadel 1 N/A 0 12d
istio-galley 1 N/A 0 12d
istio-pilot 1 N/A 0 12d
istio-policy 1 N/A 0 12d
istio-sidecar-injector 1 N/A 0 12d
istio-telemetry 1 N/A 0 12d

ALLOWED DISRUPTIONS = 0 for all components.

Thanks!

[sazzle2611]

Hi all, I have this same issue only it is on a kubernetes cluster (v 17.2) with only 2 nodes, one master and one worker, meaning as pods are only scheduled on the worker then we can't get around this problem by autoscaling.

He don't need HA but it would be nice to be able to drain the node properly, is there any way to disable the pdb (and hpa) for the ingress gateways and istiod?

We are using istio 1.6 installed with istioctl and this is the current ingress gateway config but it still creates pdb

components:
ingressGateways:
- name: istio-ingressgateway
enabled: true
k8s:
replicaCount: 1
hpaSpec:
minReplicas: 1
maxReplicas: 1

If I remove the hpaSec then it sets a hpa with max replicas 5 which is definitely not what we want.

Thanks,
Sarah

[davidkarlsen]

This can be avoided with the following spec:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-controlplane
  namespace: istio-system
spec:
  addonComponents:
    kiali:
      enabled: true
    prometheus:
      enabled: false
    tracing:
      enabled: true
  components:
    cni:
      enabled: true
    ingressGateways:
    - enabled: true
      k8s:
        hpaSpec:
          minReplicas: 2
      name: istio-ingressgateway
    pilot:
      enabled: true
      k8s:
        hpaSpec:
          minReplicas: 2
  meshConfig:
    enablePrometheusMerge: true
  profile: default
  values:
    cni:
      excludeNamespaces:
      - istio-system
      - kube-system
      logLevel: info
    kiali:
      dashboard:
        grafanaURL: http:https://monitoring-grafana.monitoring.svc
      prometheusAddr: http:https://monitoring-prometheus-server.monitoring.svc

which should probablt be default (the hpa values)

[bianpengyuan]

Related: #24000

[Vaccano]

This can be avoided with the following spec:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-controlplane
  namespace: istio-system
spec:
  addonComponents:
    kiali:
      enabled: true
    prometheus:
      enabled: false
    tracing:
      enabled: true
  components:
    cni:
      enabled: true
    ingressGateways:
    - enabled: true
      k8s:
        hpaSpec:
          minReplicas: 2
      name: istio-ingressgateway
    pilot:
      enabled: true
      k8s:
        hpaSpec:
          minReplicas: 2
  meshConfig:
    enablePrometheusMerge: true
  profile: default
  values:
    cni:
      excludeNamespaces:
      - istio-system
      - kube-system
      logLevel: info
    kiali:
      dashboard:
        grafanaURL: http:https://monitoring-grafana.monitoring.svc
      prometheusAddr: http:https://monitoring-prometheus-server.monitoring.svc

which should probablt be default (the hpa values)

It is important to note that this solution only works if you installed Istio using the Istio Operator. (At least that was my experience.)

[Kostanos]

Just scaling istiod deployment to 2 and after scaling it back to 1 worked for me.
Meanwhile the draining node is in cordoned state.
I'm using lens for it, but here is a kubectl:

kubectl cordon <your node to drain>
kubectl scale --replicas=2 deployemnt/istiod
# Make sure it was scaled, when you scale it back, it will remove pod from cordoned node
kubectl scale --replicas=1 deployment/istiod
kubectl drain <your node to drain>

PS. in my case, istiod was in separate node from other istio related services, and it worked.

[dozer75]

@Kostanos suggestion helped me during the AKS outage yesterday due to the Ubuntu 18.04 issues to heal the cluster. In addition I had to run the same command on istio-ingressgateway as well.

[des1redState]

You can save yourself a command @Kostanos. Once you've cordoned your Node, use the kubectl rollout restart deployment/... command instead of manually scaling, which will essentially do the same thing (scale up to 2 Pods, wait for the 2nd to become Ready, then remove the initial Pod).