Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster autoscaler scales up too much #43709

Closed
mwielgus opened this issue Mar 27, 2017 · 4 comments · Fixed by #43745
Closed

Cluster autoscaler scales up too much #43709

mwielgus opened this issue Mar 27, 2017 · 4 comments · Fixed by #43745
Assignees
@mwielgus
Labels
sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling.
Milestone
v1.6

Comments

@mwielgus
Copy link
Contributor

On some occasions Cluster Autoscaler may scale up the cluster two times. This results in growing the cluster too large. The size gets back to normal soon but the user will be forced to pay for the unneeded node for 10min/1h, depending on the cloud provider.

cc: @MaciekPytel @fgrzadkowski @ethernetdan @enisoc
@mwielgus mwielgus added priority/P1 sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling. labels Mar 27, 2017
@mwielgus mwielgus added this to the v1.6 milestone Mar 27, 2017
@mwielgus mwielgus self-assigned this Mar 27, 2017
@mwielgus mwielgus changed the title Cluster autoscaler scales to far Cluster autoscaler scales up too much Mar 27, 2017
@calebamiles calebamiles modified the milestones: v1.6.1, v1.6 Mar 27, 2017
@bgrant0607
Copy link
Member

@mwielgus Please provide more information.

@bgrant0607 bgrant0607 added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Mar 27, 2017
@mwielgus
Copy link
Contributor Author

@bgrant0607
Imagine that you have just created a pod that cannot be scheduled. CA expands one of the migs to accommodate it. Just when the node arrives there is a brief moment when the pod is still pending (scheduler didn't retry it yet) but the node is available. Scale up is over. Pod is pending. So we might try to scale up again, which would be incorrect. To protect against such situations we run a simulated scheduling on all nodes. If any node can accommodate any of the pending pods then the pod is not pending and should not trigger a scale up.

After a scale up we create "virtual" nodes inside CA so that we don't scale up again for the same pods. Once the node arrives its virtual placeholder is deleted.

And, if I'm correct, we have a bug here. A node with node condition ready=true but with non ready network is considered as fully started but broken. Its virtual placeholder is gone so we trigger scale up again. Seconds later the network arrives and the scale up appears to be non-needed.

I'm testing right now a 10 line fix for this issue. After the fix is merged I would like to bump CA version in 1.6 as having the old code would impact all of the current CA users on GCE and GKE (and possibly on other cloud providers too).

@mwielgus mwielgus modified the milestones: v1.6, v1.6.1 Mar 27, 2017
@liggitt
Copy link
Member

liggitt commented Mar 27, 2017

is this a regression in cluster autoscaler for 1.6, or did the same behavior exist in 1.5?

@mwielgus
Copy link
Contributor Author

It is a regression. Doesn't happen always (depends on the network setup timings) but when it does it is quite confusing.

@mwielgus mwielgus mentioned this issue Mar 27, 2017
Merged
k8s-github-robot pushed a commit to kubernetes-retired/contrib that referenced this issue Mar 28, 2017
Merge pull request #2502 from mwielgus/ca-starting-fix
a74a141 
Automatic merge from submit-queue

Cluster-autoscaler: Fix isNodeStarting

Fix for: kubernetes/kubernetes#43709

cc: @MaciekPytel @fgrzadkowski
@mwielgus mwielgus mentioned this issue Mar 28, 2017
Merged
@mwielgus mwielgus removed the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Mar 28, 2017
k8s-github-robot pushed a commit that referenced this issue Mar 28, 2017
Merge pull request #43745 from mwielgus/ca-0.5.1
075dc27 
Automatic merge from submit-queue

Bump cluster autoscaler to 0.5.1

Fixes: #43709

**Release note**:
```release-note
With Cluster Autoscaler 0.5 the cluster will be autoscaled even if there are some unready or broken nodes. Moreover the status of CA is exposed in kube-system/cluster-autoscaler-status config map.
```
mwielgus pushed a commit to kubernetes/autoscaler that referenced this issue Apr 18, 2017
Merge pull request kubernetes-retired/contrib#2502 from mwielgus/ca-s…
7bfd399 
…tarting-fix

Automatic merge from submit-queue

Cluster-autoscaler: Fix isNodeStarting

Fix for: kubernetes/kubernetes#43709

cc: @MaciekPytel @fgrzadkowski
mwielgus pushed a commit to kubernetes/autoscaler that referenced this issue Apr 18, 2017
Merge pull request kubernetes-retired/contrib#2502 from mwielgus/ca-s…
6c991c8 
…tarting-fix

Automatic merge from submit-queue

Cluster-autoscaler: Fix isNodeStarting

Fix for: kubernetes/kubernetes#43709

cc: @MaciekPytel @fgrzadkowski
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mwielgus mwielgus

Labels
sig/autoscaling Categorizes an issue or PR as relevant to SIG Autoscaling.
Projects
None yet
Milestone
v1.6
Development

Successfully merging a pull request may close this issue.

Bump cluster autoscaler to 0.5.1 mwielgus/kubernetes
4 participants
@calebamiles @liggitt @bgrant0607 @mwielgus