Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ Taint nodes with PreferNoSchedule during rollouts #10223

Merged
merged 1 commit into from
Mar 15, 2024
Merged

✨ Taint nodes with PreferNoSchedule during rollouts #10223

merged 1 commit into from
Mar 15, 2024

Conversation

chrischdi
Copy link
Member

@chrischdi chrischdi commented Mar 1, 2024 •
edited by sbueringer
Loading

What this PR does / why we need it:

This PR builds on top of the amazing work of @hiromu-a5a and finishes this feature. Thanks to @hiromu-a5a for starting the initially work at:

If the MachineSet or MachineDeployment has many replicas, and each node has many pods, changes to an existing MachineDeployment or MachineSet infrastructure template can result in unnecessary pod churn. As the first node is drained, pods previously running on that node may be scheduled onto nodes who have yet to be replaced, but will be torn down soon. This can happen over and over again.

To avoid the above problem, this PR changes the machine controller to add a corev1.TaintEffectPreferNoSchedule taint to nodes in old MachineSet. As mentioned in #7043 (comment), tainting should be triggered by Machine controller.

In the Machine's reconcileNode phase, the machine get's checked if it should have the outdated taint by checking if it is part of an machinedeployment's machineset which is in a older revision than the machine deployment.
If so, the taint gets ensured on the node object.

This behavior may be related to #493

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #7043

/area machinedeployment

@chrischdi chrischdi added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Mar 1, 2024
@k8s-ci-robot k8s-ci-robot added area/machinedeployment Issues or PRs related to machinedeployments cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 1, 2024
@k8s-ci-robot k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Mar 1, 2024
@chrischdi chrischdi mentioned this pull request Mar 1, 2024
Closed
@chrischdi
Copy link
Member Author

Note: this should address all review comments at the old PR at:

@chrischdi
Copy link
Member Author

/assign fabriziopandini

:-)

vincepri
vincepri approved these changes Mar 4, 2024
View reviewed changes
Copy link
Member

@vincepri vincepri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 4, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: fe9e6163ba3a9389e9364729393af3815366a1b4

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 5, 2024
@fabriziopandini
Copy link
Member

/lgtm
cc @sbueringer for a final pass

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 12, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: 8d7ac3394a1fc0a645111e8e20d4d0912dea60ef

@sbueringer
Copy link
Member

@chrischdi I think the PR description does not reflect the current implementation (the second paragraph)

@chrischdi
Copy link
Member Author

@chrischdi I think the PR description does not reflect the current implementation (the second paragraph)

updated it.

sbueringer
sbueringer reviewed Mar 12, 2024
View reviewed changes
Copy link
Member

@sbueringer sbueringer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few nits otherwise lgtm

util/util.go Show resolved Hide resolved
internal/controllers/machine/machine_controller_noderef.go Outdated Show resolved Hide resolved
internal/controllers/machine/machine_controller_noderef.go Outdated Show resolved Hide resolved
@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 13, 2024
@sbueringer
Copy link
Member

Thx!

Feel free to squash

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 13, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: cb0bee687d69fa53d290690c2ea8c4a353db1166

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 13, 2024
@sbueringer
Copy link
Member

If possible, please just squash instead of squash+rebase. Then the lgtm label stays there and nobody has to verify if there are rebase errors :)

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 13, 2024
@k8s-ci-robot
Copy link
Contributor

LGTM label has been added.

Git tree hash: e2adf1c8a8edfa78dbc079b1039ab606982771d4

@fabriziopandini
Copy link
Member

/lgtm

@sbueringer
Copy link
Member

/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 15, 2024
@k8s-ci-robot k8s-ci-robot merged commit 15615f6 into kubernetes-sigs:main Mar 15, 2024
20 checks passed
@k8s-ci-robot k8s-ci-robot added this to the v1.7 milestone Mar 15, 2024
@sbueringer sbueringer mentioned this pull request Apr 16, 2024
Merged
Dhairya-Arora01 pushed a commit to Dhairya-Arora01/cluster-api that referenced this pull request May 25, 2024
@chrischdi @hiromu-a5a @Dhairya-Arora01
Add Taint during rolling update (kubernetes-sigs#10223)
466ee9b 
Co-authored-by: Hiromu Asahina <[email protected]>
Dhairya-Arora01 pushed a commit to Dhairya-Arora01/cluster-api that referenced this pull request May 25, 2024
@chrischdi @hiromu-a5a @Dhairya-Arora01
Add Taint during rolling update (kubernetes-sigs#10223)
a9d239d 
Co-authored-by: Hiromu Asahina <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@killianmuldoon killianmuldoon Awaiting requested review from killianmuldoon

@fabriziopandini fabriziopandini Awaiting requested review from fabriziopandini

@vincepri vincepri Awaiting requested review from vincepri

@sbueringer sbueringer Awaiting requested review from sbueringer

Assignees

@vincepri vincepri

@sbueringer sbueringer

@fabriziopandini fabriziopandini

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/machinedeployment Issues or PRs related to machinedeployments cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.
Projects
None yet
Milestone
v1.7
Development

Successfully merging this pull request may close these issues.

CAPI controller should taint outdated nodes with PreferNoSchedule
6 participants
@chrischdi @k8s-ci-robot @fabriziopandini @sbueringer @vincepri @hiromu-a5a