Expand UnschedulablePlugins/PendingPlugins to include PreBind plugins

[sanposhiho]

Currently, the scheduler honors Unschedulable(AndUnresolvable) or Pending only from PreFilter, Filter, Reserve, and Permit (WaitOnPermit).
https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/framework/types.go#L210-L214

It doesn't expect other extension points to return those statuses and just regard all non-success status as errors, that is, immediately retrying scheduling of Pods.

Actually DRA returns Pending at PreBind, which is currently being ignored as described.
So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?)

/cc @pohly
/sig scheduling
/kind feature

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[pohly]

So, we should expand UnschedulablePlugins/PendingPlugins to include PreBind plugins. (or maybe all extension points too?)

Both seems reasonable to me.

[pohly]

Because Pending is not handled for PreBind, the following error is logged:

E0604 15:45:50.980929  306340 schedule_one.go:1048] "Error scheduling pod; retrying" err="waiting for resource driver" pod="test/test-draqld28"

[pohly]

This patch gets rid of the error. I am uncertain whether it has other implications? Performance was the same with and without it.

diff --git a/pkg/scheduler/schedule_one.go b/pkg/scheduler/schedule_one.go
index 6b6992095d8..8c7868fedec 100644
--- a/pkg/scheduler/schedule_one.go
+++ b/pkg/scheduler/schedule_one.go
@@ -292,6 +292,17 @@ func (sched *Scheduler) bindingCycle(
 
        // Run "prebind" plugins.
        if status := fwk.RunPreBindPlugins(ctx, state, assumedPod, scheduleResult.SuggestedHost); !status.IsSuccess() {
+               if status.IsRejected() {
+                       fitErr := &framework.FitError{
+                               NumAllNodes: 1,
+                               Pod:         assumedPodInfo.Pod,
+                               Diagnosis: framework.Diagnosis{
+                                       NodeToStatusMap:      framework.NodeToStatusMap{scheduleResult.SuggestedHost: status},
+                                       UnschedulablePlugins: sets.New(status.Plugin()),
+                               },
+                       }
+                       return framework.NewStatus(status.Code()).WithError(fitErr)
+               }
                return status
        }
 

[sanposhiho]

What I imagined is like that. It shouldn't have any other impact because volumebinding, which is the only PreBind plugin apart from DRA in in-tree, only returns success or error status.
https://github.com/kubernetes/kubernetes/blob/master/pkg/scheduler/framework/plugins/volumebinding/volume_binding.go#L355-L378

We have to mention the change in the release note for custom plugin developers though.

[pohly]

I've turned this into a PR: #125360