fix: terminate workflow should not get throttled Fixes #12778

[tczhao]

Fixes #12778

Motivation

To prevent cronworkflow from queueing up pending workflows when

• reached parallelism limit
• concurrenpolicy-replace already update the spec of the pending workflow to shutdown: terminated

Modifications

Allow workflow in the queue to be processed (operated) regardless of parallelism limit.

We have scenarios

• reach parallelism limit, terminate running workflow (no issue)
• reach parallelism limit, terminate pending workflow (we should not block the queue from processing these workflow, just like semaphore doesn't block the queue from processing pending workflow that got terminated

  argo-workflows/workflow/controller/operator.go

  Line 528 in 5b3909b

  func (woc *wfOperationCtx) releaseLocksForPendingShuttingdownWfs(ctx context.Context) bool {

  )

Verification

Add test,
test fail
Add code change
test succeed

[agilgur5]

• concurrenpolicy-replace already update the spec of the pending workflow to shutdown: terminated

oh interesting, I didn't know that. that does make the fix much simpler.

• reach parallelism limit, terminate running workflow (no issue)
• reach parallelism limit, terminate pending workflow (we should not block the queue from processing these workflow, just like semaphore doesn't block the queue from processing pending workflow that got terminated

this makes sense to me and sounds consistent, so I'll approve, but I'm not an expert in the synchronization area of the codebase so would appreciate if someone else could take a look. EDIT: asked on Slack for a second pair of eyes.
in particular, technically speaking this change means that terminating Workflows are no longer counted toward parallelism limits, right?

[agilgur5]

Also if you're interested, another synchronization race condition issue is #11984

[tczhao]

technically speaking this change means that terminating Workflows are no longer counted toward parallelism limits, right?

Correct,
We either

• terminate running workflow (in parallelism limit, release it from parallelism)
• terminate pending/unknown workflow (bypass parallelism)
• terminate finished workflow (no effect)

[sarabala1979]

LGTM

[agilgur5]

Backported cleanly to release-3.5 as c9dd50d