[DNM] Don't queue resource tasks; wait for availability

[gjoseph92]

Wait to schedule resource-consuming tasks until the resource is actually available. Currently, we oversaturate (queue tasks) on workers that have the resource, even if it's currently in use.

This is just a proof-of-concept and should not actually be merged—the logic for scheduling no-worker tasks in _remove_from_processing is way too inefficient. And obviously, we first need to discuss whether #6467 is intended behavior or not.

I wrote this just so @TomNicholas could try it out with #6360 and see how much it improves things.

Closes #6467

• Tests added / passed
• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

       15 files  ±0         15 suites  ±0   6h 37m 29s ⏱️ + 29m 29s
  2 854 tests +1    2 766 ✔️  -   4    82 💤 ±0    6 ❌ +  5 
21 145 runs  +8  20 174 ✔️  - 15  951 💤 +4  20 ❌ +19 

For more details on these failures, see this check.

Results for commit f05c94e. ± Comparison against base commit 3468601.

♻️ This comment has been updated with latest results.