[Distributed] make finalizer messages threadsafe

[vchuravy]

Partial revert of #41722

Fixes #42126

[Sacha0]

Tentatively marking backport-1.7, given the issue this patch targets has been observed on 1.7 previously.

[vtjnash]

From CI, this hangs

[vtjnash]

Looks like this would be vulnerable to a lock priority inversion deadlock situation

Addressed review

[tkf]

Would using a lock-free queue, stack, or "channel" for the messages help here? It sounds nice to not block and do minimal work inside a finalizer. (You can always schedule a task and so maybe this is not the issue/solution though.)

[vchuravy]

Would using a lock-free queue, stack, or "channel" for the messages help here? It sounds nice to not block and do minimal work inside a finalizer. (You can always schedule a task and so maybe this is not the issue/solution though.)

Yeah that would be a good optimization, but I don't think we have that in base right now?

But even with a lock-free queue, notifying the consumer is a problem.

[tkf]

For a queue with notification, we can use a lock-free unbounded "channel" (aka dual queue). It's possible to implement lock-free(-ish) push! even when popfirst! can wait. One catch is that push! needs to be able to run schedule ¹ but it's already usable in the finalizer. I think I can create a PR for adding the queue in this: Scherer & Scott (2004) Nonblocking Concurrent Data Structures with Condition Synchronization https://link.springer.com/chapter/10.1007/978-3-540-30186-8_13

Footnotes

1. It's also conceivable to use lock-free(-ish) queues inside the Julia runtime. It'd then make push! "more" lock-free. ↩

[vchuravy]

@KristofferC I would like to let this bake on master for a while before backporting. Should I remove the tag until a week or two from now?

[KristofferC]

Sounds good.

[iamed2]

Is this backportable to 1.6? We are experiencing #42126 (with the old error type) there.