Task scheduling can result in delays when synchronizing

[maleadt]

See https://discourse.julialang.org/t/occasional-long-delays-in-cuda-jl/81545

[physicsjoke]

I think I might have a similar problem. While testing some code that we ported from CPU, including a few kernels to draw random numbers and some array operations, Julia reproducibly gets stuck, and needs to be killed/interrupted using Ctrl+C.
Nsys shows that the computations are done within seconds, the kernel execution times being very short. I'd be glad to provide mor info as needed...

I tried this with and without the nonblocking_synchronize line commented out, as suggested in that thread. the behaviour is the same.
It occurs independently using either the Pkg.test mechanism or including the "runtests.jl" file we made for the project

[maleadt]

I tried this with and without the nonblocking_synchronize line commented out, as suggested in that thread. the behaviour is the same.

Then this is a different issue. Julia should always respond to CRTL-C, especially in the case of this issue. If it doesn't, attach GDB and see where it's stuck.

[physicsjoke]

It does respond to Ctrl-C, but the test should finish, right?

[maleadt]

Ah I misread. It cleanly reports a shutdown, right? Because if you mash CTRL-C you can get Julia killed uncleanly.

What I'm looking for, is whether there's a task that just doesn't finish (in which case a CTRL-C should be cleanly intercepted, and result in an InterruptException), or whether a CUDA API call blocks (in which case a CTRL-C will likely result in a fatal error, like a segfault). In case of the former, you could try something like JuliaLang/julia#47933 to enumerate the outstanding tasks. In case of the latter, you can also use gdb to identify the blocking API call.

Either way, this is a different issue, unrelated to the task scheduling delay reported here. And it may still be a user error; if you create a deadlock on a task, that will result in the task never finishing and Julia soft-hanging.

[physicsjoke]

You were correct and this was not your issue, sorry for the confusion :)

[maleadt]

Should be fixed by #2025