Track array ownership to avoid illegal memory accesses

[marius311]

This is one thing that I think would greatly improve the interactive single process multi-GPU workflow. Right now if you accidentally trigger an illegal memory access (like say you just forgot that some variable in your session isn't on the GPU you currently have active), then it borks the whole session and you have to restart:

julia> using CUDA

julia> device!(0)

julia> x = cu(rand(2,2))
2×2 CuArray{Float32, 2}:
 0.911817  0.814058
 0.579863  0.511812

julia> device!(1)

julia> 2 .* x  # oops forgot x is on device 0
2×2 CuArray{Float32, 2}:
Error showing value of type CuArray{Float32, 2}:
ERROR: CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)
Stacktrace:

# have to restart session now, all other (otherwise valid) GPU operations now throw illegal memory access

[maleadt]

That's a CUDA limitation, nothing we can do about it. File it with NVIDIA instead 😄

[maleadt]

Of course, we shouldn't be running into illegal memory accesses at all, CUDA.jl should be as safe to use as possible. In this case, we should probably be tracking which device owns an array.

[marius311]

Yea, I think just doing a little check on the CUDA.jl size would be pretty useful. I suppose this is already tracked, right? Far from the cleanest, but

findfirst(==(x.ctx), CUDA.__device_contexts)-1

does give you the device id that x::CuArray was created on.

[maleadt]

Note to self: it might be an idea to track the context in the buffer and disallow conversion to a pointer if the current context doesn't match the buffer's.

[maleadt]

This is implemented now.