-
Notifications
You must be signed in to change notification settings - Fork 209
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cross-device copy of wrapped arrays fails #1377
Comments
Ah yeah, that's not supported for now. This fails like any cross-device copy used to fail. It still does because I only implemented the cross-device copy between CuArrays, once you wrap them (e.g., here with a SubArray) the specialized dispatch gets lost and we fall back to the kernel-based GPUArrays implementation. That one then fails because it does not handle multiple devices. To fix this, I guess we either have to add specialized functions to CUDA.jl (but where do we draw the line then? with SubArray? or also LinearAlgebra wrappers? etc), and/or teach GPUArrays about the specific device (for which we'd also need to model unified memory, etc). |
Accessing memory from another device in a kernel works now, so the GPUArrays fallbacks should work. |
It's expected that non-continguous data sending should fail. However, it could be unexpected that it breaks CUDA / the Julia session.
v3.8.1
First we demonstrate that sending contiguous data works charmingly:
Now let's try to send non-continguous data:
CUDA and Julia are now broken:
When exiting Julia we get
The rest of the error
error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing) throw_api_error at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:91 macro expansion at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:101 [inlined] cuModuleUnload at /home/greg/.julia/packages/CUDA/t62lT/lib/utils/call.jl:26 #27 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/module.jl:82 [inlined] #context!#63 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:164 unknown function (ip: 0x7fe39c0fa1ab) _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429 context!##kw at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:161 unknown function (ip: 0x7fe39c0f9bf8) unsafe_unload! at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/module.jl:81 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined] run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278 jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365 run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394 [inlined] jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:482 jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:240 jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702 main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) _start at ./julia (unknown line) error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing) throw_api_error at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:91 macro expansion at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:101 [inlined] cuStreamDestroy_v2 at /home/greg/.julia/packages/CUDA/t62lT/lib/utils/call.jl:26 #10 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/stream.jl:86 [inlined] #context!#63 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:164 unknown function (ip: 0x7fe39c0faf8b) _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429 context!##kw at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:161 unsafe_destroy! at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/stream.jl:85 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined] run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278 jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365 run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394 [inlined] jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:482 jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:240 jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702 main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) _start at ./julia (unknown line) error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing) throw_api_error at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:91 macro expansion at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:101 [inlined] cuStreamDestroy_v2 at /home/greg/.julia/packages/CUDA/t62lT/lib/utils/call.jl:26 #10 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/stream.jl:86 [inlined] #context!#63 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:164 unknown function (ip: 0x7fe39c0faf8b) _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429 context!##kw at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:161 unsafe_destroy! at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/stream.jl:85 _jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined] jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined] run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278 jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365 run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394 [inlined] jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:482 jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:240 jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702 main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42 __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) _start at ./julia (unknown line) WARNING: Error while freeing DeviceBuffer(128 bytes at 0x0000000be8000000): CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing)Stacktrace:
[1] throw_api_error(res::CUDA.cudaError_enum)
@ Unhandled Task ERROR: EOFError: read end of file
Stacktrace:
[1] wait
@ ./asyncevent.jl:136 [inlined]
[2] sleep(sec::Int64)
@ Base ./asyncevent.jl:221
[3] pool_cleanup()
@ CUDA ~/.julia/packages/CUDA/t62lT/src/pool.jl:126
[4] (::CUDA.var"#190#191")()
@ CUDA ./threadingconstructs.jl:178
CUDA ~/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:91
[2] macro expansion
@ ~/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:101 [inlined]
[3] cuMemFreeAsync(dptr::CUDA.Mem.DeviceBuffer, hStream::CuStream)
@ CUDA ~/.julia/packages/CUDA/t62lT/lib/utils/call.jl:26
[4] #free#2
@ ~/.julia/packages/CUDA/t62lT/lib/cudadrv/memory.jl:96 [inlined]
[5] macro expansion
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:58 [inlined]
[6] macro expansion
@ ./timing.jl:299 [inlined]
[7] #actual_free#183
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:57 [inlined]
[8] #_free#199
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:263 [inlined]
[9] macro expansion
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:243 [inlined]
[10] macro expansion
@ ./timing.jl:299 [inlined]
[11] #free#198
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:242 [inlined]
[12] #204
@ ~/.julia/packages/CUDA/t62lT/src/array.jl:81 [inlined]
[13] context!(f::CUDA.var"#204#205"{CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, CuStream}, ctx::CuContext; skip_destroyed::Bool)
@ CUDA ~/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:164
[14] unsafe_free!(xs::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, stream::CuStream)
@ CUDA ~/.julia/packages/CUDA/t62lT/src/array.jl:80
[15] unsafe_finalize!(xs::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer})
@ CUDA ~/.julia/packages/CUDA/t62lT/src/array.jl:101
error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing)
throw_api_error at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:91
macro expansion at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:101 [inlined]
cuStreamDestroy_v2 at /home/greg/.julia/packages/CUDA/t62lT/lib/utils/call.jl:26
#10 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/stream.jl:86 [inlined]
#context!#63 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:164
unknown function (ip: 0x7fe39c0faf8b)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
context!##kw at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:161
unsafe_destroy! at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/stream.jl:85
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365
run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394 [inlined]
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:482
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:240
jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at ./julia (unknown line)
WARNING: Error while freeing DeviceBuffer(128 bytes at 0x0000000602000000):
CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing)
Stacktrace:
[1] throw_api_error(res::CUDA.cudaError_enum)
@ CUDA ~/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:91
[2] macro expansion
@ ~/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:101 [inlined]
[3] cuMemFreeAsync(dptr::CUDA.Mem.DeviceBuffer, hStream::CuStream)
@ CUDA ~/.julia/packages/CUDA/t62lT/lib/utils/call.jl:26
[4] #free#2
@ ~/.julia/packages/CUDA/t62lT/lib/cudadrv/memory.jl:96 [inlined]
[5] macro expansion
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:58 [inlined]
[6] macro expansion
@ ./timing.jl:299 [inlined]
[7] #actual_free#183
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:57 [inlined]
[8] #_free#199
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:263 [inlined]
[9] macro expansion
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:243 [inlined]
[10] macro expansion
@ ./timing.jl:299 [inlined]
[11] #free#198
@ ~/.julia/packages/CUDA/t62lT/src/pool.jl:242 [inlined]
[12] #204
@ ~/.julia/packages/CUDA/t62lT/src/array.jl:81 [inlined]
[13] context!(f::CUDA.var"#204#205"{CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, CuStream}, ctx::CuContext; skip_destroyed::Bool)
@ CUDA ~/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:164
[14] unsafe_free!(xs::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer}, stream::CuStream)
@ CUDA ~/.julia/packages/CUDA/t62lT/src/array.jl:80
[15] unsafe_finalize!(xs::CuArray{Float64, 2, CUDA.Mem.DeviceBuffer})
@ CUDA ~/.julia/packages/CUDA/t62lT/src/array.jl:101
error in running finalizer: CUDA.CuError(code=CUDA.cudaError_enum(0x000002bc), meta=nothing)
throw_api_error at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:91
macro expansion at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/error.jl:101 [inlined]
cuStreamDestroy_v2 at /home/greg/.julia/packages/CUDA/t62lT/lib/utils/call.jl:26
#10 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/stream.jl:86 [inlined]
#context!#63 at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:164
unknown function (ip: 0x7fe39c0faf8b)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
context!##kw at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/state.jl:161
unsafe_destroy! at /home/greg/.julia/packages/CUDA/t62lT/lib/cudadrv/stream.jl:85
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2247 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2429
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1788 [inlined]
run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:278
jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:365
run_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:394 [inlined]
jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:482
jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:240
jl_repl_entrypoint at /buildworker/worker/package_linux64/build/src/jlapi.c:702
main at /buildworker/worker/package_linux64/build/cli/loader_exe.c:42
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at ./julia (unknown line)
The text was updated successfully, but these errors were encountered: