Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track CuArray stream usage #2236

Closed
maleadt opened this issue Jan 15, 2024 · 1 comment · Fixed by #2335
Closed

Track CuArray stream usage #2236

maleadt opened this issue Jan 15, 2024 · 1 comment · Fixed by #2335
Labels
cuda array Stuff about CuArray. enhancement New feature or request speculative Not sure about this one yet.

Comments

@maleadt
Copy link
Member

maleadt commented Jan 15, 2024

We currently require that tasks where CuArrays are written to, synchronize the GPU before sending that data off to other tasks. This is fragile, difficult to explain, and a source of common errors. We should consider tracking which tasks or streams a CuArray is used in, in order to automatically synchronize on use.

@maleadt maleadt added enhancement New feature or request cuda array Stuff about CuArray. speculative Not sure about this one yet. labels Jan 15, 2024
@maleadt
Copy link
Member Author

maleadt commented Mar 25, 2024

cc @jpsamaroo; this would also help Dagger.

One potentially cheap way to do this is by keeping track of a single stream in each array, and when deriving a pointer (which dominates every use of a CuArray) check whether the current stream matches the array's. If not, synchronize the contained stream and replace it by the current one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cuda array Stuff about CuArray. enhancement New feature or request speculative Not sure about this one yet.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant