Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DTable] Negative groupby scaling with threads #8

Open
krynju opened this issue Jan 6, 2022 · 1 comment
Open

[DTable] Negative groupby scaling with threads #8

krynju opened this issue Jan 6, 2022 · 1 comment

Comments

@krynju
Copy link
Member

krynju commented Jan 6, 2022

Noticed this in my benchmarks
Table shows results for 2, 4, 8 and 16 threads and their comparisons against times from 2-thread config
Other ops mostly scale well

Either the groupby algo is messed up or the scheduler doesn't handle it well

image

@krynju
Copy link
Member Author

krynju commented Jan 7, 2022

Scaling works fine with processes though
each worker has 4 threads in this setup

4 workers time -> 696.21
8 workers time -> 234.21
speedup -> 2.97

@krynju krynju transferred this issue from JuliaParallel/Dagger.jl Jun 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant