More lock contention in Julia 1.6 to 1.8 than in 1.5.1 (I/O with parallelism strictly not the reason) #39598
Labels
Comments
|
For binarytrees, if you put |
|
Indeed, if you try something like |
JeffBezanson
added
the
kind:regression
|
Looks like probably #37816 |
|
The fix, if it's in, seems only for Distributed, not Threads. Should this hold up a new RC? EDIT: Looking at my timing, I see it's only minor regression for the latter, so no, but for the former, I would really like it confirmed that the fix is in... |
Nothing has been changed with respect to this. Removing the precompile signatures in #37816 was intentional since they caused a larger sysimage and invalidations and it isn't clear that it is worth it. |
PallHaraldsson
changed the title
|
Threading is getting noticeably slower on 1.7 (52% slower than 1.5.1; 20% slower than rc2): This time I'm testing under some load (i.e. Firefox running), but that's also a valuable test.
Maybe, but that's only for the other kind of slowdown (not fixed on 1.7). |
|
With 4 threads: I think you're mostly benchmarking I/O? Also, I know you love hyperfine (and it's a great tool to benchmark applications!), but it always includes Julia startup time. |
|
Yes, I kind of love hyperfine! And it's good to know the real cause, but it's also important to fix. When I looked into IO, Julia seemed to have too much locking (I tried unsuccessfully to lower it), and it's known to be a problem/slower than in other languages, but still it shouldn't be more of a problem in later Julia versions. The number of allocations seems to be exactly the same each time "71.69 k allocations", without threading, and even across versions 1.5.1 and 1.7, but is not with threading on, in older, or here for 1.7: |
PallHaraldsson
changed the title
PallHaraldsson
changed the title
|
FYI: Actually 1.6.0 is faster up to 2 threads, just 1.5.1 keeps getting faster up to about 7 threads while 1.6 gets slower: |
|
Strange Julia error, but first reporting in chronological order the timings: It's not better with 1.8 (and similar on 1.7): These numbers are with Firefox on, and without I get at least 1 sec better. julia-1.6.0-rc2 may be slightly faster on same conditions (I tried suspending the main firefox process, while "Web Content" processes weren't), and julia-1.5.1 almost certainly still is, getting down to 10 sec. with -O0. The strange error, first working fine: Might this be a hint to the possible slowness? I'll wait a bit with killing these processes. The fan is annoying, and I doubt something can be debugged, but let me know then. Could this have anything to do with memory running out? I see in /var/log/syslog from BEFORE those julia processes started (but I' still puzzled by this, I believe I haven't used julia today exert for benchmarking): Soon after, it's now 14:17, some minutes ago, Mozilla Crash reporter appeared. Firefox often just crashes, with memory running out (OOM in Linux). I don't see that happening in the log for julia or hyperfine (nor firefox which is unusual, I would usually notice, possibly not now since the reporter kicked in), and I stopped hyperfine myself, and julia didn't got killed. Can memory running out still be affecting? Are other processes involved or just the Linux kernel? I get "no"/"same" output for either julia process: |
|
Same problem as in previous comment, while now hung earlier: Just after, I could run with out hyperfine and then: I doubt this is a hyperfine problem, as then it worked fine. Still I can't rule it out as I've only seen the problem with it. Is it a likely interaction? Since it seems to just run the program in a shell? Actually just trying that seems might be the problem as this just hung: |
|
#41411 likely fixes threading hang (untested), but I tested commenting out printing ("write" and "unsafe_write") statements, and fasta.jl went from (min) 2.793 sec to (min) 2.401 sec. or 14% reduction. I took out some wait statements (not all) and time went further down to (min) 2.207 sec, and I still got hangs, but if I take out the last two waits, i.e. the following lines, the time went down to (min) 497.0 ms (82% reduction), but I think the program exited before the threads had completed running the code. and At some point after, i.e. I think I had eliminated all waits (but I'm not sure, I was testing back and forth) I did get: then julia processes were left running and load went up to 30, and I started seening "Measuring shell spawning time" and progress bar for long before "Initial time measurement" and high measurements: |
|
@vtjnash I did try to uncomment back the write statements (continuing from previous comment), while keeping the wait statements I had commented out, still commented out, and I predictable get the output in some garbled order, but at least it's fast... so the I/O isn't strictly slow, nor seems the cause of the hang, only the lock contention. And it seems to have eliminated the hang. |
PallHaraldsson
changed the title
PallHaraldsson
changed the title
|
I just downloaded a nightly and it seems to be getting worse, i.e. "user" time, while "real" maybe slightly better: |
|
The "real" time is getting a little better on the latest master, while with --compile=min much better: |
|
I think this is in our nanosoldier basebenchmarks, so we will notice any changes. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I ran with Firefox turned off, and all 16 virtual cores were idle when starting (strangely load still about "1,08"):
https://discourse.julialang.org/t/julia-1-6-0-rc1-and-1-7-with-threads-is-slower-considerations-for-the-benchmark-game/54957
The text was updated successfully, but these errors were encountered: