RFC: stop generating precompile signatures for Distributed

[KristofferC]

The distributed code is in general quite poorly typed so precompiling it leads to quite a lot of possible source of invalidations. Also, if you are spinning up a full distributed computation, you are probably not that worried about a short latency. This removes around 200 precompile signatures (out of 1500) from the build process.

Before:

julia> using Distributed

julia> @time     pmap(x->iseven(x) ? 1 : 0, 1:4);
  0.107108 seconds (128.56 k allocations: 7.853 MiB, 6.63% gc time, 99.45% compilation time)

julia> @time     @distributed (+) for i = 1:100 Int(rand(Bool)) end;
  0.055258 seconds (72.36 k allocations: 4.249 MiB, 98.02% compilation time)

vs now:

julia> using Distributed

julia> @time     pmap(x->iseven(x) ? 1 : 0, 1:4);
  0.510848 seconds (2.25 M allocations: 134.619 MiB, 4.30% gc time, 99.88% compilation time)

julia> @time     @distributed (+) for i = 1:100 Int(rand(Bool)) end;
  0.188521 seconds (394.07 k allocations: 23.134 MiB, 99.48% compilation time)

The only reason I added this in #33077 was that I found some random precompile files for Distributed lying around (that were probably very out of date)

[KristofferC]

Looking at the emotes it seems people are positive to try this. So let's do that and see if anyone reacts (my guess is no).

[PallHaraldsson]

One program here uses Distributed: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/binarytrees-julia-5.html

It's unusually slow, but actually it should maybe use threads instead (to be as fast as alternative languages), but that has some issues with GC.

[PallHaraldsson]

you are probably not that worried about a short latency.

It's bad:

$ time ~/julia-1.8-DEV-d1145d4569/bin/julia -p10 -e ""

real	0m10,317s
user	1m5,155s
sys	0m3,631s

$ time ~/julia-1.5.1/bin/julia -p10 -e ""

real	0m3,475s
user	0m21,953s
sys	0m3,672s

$ time ~/julia-1.8-DEV-d1145d4569/bin/julia -p1 -e ""

real	0m7,680s
user	0m12,496s
sys	0m1,192s

$ time ~/julia-1.5.1/bin/julia -p1 -e ""

real	0m2,409s
user	0m2,962s
sys	0m1,183s

And I get worse than you do:

julia> @time     pmap(x->iseven(x) ? 1 : 0, 1:4); # first run:
  1.230571 seconds (2.59 M allocations: 162.570 MiB, 5.22% gc time)

julia> @time     pmap(x->iseven(x) ? 1 : 0, 1:4); # second run, why "compilation time", still much faster?
  0.139251 seconds (80.55 k allocations: 4.913 MiB, 99.50% compilation time)

Is it possible, to even partially revert?

and this slower than 1.5.1:

julia> @time     @distributed (+) for i = 1:100 Int(rand(Bool)) end;
  0.332838 seconds (424.01 k allocations: 26.573 MiB, 2.52% gc time, 99.89% compilation time)

julia> @time     @distributed (+) for i = 1:100 Int(rand(Bool)) end;
  0.084881 seconds (46.19 k allocations: 2.655 MiB, 99.68% compilation time)

1.5.1:
julia> @time     @distributed (+) for i = 1:100 Int(rand(Bool)) end;
  0.098766 seconds (76.25 k allocations: 4.031 MiB)

julia> @time     @distributed (+) for i = 1:100 Int(rand(Bool)) end;
  0.065623 seconds (48.95 k allocations: 2.473 MiB)