Benchmark 0.5 vs 0.6 [do not merge] #20993

KristofferC · 2017-03-12T10:26:18Z

#20947 got auto closed for some reason... Oh well.

Benchmarks again now when the inlining improvements were merged.:

@nanosoldier runbenchmarks(ALL, vs = "@6445c82d0060dbe82b88436f0f4371a4ee64d918")

nanosoldier · 2017-03-12T14:04:18Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

KristofferC · 2017-03-14T08:49:44Z

Copied from comment in #20947 and updated.

Summary of stuff that got slower:

Arrays:

["array","index",("sumcolon","1.0:1.0:100000.0")]
~~["array","index",("sumeach_view","BitArray{2}")]~~
~~["array","index",("sumlinear_view","BitArray{2}")]~~
["array","index",("sumrange","1.0:1.0:100000.0")]
Some of the concatenations.

Date accessors in general

~~For example: ["dates","accessor","millisecond"]~~

Fixed by #20853?

Eig and Schur Factorizations

~~For example: ["linalg","factorization",("eig","Matrix",256)]~~

`div` with BigNumbers

For example: ["scalar","arithmetic",("div","BigFloat","Complex{Float32}")]

Maybe some new promotion paths that leads to multiple allocations of big numbers?

Yes, changed from:

/(a::Real, z::Complex) = a*inv(z)

to

/{R<:Real,S<:Complex}(a::R, z::S) = (T = promote_type(R,S); a*inv(T(z)))

Parallell remotecall

For example ["parallel","remotecall",("identity",2)]

Sorting in general

For example ["sort","insertionsort",("sort forwards","ascending")]

I think this happened when we upgraded LLVM

Sparse indexing

~~For example ["sparse","index",("spmat","col","array",10)]~~

Other

~~["problem","laplacian","laplace_sparse_matvec"]~~
["misc","bitshift",("UInt32","UInt32")]
~~["shootout","revcomp"]~~

tkelman · 2017-03-17T02:40:15Z

["scalar","arithmetic",("div","BigFloat","Complex{Float32}")]

that should be pretty easy to fix by adding a few specialized methods that avoid creating the intermediate BigFloats that the new promotion path introduced

tkelman · 2017-03-30T09:35:20Z

cc @rfourquet if you're working on BigInt / BigFloat performance tweaks

pabloferz · 2017-04-06T20:30:04Z

With all the recent changes to codegen and bugfixes for the type system would it be worth to rebase this and rerun the benchmarks?

KristofferC · 2017-04-06T21:02:26Z

@nanosoldier runbenchmarks(ALL, vs = "@6445c82d0060dbe82b88436f0f4371a4ee64d918")

nanosoldier · 2017-04-06T22:52:51Z

Something went wrong when running your job:

NanosoldierError: failed to run benchmarks against comparison commit: failed process: Process(`sudo cset shield -e su nanosoldier -- -c ./benchscript.sh`, ProcessExited(1)) [1]

Logs and partial data can be found here
cc @jrevels

KristofferC · 2017-04-07T08:48:12Z

Looks like the benchmarks I added are to blame:

ERROR: LoadError: LoadError: UndefVarError: IndexStyle not defined

https://github.com/JuliaCI/BaseBenchmarks.jl/blob/master/src/tuple/TupleBenchmarks.jl#L62

Anyone know what the problem is?

timholy · 2017-04-07T09:09:53Z

Is it using an old version of Compat? It needs at least v0.19.

KristofferC · 2017-04-07T09:13:42Z

Perhaps, the required one is too low at least. I made a PR to bump it.

jrevels · 2017-04-07T14:20:00Z

Now that Compat is updated on Nanosoldier:

@nanosoldier runbenchmarks(ALL, vs = "@6445c82d0060dbe82b88436f0f4371a4ee64d918")

nanosoldier · 2017-04-07T18:16:38Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

jebej · 2017-04-07T21:43:47Z

Factorization regressions are still there :(

jrevels · 2017-04-13T20:11:31Z

Let's see if benchmarks vs. v0.4.7 work now (I did a Pkg.update() on Nanosoldier's v0.4 installation):

@nanosoldier runbenchmarks(ALL, vs = "@ae26b25d43317d7dd3ca05f60b70677aab9c0e08")

nanosoldier · 2017-04-13T23:09:10Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

This works around some of the performance regressions flagged in #20993. The instability itself is not new nor is it the cause of the regression. The trouble is just that there are a few more inlined functions now, each of which needs a dynamic dispatch. It is worth noting that 0.6 (with this patch) is significantly faster than 0.5 (with this patch). I measure an improvement of ~30%.

This works around some of the performance regressions flagged in #20993. The instability itself is not new nor is it the cause of the regression. The trouble is just that there are a few more inlined functions now, each of which needs a dynamic dispatch. It is worth noting that 0.6 (with this patch) is significantly faster than 0.5 (with this patch). I measure an improvement of ~30%. (cherry picked from commit 253fa74)

KristofferC · 2017-05-05T08:18:03Z

How should this be continued? Should I open a new PR against some release-0.6 branch or just keep benchmarking against master?

tkelman · 2017-05-05T09:01:21Z

at the moment the latest wip state is at tk/backports-0.6.0-rc1

KristofferC · 2017-05-05T09:08:05Z

Ah, I can change the branch to merge into from an existing PR. Cool

KristofferC · 2017-05-05T09:09:15Z

tk/backports-0.6.0-rc1 vs 0.4.7:

@nanosoldier runbenchmarks(ALL, vs = "@ae26b25d43317d7dd3ca05f60b70677aab9c0e08")

KristofferC · 2017-05-05T09:11:01Z

tk/backports-0.6.0-rc1 vs 0.5.1:

@nanosoldier runbenchmarks(ALL, vs = "@6445c82d0060dbe82b88436f0f4371a4ee64d918")

nanosoldier · 2017-05-05T10:42:42Z

Something went wrong when running your job:

NanosoldierError: failed to run benchmarks against comparison commit: failed process: Process(`make -j3`, ProcessExited(2)) [2]

Logs and partial data can be found here
cc @jrevels

nanosoldier · 2017-05-05T12:14:35Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

oscardssmith · 2017-05-05T13:08:10Z

This looks really good overall. Does anyone know what caused the regression in div for complex bigfloats?

KristofferC · 2017-05-05T13:13:03Z

Yes, read the first post.

KristofferC · 2017-05-14T21:54:14Z

0.6-rc2 vs 0.5.2:

@nanosoldier runbenchmarks(ALL, vs = "@6445c82d0060dbe82b88436f0f4371a4ee64d918")

nanosoldier · 2017-05-15T00:59:21Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @jrevels

tkelman · 2017-05-15T01:05:59Z

dates constructors, uint bitshift, and some of the simd regressions might be worth looking at to see if they're fixable

I looked a bit at bigint divided by complex (and bigfloat divided by complex) but couldn't find a way of preserving the more accurate result with fewer allocations. I think it would require writing some careful in-place code, which would be really messy without the module reorganization @rfourquet did recently (which for now is 0.7-only)

jebej · 2017-06-03T19:46:02Z

Should the benchmarks be run for the upcoming rc3 as well?

KristofferC · 2017-06-03T19:47:48Z

They are run for regressions against the last rc but she how useful it is to benchmark here. Could probably close it.

dummy commit [ci skip]

977e0ca

KristofferC force-pushed the benchmark_05_06 branch from 64b3224 to 977e0ca Compare April 6, 2017 21:02

KristofferC mentioned this pull request Apr 7, 2017

bump compat req JuliaCI/BaseBenchmarks.jl#71

Merged

fredrikekre mentioned this pull request Apr 13, 2017

Sparse-sparse matrix multiplication 4x slower on 0.5 than 0.4, 6x slower on master #21370

Closed

fredrikekre mentioned this pull request Apr 29, 2017

Factorization twice as slow on 0.6 than 0.5 #21624

Closed

mbauman mentioned this pull request May 2, 2017

Make reshape(::BitArray,...) type stable #21670

Merged

fredrikekre mentioned this pull request May 3, 2017

Recover some lost build flags for openblas #21688

Merged

KristofferC changed the base branch from master to tk/backports-0.6.0-rc1 May 5, 2017 09:08

tkelman changed the base branch from tk/backports-0.6.0-rc1 to release-0.6 May 6, 2017 15:16

KristofferC changed the base branch from release-0.6 to tk/backports-0.6.0-rc2 May 14, 2017 21:53

tkelman changed the base branch from tk/backports-0.6.0-rc2 to release-0.6 May 18, 2017 02:29

KristofferC closed this Jun 25, 2017

tkelman deleted the benchmark_05_06 branch June 25, 2017 18:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark 0.5 vs 0.6 [do not merge] #20993

Benchmark 0.5 vs 0.6 [do not merge] #20993

KristofferC commented Mar 12, 2017

nanosoldier commented Mar 12, 2017

KristofferC commented Mar 14, 2017 •

edited

Loading

tkelman commented Mar 17, 2017

tkelman commented Mar 30, 2017

pabloferz commented Apr 6, 2017

KristofferC commented Apr 6, 2017

nanosoldier commented Apr 6, 2017

KristofferC commented Apr 7, 2017

timholy commented Apr 7, 2017

KristofferC commented Apr 7, 2017

jrevels commented Apr 7, 2017

nanosoldier commented Apr 7, 2017

jebej commented Apr 7, 2017

jrevels commented Apr 13, 2017

nanosoldier commented Apr 13, 2017

KristofferC commented May 5, 2017

tkelman commented May 5, 2017

KristofferC commented May 5, 2017 •

edited

Loading

KristofferC commented May 5, 2017 •

edited

Loading

KristofferC commented May 5, 2017

nanosoldier commented May 5, 2017

nanosoldier commented May 5, 2017

oscardssmith commented May 5, 2017

KristofferC commented May 5, 2017

KristofferC commented May 14, 2017 •

edited

Loading

nanosoldier commented May 15, 2017

tkelman commented May 15, 2017 •

edited

Loading

jebej commented Jun 3, 2017

KristofferC commented Jun 3, 2017 •

edited

Loading

Benchmark 0.5 vs 0.6 [do not merge] #20993

Benchmark 0.5 vs 0.6 [do not merge] #20993

Conversation

KristofferC commented Mar 12, 2017

nanosoldier commented Mar 12, 2017

KristofferC commented Mar 14, 2017 • edited Loading

Arrays:

Date accessors in general

Eig and Schur Factorizations

div with BigNumbers

Parallell remotecall

Sorting in general

Sparse indexing

Other

tkelman commented Mar 17, 2017

tkelman commented Mar 30, 2017

pabloferz commented Apr 6, 2017

KristofferC commented Apr 6, 2017

nanosoldier commented Apr 6, 2017

KristofferC commented Apr 7, 2017

timholy commented Apr 7, 2017

KristofferC commented Apr 7, 2017

jrevels commented Apr 7, 2017

nanosoldier commented Apr 7, 2017

jebej commented Apr 7, 2017

jrevels commented Apr 13, 2017

nanosoldier commented Apr 13, 2017

KristofferC commented May 5, 2017

tkelman commented May 5, 2017

KristofferC commented May 5, 2017 • edited Loading

KristofferC commented May 5, 2017 • edited Loading

KristofferC commented May 5, 2017

nanosoldier commented May 5, 2017

nanosoldier commented May 5, 2017

oscardssmith commented May 5, 2017

KristofferC commented May 5, 2017

KristofferC commented May 14, 2017 • edited Loading

nanosoldier commented May 15, 2017

tkelman commented May 15, 2017 • edited Loading

jebej commented Jun 3, 2017

KristofferC commented Jun 3, 2017 • edited Loading

KristofferC commented Mar 14, 2017 •

edited

Loading

`div` with BigNumbers

KristofferC commented May 5, 2017 •

edited

Loading

KristofferC commented May 5, 2017 •

edited

Loading

KristofferC commented May 14, 2017 •

edited

Loading

tkelman commented May 15, 2017 •

edited

Loading

KristofferC commented Jun 3, 2017 •

edited

Loading