Recording performance improvements/regressions somewhere?

[IainNZ]

So I've been looking at https://julia-codespeed.csail.mit.edu/ (I find it fascinating!) and there are some interesting things going on there, e.g.

• fasta test got a lot faster on Oct 18 (https://julia-codespeed.csail.mit.edu/timeline/#/?base=none&ben=fasta&env=1&revs=1000&equid=off&exe=1)
• go_benchmark test got a fair bit faster on Oct 31 (https://julia-codespeed.csail.mit.edu/timeline/#/?base=none&ben=go_benchmark&env=1&revs=1000&equid=off&exe=1)
• pidigits got a fair bit slower on Oct 16 (https://julia-codespeed.csail.mit.edu/timeline/#/?base=none&ben=pidigits&env=1&revs=1000&equid=off&exe=1)
• splitline got slower on Oct 29 (https://julia-codespeed.csail.mit.edu/timeline/#/?base=none&ben=splitline&env=1&revs=1000&equid=off&exe=1)

Should we be "annotating" these changes to explain what happened as part of long-term performance tracking? E.g. the splitline change seems to be a change in the test itself, but the commit that corresponds with the changes in pidigits seems to be related to "LUFACT fixes for sparse complex matrices" (cdcc91d), which is odd.

[JeffBezanson]

The splitline benchmark was changed to do more work; i have no idea why. @jiahao What motivated this?

The fasta benchmark was rewritten on Oct 18 to be faster.

[jiahao]

@JeffBezanson probably a typo on my part somewhere...

[JeffBezanson]

Somebody should watch out for regressions and report them here as you have done. Is there any way to annotate the data?

[staticfloat]

We could come up with some method of doing so. Codespeed as it is attempts to do some basic statistical analysis on the data to identify "trends" or sharp changes based on statistics of the last 5 or so builds.

We can fiddle around with what Codespeed does; that's the beauty of open source. :)