Clean up Continuous Benchmarking #675

avik-pal · 2024-06-02T20:00:13Z

TODOs

Write a script to consolidate the data from bson files into a dataframe or something easy to extend later on
Write a plotting code to plot the results from benchmark-results and push it to gh-pages

codecov · 2024-06-02T20:09:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.10%. Comparing base (3610b73) to head (57def39).

❗ Current head 57def39 differs from pull request most recent head d561800

Please upload reports for the commit d561800 to get more accurate results.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #675      +/-   ##
==========================================
- Coverage   87.25%   81.10%   -6.15%     
==========================================
  Files          50       50              
  Lines        2511     2509       -2     
==========================================
- Hits         2191     2035     -156     
- Misses        320      474     +154

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions

Benchmark Results

Benchmark suite	Current: `945dc6f`	Previous: `3610b73`	Ratio
`Dense(2 => 2)/cpu/reverse/ReverseDiff (compiled)/(2, 128)`	`3670.625` ns	`3658.125` ns	`1.00`
`Dense(2 => 2)/cpu/reverse/Zygote/(2, 128)`	`7185.4` ns	`7232` ns	`0.99`
`Dense(2 => 2)/cpu/reverse/Tracker/(2, 128)`	`19942.5` ns	`21060` ns	`0.95`
`Dense(2 => 2)/cpu/reverse/ReverseDiff/(2, 128)`	`9603.9` ns	`9628.2` ns	`1.00`
`Dense(2 => 2)/cpu/reverse/Flux/(2, 128)`	`8916.75` ns	`8960.8` ns	`1.00`
`Dense(2 => 2)/cpu/reverse/SimpleChains/(2, 128)`	`4509.0625` ns	`4460.875` ns	`1.01`
`Dense(2 => 2)/cpu/reverse/Enzyme/(2, 128)`	`1174.2740740740742` ns	`1184.7829457364342` ns	`0.99`
`Dense(2 => 2)/cpu/forward/NamedTuple/(2, 128)`	`1116.9583333333335` ns	`1125.7961783439491` ns	`0.99`
`Dense(2 => 2)/cpu/forward/ComponentArray/(2, 128)`	`1186.014598540146` ns	`1198.031746031746` ns	`0.99`
`Dense(2 => 2)/cpu/forward/Flux/(2, 128)`	`1814.468085106383` ns	`1827.8139534883721` ns	`0.99`
`Dense(2 => 2)/cpu/forward/SimpleChains/(2, 128)`	`179.98739495798318` ns	`179.58815232722145` ns	`1.00`
`Dense(20 => 20)/cpu/reverse/ReverseDiff (compiled)/(20, 128)`	`17532` ns	`17263` ns	`1.02`
`Dense(20 => 20)/cpu/reverse/Zygote/(20, 128)`	`17292` ns	`17122` ns	`1.01`
`Dense(20 => 20)/cpu/reverse/Tracker/(20, 128)`	`37555.5` ns	`36889` ns	`1.02`
`Dense(20 => 20)/cpu/reverse/ReverseDiff/(20, 128)`	`28563` ns	`27992` ns	`1.02`
`Dense(20 => 20)/cpu/reverse/Flux/(20, 128)`	`19808` ns	`19667` ns	`1.01`
`Dense(20 => 20)/cpu/reverse/SimpleChains/(20, 128)`	`17122` ns	`16976.5` ns	`1.01`
`Dense(20 => 20)/cpu/reverse/Enzyme/(20, 128)`	`4355.285714285715` ns	`4345.285714285715` ns	`1.00`
`Dense(20 => 20)/cpu/forward/NamedTuple/(20, 128)`	`3897.25` ns	`3866` ns	`1.01`
`Dense(20 => 20)/cpu/forward/ComponentArray/(20, 128)`	`3982.5` ns	`3944.875` ns	`1.01`
`Dense(20 => 20)/cpu/forward/Flux/(20, 128)`	`5002.285714285715` ns	`4863.428571428572` ns	`1.03`
`Dense(20 => 20)/cpu/forward/SimpleChains/(20, 128)`	`1655.1` ns	`1655.1` ns	`1`
`Conv((3, 3), 3 => 3)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 3, 128)`	`49266645` ns	`39915572` ns	`1.23`
`Conv((3, 3), 3 => 3)/cpu/reverse/Zygote/(64, 64, 3, 128)`	`57845468` ns	`57483129` ns	`1.01`
`Conv((3, 3), 3 => 3)/cpu/reverse/Tracker/(64, 64, 3, 128)`	`69225030.5` ns	`77827690.5` ns	`0.89`
`Conv((3, 3), 3 => 3)/cpu/reverse/ReverseDiff/(64, 64, 3, 128)`	`90783400` ns	`91917891` ns	`0.99`
`Conv((3, 3), 3 => 3)/cpu/reverse/Flux/(64, 64, 3, 128)`	`64831488` ns	`77949027.5` ns	`0.83`
`Conv((3, 3), 3 => 3)/cpu/reverse/SimpleChains/(64, 64, 3, 128)`	`11933347` ns	`12071109.5` ns	`0.99`
`Conv((3, 3), 3 => 3)/cpu/reverse/Enzyme/(64, 64, 3, 128)`	`8426666` ns	`8366464` ns	`1.01`
`Conv((3, 3), 3 => 3)/cpu/forward/NamedTuple/(64, 64, 3, 128)`	`7004567` ns	`6945348` ns	`1.01`
`Conv((3, 3), 3 => 3)/cpu/forward/ComponentArray/(64, 64, 3, 128)`	`6964729` ns	`6899807` ns	`1.01`
`Conv((3, 3), 3 => 3)/cpu/forward/Flux/(64, 64, 3, 128)`	`10755596` ns	`11614581` ns	`0.93`
`Conv((3, 3), 3 => 3)/cpu/forward/SimpleChains/(64, 64, 3, 128)`	`6383174.5` ns	`6380416` ns	`1.00`
`vgg16/cpu/reverse/Zygote/(32, 32, 3, 16)`	`707549512` ns	`695496825` ns	`1.02`
`vgg16/cpu/reverse/Zygote/(32, 32, 3, 64)`	`2962051209` ns	`2804547141` ns	`1.06`
`vgg16/cpu/reverse/Zygote/(32, 32, 3, 2)`	`147109581.5` ns	`154892659` ns	`0.95`
`vgg16/cpu/reverse/Tracker/(32, 32, 3, 16)`	`887340902` ns	`967247411` ns	`0.92`
`vgg16/cpu/reverse/Tracker/(32, 32, 3, 64)`	`2959355202` ns	`3242800947` ns	`0.91`
`vgg16/cpu/reverse/Tracker/(32, 32, 3, 2)`	`243832042` ns	`203869077.5` ns	`1.20`
`vgg16/cpu/reverse/Flux/(32, 32, 3, 16)`	`737740608` ns	`688694506` ns	`1.07`
`vgg16/cpu/reverse/Flux/(32, 32, 3, 64)`	`3119802783` ns	`2898534015` ns	`1.08`
`vgg16/cpu/reverse/Flux/(32, 32, 3, 2)`	`149215566` ns	`143258580.5` ns	`1.04`
`vgg16/cpu/forward/NamedTuple/(32, 32, 3, 16)`	`177976650.5` ns	`173528994` ns	`1.03`
`vgg16/cpu/forward/NamedTuple/(32, 32, 3, 64)`	`662415886` ns	`648298149` ns	`1.02`
`vgg16/cpu/forward/NamedTuple/(32, 32, 3, 2)`	`38916494` ns	`34150612` ns	`1.14`
`vgg16/cpu/forward/ComponentArray/(32, 32, 3, 16)`	`172675765` ns	`163561257.5` ns	`1.06`
`vgg16/cpu/forward/ComponentArray/(32, 32, 3, 64)`	`649014676` ns	`639345113` ns	`1.02`
`vgg16/cpu/forward/ComponentArray/(32, 32, 3, 2)`	`34152130` ns	`29520830.5` ns	`1.16`
`vgg16/cpu/forward/Flux/(32, 32, 3, 16)`	`224599617` ns	`227374451.5` ns	`0.99`
`vgg16/cpu/forward/Flux/(32, 32, 3, 64)`	`850956502` ns	`853156689` ns	`1.00`
`vgg16/cpu/forward/Flux/(32, 32, 3, 2)`	`42407878` ns	`37982328.5` ns	`1.12`
`Conv((3, 3), 64 => 64)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 64, 128)`	`1456822714` ns	`1207786302.5` ns	`1.21`
`Conv((3, 3), 64 => 64)/cpu/reverse/Zygote/(64, 64, 64, 128)`	`1862976425.5` ns	`1847921696` ns	`1.01`
`Conv((3, 3), 64 => 64)/cpu/reverse/Tracker/(64, 64, 64, 128)`	`2272595123.5` ns	`2060352063.5` ns	`1.10`
`Conv((3, 3), 64 => 64)/cpu/reverse/ReverseDiff/(64, 64, 64, 128)`	`2344898675` ns	`2382201115` ns	`0.98`
`Conv((3, 3), 64 => 64)/cpu/reverse/Flux/(64, 64, 64, 128)`	`2105028068` ns	`1902804893.5` ns	`1.11`
`Conv((3, 3), 64 => 64)/cpu/reverse/Enzyme/(64, 64, 64, 128)`	`346542384.5` ns	`345670316` ns	`1.00`
`Conv((3, 3), 64 => 64)/cpu/forward/NamedTuple/(64, 64, 64, 128)`	`317507097.5` ns	`318483653.5` ns	`1.00`
`Conv((3, 3), 64 => 64)/cpu/forward/ComponentArray/(64, 64, 64, 128)`	`319603720.5` ns	`317019643` ns	`1.01`
`Conv((3, 3), 64 => 64)/cpu/forward/Flux/(64, 64, 64, 128)`	`459870622.5` ns	`451482707` ns	`1.02`
`Conv((3, 3), 1 => 1)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 1, 128)`	`11935784.5` ns	`11823865` ns	`1.01`
`Conv((3, 3), 1 => 1)/cpu/reverse/Zygote/(64, 64, 1, 128)`	`17910479` ns	`17713044` ns	`1.01`
`Conv((3, 3), 1 => 1)/cpu/reverse/Tracker/(64, 64, 1, 128)`	`19142783` ns	`18879877.5` ns	`1.01`
`Conv((3, 3), 1 => 1)/cpu/reverse/ReverseDiff/(64, 64, 1, 128)`	`23955586` ns	`23646980` ns	`1.01`
`Conv((3, 3), 1 => 1)/cpu/reverse/Flux/(64, 64, 1, 128)`	`17954014.5` ns	`17741815` ns	`1.01`
`Conv((3, 3), 1 => 1)/cpu/reverse/SimpleChains/(64, 64, 1, 128)`	`1156535` ns	`1161321` ns	`1.00`
`Conv((3, 3), 1 => 1)/cpu/reverse/Enzyme/(64, 64, 1, 128)`	`2519823.5` ns	`2511195` ns	`1.00`
`Conv((3, 3), 1 => 1)/cpu/forward/NamedTuple/(64, 64, 1, 128)`	`2039347` ns	`2037509` ns	`1.00`
`Conv((3, 3), 1 => 1)/cpu/forward/ComponentArray/(64, 64, 1, 128)`	`2022156` ns	`2021941` ns	`1.00`
`Conv((3, 3), 1 => 1)/cpu/forward/Flux/(64, 64, 1, 128)`	`2064645.5` ns	`2061425` ns	`1.00`
`Conv((3, 3), 1 => 1)/cpu/forward/SimpleChains/(64, 64, 1, 128)`	`194164` ns	`200374` ns	`0.97`
`Dense(200 => 200)/cpu/reverse/ReverseDiff (compiled)/(200, 128)`	`294451` ns	`292156` ns	`1.01`
`Dense(200 => 200)/cpu/reverse/Zygote/(200, 128)`	`267781.5` ns	`265036` ns	`1.01`
`Dense(200 => 200)/cpu/reverse/Tracker/(200, 128)`	`370253` ns	`362265.5` ns	`1.02`
`Dense(200 => 200)/cpu/reverse/ReverseDiff/(200, 128)`	`411324.5` ns	`406235` ns	`1.01`
`Dense(200 => 200)/cpu/reverse/Flux/(200, 128)`	`277103.5` ns	`272820.5` ns	`1.02`
`Dense(200 => 200)/cpu/reverse/SimpleChains/(200, 128)`	`416159` ns	`405247.5` ns	`1.03`
`Dense(200 => 200)/cpu/reverse/Enzyme/(200, 128)`	`83386` ns	`83476` ns	`1.00`
`Dense(200 => 200)/cpu/forward/NamedTuple/(200, 128)`	`82915` ns	`81162` ns	`1.02`
`Dense(200 => 200)/cpu/forward/ComponentArray/(200, 128)`	`85871` ns	`81632` ns	`1.05`
`Dense(200 => 200)/cpu/forward/Flux/(200, 128)`	`87714` ns	`86611` ns	`1.01`
`Dense(200 => 200)/cpu/forward/SimpleChains/(200, 128)`	`104947` ns	`104535` ns	`1.00`
`Conv((3, 3), 16 => 16)/cpu/reverse/ReverseDiff (compiled)/(64, 64, 16, 128)`	`204951314.5` ns	`193250179` ns	`1.06`
`Conv((3, 3), 16 => 16)/cpu/reverse/Zygote/(64, 64, 16, 128)`	`323359309` ns	`323860890.5` ns	`1.00`
`Conv((3, 3), 16 => 16)/cpu/reverse/Tracker/(64, 64, 16, 128)`	`402003241` ns	`427466270` ns	`0.94`
`Conv((3, 3), 16 => 16)/cpu/reverse/ReverseDiff/(64, 64, 16, 128)`	`426588502` ns	`434737998` ns	`0.98`
`Conv((3, 3), 16 => 16)/cpu/reverse/Flux/(64, 64, 16, 128)`	`373624838` ns	`377323893.5` ns	`0.99`
`Conv((3, 3), 16 => 16)/cpu/reverse/SimpleChains/(64, 64, 16, 128)`	`334380331` ns	`321116314.5` ns	`1.04`
`Conv((3, 3), 16 => 16)/cpu/reverse/Enzyme/(64, 64, 16, 128)`	`51327536` ns	`51261538.5` ns	`1.00`
`Conv((3, 3), 16 => 16)/cpu/forward/NamedTuple/(64, 64, 16, 128)`	`43735393` ns	`43626588` ns	`1.00`
`Conv((3, 3), 16 => 16)/cpu/forward/ComponentArray/(64, 64, 16, 128)`	`43642635` ns	`43365628` ns	`1.01`
`Conv((3, 3), 16 => 16)/cpu/forward/Flux/(64, 64, 16, 128)`	`60237148.5` ns	`49421158` ns	`1.22`
`Conv((3, 3), 16 => 16)/cpu/forward/SimpleChains/(64, 64, 16, 128)`	`28055894.5` ns	`28683332` ns	`0.98`
`Dense(2000 => 2000)/cpu/reverse/ReverseDiff (compiled)/(2000, 128)`	`19091675` ns	`18551795` ns	`1.03`
`Dense(2000 => 2000)/cpu/reverse/Zygote/(2000, 128)`	`19631799` ns	`19431895` ns	`1.01`
`Dense(2000 => 2000)/cpu/reverse/Tracker/(2000, 128)`	`23586888` ns	`23018479.5` ns	`1.02`
`Dense(2000 => 2000)/cpu/reverse/ReverseDiff/(2000, 128)`	`45382444` ns	`23926235.5` ns	`1.90`
`Dense(2000 => 2000)/cpu/reverse/Flux/(2000, 128)`	`36440406` ns	`19524692.5` ns	`1.87`
`Dense(2000 => 2000)/cpu/reverse/Enzyme/(2000, 128)`	`12615242` ns	`6482744` ns	`1.95`
`Dense(2000 => 2000)/cpu/forward/NamedTuple/(2000, 128)`	`6591061.5` ns	`6461434` ns	`1.02`
`Dense(2000 => 2000)/cpu/forward/ComponentArray/(2000, 128)`	`6577579` ns	`6462661` ns	`1.02`
`Dense(2000 => 2000)/cpu/forward/Flux/(2000, 128)`	`6526774` ns	`6479762.5` ns	`1.01`

This comment was automatically generated by workflow using github-action-benchmark.

bench/layers.jl

bench/runbenchmarks.jl

bench/helpers.jl

bench/runbenchmarks.jl

bench/helpers.jl

bench/runbenchmarks.jl

avik-pal force-pushed the ap/rework_benchmarking branch from 531c1a2 to 57def39 Compare June 2, 2024 20:00

github-actions bot reviewed Jun 2, 2024

View reviewed changes

avik-pal force-pushed the ap/rework_benchmarking branch 10 times, most recently from 72d3a46 to 3fc2576 Compare June 3, 2024 02:16

github-actions bot reviewed Jun 3, 2024

View reviewed changes

bench/layers.jl Outdated Show resolved Hide resolved

bench/runbenchmarks.jl Outdated Show resolved Hide resolved

bench/runbenchmarks.jl Outdated Show resolved Hide resolved

avik-pal force-pushed the ap/rework_benchmarking branch from 3fc2576 to 303ea94 Compare June 3, 2024 03:01

github-actions bot reviewed Jun 3, 2024

View reviewed changes

bench/helpers.jl Outdated Show resolved Hide resolved

bench/helpers.jl Outdated Show resolved Hide resolved

bench/runbenchmarks.jl Outdated Show resolved Hide resolved

avik-pal force-pushed the ap/rework_benchmarking branch 2 times, most recently from 56a2ba8 to b6cbbae Compare June 3, 2024 04:32

github-actions bot reviewed Jun 4, 2024

View reviewed changes

bench/helpers.jl Outdated Show resolved Hide resolved

avik-pal force-pushed the ap/rework_benchmarking branch from 2398571 to 8f86ec5 Compare June 4, 2024 02:01

github-actions bot reviewed Jun 4, 2024

View reviewed changes

bench/runbenchmarks.jl Outdated Show resolved Hide resolved

avik-pal force-pushed the ap/rework_benchmarking branch 3 times, most recently from 0a4b4f0 to 3045b08 Compare June 4, 2024 02:35

github-actions bot reviewed Jun 4, 2024

View reviewed changes

bench/runbenchmarks.jl Outdated Show resolved Hide resolved

avik-pal force-pushed the ap/rework_benchmarking branch 5 times, most recently from cfd6a54 to 22cc24c Compare June 4, 2024 03:34

avik-pal force-pushed the ap/rework_benchmarking branch 20 times, most recently from d5f9d22 to f68e022 Compare June 24, 2024 02:50

avik-pal added 7 commits June 23, 2024 21:44

[skip tests] [skip docs] Relax ADTypes for benchmarks

390551a

[skip tests] [skip docs] Migrate to Buildkite for benchmarks

5c68794

[skip tests] [skip docs] Add GPU benchmarking

9a5265c

[skip tests] [skip docs] Try caching the parameter file

5f2ba27

[skip tests] [skip docs] Benchmark using PkgJogger

87de081

[skip tests] [skip docs] Try reusing the cached tuning

7522dbc

[skip tests] [skip docs] More benchmarks

861adf4

avik-pal force-pushed the ap/rework_benchmarking branch from f68e022 to 861adf4 Compare June 24, 2024 04:44

avik-pal force-pushed the main branch from dcab297 to ced869f Compare June 28, 2024 17:00

avik-pal marked this pull request as draft July 25, 2024 01:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Clean up Continuous Benchmarking #675

Clean up Continuous Benchmarking #675

avik-pal commented Jun 2, 2024 •

edited

Loading

codecov bot commented Jun 2, 2024 •

edited

Loading

github-actions bot left a comment •

edited

Loading

Clean up Continuous Benchmarking #675

Are you sure you want to change the base?

Clean up Continuous Benchmarking #675

Conversation

avik-pal commented Jun 2, 2024 • edited Loading

TODOs

codecov bot commented Jun 2, 2024 • edited Loading

Codecov Report

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

Benchmark Results

avik-pal commented Jun 2, 2024 •

edited

Loading

codecov bot commented Jun 2, 2024 •

edited

Loading

github-actions bot left a comment •

edited

Loading