Repeating Tracemalloc Benchmark for accuracy #5981

trexfeathers · 2024-05-28T08:34:58Z

~~A demonstration of how we can use ASV's built-in repetition logic to improve the accuracy of tracemalloc benchmarking.~~

I welcome anyone who wants to finish this off - I have some other things I need to focus on at the moment.

Edit `2024-07-05`

Recovering from a disastrous afternoon yesterday, I believe I now have this ready to go. I have tested it and it definitely does benefit from the repeat behaviour by detecting when accuracy prevents certainty about a regression. The in-built reporting for byte measurements is also quite nice, including automatic choice between KB / MB / etcetera.

for more information, see https://pre-commit.ci

trexfeathers · 2024-06-26T14:33:31Z

I will have this reviewable by Thursday 2024-07-04

…nch_poc

benchmarks/custom_bms/README.md

codecov · 2024-07-05T11:10:59Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.78%. Comparing base (1bfe311) to head (b4cb41b).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5981   +/-   ##
=======================================
  Coverage   89.78%   89.78%           
=======================================
  Files          88       88           
  Lines       23056    23056           
  Branches     5027     5027           
=======================================
  Hits        20700    20700           
  Misses       1624     1624           
  Partials      732      732

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-07-05T11:57:23Z

⏱️ Performance Benchmark Report: `6e499f4`

Performance shifts

Full benchmark results


Benchmarks that have stayed the same:

| Change   | Before [b8571748]    | After [6e499f42]    |   Ratio | Benchmark (Parameter)                                                                        |
|----------|----------------------|---------------------|---------|----------------------------------------------------------------------------------------------|
|          | 53.1±0.6ms           | 52.8±0.5ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_COUNT(False)                               |
|          | 54.5±0.8ms           | 53.5±0.4ms          |    0.98 | aggregate_collapse.Aggregation.time_aggregated_by_COUNT(True)                                |
|          | 188±1ms              | 189±3ms             |    1    | aggregate_collapse.Aggregation.time_aggregated_by_FAST_PERCENTILE(False)                     |
|          | 190±1ms              | 189±1ms             |    1    | aggregate_collapse.Aggregation.time_aggregated_by_FAST_PERCENTILE(True)                      |
|          | 36.2±0.2ms           | 36.3±0.4ms          |    1.01 | aggregate_collapse.Aggregation.time_aggregated_by_GMEAN(False)                               |
|          | 36.9±0.5ms           | 37.1±0.3ms          |    1.01 | aggregate_collapse.Aggregation.time_aggregated_by_GMEAN(True)                                |
|          | 36.3±0.2ms           | 36.6±0.5ms          |    1.01 | aggregate_collapse.Aggregation.time_aggregated_by_HMEAN(False)                               |
|          | 37.3±0.4ms           | 37.1±0.3ms          |    1    | aggregate_collapse.Aggregation.time_aggregated_by_HMEAN(True)                                |
|          | 46.3±0.8ms           | 46.1±0.7ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_MAX(False)                                 |
|          | 46.9±0.5ms           | 47.3±0.6ms          |    1.01 | aggregate_collapse.Aggregation.time_aggregated_by_MAX(True)                                  |
|          | 119±2ms              | 119±0.7ms           |    1    | aggregate_collapse.Aggregation.time_aggregated_by_MAX_RUN(False)                             |
|          | 121±2ms              | 121±2ms             |    1    | aggregate_collapse.Aggregation.time_aggregated_by_MAX_RUN(True)                              |
|          | 51.1±0.7ms           | 50.6±0.6ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_MEAN(False)                                |
|          | 51.9±0.9ms           | 51.4±0.8ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_MEAN(True)                                 |
|          | 36.5±0.3ms           | 35.9±0.6ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_MEDIAN(False)                              |
|          | 37.1±0.2ms           | 36.8±0.2ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_MEDIAN(True)                               |
|          | 46.2±1ms             | 45.7±0.7ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_MIN(False)                                 |
|          | 47.2±0.8ms           | 46.9±0.6ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_MIN(True)                                  |
|          | 1.34±0.02s           | 1.33±0.01s          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_PEAK(False)                                |
|          | 1.33±0.01s           | 1.33±0.02s          |    1    | aggregate_collapse.Aggregation.time_aggregated_by_PEAK(True)                                 |
|          | 679±10ms             | 705±20ms            |    1.04 | aggregate_collapse.Aggregation.time_aggregated_by_PERCENTILE(False)                          |
|          | 678±10ms             | 687±10ms            |    1.01 | aggregate_collapse.Aggregation.time_aggregated_by_PERCENTILE(True)                           |
|          | 34.9±0.6ms           | 34.9±0.4ms          |    1    | aggregate_collapse.Aggregation.time_aggregated_by_PROPORTION(False)                          |
|          | 35.4±0.3ms           | 35.7±0.5ms          |    1.01 | aggregate_collapse.Aggregation.time_aggregated_by_PROPORTION(True)                           |
|          | 61.8±0.8ms           | 61.5±0.5ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_RMS(False)                                 |
|          | 62.4±0.5ms           | 62.3±0.3ms          |    1    | aggregate_collapse.Aggregation.time_aggregated_by_RMS(True)                                  |
|          | 65.6±0.6ms           | 65.7±0.6ms          |    1    | aggregate_collapse.Aggregation.time_aggregated_by_STD_DEV(False)                             |
|          | 66.8±2ms             | 66.1±0.9ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_STD_DEV(True)                              |
|          | 61.2±0.5ms           | 60.6±0.7ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_VARIANCE(False)                            |
|          | 62.2±0.9ms           | 61.7±0.6ms          |    0.99 | aggregate_collapse.Aggregation.time_aggregated_by_VARIANCE(True)                             |
|          | 20.0±1ms             | 19.7±1ms            |    0.98 | aggregate_collapse.Aggregation.time_collapsed_by_COUNT(False)                                |
|          | 23.7±4ms             | 23.1±0.3ms          |    0.98 | aggregate_collapse.Aggregation.time_collapsed_by_COUNT(True)                                 |
|          | 132±2ms              | 131±2ms             |    1    | aggregate_collapse.Aggregation.time_collapsed_by_FAST_PERCENTILE(False)                      |
|          | 145±1ms              | 144±0.9ms           |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_FAST_PERCENTILE(True)                       |
|          | 18.0±0.8ms           | 17.8±0.2ms          |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_GMEAN(False)                                |
|          | 21.6±1ms             | 21.7±1ms            |    1.01 | aggregate_collapse.Aggregation.time_collapsed_by_GMEAN(True)                                 |
|          | 18.0±1ms             | 17.8±0.4ms          |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_HMEAN(False)                                |
|          | 21.5±0.2ms           | 21.6±0.3ms          |    1.01 | aggregate_collapse.Aggregation.time_collapsed_by_HMEAN(True)                                 |
|          | 18.9±0.7ms           | 18.6±0.5ms          |    0.98 | aggregate_collapse.Aggregation.time_collapsed_by_MAX(False)                                  |
|          | 22.5±0.4ms           | 22.3±0.4ms          |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_MAX(True)                                   |
|          | 34.6±1ms             | 34.2±1ms            |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_MAX_RUN(False)                              |
|          | 38.3±0.8ms           | 37.8±0.7ms          |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_MAX_RUN(True)                               |
|          | 19.1±0.2ms           | 18.7±0.3ms          |    0.98 | aggregate_collapse.Aggregation.time_collapsed_by_MEAN(False)                                 |
|          | 22.6±0.4ms           | 22.5±0.5ms          |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_MEAN(True)                                  |
|          | 18.6±0.5ms           | 18.7±0.6ms          |    1.01 | aggregate_collapse.Aggregation.time_collapsed_by_MEDIAN(False)                               |
|          | 22.2±0.2ms           | 22.3±0.4ms          |    1.01 | aggregate_collapse.Aggregation.time_collapsed_by_MEDIAN(True)                                |
|          | 18.4±0.7ms           | 18.3±0.4ms          |    1    | aggregate_collapse.Aggregation.time_collapsed_by_MIN(False)                                  |
|          | 22.5±0.5ms           | 22.1±0.4ms          |    0.98 | aggregate_collapse.Aggregation.time_collapsed_by_MIN(True)                                   |
|          | 552±5ms              | 560±10ms            |    1.02 | aggregate_collapse.Aggregation.time_collapsed_by_PEAK(False)                                 |
|          | 557±6ms              | 557±5ms             |    1    | aggregate_collapse.Aggregation.time_collapsed_by_PEAK(True)                                  |
|          | 150±3ms              | 148±2ms             |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_PERCENTILE(False)                           |
|          | 168±2ms              | 167±2ms             |    1    | aggregate_collapse.Aggregation.time_collapsed_by_PERCENTILE(True)                            |
|          | 17.8±0.4ms           | 17.6±0.5ms          |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_PROPORTION(False)                           |
|          | 21.3±1ms             | 21.2±0.5ms          |    1    | aggregate_collapse.Aggregation.time_collapsed_by_PROPORTION(True)                            |
|          | 21.4±0.4ms           | 20.9±0.3ms          |    0.98 | aggregate_collapse.Aggregation.time_collapsed_by_RMS(False)                                  |
|          | 24.4±0.5ms           | 24.8±1ms            |    1.02 | aggregate_collapse.Aggregation.time_collapsed_by_RMS(True)                                   |
|          | 21.0±0.4ms           | 21.1±0.7ms          |    1.01 | aggregate_collapse.Aggregation.time_collapsed_by_STD_DEV(False)                              |
|          | 24.7±0.3ms           | 24.6±0.6ms          |    1    | aggregate_collapse.Aggregation.time_collapsed_by_STD_DEV(True)                               |
|          | 20.6±0.6ms           | 20.4±0.4ms          |    0.99 | aggregate_collapse.Aggregation.time_collapsed_by_VARIANCE(False)                             |
|          | 24.2±0.6ms           | 24.2±0.5ms          |    1    | aggregate_collapse.Aggregation.time_collapsed_by_VARIANCE(True)                              |
|          | 84.2±2ms             | 82.3±0.9ms          |    0.98 | aggregate_collapse.WeightedAggregation.time_w_aggregated_by_MEAN(False)                      |
|          | 83.5±0.6ms           | 83.1±1ms            |    1    | aggregate_collapse.WeightedAggregation.time_w_aggregated_by_MEAN(True)                       |
|          | 95.3±1ms             | 94.7±0.8ms          |    0.99 | aggregate_collapse.WeightedAggregation.time_w_aggregated_by_RMS(False)                       |
|          | 95.4±2ms             | 94.9±1ms            |    0.99 | aggregate_collapse.WeightedAggregation.time_w_aggregated_by_RMS(True)                        |
|          | 58.5±0.9ms           | 58.2±0.8ms          |    0.99 | aggregate_collapse.WeightedAggregation.time_w_aggregated_by_SUM(False)                       |
|          | 58.7±0.7ms           | 57.9±0.5ms          |    0.99 | aggregate_collapse.WeightedAggregation.time_w_aggregated_by_SUM(True)                        |
|          | 29.1±0.4ms           | 28.8±0.5ms          |    0.99 | aggregate_collapse.WeightedAggregation.time_w_collapsed_by_MEAN(False)                       |
|          | 32.5±0.7ms           | 32.6±0.7ms          |    1    | aggregate_collapse.WeightedAggregation.time_w_collapsed_by_MEAN(True)                        |
|          | 31.1±0.5ms           | 30.8±0.7ms          |    0.99 | aggregate_collapse.WeightedAggregation.time_w_collapsed_by_RMS(False)                        |
|          | 34.4±0.7ms           | 34.8±0.5ms          |    1.01 | aggregate_collapse.WeightedAggregation.time_w_collapsed_by_RMS(True)                         |
|          | 25.5±0.2ms           | 25.7±0.9ms          |    1    | aggregate_collapse.WeightedAggregation.time_w_collapsed_by_SUM(False)                        |
|          | 29.5±0.3ms           | 29.0±0.5ms          |    0.98 | aggregate_collapse.WeightedAggregation.time_w_collapsed_by_SUM(True)                         |
|          | 324±6ms              | 318±3ms             |    0.98 | aggregate_collapse.WeightedAggregation.time_w_collapsed_by_WPERCENTILE(False)                |
|          | 350±10ms             | 340±5ms             |    0.97 | aggregate_collapse.WeightedAggregation.time_w_collapsed_by_WPERCENTILE(True)                 |
|          | 1.13±0.01ms          | 1.10±0.01ms         |    0.97 | cube.CubeCreation.time_create(False, 'construct')                                            |
|          | 399±8μs              | 391±7μs             |    0.98 | cube.CubeCreation.time_create(False, 'instantiate')                                          |
|          | 952±20μs             | 939±6μs             |    0.99 | cube.CubeCreation.time_create(True, 'construct')                                             |
|          | 572±9μs              | 567±3μs             |    0.99 | cube.CubeCreation.time_create(True, 'instantiate')                                           |
|          | 224±6ms              | 222±3ms             |    0.99 | cube.CubeEquality.time_equality(False, False, 'all_equal')                                   |
|          | 113±2ms              | 112±1ms             |    0.99 | cube.CubeEquality.time_equality(False, False, 'coord_inequality')                            |
|          | 235±3ms              | 236±5ms             |    1    | cube.CubeEquality.time_equality(False, False, 'data_inequality')                             |
|          | 16.6±0.1μs           | 16.5±0.2μs          |    0.99 | cube.CubeEquality.time_equality(False, False, 'metadata_inequality')                         |
|          | 310±20ms             | 307±4ms             |    0.99 | cube.CubeEquality.time_equality(False, True, 'all_equal')                                    |
|          | 203±4ms              | 202±3ms             |    0.99 | cube.CubeEquality.time_equality(False, True, 'coord_inequality')                             |
|          | 321±8ms              | 317±2ms             |    0.99 | cube.CubeEquality.time_equality(False, True, 'data_inequality')                              |
|          | 16.5±0.1μs           | 16.4±0.2μs          |    1    | cube.CubeEquality.time_equality(False, True, 'metadata_inequality')                          |
|          | 222±3ms              | 221±3ms             |    0.99 | cube.CubeEquality.time_equality(True, False, 'all_equal')                                    |
|          | 113±3ms              | 112±2ms             |    0.99 | cube.CubeEquality.time_equality(True, False, 'coord_inequality')                             |
|          | 236±10ms             | 232±3ms             |    0.99 | cube.CubeEquality.time_equality(True, False, 'data_inequality')                              |
|          | 54.4±0.6μs           | 52.9±0.3μs          |    0.97 | cube.CubeEquality.time_equality(True, False, 'metadata_inequality')                          |
|          | 307±3ms              | 306±3ms             |    1    | cube.CubeEquality.time_equality(True, True, 'all_equal')                                     |
|          | 199±2ms              | 198±2ms             |    1    | cube.CubeEquality.time_equality(True, True, 'coord_inequality')                              |
|          | 317±4ms              | 315±4ms             |    1    | cube.CubeEquality.time_equality(True, True, 'data_inequality')                               |
|          | 54.0±0.8μs           | 54.9±0.4μs          |    1.02 | cube.CubeEquality.time_equality(True, True, 'metadata_inequality')                           |
|          | 403±1ns              | 410±3ns             |    1.02 | experimental.ugrid.regions_combine.CombineRegionsComputeRealData.time_compute_data(50)       |
|          | 281±2ms              | 278±2ms             |    0.99 | experimental.ugrid.regions_combine.CombineRegionsComputeRealData.time_compute_data(500)      |
|          | 14.2±0.09ms          | 14.3±0.1ms          |    1.01 | experimental.ugrid.regions_combine.CombineRegionsCreateCube.time_create_combined_cube(50)    |
|          | 16.3±0.9ms           | 16.0±0.4ms          |    0.98 | experimental.ugrid.regions_combine.CombineRegionsCreateCube.time_create_combined_cube(500)   |
|          | 104±0.4ms            | 105±1ms             |    1.01 | experimental.ugrid.regions_combine.CombineRegionsFileStreamedCalc.time_stream_file2file(50)  |
|          | 719±8ms              | 726±6ms             |    1.01 | experimental.ugrid.regions_combine.CombineRegionsFileStreamedCalc.time_stream_file2file(500) |
|          | 66.0±0.6ms           | 66.7±0.4ms          |    1.01 | experimental.ugrid.regions_combine.CombineRegionsSaveData.time_save(50)                      |
|          | 677±9ms              | 679±4ms             |    1    | experimental.ugrid.regions_combine.CombineRegionsSaveData.time_save(500)                     |
|          | 2.1752849999999997   | 2.1752849999999997  |    1    | experimental.ugrid.regions_combine.CombineRegionsSaveData.track_filesize_saved(50)           |
|          | 216.01528499999998   | 216.01528499999998  |    1    | experimental.ugrid.regions_combine.CombineRegionsSaveData.track_filesize_saved(500)          |
|          | 659±8μs              | 665±7μs             |    1.01 | import_iris.Iris.time__concatenate                                                           |
|          | 181±4μs              | 181±4μs             |    1    | import_iris.Iris.time__constraints                                                           |
|          | 111±1μs              | 113±2μs             |    1.02 | import_iris.Iris.time__data_manager                                                          |
|          | 94.2±0.9μs           | 93.0±1μs            |    0.99 | import_iris.Iris.time__deprecation                                                           |
|          | 137±1μs              | 138±1μs             |    1.01 | import_iris.Iris.time__lazy_data                                                             |
|          | 886±10μs             | 899±5μs             |    1.01 | import_iris.Iris.time__merge                                                                 |
|          | 76.4±1μs             | 76.4±0.7μs          |    1    | import_iris.Iris.time__representation                                                        |
|          | 481±4μs              | 486±4μs             |    1.01 | import_iris.Iris.time_analysis                                                               |
|          | 140±1μs              | 140±0.9μs           |    1    | import_iris.Iris.time_analysis__area_weighted                                                |
|          | 110±1μs              | 110±2μs             |    1    | import_iris.Iris.time_analysis__grid_angles                                                  |
|          | 244±3μs              | 243±1μs             |    1    | import_iris.Iris.time_analysis__interpolation                                                |
|          | 187±1μs              | 189±2μs             |    1.01 | import_iris.Iris.time_analysis__regrid                                                       |
|          | 110±0.5μs            | 112±0.8μs           |    1.01 | import_iris.Iris.time_analysis__scipy_interpolate                                            |
|          | 139±2μs              | 141±3μs             |    1.01 | import_iris.Iris.time_analysis_calculus                                                      |
|          | 324±2μs              | 327±3μs             |    1.01 | import_iris.Iris.time_analysis_cartography                                                   |
|          | 94.1±0.9μs           | 93.6±1μs            |    0.99 | import_iris.Iris.time_analysis_geomerty                                                      |
|          | 216±2μs              | 217±2μs             |    1    | import_iris.Iris.time_analysis_maths                                                         |
|          | 98.0±0.8μs           | 97.6±0.8μs          |    1    | import_iris.Iris.time_analysis_stats                                                         |
|          | 175±1μs              | 174±2μs             |    0.99 | import_iris.Iris.time_analysis_trajectory                                                    |
|          | 305±3μs              | 302±2μs             |    0.99 | import_iris.Iris.time_aux_factory                                                            |
|          | 83.1±0.8μs           | 83.6±0.5μs          |    1.01 | import_iris.Iris.time_common                                                                 |
|          | 162±1μs              | 162±4μs             |    1    | import_iris.Iris.time_common_lenient                                                         |
|          | 977±7μs              | 981±4μs             |    1    | import_iris.Iris.time_common_metadata                                                        |
|          | 133±2μs              | 132±0.6μs           |    0.99 | import_iris.Iris.time_common_mixin                                                           |
|          | 1.17±0.01ms          | 1.18±0ms            |    1.01 | import_iris.Iris.time_common_resolve                                                         |
|          | 200±1μs              | 199±2μs             |    0.99 | import_iris.Iris.time_config                                                                 |
|          | 116±1μs              | 115±0.6μs           |    0.99 | import_iris.Iris.time_coord_categorisation                                                   |
|          | 361±4μs              | 361±2μs             |    1    | import_iris.Iris.time_coord_systems                                                          |
|          | 742±10μs             | 736±3μs             |    0.99 | import_iris.Iris.time_coords                                                                 |
|          | 666±7μs              | 661±7μs             |    0.99 | import_iris.Iris.time_cube                                                                   |
|          | 227±4μs              | 226±4μs             |    0.99 | import_iris.Iris.time_exceptions                                                             |
|          | 77.9±0.5μs           | 77.3±0.5μs          |    0.99 | import_iris.Iris.time_experimental                                                           |
|          | 191±3μs              | 188±2μs             |    0.98 | import_iris.Iris.time_fileformats                                                            |
|          | 248±2μs              | 248±2μs             |    1    | import_iris.Iris.time_fileformats__ff                                                        |
|          | 2.68±0.02ms          | 2.69±0.01ms         |    1    | import_iris.Iris.time_fileformats__ff_cross_references                                       |
|          | 79.4±0.6μs           | 79.0±0.6μs          |    1    | import_iris.Iris.time_fileformats__pp_lbproc_pairs                                           |
|          | 116±1μs              | 115±0.9μs           |    0.99 | import_iris.Iris.time_fileformats_abf                                                        |
|          | 361±4μs              | 361±1μs             |    1    | import_iris.Iris.time_fileformats_cf                                                         |
|          | 5.36±0.04ms          | 5.32±0.07ms         |    0.99 | import_iris.Iris.time_fileformats_dot                                                        |
|          | 75.3±0.4μs           | 74.1±0.6μs          |    0.99 | import_iris.Iris.time_fileformats_name                                                       |
|          | 258±4μs              | 256±4μs             |    0.99 | import_iris.Iris.time_fileformats_name_loaders                                               |
|          | 119±1μs              | 118±2μs             |    0.99 | import_iris.Iris.time_fileformats_netcdf                                                     |
|          | 122±1μs              | 121±2μs             |    0.99 | import_iris.Iris.time_fileformats_nimrod                                                     |
|          | 212±4μs              | 213±3μs             |    1    | import_iris.Iris.time_fileformats_nimrod_load_rules                                          |
|          | 776±7μs              | 774±5μs             |    1    | import_iris.Iris.time_fileformats_pp                                                         |
|          | 181±1μs              | 182±3μs             |    1.01 | import_iris.Iris.time_fileformats_pp_load_rules                                              |
|          | 135±1μs              | 133±1μs             |    0.99 | import_iris.Iris.time_fileformats_pp_save_rules                                              |
|          | 509±2μs              | 512±2μs             |    1.01 | import_iris.Iris.time_fileformats_rules                                                      |
|          | 219±4μs              | 219±2μs             |    1    | import_iris.Iris.time_fileformats_structured_array_identification                            |
|          | 82.9±0.9μs           | 83.1±5μs            |    1    | import_iris.Iris.time_fileformats_um                                                         |
|          | 161±2μs              | 159±1μs             |    0.99 | import_iris.Iris.time_fileformats_um__fast_load                                              |
|          | 139±2μs              | 137±1μs             |    0.99 | import_iris.Iris.time_fileformats_um__fast_load_structured_fields                            |
|          | 77.0±0.8μs           | 75.3±0.5μs          |    0.98 | import_iris.Iris.time_fileformats_um__ff_replacement                                         |
|          | 81.8±0.7μs           | 81.4±0.9μs          |    0.99 | import_iris.Iris.time_fileformats_um__optimal_array_structuring                              |
|          | 972±5μs              | 984±60μs            |    1.01 | import_iris.Iris.time_fileformats_um_cf_map                                                  |
|          | 138±2μs              | 136±1μs             |    0.98 | import_iris.Iris.time_io                                                                     |
|          | 172±2μs              | 171±0.9μs           |    1    | import_iris.Iris.time_io_format_picker                                                       |
|          | 228±2μs              | 230±2μs             |    1.01 | import_iris.Iris.time_iris                                                                   |
|          | 130±8μs              | 128±0.8μs           |    0.99 | import_iris.Iris.time_iterate                                                                |
|          | 8.41±0.1ms           | 8.42±0.05ms         |    1    | import_iris.Iris.time_palette                                                                |
|          | 2.22±0.06ms          | 2.21±0.05ms         |    1    | import_iris.Iris.time_plot                                                                   |
|          | 105±2μs              | 104±1μs             |    1    | import_iris.Iris.time_quickplot                                                              |
|          | 2.16±0.05ms          | 2.14±0.02ms         |    0.99 | import_iris.Iris.time_std_names                                                              |
|          | 1.76±0.01ms          | 1.77±0.08ms         |    1    | import_iris.Iris.time_symbols                                                                |
|          | 34.2±1ms             | 34.0±0.7ms          |    1    | import_iris.Iris.time_tests                                                                  |
|          | 255±0.9μs            | 256±2μs             |    1    | import_iris.Iris.time_third_party_cartopy                                                    |
|          | 4.83±0.02ms          | 4.84±0.03ms         |    1    | import_iris.Iris.time_third_party_cf_units                                                   |
|          | 119±2μs              | 119±0.9μs           |    0.99 | import_iris.Iris.time_third_party_cftime                                                     |
|          | 2.77±0.01ms          | 2.78±0.01ms         |    1    | import_iris.Iris.time_third_party_matplotlib                                                 |
|          | 1.07±0.04ms          | 1.07±0.01ms         |    0.99 | import_iris.Iris.time_third_party_numpy                                                      |
|          | 170±0.6μs            | 170±1μs             |    1    | import_iris.Iris.time_third_party_scipy                                                      |
|          | 99.8±1μs             | 99.6±0.6μs          |    1    | import_iris.Iris.time_time                                                                   |
|          | 320±3μs              | 325±30μs            |    1.02 | import_iris.Iris.time_util                                                                   |
|          | 72.9±0.4μs           | 73.2±1μs            |    1    | iterate.IZip.time_izip                                                                       |
|          | 8.11±0.2ms           | 8.03±0.05ms         |    0.99 | load.LoadAndRealise.time_load((1280, 960, 5), False, 'FF')                                   |
|          | 23.6±0.8ms           | 23.9±0.2ms          |    1.01 | load.LoadAndRealise.time_load((1280, 960, 5), False, 'NetCDF')                               |
|          | 8.86±0.1ms           | 8.83±0.08ms         |    1    | load.LoadAndRealise.time_load((1280, 960, 5), False, 'PP')                                   |
|          | 8.07±0.06ms          | 8.05±0.05ms         |    1    | load.LoadAndRealise.time_load((1280, 960, 5), True, 'FF')                                    |
|          | 21.3±0.3ms           | 21.3±0.2ms          |    1    | load.LoadAndRealise.time_load((1280, 960, 5), True, 'NetCDF')                                |
|          | 8.83±0.07ms          | 8.79±0.09ms         |    1    | load.LoadAndRealise.time_load((1280, 960, 5), True, 'PP')                                    |
|          | 1.38±0.02s           | 1.36±0.03s          |    0.99 | load.LoadAndRealise.time_load((2, 2, 1000), False, 'FF')                                     |
|          | 21.0±0.2ms           | 20.7±0.3ms          |    0.98 | load.LoadAndRealise.time_load((2, 2, 1000), False, 'NetCDF')                                 |
|          | 1.52±0.01s           | 1.53±0.04s          |    1.01 | load.LoadAndRealise.time_load((2, 2, 1000), False, 'PP')                                     |
|          | 1.39±0.03s           | 1.37±0.03s          |    0.99 | load.LoadAndRealise.time_load((2, 2, 1000), True, 'FF')                                      |
|          | 20.7±0.2ms           | 20.5±0.09ms         |    0.99 | load.LoadAndRealise.time_load((2, 2, 1000), True, 'NetCDF')                                  |
|          | 1.51±0.02s           | 1.51±0.04s          |    1    | load.LoadAndRealise.time_load((2, 2, 1000), True, 'PP')                                      |
|          | 3.91±0.03ms          | 3.92±0.02ms         |    1    | load.LoadAndRealise.time_load((50, 50, 2), False, 'FF')                                      |
|          | 20.0±0.4ms           | 20.0±0.3ms          |    1    | load.LoadAndRealise.time_load((50, 50, 2), False, 'NetCDF')                                  |
|          | 4.25±0.06ms          | 4.14±0.03ms         |    0.98 | load.LoadAndRealise.time_load((50, 50, 2), False, 'PP')                                      |
|          | 3.88±0.05ms          | 3.91±0.01ms         |    1.01 | load.LoadAndRealise.time_load((50, 50, 2), True, 'FF')                                       |
|          | 20.0±0.5ms           | 19.9±0.3ms          |    0.99 | load.LoadAndRealise.time_load((50, 50, 2), True, 'NetCDF')                                   |
|          | 4.16±0.07ms          | 4.16±0.04ms         |    1    | load.LoadAndRealise.time_load((50, 50, 2), True, 'PP')                                       |
|          | 32.0±1ms             | 35.3±3ms            |    1.1  | load.LoadAndRealise.time_realise((1280, 960, 5), False, 'FF')                                |
|          | 19.1±0.2ms           | 18.9±0.4ms          |    0.99 | load.LoadAndRealise.time_realise((1280, 960, 5), False, 'NetCDF')                            |
|          | 13.4±1ms             | 14.4±4ms            |    1.08 | load.LoadAndRealise.time_realise((1280, 960, 5), False, 'PP')                                |
|          | 26.2±2ms             | 25.4±2ms            |    0.97 | load.LoadAndRealise.time_realise((1280, 960, 5), True, 'FF')                                 |
|          | 70.2±2ms             | 70.4±2ms            |    1    | load.LoadAndRealise.time_realise((1280, 960, 5), True, 'NetCDF')                             |
|          | 25.6±1ms             | 25.6±1ms            |    1    | load.LoadAndRealise.time_realise((1280, 960, 5), True, 'PP')                                 |
|          | 435±3ms              | 439±10ms            |    1.01 | load.LoadAndRealise.time_realise((2, 2, 1000), False, 'FF')                                  |
|          | 2.77±0.08ms          | 2.84±0.1ms          |    1.03 | load.LoadAndRealise.time_realise((2, 2, 1000), False, 'NetCDF')                              |
|          | 441±3ms              | 441±5ms             |    1    | load.LoadAndRealise.time_realise((2, 2, 1000), False, 'PP')                                  |
|          | 441±4ms              | 440±4ms             |    1    | load.LoadAndRealise.time_realise((2, 2, 1000), True, 'FF')                                   |
|          | 2.89±0.06ms          | 2.90±0.2ms          |    1    | load.LoadAndRealise.time_realise((2, 2, 1000), True, 'NetCDF')                               |
|          | 448±2ms              | 449±7ms             |    1    | load.LoadAndRealise.time_realise((2, 2, 1000), True, 'PP')                                   |
|          | 1.55±0.1ms           | 1.55±0.08ms         |    1    | load.LoadAndRealise.time_realise((50, 50, 2), False, 'FF')                                   |
|          | 2.80±0.05ms          | 2.77±0.1ms          |    0.99 | load.LoadAndRealise.time_realise((50, 50, 2), False, 'NetCDF')                               |
|          | 1.51±0.1ms           | 1.57±0.03ms         |    1.04 | load.LoadAndRealise.time_realise((50, 50, 2), False, 'PP')                                   |
|          | 1.53±0.06ms          | 1.52±0.1ms          |    0.99 | load.LoadAndRealise.time_realise((50, 50, 2), True, 'FF')                                    |
|          | 2.77±0.1ms           | 2.85±0.1ms          |    1.03 | load.LoadAndRealise.time_realise((50, 50, 2), True, 'NetCDF')                                |
|          | 1.54±0.09ms          | 1.59±0.1ms          |    1.03 | load.LoadAndRealise.time_realise((50, 50, 2), True, 'PP')                                    |
|          | 361±4ms              | 358±7ms             |    0.99 | load.ManyVars.time_many_var_load                                                             |
|          | 8.23±0.08ms          | 8.29±0.1ms          |    1.01 | load.STASHConstraint.time_stash_constraint((1280, 960, 5), 'FF')                             |
|          | 9.02±0.08ms          | 8.95±0.09ms         |    0.99 | load.STASHConstraint.time_stash_constraint((1280, 960, 5), 'PP')                             |
|          | 1.36±0.01s           | 1.37±0.02s          |    1.01 | load.STASHConstraint.time_stash_constraint((2, 2, 1000), 'FF')                               |
|          | 1.53±0.02s           | 1.53±0.03s          |    1    | load.STASHConstraint.time_stash_constraint((2, 2, 1000), 'PP')                               |
|          | 3.93±0.03ms          | 3.99±0.2ms          |    1.01 | load.STASHConstraint.time_stash_constraint((2, 2, 2), 'FF')                                  |
|          | 4.21±0.03ms          | 4.25±0.04ms         |    1.01 | load.STASHConstraint.time_stash_constraint((2, 2, 2), 'PP')                                  |
|          | 8.05±0.1ms           | 8.05±0.04ms         |    1    | load.StructuredFF.time_structured_load((1280, 960, 5), False)                                |
|          | 4.70±0.06ms          | 4.72±0.03ms         |    1    | load.StructuredFF.time_structured_load((1280, 960, 5), True)                                 |
|          | 1.35±0.02s           | 1.37±0.03s          |    1.01 | load.StructuredFF.time_structured_load((2, 2, 1000), False)                                  |
|          | 363±4ms              | 364±3ms             |    1    | load.StructuredFF.time_structured_load((2, 2, 1000), True)                                   |
|          | 3.89±0.04ms          | 3.91±0.02ms         |    1.01 | load.StructuredFF.time_structured_load((2, 2, 2), False)                                     |
|          | 3.52±0.02ms          | 3.55±0.01ms         |    1.01 | load.StructuredFF.time_structured_load((2, 2, 2), True)                                      |
|          | 144±1ms              | 145±4ms             |    1.01 | load.TimeConstraint.time_time_constraint(20, 'FF')                                           |
|          | 22.9±0.2ms           | 23.3±0.2ms          |    1.02 | load.TimeConstraint.time_time_constraint(20, 'NetCDF')                                       |
|          | 161±1ms              | 160±1ms             |    1    | load.TimeConstraint.time_time_constraint(20, 'PP')                                           |
|          | 28.6±0.1ms           | 28.9±0.1ms          |    1.01 | load.TimeConstraint.time_time_constraint(3, 'FF')                                            |
|          | 22.8±0.2ms           | 22.6±0.2ms          |    0.99 | load.TimeConstraint.time_time_constraint(3, 'NetCDF')                                        |
|          | 31.0±0.08ms          | 31.0±0.1ms          |    1    | load.TimeConstraint.time_time_constraint(3, 'PP')                                            |
|          | 18.6±1ms             | 17.1±0.3ms          |    0.92 | load.ugrid.BasicLoading.time_load_file(1)                                                    |
|          | 41.1±1ms             | 41.2±0.4ms          |    1    | load.ugrid.BasicLoading.time_load_file(200000)                                               |
|          | 14.1±0.3ms           | 14.0±0.3ms          |    0.99 | load.ugrid.BasicLoading.time_load_mesh(1)                                                    |
|          | 22.2±0.6ms           | 21.5±0.2ms          |    0.97 | load.ugrid.BasicLoading.time_load_mesh(200000)                                               |
|          | 17.4±0.3ms           | 17.2±0.3ms          |    0.99 | load.ugrid.BasicLoadingTime.time_load_file(1)                                                |
|          | 20.3±0.2ms           | 19.9±0.5ms          |    0.98 | load.ugrid.BasicLoadingTime.time_load_file(200000)                                           |
|          | 14.2±0.4ms           | 14.3±0.2ms          |    1.01 | load.ugrid.BasicLoadingTime.time_load_mesh(1)                                                |
|          | 17.0±0.2ms           | 16.6±0.5ms          |    0.97 | load.ugrid.BasicLoadingTime.time_load_mesh(200000)                                           |
|          | 18.2±0.3ms           | 18.2±0.2ms          |    1    | load.ugrid.Callback.time_load_file_callback(1)                                               |
|          | 49.7±0.4ms           | 49.7±0.9ms          |    1    | load.ugrid.Callback.time_load_file_callback(200000)                                          |
|          | 18.3±0.3ms           | 18.4±0.5ms          |    1.01 | load.ugrid.CallbackTime.time_load_file_callback(1)                                           |
|          | 22.0±0.5ms           | 21.6±0.2ms          |    0.98 | load.ugrid.CallbackTime.time_load_file_callback(200000)                                      |
|          | 2.74±0.1ms           | 2.67±0.1ms          |    0.97 | load.ugrid.DataRealisation.time_realise_data(10000)                                          |
|          | 5.42±0.8ms           | 5.39±0.1ms          |    0.99 | load.ugrid.DataRealisation.time_realise_data(200000)                                         |
|          | 37.9±1ms             | 37.0±1ms            |    0.98 | load.ugrid.DataRealisationTime.time_realise_data(10000)                                      |
|          | 799±10ms             | 799±6ms             |    1    | load.ugrid.DataRealisationTime.time_realise_data(200000)                                     |
|          | 132±1ms              | 131±1ms             |    0.99 | merge_concat.Concatenate.time_concatenate                                                    |
|          | 47.4±0.4ms           | 47.5±0.8ms          |    1    | merge_concat.Merge.time_merge                                                                |
|          | 6.69±0.09ms          | 6.56±0.07ms         |    0.98 | plot.AuxSort.time_aux_sort                                                                   |
|          | 73.7±5ms             | 76.4±3ms            |    1.04 | regridding.CurvilinearRegridding.time_regrid_pic                                             |
|          | 97.1±1ms             | 97.4±0.5ms          |    1    | regridding.HorizontalChunkedRegridding.time_regrid_area_w                                    |
|          | 47.1±2ms             | 47.5±2ms            |    1.01 | regridding.HorizontalChunkedRegridding.time_regrid_area_w_new_grid                           |
|          | 4.07±0.04ms          | 4.03±0.03ms         |    0.99 | save.NetcdfSave.time_netcdf_save_cube(50, False)                                             |
|          | 71.0±0.5ms           | 71.3±0.5ms          |    1    | save.NetcdfSave.time_netcdf_save_cube(50, True)                                              |
|          | 52.1±2ms             | 52.0±0.5ms          |    1    | save.NetcdfSave.time_netcdf_save_cube(600, False)                                            |
|          | 564±5ms              | 563±3ms             |    1    | save.NetcdfSave.time_netcdf_save_cube(600, True)                                             |
|          | 89.2±0.5ns           | 90.6±0.9ns          |    1.02 | save.NetcdfSave.time_netcdf_save_mesh(50, False)                                             |
|          | 54.6±0.3ms           | 55.1±0.4ms          |    1.01 | save.NetcdfSave.time_netcdf_save_mesh(50, True)                                              |
|          | 92.4±1ns             | 92.1±2ns            |    1    | save.NetcdfSave.time_netcdf_save_mesh(600, False)                                            |
|          | 494±2ms              | 496±3ms             |    1    | save.NetcdfSave.time_netcdf_save_mesh(600, True)                                             |
|          | 42.2±2ms             | 42.5±1ms            |    1.01 | stats.PearsonR.time_lazy                                                                     |
|          | 18.8±0.4ms           | 19.0±0.3ms          |    1.01 | stats.PearsonR.time_real                                                                     |
|          | 23.2±1ms             | 22.7±1ms            |    0.98 | trajectory.TrajectoryInterpolation.time_trajectory_linear                                    |
|          | 60.8±0.3ms           | 60.6±0.5ms          |    1    | trajectory.TrajectoryInterpolation.time_trajectory_nearest                                   |

Generated by GHA run 9807338050

…nch_poc

github-actions · 2024-07-08T12:54:10Z

⏱️ Performance Benchmark Report: `027501f`

Performance shifts

Full benchmark results


Benchmarks that have stayed the same:

| Change   | Before [d3ec6174]    | After [027501ff]    |   Ratio | Benchmark (Parameter)                |
|----------|----------------------|---------------------|---------|--------------------------------------|
|          | 1.2±0.3M             | 1.2±0.3M            |       1 | merge_concat.Merge.tracemalloc_merge |

Generated by GHA run 9839942792

This reverts commit 9dffa1b.

trexfeathers · 2024-08-15T08:52:57Z

We're up to 7 issues that this will close 📈

HGWright

Thanks @trexfeathers hopefully this removes some of the noise from memory benchmarks going forward!

trexfeathers · 2024-09-02T09:55:52Z

Thanks @HGWright!

trexfeathers added 3 commits May 23, 2024 14:11

PoC custom benchmark.

bc74b1c

Working tracemalloc subclass.

74c755a

Docstrings and comments.

926a8ac

trexfeathers added Type: Performance Type: Testing labels May 28, 2024

trexfeathers mentioned this pull request May 28, 2024

Holding space for memory benchmarking issues #5967

Closed

[pre-commit.ci] auto fixes from pre-commit.com hooks

ce5f630

for more information, see https://pre-commit.ci

trexfeathers mentioned this pull request Jun 14, 2024

Performance Shift(s): e92e36ac #6001

Closed

trexfeathers assigned trexfeathers and HGWright Jun 26, 2024

trexfeathers added 17 commits July 4, 2024 10:06

Merge remote-tracking branch 'upstream/main' into custom_bench_poc

c120acc

Finalised code.

0b9fc98

Make custom benchmarks installable.

bd2921d

Experiment.

f55b1e9

Re-write following testing.

0c1938e

Replace pyproject with setup.py.

94c27ef

Testing.

db5bd2f

Make testing smaller.

9928094

Better testing.

5b0f3e7

Better testing 2.

05e6115

Better testing 3.

3f6ae4e

Better testing 4.

c0199e5

Better testing 5.

650e870

Remove testing.

8d26eab

Remove tracemalloc decorator.

9e9aa06

Docs tidy-up.

291addb

Merge remote-tracking branch 'origin/custom_bench_poc' into custom_be…

f095b3c

…nch_poc

trexfeathers marked this pull request as ready for review July 5, 2024 11:01

Merge branch 'main' into custom_bench_poc

4733274

trexfeathers changed the title ~~Custom Tracemalloc Benchmark POC~~ Repeating Tracemalloc Benchmark for accuracy Jul 5, 2024

trexfeathers added the benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts label Jul 5, 2024

trexfeathers commented Jul 5, 2024

View reviewed changes

benchmarks/custom_bms/README.md Outdated Show resolved Hide resolved

trexfeathers added 4 commits July 8, 2024 13:48

Restructure and use custom install script.

4e72cef

Temporary quick demo.

9dffa1b

Merge remote-tracking branch 'origin/custom_bench_poc' into custom_be…

8ec3d98

…nch_poc

Merge remote-tracking branch 'upstream/main' into custom_bench_poc

c957f5b

trexfeathers added benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts and removed benchmark_this Request that this pull request be benchmarked to check if it introduces performance shifts labels Jul 8, 2024

Revert "Temporary quick demo."

2ef421b

This reverts commit 9dffa1b.

This was linked to issues Jul 18, 2024

Performance Shift(s): c66703a0 #6060

Closed

Performance Shift(s): 17dd34a7 #6059

Closed

Performance Shift(s): 8604bb80 #6058

Closed

trexfeathers linked an issue Aug 5, 2024 that may be closed by this pull request

Performance Shift(s): db77e6ec #6103

Closed

bjlittle unassigned trexfeathers Aug 7, 2024

This was linked to issues Aug 7, 2024

Holding space for memory benchmarking issues #5967

Closed

Performance Shift(s): c88b4076 #6114

Closed

trexfeathers linked an issue Aug 15, 2024 that may be closed by this pull request

Performance Shift(s): b8f554f7 #6118

Open

pp-mo mentioned this pull request Aug 16, 2024

Performance Shift(s): b8f554f7 #6118

Open

This was linked to issues Aug 27, 2024

Performance Shift(s): 773c55ba #6130

Closed

Performance Shift(s): 9a23aa0f #6132

Closed

Merge remote-tracking branch 'upstream/main' into custom_bench_poc

b4cb41b

HGWright approved these changes Aug 30, 2024

View reviewed changes

HGWright merged commit 5cc6737 into SciTools:main Aug 30, 2024
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repeating Tracemalloc Benchmark for accuracy #5981

Repeating Tracemalloc Benchmark for accuracy #5981

trexfeathers commented May 28, 2024 •

edited

Loading

trexfeathers commented Jun 26, 2024 •

edited

Loading

codecov bot commented Jul 5, 2024 •

edited

Loading

github-actions bot commented Jul 5, 2024

github-actions bot commented Jul 8, 2024

trexfeathers commented Aug 15, 2024

HGWright left a comment

trexfeathers commented Sep 2, 2024

Repeating Tracemalloc Benchmark for accuracy #5981

Repeating Tracemalloc Benchmark for accuracy #5981

Conversation

trexfeathers commented May 28, 2024 • edited Loading

Edit 2024-07-05

trexfeathers commented Jun 26, 2024 • edited Loading

codecov bot commented Jul 5, 2024 • edited Loading

Codecov Report

github-actions bot commented Jul 5, 2024

⏱️ Performance Benchmark Report: 6e499f4

github-actions bot commented Jul 8, 2024

⏱️ Performance Benchmark Report: 027501f

trexfeathers commented Aug 15, 2024

HGWright left a comment

Choose a reason for hiding this comment

trexfeathers commented Sep 2, 2024

trexfeathers commented May 28, 2024 •

edited

Loading

Edit `2024-07-05`

trexfeathers commented Jun 26, 2024 •

edited

Loading

codecov bot commented Jul 5, 2024 •

edited

Loading

⏱️ Performance Benchmark Report: `6e499f4`

⏱️ Performance Benchmark Report: `027501f`