changed
CHANGELOG.md
|
|
@@ -5,6 +5,18 @@ All notable changes to this project will be documented in this file.
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+ ## 1.2.0 (2023-11-09)
|
|
9
|
+
|
|
10
|
+ Long time, huh? 😉 I'm not gonna repeat all that, but I'm happy benchee is in a place where it just works and doesn't need too much support. Biggest feature here is the implementation of the `Table.Reader` protocol for a better Livebook experience.
|
|
11
|
+
|
|
12
|
+ ### Features (User Facing)
|
|
13
|
+ * Trying to benchmark evaluated functions will now result in a warning. Thanks [@BrooklinJazz](https://github.com/BrooklinJazz), [@czrpb](https://github.com/czrpb), [@aar2dee2](https://github.com/aar2dee2),[@ReecesPeanutButterCodes](https://github.com/ReecesPeanutButterCodes).
|
|
14
|
+ * Add support for the [`Table.Reader`](https://hexdocs.pm/table/Table.html) protocol so that benchee works out of the box in Livebook. See [#369](https://github.com/bencheeorg/benchee/pull/369), big shoutout to [@akoutmos](https://github.com/akoutmos) and sorry for keeping it unreleased for so long.
|
|
15
|
+
|
|
16
|
+ ### Bugfixes (User Facing)
|
|
17
|
+ * Removed Elixir 1.16 compiler warnings around +0.0 and -0.0. Thanks [@tomciop](https://github.com/tomciopp).
|
|
18
|
+ * Building an escript and running it is fixed. See [#384](https://github.com/bencheeorg/benchee/issues/384) and thanks [@Munksgaard](https://github.com/Munksgaard)!
|
|
19
|
+
|
|
8
20
|
## 1.1.0 (2022-03-08)
|
|
9
21
|
|
|
10
22
|
Long time, huh? I'm sorry, combination of priorities, difficult to fix bugs, stress and arm problems kept me away too long.
|
|
|
@@ -14,8 +26,11 @@ This release brings major features long developed and now finally released (redu
|
|
14
26
|
* Reduction counting/measurements was implemented. Basically, it's a rather stable unit of execution implemented by the BEAM that measures in more abstract manner how much work was done. It's helpful, as it shouldn't be affected by load on the system. Check out [the docs](https://github.com/bencheeorg/benchee#measuring-reductions).
|
|
15
27
|
* You can now dive straight into profiling from your benchmarks by using the profiling feature. See [the docs](https://github.com/bencheeorg/benchee#profiling-after-a-run) - thanks [@pablocostass](https://github.com/pablocostass)
|
|
16
28
|
|
|
29
|
+ ### Default Behavior Changes (User Facing)
|
|
30
|
+ * measuring function call overhead is now turned off by default, as it only helps with extreme nano benchmarks and has some potential for causing wrong results, so it should only be opt in. We now also print out the measured overhead for vsibility.
|
|
31
|
+
|
|
17
32
|
### Bugfixes (User Facing)
|
|
18
|
- * Benchee now correctly looks for the time resolution as reported by `:erlang.system_info(:os_monotonic_time_source)` to accomodate when determining if a measurement is "precise" enough. Benche also works around an erlang bug we discovered present in erlang <= 22.2. [Issue for reference](https://github.com/bencheeorg/benchee/issues/313).
|
|
33
|
+ * Benchee now correctly looks for the time resolution as reported by `:erlang.system_info(:os_monotonic_time_source)` to accommodate when determining if a measurement is "precise" enough. Benche also works around an erlang bug we discovered present in erlang <= 22.2. [Issue for reference](https://github.com/bencheeorg/benchee/issues/313).
|
|
19
34
|
* The annoying stacktrace warning has been removed - thanks [@mad42](https://github.com/mad42)
|
|
20
35
|
|
|
21
36
|
### Noteworthy
|
changed
README.md
|
|
@@ -67,7 +67,7 @@ The aforementioned [plugins](#plugins) like [benchee_html](https://github.com/be
|
|
67
67
|
## Table of Contents
|
|
68
68
|
|
|
69
69
|
- [Features](#features)
|
|
70
|
- - [Statitics](#statitics)
|
|
70
|
+ - [Statistics](#statistics)
|
|
71
71
|
- [Installation](#installation)
|
|
72
72
|
- [Usage](#usage)
|
|
73
73
|
- [Configuration](#configuration)
|
|
|
@@ -104,7 +104,7 @@ The aforementioned [plugins](#plugins) like [benchee_html](https://github.com/be
|
|
104
104
|
|
|
105
105
|
* first runs the functions for a given warmup time without recording the results, to simulate a _"warm"/running_ system
|
|
106
106
|
* [measures memory usage](#measuring-memory-consumption)
|
|
107
|
- * provides you with lots of [statistics](#statitics)
|
|
107
|
+ * provides you with lots of [statistics](#statistics)
|
|
108
108
|
* plugin/extensible friendly architecture so you can use different formatters to display benchmarking results as [HTML, markdown, JSON and more](#plugins)
|
|
109
109
|
* measure not only [time](#measuring-time), but also measure [memory](#measuring-memory-consumption) and [reductions](#measuring-reductions)
|
|
110
110
|
* [profile](#profiling-after-a-run) right after benchmarking
|
|
|
@@ -115,7 +115,7 @@ The aforementioned [plugins](#plugins) like [benchee_html](https://github.com/be
|
|
115
115
|
* execute benchmark jobs in parallel to gather more results in the same time, or simulate a system under load
|
|
116
116
|
* well documented & well tested
|
|
117
117
|
|
|
118
|
- ### Statitics
|
|
118
|
+ ### Statistics
|
|
119
119
|
|
|
120
120
|
Provides you with the following **statistical data**:
|
|
121
121
|
|
|
|
@@ -146,7 +146,7 @@ end
|
|
146
146
|
|
|
147
147
|
Install via `mix deps.get` and then happy benchmarking as described in [Usage](#usage) :)
|
|
148
148
|
|
|
149
|
- Elixir versions supported/tested against are 1.6+. It _should_ theoretically still work with 1.4 & 1.5 but we don't actively test/support it.
|
|
149
|
+ Elixir versions supported/tested against are 1.7+. It _should_ theoretically still work with 1.4, 1.5 & 1.6 but we don't actively test/support it as installing them on current CI systems is not supported out of the box.
|
|
150
150
|
|
|
151
151
|
## Usage
|
|
152
152
|
|
|
|
@@ -338,7 +338,7 @@ Benchee.run(
|
|
338
338
|
%{
|
|
339
339
|
"something_great" => fn -> cool_stuff end
|
|
340
340
|
},
|
|
341
|
- memory_time: 2
|
|
341
|
+ reduction_time: 2
|
|
342
342
|
)
|
|
343
343
|
```
|
|
344
344
|
|
|
|
@@ -526,6 +526,8 @@ Enum."-map/2-lists^map/1-0-"/2 10001 26.38 2282 0.23
|
|
526
526
|
:lists.do_flatten/2 40001 57.24 4951 0.12
|
|
527
527
|
```
|
|
528
528
|
|
|
529
|
+ **Note about after_each hooks:** `after_each` hooks currently don't work when profiling a function, as they are not passed the return value of the function after the profiling run. It's already fixed on the elixir side and is waiting for release, likely in 1.14. It should then just work.
|
|
530
|
+
|
|
529
531
|
### Saving, loading and comparing previous runs
|
|
530
532
|
|
|
531
533
|
Benchee can store the results of previous runs in a file and then load them again to compare them. For example this is useful to compare what was recorded on the main branch against a branch with performance improvements.
|
changed
hex_metadata.config
|
|
@@ -1,63 +1,69 @@
|
|
1
|
- {<<"app">>,<<"benchee">>}.
|
|
2
|
- {<<"build_tools">>,[<<"mix">>]}.
|
|
3
|
- {<<"description">>,
|
|
4
|
- <<"Versatile (micro) benchmarking that is extensible. Get statistics such as:\naverage, iterations per second, standard deviation and the median.">>}.
|
|
5
|
- {<<"elixir">>,<<"~> 1.6">>}.
|
|
6
|
- {<<"files">>,
|
|
7
|
- [<<"lib">>,<<"lib/benchee.ex">>,<<"lib/benchee">>,
|
|
8
|
- <<"lib/benchee/scenario_loader.ex">>,
|
|
9
|
- <<"lib/benchee/relative_statistics.ex">>,<<"lib/benchee/scenario.ex">>,
|
|
10
|
- <<"lib/benchee/collection_data.ex">>,<<"lib/benchee/statistics.ex">>,
|
|
11
|
- <<"lib/benchee/system.ex">>,<<"lib/benchee/formatter.ex">>,
|
|
12
|
- <<"lib/benchee/formatters">>,<<"lib/benchee/formatters/tagged_save.ex">>,
|
|
13
|
- <<"lib/benchee/formatters/console.ex">>,
|
|
14
|
- <<"lib/benchee/formatters/console">>,
|
|
15
|
- <<"lib/benchee/formatters/console/reductions.ex">>,
|
|
16
|
- <<"lib/benchee/formatters/console/memory.ex">>,
|
|
17
|
- <<"lib/benchee/formatters/console/helpers.ex">>,
|
|
18
|
- <<"lib/benchee/formatters/console/run_time.ex">>,
|
|
19
|
- <<"lib/benchee/configuration.ex">>,<<"lib/benchee/profile.ex">>,
|
|
20
|
- <<"lib/benchee/conversion">>,<<"lib/benchee/conversion/duration.ex">>,
|
|
21
|
- <<"lib/benchee/conversion/format.ex">>,
|
|
22
|
- <<"lib/benchee/conversion/count.ex">>,
|
|
23
|
- <<"lib/benchee/conversion/memory.ex">>,<<"lib/benchee/conversion/unit.ex">>,
|
|
24
|
- <<"lib/benchee/conversion/deviation_percent.ex">>,
|
|
25
|
- <<"lib/benchee/conversion/scale.ex">>,<<"lib/benchee/benchmark.ex">>,
|
|
26
|
- <<"lib/benchee/conversion.ex">>,<<"lib/benchee/output">>,
|
|
27
|
- <<"lib/benchee/output/benchmark_printer.ex">>,
|
|
28
|
- <<"lib/benchee/output/profile_printer.ex">>,<<"lib/benchee/suite.ex">>,
|
|
29
|
- <<"lib/benchee/benchmark">>,<<"lib/benchee/benchmark/collect">>,
|
|
30
|
- <<"lib/benchee/benchmark/collect/reductions.ex">>,
|
|
31
|
- <<"lib/benchee/benchmark/collect/memory.ex">>,
|
|
32
|
- <<"lib/benchee/benchmark/collect/profile.ex">>,
|
|
33
|
- <<"lib/benchee/benchmark/collect/time.ex">>,
|
|
34
|
- <<"lib/benchee/benchmark/hooks.ex">>,
|
|
35
|
- <<"lib/benchee/benchmark/repeated_measurement.ex">>,
|
|
36
|
- <<"lib/benchee/benchmark/run_once.ex">>,
|
|
37
|
- <<"lib/benchee/benchmark/runner.ex">>,
|
|
38
|
- <<"lib/benchee/benchmark/scenario_context.ex">>,
|
|
39
|
- <<"lib/benchee/benchmark/function_call_overhead.ex">>,
|
|
40
|
- <<"lib/benchee/benchmark/collect.ex">>,<<"lib/benchee/utility">>,
|
|
41
|
- <<"lib/benchee/utility/parallel.ex">>,<<"lib/benchee/utility/repeat_n.ex">>,
|
|
42
|
- <<"lib/benchee/utility/erlang_version.ex">>,
|
|
43
|
- <<"lib/benchee/utility/file_creation.ex">>,
|
|
44
|
- <<"lib/benchee/utility/deep_convert.ex">>,<<".formatter.exs">>,
|
|
45
|
- <<"mix.exs">>,<<"README.md">>,<<"LICENSE.md">>,<<"CHANGELOG.md">>]}.
|
|
46
|
- {<<"licenses">>,[<<"MIT">>]}.
|
|
47
1
|
{<<"links">>,
|
|
48
2
|
[{<<"Blog posts">>,<<"https://pragtob.wordpress.com/tag/benchee/">>},
|
|
49
3
|
{<<"Changelog">>,<<"https://hexdocs.pm/benchee/changelog.html">>},
|
|
50
4
|
{<<"GitHub">>,<<"https://github.com/bencheeorg/benchee">>}]}.
|
|
51
5
|
{<<"name">>,<<"benchee">>}.
|
|
6
|
+ {<<"version">>,<<"1.2.0">>}.
|
|
7
|
+ {<<"description">>,
|
|
8
|
+ <<"Versatile (micro) benchmarking that is extensible. Get statistics such as:\naverage, iterations per second, standard deviation and the median.">>}.
|
|
9
|
+ {<<"elixir">>,<<"~> 1.6">>}.
|
|
10
|
+ {<<"app">>,<<"benchee">>}.
|
|
11
|
+ {<<"licenses">>,[<<"MIT">>]}.
|
|
52
12
|
{<<"requirements">>,
|
|
53
|
- [[{<<"app">>,<<"deep_merge">>},
|
|
54
|
- {<<"name">>,<<"deep_merge">>},
|
|
13
|
+ [[{<<"name">>,<<"table">>},
|
|
14
|
+ {<<"app">>,<<"table">>},
|
|
15
|
+ {<<"optional">>,true},
|
|
16
|
+ {<<"requirement">>,<<"~> 0.1.0">>},
|
|
17
|
+ {<<"repository">>,<<"hexpm">>}],
|
|
18
|
+ [{<<"name">>,<<"deep_merge">>},
|
|
19
|
+ {<<"app">>,<<"deep_merge">>},
|
|
55
20
|
{<<"optional">>,false},
|
|
56
|
- {<<"repository">>,<<"hexpm">>},
|
|
57
|
- {<<"requirement">>,<<"~> 1.0">>}],
|
|
58
|
- [{<<"app">>,<<"statistex">>},
|
|
59
|
- {<<"name">>,<<"statistex">>},
|
|
21
|
+ {<<"requirement">>,<<"~> 1.0">>},
|
|
22
|
+ {<<"repository">>,<<"hexpm">>}],
|
|
23
|
+ [{<<"name">>,<<"statistex">>},
|
|
24
|
+ {<<"app">>,<<"statistex">>},
|
|
60
25
|
{<<"optional">>,false},
|
|
61
|
- {<<"repository">>,<<"hexpm">>},
|
|
62
|
- {<<"requirement">>,<<"~> 1.0">>}]]}.
|
|
63
|
- {<<"version">>,<<"1.1.0">>}.
|
|
26
|
+ {<<"requirement">>,<<"~> 1.0">>},
|
|
27
|
+ {<<"repository">>,<<"hexpm">>}]]}.
|
|
28
|
+ {<<"files">>,
|
|
29
|
+ [<<"lib">>,<<"lib/benchee">>,<<"lib/benchee/conversion.ex">>,
|
|
30
|
+ <<"lib/benchee/scenario.ex">>,<<"lib/benchee/suite.ex">>,
|
|
31
|
+ <<"lib/benchee/configuration.ex">>,<<"lib/benchee/formatter.ex">>,
|
|
32
|
+ <<"lib/benchee/system.ex">>,<<"lib/benchee/profile.ex">>,
|
|
33
|
+ <<"lib/benchee/conversion">>,<<"lib/benchee/conversion/memory.ex">>,
|
|
34
|
+ <<"lib/benchee/conversion/unit.ex">>,
|
|
35
|
+ <<"lib/benchee/conversion/duration.ex">>,
|
|
36
|
+ <<"lib/benchee/conversion/scale.ex">>,<<"lib/benchee/conversion/count.ex">>,
|
|
37
|
+ <<"lib/benchee/conversion/deviation_percent.ex">>,
|
|
38
|
+ <<"lib/benchee/conversion/format.ex">>,<<"lib/benchee/utility">>,
|
|
39
|
+ <<"lib/benchee/utility/deep_convert.ex">>,
|
|
40
|
+ <<"lib/benchee/utility/repeat_n.ex">>,<<"lib/benchee/utility/parallel.ex">>,
|
|
41
|
+ <<"lib/benchee/utility/erlang_version.ex">>,
|
|
42
|
+ <<"lib/benchee/utility/file_creation.ex">>,
|
|
43
|
+ <<"lib/benchee/scenario_loader.ex">>,<<"lib/benchee/statistics.ex">>,
|
|
44
|
+ <<"lib/benchee/collection_data.ex">>,
|
|
45
|
+ <<"lib/benchee/relative_statistics.ex">>,<<"lib/benchee/benchmark.ex">>,
|
|
46
|
+ <<"lib/benchee/output">>,<<"lib/benchee/output/profile_printer.ex">>,
|
|
47
|
+ <<"lib/benchee/output/benchmark_printer.ex">>,<<"lib/benchee/formatters">>,
|
|
48
|
+ <<"lib/benchee/formatters/console">>,
|
|
49
|
+ <<"lib/benchee/formatters/console/run_time.ex">>,
|
|
50
|
+ <<"lib/benchee/formatters/console/helpers.ex">>,
|
|
51
|
+ <<"lib/benchee/formatters/console/memory.ex">>,
|
|
52
|
+ <<"lib/benchee/formatters/console/reductions.ex">>,
|
|
53
|
+ <<"lib/benchee/formatters/console.ex">>,
|
|
54
|
+ <<"lib/benchee/formatters/tagged_save.ex">>,<<"lib/benchee/benchmark">>,
|
|
55
|
+ <<"lib/benchee/benchmark/runner.ex">>,
|
|
56
|
+ <<"lib/benchee/benchmark/scenario_context.ex">>,
|
|
57
|
+ <<"lib/benchee/benchmark/run_once.ex">>,
|
|
58
|
+ <<"lib/benchee/benchmark/collect.ex">>,
|
|
59
|
+ <<"lib/benchee/benchmark/repeated_measurement.ex">>,
|
|
60
|
+ <<"lib/benchee/benchmark/function_call_overhead.ex">>,
|
|
61
|
+ <<"lib/benchee/benchmark/collect">>,
|
|
62
|
+ <<"lib/benchee/benchmark/collect/memory.ex">>,
|
|
63
|
+ <<"lib/benchee/benchmark/collect/profile.ex">>,
|
|
64
|
+ <<"lib/benchee/benchmark/collect/reductions.ex">>,
|
|
65
|
+ <<"lib/benchee/benchmark/collect/time.ex">>,
|
|
66
|
+ <<"lib/benchee/benchmark/hooks.ex">>,<<"lib/benchee.ex">>,
|
|
67
|
+ <<".formatter.exs">>,<<"mix.exs">>,<<"README.md">>,<<"LICENSE.md">>,
|
|
68
|
+ <<"CHANGELOG.md">>]}.
|
|
69
|
+ {<<"build_tools">>,[<<"mix">>]}.
|
changed
lib/benchee.ex
|
|
@@ -39,7 +39,7 @@ for {module, moduledoc} <- [{Benchee, elixir_doc}, {:benchee, erlang_doc}] do
|
|
39
39
|
time: 3
|
|
40
40
|
)
|
|
41
41
|
"""
|
|
42
|
- @spec run(map, keyword) :: any
|
|
42
|
+ @spec run(map, keyword) :: Benchee.Suite.t()
|
|
43
43
|
def run(jobs, config \\ []) when is_list(config) do
|
|
44
44
|
config
|
|
45
45
|
|> Benchee.init()
|
changed
lib/benchee/benchmark.ex
|
|
@@ -23,15 +23,34 @@ defmodule Benchee.Benchmark do
|
|
23
23
|
If there are inputs in the suite's config, a scenario will be added for the given
|
|
24
24
|
function for each input.
|
|
25
25
|
"""
|
|
26
|
- @spec benchmark(Suite.t(), Suite.key(), Scenario.benchmarking_function(), module) :: Suite.t()
|
|
27
|
- def benchmark(suite = %Suite{scenarios: scenarios}, job_name, function, printer \\ Printer) do
|
|
26
|
+ @spec benchmark(Suite.t(), Suite.key(), Scenario.to_benchmark(), module) :: Suite.t()
|
|
27
|
+ def benchmark(
|
|
28
|
+ suite = %Suite{scenarios: scenarios},
|
|
29
|
+ job_name,
|
|
30
|
+ to_be_benchmark,
|
|
31
|
+ printer \\ Printer
|
|
32
|
+ ) do
|
|
33
|
+ warn_if_evaluated(to_be_benchmark, job_name, printer)
|
|
34
|
+
|
|
28
35
|
normalized_name = to_string(job_name)
|
|
29
36
|
|
|
30
37
|
if duplicate?(scenarios, normalized_name) do
|
|
31
38
|
printer.duplicate_benchmark_warning(normalized_name)
|
|
32
39
|
suite
|
|
33
40
|
else
|
|
34
|
- add_scenario(suite, normalized_name, function)
|
|
41
|
+ add_scenario(suite, normalized_name, to_be_benchmark)
|
|
42
|
+ end
|
|
43
|
+ end
|
|
44
|
+
|
|
45
|
+ defp warn_if_evaluated(to_be_benchmark, job_name, printer) do
|
|
46
|
+ function =
|
|
47
|
+ case to_be_benchmark do
|
|
48
|
+ {function, _} -> function
|
|
49
|
+ function -> function
|
|
50
|
+ end
|
|
51
|
+
|
|
52
|
+ if :erlang.fun_info(function, :module) == {:module, :erl_eval} do
|
|
53
|
+ printer.evaluated_function_warning(job_name)
|
|
35
54
|
end
|
|
36
55
|
end
|
changed
lib/benchee/benchmark/collect.ex
|
|
@@ -12,7 +12,7 @@ defmodule Benchee.Benchmark.Collect do
|
|
12
12
|
reason - it will then be ignored and not counted.
|
|
13
13
|
"""
|
|
14
14
|
@type return_value :: {non_neg_integer | nil, any}
|
|
15
|
- @type zero_arity_function :: (() -> any)
|
|
15
|
+ @type zero_arity_function :: (-> any)
|
|
16
16
|
@type opts :: any
|
|
17
17
|
@callback collect(zero_arity_function()) :: return_value()
|
|
18
18
|
@callback collect(zero_arity_function(), opts) :: return_value()
|
changed
lib/benchee/benchmark/collect/memory.ex
|
|
@@ -27,7 +27,7 @@ defmodule Benchee.Benchmark.Collect.Memory do
|
|
27
27
|
end
|
|
28
28
|
end
|
|
29
29
|
|
|
30
|
- @spec collect((() -> any)) :: {nil | non_neg_integer, any}
|
|
30
|
+ @spec collect((-> any)) :: {nil | non_neg_integer, any}
|
|
31
31
|
def collect(fun) do
|
|
32
32
|
ref = make_ref()
|
|
33
33
|
Process.flag(:trap_exit, true)
|
changed
lib/benchee/benchmark/collect/time.ex
|
|
@@ -11,7 +11,7 @@ defmodule Benchee.Benchmark.Collect.Time do
|
|
11
11
|
|
|
12
12
|
@behaviour Benchee.Benchmark.Collect
|
|
13
13
|
|
|
14
|
- @spec collect((() -> any)) :: {non_neg_integer, any}
|
|
14
|
+ @spec collect((-> any)) :: {non_neg_integer, any}
|
|
15
15
|
def collect(function) do
|
|
16
16
|
start = :erlang.monotonic_time()
|
|
17
17
|
result = function.()
|
changed
lib/benchee/benchmark/function_call_overhead.ex
|
|
@@ -40,7 +40,7 @@ defmodule Benchee.Benchmark.FunctionCallOverhead do
|
|
40
40
|
do_run(function, [], end_time)
|
|
41
41
|
end
|
|
42
42
|
|
|
43
|
- @spec do_run((() -> any), [number], number) :: [number, ...]
|
|
43
|
+ @spec do_run((-> any), [number], number) :: [number, ...]
|
|
44
44
|
defp do_run(function, durations, end_time) do
|
|
45
45
|
{duration, _} = Time.collect(function)
|
changed
lib/benchee/benchmark/runner.ex
|
|
@@ -168,7 +168,7 @@ defmodule Benchee.Benchmark.Runner do
|
|
168
168
|
Enum.map(reductions, &(&1 - offset))
|
|
169
169
|
end
|
|
170
170
|
|
|
171
|
- defp run_reductions_benchmark(_, %ScenarioContext{config: %{reduction_time: 0.0}}) do
|
|
171
|
+ defp run_reductions_benchmark(_, %ScenarioContext{config: %{reduction_time: +0.0}}) do
|
|
172
172
|
[]
|
|
173
173
|
end
|
|
174
174
|
|
|
|
@@ -191,7 +191,7 @@ defmodule Benchee.Benchmark.Runner do
|
|
191
191
|
do_benchmark(scenario, new_context, Collect.Reductions, [])
|
|
192
192
|
end
|
|
193
193
|
|
|
194
|
- defp run_memory_benchmark(_, %ScenarioContext{config: %{memory_time: 0.0}}) do
|
|
194
|
+ defp run_memory_benchmark(_, %ScenarioContext{config: %{memory_time: +0.0}}) do
|
|
195
195
|
[]
|
|
196
196
|
end
|
|
197
197
|
|
|
|
@@ -216,7 +216,7 @@ defmodule Benchee.Benchmark.Runner do
|
|
216
216
|
|
|
217
217
|
@spec measure_runtimes(Scenario.t(), ScenarioContext.t(), number, boolean) :: [number]
|
|
218
218
|
defp measure_runtimes(scenario, context, run_time, fast_warning)
|
|
219
|
- defp measure_runtimes(_, _, 0.0, _), do: []
|
|
219
|
+ defp measure_runtimes(_, _, +0.0, _), do: []
|
|
220
220
|
|
|
221
221
|
defp measure_runtimes(scenario, scenario_context, run_time, fast_warning) do
|
|
222
222
|
end_time = current_time() + run_time
|
changed
lib/benchee/configuration.ex
|
|
@@ -176,7 +176,7 @@ defmodule Benchee.Configuration do
|
|
176
176
|
|
|
177
177
|
## Examples
|
|
178
178
|
|
|
179
|
- iex> Benchee.init
|
|
179
|
+ iex> init()
|
|
180
180
|
%Benchee.Suite{
|
|
181
181
|
configuration:
|
|
182
182
|
%Benchee.Configuration{
|
|
|
@@ -204,7 +204,7 @@ defmodule Benchee.Configuration do
|
|
204
204
|
scenarios: []
|
|
205
205
|
}
|
|
206
206
|
|
|
207
|
- iex> Benchee.init time: 1, warmup: 0.2
|
|
207
|
+ iex> init(time: 1, warmup: 0.2)
|
|
208
208
|
%Benchee.Suite{
|
|
209
209
|
configuration:
|
|
210
210
|
%Benchee.Configuration{
|
|
|
@@ -232,7 +232,7 @@ defmodule Benchee.Configuration do
|
|
232
232
|
scenarios: []
|
|
233
233
|
}
|
|
234
234
|
|
|
235
|
- iex> Benchee.init %{time: 1, warmup: 0.2}
|
|
235
|
+ iex> init(%{time: 1, warmup: 0.2})
|
|
236
236
|
%Benchee.Suite{
|
|
237
237
|
configuration:
|
|
238
238
|
%Benchee.Configuration{
|
|
|
@@ -260,7 +260,7 @@ defmodule Benchee.Configuration do
|
|
260
260
|
scenarios: []
|
|
261
261
|
}
|
|
262
262
|
|
|
263
|
- iex> Benchee.init(
|
|
263
|
+ iex> init(
|
|
264
264
|
...> parallel: 2,
|
|
265
265
|
...> time: 1,
|
|
266
266
|
...> warmup: 0.2,
|
changed
lib/benchee/conversion.ex
|
|
@@ -52,9 +52,9 @@ defmodule Benchee.Conversion do
|
|
52
52
|
}
|
|
53
53
|
"""
|
|
54
54
|
def units(scenarios, scaling_strategy) do
|
|
55
|
- run_time_measurements = measurments_for(scenarios, :run_time_data)
|
|
56
|
- reductions_measurements = measurments_for(scenarios, :reductions_data)
|
|
57
|
- memory_measurements = measurments_for(scenarios, :memory_usage_data)
|
|
55
|
+ run_time_measurements = measurements_for(scenarios, :run_time_data)
|
|
56
|
+ reductions_measurements = measurements_for(scenarios, :reductions_data)
|
|
57
|
+ memory_measurements = measurements_for(scenarios, :memory_usage_data)
|
|
58
58
|
|
|
59
59
|
%{
|
|
60
60
|
run_time: Duration.best(run_time_measurements.average, strategy: scaling_strategy),
|
|
|
@@ -64,7 +64,7 @@ defmodule Benchee.Conversion do
|
|
64
64
|
}
|
|
65
65
|
end
|
|
66
66
|
|
|
67
|
- defp measurments_for(scenarios, path) do
|
|
67
|
+ defp measurements_for(scenarios, path) do
|
|
68
68
|
paths = [Access.key(path), Access.key(:statistics)]
|
|
69
69
|
|
|
70
70
|
scenarios
|
changed
lib/benchee/conversion/count.ex
|
|
@@ -49,13 +49,13 @@ defmodule Benchee.Conversion.Count do
|
|
49
49
|
|
|
50
50
|
## Examples
|
|
51
51
|
|
|
52
|
- iex> {value, unit} = Benchee.Conversion.Count.scale(4_321.09)
|
|
52
|
+ iex> {value, unit} = scale(4_321.09)
|
|
53
53
|
iex> value
|
|
54
54
|
4.32109
|
|
55
55
|
iex> unit.name
|
|
56
56
|
:thousand
|
|
57
57
|
|
|
58
|
- iex> {value, unit} = Benchee.Conversion.Count.scale(0.0045)
|
|
58
|
+ iex> {value, unit} = scale(0.0045)
|
|
59
59
|
iex> value
|
|
60
60
|
0.0045
|
|
61
61
|
iex> unit.name
|
|
|
@@ -89,7 +89,7 @@ defmodule Benchee.Conversion.Count do
|
|
89
89
|
|
|
90
90
|
## Examples
|
|
91
91
|
|
|
92
|
- iex> Benchee.Conversion.Count.unit_for :thousand
|
|
92
|
+ iex> unit_for :thousand
|
|
93
93
|
%Benchee.Conversion.Unit{
|
|
94
94
|
name: :thousand,
|
|
95
95
|
magnitude: 1_000,
|
|
|
@@ -97,7 +97,7 @@ defmodule Benchee.Conversion.Count do
|
|
97
97
|
long: "Thousand"
|
|
98
98
|
}
|
|
99
99
|
|
|
100
|
- iex> Benchee.Conversion.Count.unit_for(%Benchee.Conversion.Unit{
|
|
100
|
+ iex> unit_for(%Benchee.Conversion.Unit{
|
|
101
101
|
...> name: :thousand,
|
|
102
102
|
...> magnitude: 1_000,
|
|
103
103
|
...> label: "K",
|
|
|
@@ -119,16 +119,16 @@ defmodule Benchee.Conversion.Count do
|
|
119
119
|
|
|
120
120
|
## Examples
|
|
121
121
|
|
|
122
|
- iex> Benchee.Conversion.Count.scale(12345, :one)
|
|
122
|
+ iex> scale(12345, :one)
|
|
123
123
|
12345.0
|
|
124
124
|
|
|
125
|
- iex> Benchee.Conversion.Count.scale(12345, :thousand)
|
|
125
|
+ iex> scale(12345, :thousand)
|
|
126
126
|
12.345
|
|
127
127
|
|
|
128
|
- iex> Benchee.Conversion.Count.scale(12345, :billion)
|
|
128
|
+ iex> scale(12345, :billion)
|
|
129
129
|
1.2345e-5
|
|
130
130
|
|
|
131
|
- iex> Benchee.Conversion.Count.scale(12345, :million)
|
|
131
|
+ iex> scale(12345, :million)
|
|
132
132
|
0.012345
|
|
133
133
|
|
|
134
134
|
"""
|
|
|
@@ -141,7 +141,7 @@ defmodule Benchee.Conversion.Count do
|
|
141
141
|
|
|
142
142
|
## Examples
|
|
143
143
|
|
|
144
|
- iex> {value, unit} = Benchee.Conversion.Count.convert({2500, :thousand}, :million)
|
|
144
|
+ iex> {value, unit} = convert({2500, :thousand}, :million)
|
|
145
145
|
iex> value
|
|
146
146
|
2.5
|
|
147
147
|
iex> unit.name
|
|
|
@@ -160,16 +160,16 @@ defmodule Benchee.Conversion.Count do
|
|
160
160
|
|
|
161
161
|
## Examples
|
|
162
162
|
|
|
163
|
- iex> Benchee.Conversion.Count.best([23, 23_000, 34_000, 2_340_000]).name
|
|
163
|
+ iex> best([23, 23_000, 34_000, 2_340_000]).name
|
|
164
164
|
:thousand
|
|
165
165
|
|
|
166
|
- iex> Benchee.Conversion.Count.best([23, 23_000, 34_000, 2_340_000, 3_450_000]).name
|
|
166
|
+ iex> best([23, 23_000, 34_000, 2_340_000, 3_450_000]).name
|
|
167
167
|
:million
|
|
168
168
|
|
|
169
|
- iex> Benchee.Conversion.Count.best([23, 23_000, 34_000, 2_340_000], strategy: :smallest).name
|
|
169
|
+ iex> best([23, 23_000, 34_000, 2_340_000], strategy: :smallest).name
|
|
170
170
|
:one
|
|
171
171
|
|
|
172
|
- iex> Benchee.Conversion.Count.best([23, 23_000, 34_000, 2_340_000], strategy: :largest).name
|
|
172
|
+ iex> best([23, 23_000, 34_000, 2_340_000], strategy: :largest).name
|
|
173
173
|
:million
|
|
174
174
|
|
|
175
175
|
"""
|
|
|
@@ -184,7 +184,7 @@ defmodule Benchee.Conversion.Count do
|
|
184
184
|
|
|
185
185
|
## Examples
|
|
186
186
|
|
|
187
|
- iex> Benchee.Conversion.Count.base_unit.name
|
|
187
|
+ iex> base_unit().name
|
|
188
188
|
:one
|
|
189
189
|
|
|
190
190
|
"""
|
|
|
@@ -196,16 +196,16 @@ defmodule Benchee.Conversion.Count do
|
|
196
196
|
|
|
197
197
|
## Examples
|
|
198
198
|
|
|
199
|
- iex> Benchee.Conversion.Count.format(45_678.9)
|
|
199
|
+ iex> format(45_678.9)
|
|
200
200
|
"45.68 K"
|
|
201
201
|
|
|
202
|
- iex> Benchee.Conversion.Count.format(45.6789)
|
|
202
|
+ iex> format(45.6789)
|
|
203
203
|
"45.68"
|
|
204
204
|
|
|
205
|
- iex> Benchee.Conversion.Count.format({45.6789, :thousand})
|
|
205
|
+ iex> format({45.6789, :thousand})
|
|
206
206
|
"45.68 K"
|
|
207
207
|
|
|
208
|
- iex> Benchee.Conversion.Count.format({45.6789, %Benchee.Conversion.Unit{long: "Thousand", magnitude: "1_000", label: "K"}})
|
|
208
|
+ iex> format({45.6789, %Benchee.Conversion.Unit{long: "Thousand", magnitude: "1_000", label: "K"}})
|
|
209
209
|
"45.68 K"
|
|
210
210
|
"""
|
|
211
211
|
def format(count) do
|
changed
lib/benchee/conversion/deviation_percent.ex
|
|
@@ -17,10 +17,10 @@ defmodule Benchee.Conversion.DeviationPercent do
|
|
17
17
|
|
|
18
18
|
## Examples
|
|
19
19
|
|
|
20
|
- iex> Benchee.Conversion.DeviationPercent.format(0.12345)
|
|
20
|
+ iex> format(0.12345)
|
|
21
21
|
"±12.35%"
|
|
22
22
|
|
|
23
|
- iex> Benchee.Conversion.DeviationPercent.format(1)
|
|
23
|
+ iex> format(1)
|
|
24
24
|
"±100.00%"
|
|
25
25
|
"""
|
|
26
26
|
def format(std_dev_ratio) do
|
changed
lib/benchee/conversion/duration.ex
|
|
@@ -65,19 +65,19 @@ defmodule Benchee.Conversion.Duration do
|
|
65
65
|
|
|
66
66
|
## Examples
|
|
67
67
|
|
|
68
|
- iex> {value, unit} = Benchee.Conversion.Duration.scale(1)
|
|
68
|
+ iex> {value, unit} = scale(1)
|
|
69
69
|
iex> value
|
|
70
70
|
1.0
|
|
71
71
|
iex> unit.name
|
|
72
72
|
:nanosecond
|
|
73
73
|
|
|
74
|
- iex> {value, unit} = Benchee.Conversion.Duration.scale(1_234)
|
|
74
|
+ iex> {value, unit} = scale(1_234)
|
|
75
75
|
iex> value
|
|
76
76
|
1.234
|
|
77
77
|
iex> unit.name
|
|
78
78
|
:microsecond
|
|
79
79
|
|
|
80
|
- iex> {value, unit} = Benchee.Conversion.Duration.scale(11_234_567_890_123)
|
|
80
|
+ iex> {value, unit} = scale(11_234_567_890_123)
|
|
81
81
|
iex> value
|
|
82
82
|
3.1207133028119443
|
|
83
83
|
iex> unit.name
|
|
|
@@ -118,7 +118,7 @@ defmodule Benchee.Conversion.Duration do
|
|
118
118
|
|
|
119
119
|
## Examples
|
|
120
120
|
|
|
121
|
- iex> Benchee.Conversion.Duration.unit_for :hour
|
|
121
|
+ iex> unit_for :hour
|
|
122
122
|
%Benchee.Conversion.Unit{
|
|
123
123
|
name: :hour,
|
|
124
124
|
magnitude: 3_600_000_000_000,
|
|
|
@@ -126,7 +126,7 @@ defmodule Benchee.Conversion.Duration do
|
|
126
126
|
long: "Hours"
|
|
127
127
|
}
|
|
128
128
|
|
|
129
|
- iex> Benchee.Conversion.Duration.unit_for(%Benchee.Conversion.Unit{
|
|
129
|
+ iex> unit_for(%Benchee.Conversion.Unit{
|
|
130
130
|
...> name: :hour,
|
|
131
131
|
...> magnitude: 3_600_000_000_000,
|
|
132
132
|
...> label: "h",
|
|
|
@@ -148,13 +148,13 @@ defmodule Benchee.Conversion.Duration do
|
|
148
148
|
|
|
149
149
|
## Examples
|
|
150
150
|
|
|
151
|
- iex> Benchee.Conversion.Duration.scale(12345, :nanosecond)
|
|
151
|
+ iex> scale(12345, :nanosecond)
|
|
152
152
|
12345.0
|
|
153
153
|
|
|
154
|
- iex> Benchee.Conversion.Duration.scale(12345, :microsecond)
|
|
154
|
+ iex> scale(12345, :microsecond)
|
|
155
155
|
12.345
|
|
156
156
|
|
|
157
|
- iex> Benchee.Conversion.Duration.scale(12345, :minute)
|
|
157
|
+ iex> scale(12345, :minute)
|
|
158
158
|
2.0575e-7
|
|
159
159
|
|
|
160
160
|
"""
|
|
|
@@ -167,7 +167,7 @@ defmodule Benchee.Conversion.Duration do
|
|
167
167
|
|
|
168
168
|
## Examples
|
|
169
169
|
|
|
170
|
- iex> {value, unit} = Benchee.Conversion.Duration.convert({90, :minute}, :hour)
|
|
170
|
+ iex> {value, unit} = convert({90, :minute}, :hour)
|
|
171
171
|
iex> value
|
|
172
172
|
1.5
|
|
173
173
|
iex> unit.name
|
|
|
@@ -182,14 +182,14 @@ defmodule Benchee.Conversion.Duration do
|
|
182
182
|
|
|
183
183
|
## Examples
|
|
184
184
|
|
|
185
|
- iex> Benchee.Conversion.Duration.convert_value({1.234, :second}, :microsecond)
|
|
185
|
+ iex> convert_value({1.234, :second}, :microsecond)
|
|
186
186
|
1_234_000.0
|
|
187
187
|
|
|
188
|
- iex> Benchee.Conversion.Duration.convert_value({1.234, :minute}, :microsecond)
|
|
188
|
+ iex> convert_value({1.234, :minute}, :microsecond)
|
|
189
189
|
7.404e7
|
|
190
190
|
|
|
191
|
- iex> microseconds = Benchee.Conversion.Duration.convert_value({1.234, :minute}, :microsecond)
|
|
192
|
- iex> {value, _} = Benchee.Conversion.Duration.convert({microseconds, :microsecond}, :minute)
|
|
191
|
+ iex> microseconds = convert_value({1.234, :minute}, :microsecond)
|
|
192
|
+ iex> {value, _} = convert({microseconds, :microsecond}, :minute)
|
|
193
193
|
iex> value
|
|
194
194
|
1.234
|
|
195
195
|
|
|
|
@@ -208,16 +208,16 @@ defmodule Benchee.Conversion.Duration do
|
|
208
208
|
|
|
209
209
|
## Examples
|
|
210
210
|
|
|
211
|
- iex> Benchee.Conversion.Duration.best([23, 23_000, 34_000, 2_340_000]).name
|
|
211
|
+ iex> best([23, 23_000, 34_000, 2_340_000]).name
|
|
212
212
|
:microsecond
|
|
213
213
|
|
|
214
|
- iex> Benchee.Conversion.Duration.best([23, 23_000, 34_000_000, 2_340_000_000, 3_450_000_000]).name
|
|
214
|
+ iex> best([23, 23_000, 34_000_000, 2_340_000_000, 3_450_000_000]).name
|
|
215
215
|
:second
|
|
216
216
|
|
|
217
|
- iex> Benchee.Conversion.Duration.best([23, 23_000, 34_000, 2_340_000], strategy: :smallest).name
|
|
217
|
+ iex> best([23, 23_000, 34_000, 2_340_000], strategy: :smallest).name
|
|
218
218
|
:nanosecond
|
|
219
219
|
|
|
220
|
- iex> Benchee.Conversion.Duration.best([23, 23_000, 34_000, 2_340_000_000], strategy: :largest).name
|
|
220
|
+ iex> best([23, 23_000, 34_000, 2_340_000_000], strategy: :largest).name
|
|
221
221
|
:second
|
|
222
222
|
"""
|
|
223
223
|
def best(list, opts \\ [strategy: :best])
|
|
|
@@ -231,7 +231,7 @@ defmodule Benchee.Conversion.Duration do
|
|
231
231
|
|
|
232
232
|
## Examples
|
|
233
233
|
|
|
234
|
- iex> Benchee.Conversion.Duration.base_unit.name
|
|
234
|
+ iex> base_unit().name
|
|
235
235
|
:nanosecond
|
|
236
236
|
|
|
237
237
|
"""
|
|
|
@@ -243,16 +243,16 @@ defmodule Benchee.Conversion.Duration do
|
|
243
243
|
|
|
244
244
|
## Examples
|
|
245
245
|
|
|
246
|
- iex> Benchee.Conversion.Duration.format(45_678.9)
|
|
246
|
+ iex> format(45_678.9)
|
|
247
247
|
"45.68 μs"
|
|
248
248
|
|
|
249
|
- iex> Benchee.Conversion.Duration.format(45.6789)
|
|
249
|
+ iex> format(45.6789)
|
|
250
250
|
"45.68 ns"
|
|
251
251
|
|
|
252
|
- iex> Benchee.Conversion.Duration.format({45.6789, :millisecond})
|
|
252
|
+ iex> format({45.6789, :millisecond})
|
|
253
253
|
"45.68 ms"
|
|
254
254
|
|
|
255
|
- iex> Benchee.Conversion.Duration.format {45.6789,
|
|
255
|
+ iex> format {45.6789,
|
|
256
256
|
...> %Benchee.Conversion.Unit{
|
|
257
257
|
...> long: "Milliseconds", magnitude: 1000, label: "ms"}
|
|
258
258
|
...> }
|
changed
lib/benchee/conversion/format.ex
|
|
@@ -41,7 +41,7 @@ defmodule Benchee.Conversion.Format do
|
|
41
41
|
formatted output. If no `separator/0` function exists, the default separator
|
|
42
42
|
(a single space) will be used.
|
|
43
43
|
|
|
44
|
- iex> Benchee.Conversion.Format.format({1.0, :kilobyte}, Benchee.Conversion.Memory)
|
|
44
|
+ iex> format({1.0, :kilobyte}, Benchee.Conversion.Memory)
|
|
45
45
|
"1 KB"
|
|
46
46
|
|
|
47
47
|
"""
|
changed
lib/benchee/conversion/memory.ex
|
|
@@ -56,14 +56,14 @@ defmodule Benchee.Conversion.Memory do
|
|
56
56
|
|
|
57
57
|
## Examples
|
|
58
58
|
|
|
59
|
- iex> {value, unit} = Benchee.Conversion.Memory.convert({1024, :kilobyte}, :megabyte)
|
|
59
|
+ iex> {value, unit} = convert({1024, :kilobyte}, :megabyte)
|
|
60
60
|
iex> value
|
|
61
61
|
1.0
|
|
62
62
|
iex> unit.name
|
|
63
63
|
:megabyte
|
|
64
64
|
|
|
65
|
- iex> current_unit = Benchee.Conversion.Memory.unit_for :kilobyte
|
|
66
|
- iex> {value, unit} = Benchee.Conversion.Memory.convert({1024, current_unit}, :megabyte)
|
|
65
|
+ iex> current_unit = unit_for :kilobyte
|
|
66
|
+ iex> {value, unit} = convert({1024, current_unit}, :megabyte)
|
|
67
67
|
iex> value
|
|
68
68
|
1.0
|
|
69
69
|
iex> unit.name
|
|
|
@@ -81,25 +81,25 @@ defmodule Benchee.Conversion.Memory do
|
|
81
81
|
|
|
82
82
|
## Examples
|
|
83
83
|
|
|
84
|
- iex> {value, unit} = Benchee.Conversion.Memory.scale(1)
|
|
84
|
+ iex> {value, unit} = scale(1)
|
|
85
85
|
iex> value
|
|
86
86
|
1.0
|
|
87
87
|
iex> unit.name
|
|
88
88
|
:byte
|
|
89
89
|
|
|
90
|
- iex> {value, unit} = Benchee.Conversion.Memory.scale(1_234)
|
|
90
|
+ iex> {value, unit} = scale(1_234)
|
|
91
91
|
iex> value
|
|
92
92
|
1.205078125
|
|
93
93
|
iex> unit.name
|
|
94
94
|
:kilobyte
|
|
95
95
|
|
|
96
|
- iex> {value, unit} = Benchee.Conversion.Memory.scale(11_234_567_890.123)
|
|
96
|
+ iex> {value, unit} = scale(11_234_567_890.123)
|
|
97
97
|
iex> value
|
|
98
98
|
10.463006692121736
|
|
99
99
|
iex> unit.name
|
|
100
100
|
:gigabyte
|
|
101
101
|
|
|
102
|
- iex> {value, unit} = Benchee.Conversion.Memory.scale(1_111_234_567_890.123)
|
|
102
|
+ iex> {value, unit} = scale(1_111_234_567_890.123)
|
|
103
103
|
iex> value
|
|
104
104
|
1.0106619519229962
|
|
105
105
|
iex> unit.name
|
|
|
@@ -140,7 +140,7 @@ defmodule Benchee.Conversion.Memory do
|
|
140
140
|
|
|
141
141
|
## Examples
|
|
142
142
|
|
|
143
|
- iex> Benchee.Conversion.Memory.unit_for :gigabyte
|
|
143
|
+ iex> unit_for :gigabyte
|
|
144
144
|
%Benchee.Conversion.Unit{
|
|
145
145
|
name: :gigabyte,
|
|
146
146
|
magnitude: 1_073_741_824,
|
|
|
@@ -148,7 +148,7 @@ defmodule Benchee.Conversion.Memory do
|
|
148
148
|
long: "Gigabytes"
|
|
149
149
|
}
|
|
150
150
|
|
|
151
|
- iex> Benchee.Conversion.Memory.unit_for(%Benchee.Conversion.Unit{
|
|
151
|
+ iex> unit_for(%Benchee.Conversion.Unit{
|
|
152
152
|
...> name: :gigabyte,
|
|
153
153
|
...> magnitude: 1_073_741_824,
|
|
154
154
|
...> label: "GB",
|
|
|
@@ -170,13 +170,13 @@ defmodule Benchee.Conversion.Memory do
|
|
170
170
|
|
|
171
171
|
## Examples
|
|
172
172
|
|
|
173
|
- iex> Benchee.Conversion.Memory.scale(12345, :kilobyte)
|
|
173
|
+ iex> scale(12345, :kilobyte)
|
|
174
174
|
12.0556640625
|
|
175
175
|
|
|
176
|
- iex> Benchee.Conversion.Memory.scale(12345, :megabyte)
|
|
176
|
+ iex> scale(12345, :megabyte)
|
|
177
177
|
0.011773109436035156
|
|
178
178
|
|
|
179
|
- iex> Benchee.Conversion.Memory.scale(123_456_789, :gigabyte)
|
|
179
|
+ iex> scale(123_456_789, :gigabyte)
|
|
180
180
|
0.11497809458523989
|
|
181
181
|
|
|
182
182
|
"""
|
|
|
@@ -195,16 +195,16 @@ defmodule Benchee.Conversion.Memory do
|
|
195
195
|
|
|
196
196
|
## Examples
|
|
197
197
|
|
|
198
|
- iex> Benchee.Conversion.Memory.best([23, 23_000, 34_000, 2_340_000]).name
|
|
198
|
+ iex> best([23, 23_000, 34_000, 2_340_000]).name
|
|
199
199
|
:kilobyte
|
|
200
200
|
|
|
201
|
- iex> Benchee.Conversion.Memory.best([23, 23_000, 34_000, 2_340_000, 3_450_000]).name
|
|
201
|
+ iex> best([23, 23_000, 34_000, 2_340_000, 3_450_000]).name
|
|
202
202
|
:megabyte
|
|
203
203
|
|
|
204
|
- iex> Benchee.Conversion.Memory.best([23, 23_000, 34_000, 2_340_000], strategy: :smallest).name
|
|
204
|
+ iex> best([23, 23_000, 34_000, 2_340_000], strategy: :smallest).name
|
|
205
205
|
:byte
|
|
206
206
|
|
|
207
|
- iex> Benchee.Conversion.Memory.best([23, 23_000, 34_000, 2_340_000], strategy: :largest).name
|
|
207
|
+ iex> best([23, 23_000, 34_000, 2_340_000], strategy: :largest).name
|
|
208
208
|
:megabyte
|
|
209
209
|
"""
|
|
210
210
|
def best(list, opts \\ [strategy: :best])
|
|
|
@@ -218,7 +218,7 @@ defmodule Benchee.Conversion.Memory do
|
|
218
218
|
|
|
219
219
|
## Examples
|
|
220
220
|
|
|
221
|
- iex> Benchee.Conversion.Memory.base_unit.name
|
|
221
|
+ iex> base_unit().name
|
|
222
222
|
:byte
|
|
223
223
|
|
|
224
224
|
"""
|
|
|
@@ -230,16 +230,16 @@ defmodule Benchee.Conversion.Memory do
|
|
230
230
|
|
|
231
231
|
## Examples
|
|
232
232
|
|
|
233
|
- iex> Benchee.Conversion.Memory.format(45_678.9)
|
|
233
|
+ iex> format(45_678.9)
|
|
234
234
|
"44.61 KB"
|
|
235
235
|
|
|
236
|
- iex> Benchee.Conversion.Memory.format(45.6789)
|
|
236
|
+ iex> format(45.6789)
|
|
237
237
|
"45.68 B"
|
|
238
238
|
|
|
239
|
- iex> Benchee.Conversion.Memory.format({45.6789, :kilobyte})
|
|
239
|
+ iex> format({45.6789, :kilobyte})
|
|
240
240
|
"45.68 KB"
|
|
241
241
|
|
|
242
|
- iex> Benchee.Conversion.Memory.format {45.6789,
|
|
242
|
+ iex> format {45.6789,
|
|
243
243
|
...> %Benchee.Conversion.Unit{
|
|
244
244
|
...> long: "Kilobytes", magnitude: 1024, label: "KB"}
|
|
245
245
|
...> }
|
changed
lib/benchee/conversion/scale.ex
|
|
@@ -61,7 +61,7 @@ defmodule Benchee.Conversion.Scale do
|
|
61
61
|
|
|
62
62
|
## Examples
|
|
63
63
|
|
|
64
|
- iex> Benchee.Conversion.Scale.scale(12345, :thousand, Benchee.Conversion.Count)
|
|
64
|
+ iex> scale(12345, :thousand, Benchee.Conversion.Count)
|
|
65
65
|
12.345
|
|
66
66
|
"""
|
|
67
67
|
def scale(value, unit = %Unit{}, _module) do
|
|
|
@@ -78,7 +78,7 @@ defmodule Benchee.Conversion.Scale do
|
|
78
78
|
## Examples
|
|
79
79
|
|
|
80
80
|
iex> unit = %Benchee.Conversion.Unit{magnitude: 1000}
|
|
81
|
- iex> Benchee.Conversion.Scale.scale 12345, unit
|
|
81
|
+ iex> scale 12345, unit
|
|
82
82
|
12.345
|
|
83
83
|
"""
|
|
84
84
|
def scale(value, %Unit{magnitude: magnitude}) do
|
|
|
@@ -130,31 +130,31 @@ defmodule Benchee.Conversion.Scale do
|
|
130
130
|
## Examples
|
|
131
131
|
|
|
132
132
|
iex> list = [1, 101, 1_001, 10_001, 100_001, 1_000_001]
|
|
133
|
- iex> Benchee.Conversion.Scale.best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
133
|
+ iex> best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
134
134
|
:thousand
|
|
135
135
|
|
|
136
136
|
iex> list = [1, 101, 1_001, 10_001, 100_001, 1_000_001]
|
|
137
|
- iex> Benchee.Conversion.Scale.best_unit(list, Benchee.Conversion.Count, strategy: :smallest).name
|
|
137
|
+ iex> best_unit(list, Benchee.Conversion.Count, strategy: :smallest).name
|
|
138
138
|
:one
|
|
139
139
|
|
|
140
140
|
iex> list = [1, 101, 1_001, 10_001, 100_001, 1_000_001]
|
|
141
|
- iex> Benchee.Conversion.Scale.best_unit(list, Benchee.Conversion.Count, strategy: :largest).name
|
|
141
|
+ iex> best_unit(list, Benchee.Conversion.Count, strategy: :largest).name
|
|
142
142
|
:million
|
|
143
143
|
|
|
144
144
|
iex> list = []
|
|
145
|
- iex> Benchee.Conversion.Scale.best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
145
|
+ iex> best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
146
146
|
:one
|
|
147
147
|
|
|
148
148
|
iex> list = [nil]
|
|
149
|
- iex> Benchee.Conversion.Scale.best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
149
|
+ iex> best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
150
150
|
:one
|
|
151
151
|
|
|
152
152
|
iex> list = [nil, nil, nil, nil]
|
|
153
|
- iex> Benchee.Conversion.Scale.best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
153
|
+ iex> best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
154
154
|
:one
|
|
155
155
|
|
|
156
156
|
iex> list = [nil, nil, nil, nil, 2_000]
|
|
157
|
- iex> Benchee.Conversion.Scale.best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
157
|
+ iex> best_unit(list, Benchee.Conversion.Count, strategy: :best).name
|
|
158
158
|
:thousand
|
|
159
159
|
"""
|
|
160
160
|
def best_unit(measurements, module, options) do
|
changed
lib/benchee/formatters/console.ex
|
|
@@ -80,7 +80,7 @@ defmodule Benchee.Formatters.Console do
|
|
80
80
|
...> unit_scaling: :best,
|
|
81
81
|
...> }
|
|
82
82
|
...> }
|
|
83
|
- iex> Benchee.Formatters.Console.format(suite, %{comparison: false, extended_statistics: false})
|
|
83
|
+ iex> format(suite, %{comparison: false, extended_statistics: false})
|
|
84
84
|
[["\n##### With input My input #####", "\nName ips average deviation median 99th %\n",
|
|
85
85
|
"My Job 5 K 200 ns ±10.00% 190 ns 300.10 ns\n",
|
|
86
86
|
"Job 2 2.50 K 400 ns ±20.00% 390 ns 500.10 ns\n"]]
|
changed
lib/benchee/formatters/console/run_time.ex
|
|
@@ -58,7 +58,7 @@ defmodule Benchee.Formatters.Console.RunTime do
|
|
58
58
|
...> }
|
|
59
59
|
...> ]
|
|
60
60
|
iex> configuration = %{comparison: false, unit_scaling: :best, extended_statistics: true}
|
|
61
|
- iex> Benchee.Formatters.Console.RunTime.format_scenarios(scenarios, configuration)
|
|
61
|
+ iex> format_scenarios(scenarios, configuration)
|
|
62
62
|
["\nName ips average deviation median 99th %\n",
|
|
63
63
|
"My Job 5 K 200 ns ±10.00% 190 ns 300.10 ns\n",
|
|
64
64
|
"Job 2 2.50 K 400 ns ±20.00% 390 ns 500.10 ns\n",
|
changed
lib/benchee/output/benchmark_printer.ex
|
|
@@ -3,6 +3,20 @@ defmodule Benchee.Output.BenchmarkPrinter do
|
|
3
3
|
|
|
4
4
|
alias Benchee.{Benchmark, Conversion.Duration}
|
|
5
5
|
|
|
6
|
+ @doc """
|
|
7
|
+ Shown when you try benchmark an evaluated function.
|
|
8
|
+
|
|
9
|
+ Compiled functions should be preferred as they are less likely to introduce additional overhead to your benchmark timing.
|
|
10
|
+ """
|
|
11
|
+ def evaluated_function_warning(job_name) do
|
|
12
|
+ IO.puts("""
|
|
13
|
+ Warning: the benchmark #{job_name} is using an evaluated function.
|
|
14
|
+ Evaluated functions perform slower than compiled functions.
|
|
15
|
+ You can move the Benchee caller to a function in a module and invoke `Mod.fun()` instead.
|
|
16
|
+ Alternatively, you can move the benchmark into a benchmark.exs file and run mix run benchmark.exs
|
|
17
|
+ """)
|
|
18
|
+ end
|
|
19
|
+
|
|
6
20
|
@doc """
|
|
7
21
|
Shown when you try to define a benchmark with the same name twice.
|
|
8
22
|
|
|
|
@@ -80,7 +94,7 @@ defmodule Benchee.Output.BenchmarkPrinter do
|
|
80
94
|
"""
|
|
81
95
|
def benchmarking(_, _, %{print: %{benchmarking: false}}), do: nil
|
|
82
96
|
|
|
83
|
- def benchmarking(_, _, %{time: 0.0, warmup: 0.0, memory_time: 0.0, reduction_time: 0.0}),
|
|
97
|
+ def benchmarking(_, _, %{time: +0.0, warmup: +0.0, memory_time: +0.0, reduction_time: +0.0}),
|
|
84
98
|
do: nil
|
|
85
99
|
|
|
86
100
|
def benchmarking(name, input_name, _config) do
|
changed
lib/benchee/relative_statistics.ex
|
|
@@ -95,8 +95,8 @@ defmodule Benchee.RelativeStatistics do
|
|
95
95
|
}
|
|
96
96
|
end
|
|
97
97
|
|
|
98
|
- defp zero_safe_division(0.0, 0.0), do: 1.0
|
|
98
|
+ defp zero_safe_division(+0.0, +0.0), do: 1.0
|
|
99
99
|
defp zero_safe_division(_, 0), do: :infinity
|
|
100
|
- defp zero_safe_division(_, 0.0), do: :infinity
|
|
100
|
+ defp zero_safe_division(_, +0.0), do: :infinity
|
|
101
101
|
defp zero_safe_division(a, b), do: a / b
|
|
102
102
|
end
|
changed
lib/benchee/scenario.ex
|
|
@@ -40,7 +40,14 @@ defmodule Benchee.Scenario do
|
|
40
40
|
|
|
41
41
|
No arguments if no inputs are used, one argument if inputs are used.
|
|
42
42
|
"""
|
|
43
|
- @type benchmarking_function :: (() -> any) | (any -> any)
|
|
43
|
+ @type benchmarking_function :: (-> any) | (any -> any)
|
|
44
|
+
|
|
45
|
+ @typedoc """
|
|
46
|
+ What shall be benchmarked, mostly a function but can contain options.
|
|
47
|
+
|
|
48
|
+ Options are there for hooks (`after_each`, `before_each` etc.)
|
|
49
|
+ """
|
|
50
|
+ @type to_benchmark :: benchmarking_function() | {benchmarking_function(), keyword()}
|
|
44
51
|
|
|
45
52
|
@typedoc """
|
|
46
53
|
All the data collected for a scenario (combination of function and input)
|
|
|
@@ -73,12 +80,13 @@ defmodule Benchee.Scenario do
|
|
73
80
|
|
|
74
81
|
## Examples
|
|
75
82
|
|
|
76
|
- iex> alias Benchee.Scenario
|
|
77
|
- iex> Scenario.display_name(%Scenario{job_name: "flat_map"})
|
|
83
|
+ iex> display_name(%Benchee.Scenario{job_name: "flat_map"})
|
|
78
84
|
"flat_map"
|
|
79
|
- iex> Scenario.display_name(%Scenario{job_name: "flat_map", tag: "main"})
|
|
85
|
+
|
|
86
|
+ iex> display_name(%Benchee.Scenario{job_name: "flat_map", tag: "main"})
|
|
80
87
|
"flat_map (main)"
|
|
81
|
- iex> Scenario.display_name(%{job_name: "flat_map"})
|
|
88
|
+
|
|
89
|
+ iex> display_name(%{job_name: "flat_map"})
|
|
82
90
|
"flat_map"
|
|
83
91
|
"""
|
|
84
92
|
@spec display_name(t) :: String.t()
|
|
|
@@ -86,6 +94,53 @@ defmodule Benchee.Scenario do
|
|
86
94
|
def display_name(%{job_name: job_name, tag: tag}), do: "#{job_name} (#{tag})"
|
|
87
95
|
def display_name(%{job_name: job_name}), do: job_name
|
|
88
96
|
|
|
97
|
+ @doc """
|
|
98
|
+ Returns the different measurement types supported.
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+ ## Examples
|
|
102
|
+
|
|
103
|
+ iex> measurement_types()
|
|
104
|
+ [:run_time, :memory, :reductions]
|
|
105
|
+ """
|
|
106
|
+ @spec measurement_types :: [:memory | :reductions | :run_time, ...]
|
|
107
|
+ def measurement_types, do: [:run_time, :memory, :reductions]
|
|
108
|
+
|
|
109
|
+ @doc """
|
|
110
|
+ Given the measurement type name given by `measurement_types/0`, get the associated data.
|
|
111
|
+
|
|
112
|
+ Raises if no correct measurement type was specified.
|
|
113
|
+
|
|
114
|
+ ## Examples
|
|
115
|
+
|
|
116
|
+ iex> scenario = %Benchee.Scenario{run_time_data: %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 1}}}
|
|
117
|
+ iex> measurement_data(scenario, :run_time)
|
|
118
|
+ %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 1}}
|
|
119
|
+
|
|
120
|
+ iex> scenario = %Benchee.Scenario{memory_usage_data: %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 2}}}
|
|
121
|
+ iex> measurement_data(scenario, :memory)
|
|
122
|
+ %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 2}}
|
|
123
|
+
|
|
124
|
+ iex> scenario = %Benchee.Scenario{reductions_data: %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 3}}}
|
|
125
|
+ iex> measurement_data(scenario, :reductions)
|
|
126
|
+ %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 3}}
|
|
127
|
+
|
|
128
|
+ iex> measurement_data(%Benchee.Scenario{}, :memory)
|
|
129
|
+ %Benchee.CollectionData{}
|
|
130
|
+
|
|
131
|
+ iex> measurement_data(%Benchee.Scenario{}, :invalid)
|
|
132
|
+ ** (FunctionClauseError) no function clause matching in Benchee.Scenario.measurement_data/2
|
|
133
|
+ """
|
|
134
|
+ @spec measurement_data(t, :memory | :reductions | :run_time) :: CollectionData.t()
|
|
135
|
+ # Arguably this access is OO-ish/not great. However, with the incosistency we have in naming
|
|
136
|
+ # between the scenario struct fields and the measurement types this should be good/covnenient.
|
|
137
|
+ # Technically, we could/should move to a map structure from "measurement_type" to collection
|
|
138
|
+ # data but it's probably not worth breaking compatibility for right now.
|
|
139
|
+ def measurement_data(scenario, measurement_type)
|
|
140
|
+ def measurement_data(scenario, :run_time), do: scenario.run_time_data
|
|
141
|
+ def measurement_data(scenario, :memory), do: scenario.memory_usage_data
|
|
142
|
+ def measurement_data(scenario, :reductions), do: scenario.reductions_data
|
|
143
|
+
|
|
89
144
|
@doc """
|
|
90
145
|
Returns `true` if data of the provided type has been fully procsessed, `false` otherwise.
|
|
91
146
|
|
|
|
@@ -97,16 +152,20 @@ defmodule Benchee.Scenario do
|
|
97
152
|
|
|
98
153
|
## Examples
|
|
99
154
|
|
|
100
|
- iex> alias Benchee.Scenario
|
|
101
|
- iex> alias Benchee.Statistics
|
|
102
|
- iex> scenario = %Scenario{run_time_data: %Benchee.CollectionData{statistics: %Statistics{sample_size: 100}}}
|
|
103
|
- iex> Scenario.data_processed?(scenario, :run_time)
|
|
155
|
+ iex> scenario = %Benchee.Scenario{run_time_data: %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 100}}}
|
|
156
|
+ iex> data_processed?(scenario, :run_time)
|
|
104
157
|
true
|
|
105
|
- iex> scenario = %Scenario{memory_usage_data: %Benchee.CollectionData{statistics: %Statistics{sample_size: 1}}}
|
|
106
|
- iex> Scenario.data_processed?(scenario, :memory)
|
|
158
|
+
|
|
159
|
+ iex> scenario = %Benchee.Scenario{memory_usage_data: %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 1}}}
|
|
160
|
+ iex> data_processed?(scenario, :memory)
|
|
107
161
|
true
|
|
108
|
- iex> scenario = %Scenario{memory_usage_data: %Benchee.CollectionData{statistics: %Statistics{sample_size: 0}}}
|
|
109
|
- iex> Scenario.data_processed?(scenario, :memory)
|
|
162
|
+
|
|
163
|
+ iex> scenario = %Benchee.Scenario{reductions_data: %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 1}}}
|
|
164
|
+ iex> data_processed?(scenario, :reductions)
|
|
165
|
+ true
|
|
166
|
+
|
|
167
|
+ iex> scenario = %Benchee.Scenario{memory_usage_data: %Benchee.CollectionData{statistics: %Benchee.Statistics{sample_size: 0}}}
|
|
168
|
+ iex> data_processed?(scenario, :memory)
|
|
110
169
|
false
|
|
111
170
|
"""
|
|
112
171
|
@spec data_processed?(t, :run_time | :memory) :: boolean
|
|
|
@@ -117,4 +176,8 @@ defmodule Benchee.Scenario do
|
|
117
176
|
def data_processed?(scenario, :memory) do
|
|
118
177
|
scenario.memory_usage_data.statistics.sample_size > 0
|
|
119
178
|
end
|
|
179
|
+
|
|
180
|
+ def data_processed?(scenario, :reductions) do
|
|
181
|
+ scenario.reductions_data.statistics.sample_size > 0
|
|
182
|
+ end
|
|
120
183
|
end
|
changed
lib/benchee/statistics.ex
|
|
@@ -115,7 +115,7 @@ defmodule Benchee.Statistics do
|
|
115
115
|
...> }
|
|
116
116
|
...> ]
|
|
117
117
|
iex> suite = %Benchee.Suite{scenarios: scenarios}
|
|
118
|
- iex> Benchee.Statistics.statistics(suite)
|
|
118
|
+ iex> statistics(suite)
|
|
119
119
|
%Benchee.Suite{
|
|
120
120
|
scenarios: [
|
|
121
121
|
%Benchee.Scenario{
|
|
|
@@ -224,7 +224,7 @@ defmodule Benchee.Statistics do
|
|
224
224
|
end
|
|
225
225
|
|
|
226
226
|
defp add_ips(statistics = %__MODULE__{sample_size: 0}), do: statistics
|
|
227
|
- defp add_ips(statistics = %__MODULE__{average: 0.0}), do: statistics
|
|
227
|
+ defp add_ips(statistics = %__MODULE__{average: +0.0}), do: statistics
|
|
228
228
|
|
|
229
229
|
defp add_ips(statistics) do
|
|
230
230
|
ips = Duration.convert_value({1, :second}, :nanosecond) / statistics.average
|
changed
lib/benchee/suite.ex
|
|
@@ -47,3 +47,103 @@ defimpl DeepMerge.Resolver, for: Benchee.Suite do
|
|
47
47
|
Map.merge(original, override, resolver)
|
|
48
48
|
end
|
|
49
49
|
end
|
|
50
|
+
|
|
51
|
+ if Code.ensure_loaded?(Table.Reader) do
|
|
52
|
+ defimpl Table.Reader, for: Benchee.Suite do
|
|
53
|
+ alias Benchee.CollectionData
|
|
54
|
+ alias Benchee.Scenario
|
|
55
|
+
|
|
56
|
+ def init(suite) do
|
|
57
|
+ measurements_processed = map_measurements_processed(suite)
|
|
58
|
+ columns = get_columns_from_suite(suite, measurements_processed)
|
|
59
|
+ {rows, count} = extract_rows_from_suite(suite, measurements_processed)
|
|
60
|
+
|
|
61
|
+ {:rows, %{columns: columns, count: count}, rows}
|
|
62
|
+ end
|
|
63
|
+
|
|
64
|
+ defp map_measurements_processed(suite) do
|
|
65
|
+ Enum.filter(Scenario.measurement_types(), fn type ->
|
|
66
|
+ Enum.any?(suite.scenarios, fn scenario -> Scenario.data_processed?(scenario, type) end)
|
|
67
|
+ end)
|
|
68
|
+ end
|
|
69
|
+
|
|
70
|
+ @run_time_fields [
|
|
71
|
+ "samples",
|
|
72
|
+ "ips",
|
|
73
|
+ "average",
|
|
74
|
+ "std_dev",
|
|
75
|
+ "median",
|
|
76
|
+ "minimum",
|
|
77
|
+ "maximum",
|
|
78
|
+ "mode",
|
|
79
|
+ "sample_size"
|
|
80
|
+ ]
|
|
81
|
+
|
|
82
|
+ @non_run_time_fields List.delete(@run_time_fields, "ips")
|
|
83
|
+
|
|
84
|
+ defp get_columns_from_suite(suite, measurements_processed) do
|
|
85
|
+ config_percentiles = suite.configuration.percentiles
|
|
86
|
+
|
|
87
|
+ percentile_labels =
|
|
88
|
+ Enum.map(config_percentiles, fn percentile ->
|
|
89
|
+ "p_#{percentile}"
|
|
90
|
+ end)
|
|
91
|
+
|
|
92
|
+ measurement_headers =
|
|
93
|
+ Enum.flat_map(measurements_processed, fn measurement_type ->
|
|
94
|
+ fields = fields_for(measurement_type) ++ percentile_labels
|
|
95
|
+
|
|
96
|
+ Enum.map(fields, fn field -> "#{measurement_type}_#{field}" end)
|
|
97
|
+ end)
|
|
98
|
+
|
|
99
|
+ ["job_name" | measurement_headers]
|
|
100
|
+ end
|
|
101
|
+
|
|
102
|
+ defp fields_for(:run_time), do: @run_time_fields
|
|
103
|
+ defp fields_for(_), do: @non_run_time_fields
|
|
104
|
+
|
|
105
|
+ defp extract_rows_from_suite(suite, measurements_processed) do
|
|
106
|
+ config_percentiles = suite.configuration.percentiles
|
|
107
|
+
|
|
108
|
+ Enum.map_reduce(suite.scenarios, 0, fn %Scenario{} = scenario, count ->
|
|
109
|
+ secenario_data =
|
|
110
|
+ Enum.flat_map(measurements_processed, fn measurement_type ->
|
|
111
|
+ scenario
|
|
112
|
+ |> Scenario.measurement_data(measurement_type)
|
|
113
|
+ |> get_stats_from_collection_data(measurement_type, config_percentiles)
|
|
114
|
+ end)
|
|
115
|
+
|
|
116
|
+ row = [scenario.job_name | secenario_data]
|
|
117
|
+
|
|
118
|
+ {row, count + 1}
|
|
119
|
+ end)
|
|
120
|
+ end
|
|
121
|
+
|
|
122
|
+ defp get_stats_from_collection_data(
|
|
123
|
+ %CollectionData{statistics: statistics, samples: samples},
|
|
124
|
+ measurement_type,
|
|
125
|
+ percentiles
|
|
126
|
+ ) do
|
|
127
|
+ percentile_data =
|
|
128
|
+ Enum.map(percentiles, fn percentile -> statistics.percentiles[percentile] end)
|
|
129
|
+
|
|
130
|
+ Enum.concat([
|
|
131
|
+ [samples],
|
|
132
|
+ maybe_ips(statistics, measurement_type),
|
|
133
|
+ [
|
|
134
|
+ statistics.average,
|
|
135
|
+ statistics.std_dev,
|
|
136
|
+ statistics.median,
|
|
137
|
+ statistics.minimum,
|
|
138
|
+ statistics.maximum,
|
|
139
|
+ statistics.mode,
|
|
140
|
+ statistics.sample_size
|
|
141
|
+ ],
|
|
142
|
+ percentile_data
|
|
143
|
+ ])
|
|
144
|
+ end
|
|
145
|
+
|
|
146
|
+ defp maybe_ips(statistics, :run_time), do: [statistics.ips]
|
|
147
|
+ defp maybe_ips(_, _not_run_time), do: []
|
|
148
|
+ end
|
|
149
|
+ end
|
changed
lib/benchee/system.ex
|
|
@@ -37,8 +37,14 @@ defmodule Benchee.System do
|
|
37
37
|
{:ok, version} ->
|
|
38
38
|
String.trim(version)
|
|
39
39
|
|
|
40
|
+ # Livebook seemingly doesn't have the file where we expect it to be:
|
|
41
|
+ # https://github.com/bencheeorg/benchee/issues/367
|
|
40
42
|
{:error, reason} ->
|
|
41
|
- IO.puts("Error trying to determine erlang version #{reason}")
|
|
43
|
+ IO.puts(
|
|
44
|
+ "Error trying to determine erlang version #{reason}, falling back to overall OTP version"
|
|
45
|
+ )
|
|
46
|
+
|
|
47
|
+ to_string(otp_release)
|
|
42
48
|
end
|
|
43
49
|
end
|
|
44
50
|
|
|
|
@@ -179,11 +185,22 @@ defmodule Benchee.System do
|
|
179
185
|
end
|
|
180
186
|
end
|
|
181
187
|
|
|
182
|
- defp all_protocols_consolidated? do
|
|
183
|
- path = :code.lib_dir(:elixir, :ebin)
|
|
188
|
+ # just made public for easy testing purposes
|
|
189
|
+ @doc false
|
|
190
|
+ def all_protocols_consolidated?(lib_dir_fun \\ &:code.lib_dir/2) do
|
|
191
|
+ case lib_dir_fun.(:elixir, :ebin) do
|
|
192
|
+ # do we get a good old erlang charlist?
|
|
193
|
+ path when is_list(path) ->
|
|
194
|
+ [path]
|
|
195
|
+ |> Protocol.extract_protocols()
|
|
196
|
+ |> Enum.all?(&Protocol.consolidated?/1)
|
|
184
197
|
|
|
185
|
- [path]
|
|
186
|
- |> Protocol.extract_protocols()
|
|
187
|
- |> Enum.all?(&Protocol.consolidated?/1)
|
|
198
|
+ _error ->
|
|
199
|
+ IO.puts(
|
|
200
|
+ "Could not check if protocols are consolidated. Running as escript? Defaulting to they are consolidated."
|
|
201
|
+ )
|
|
202
|
+
|
|
203
|
+ true
|
|
204
|
+ end
|
|
188
205
|
end
|
|
189
206
|
end
|
changed
lib/benchee/utility/deep_convert.ex
|
|
@@ -6,25 +6,25 @@ defmodule Benchee.Utility.DeepConvert do
|
|
6
6
|
|
|
7
7
|
## Examples
|
|
8
8
|
|
|
9
|
- iex> Benchee.Utility.DeepConvert.to_map([a: 1, b: 2])
|
|
9
|
+ iex> to_map([a: 1, b: 2])
|
|
10
10
|
%{a: 1, b: 2}
|
|
11
11
|
|
|
12
|
- iex> Benchee.Utility.DeepConvert.to_map([a: [b: 2], c: [d: 3, e: 4, e: 5]])
|
|
12
|
+ iex> to_map([a: [b: 2], c: [d: 3, e: 4, e: 5]])
|
|
13
13
|
%{a: %{b: 2}, c: %{d: 3, e: 5}}
|
|
14
14
|
|
|
15
|
- iex> Benchee.Utility.DeepConvert.to_map([a: [b: 2], c: [1, 2, 3], d: []])
|
|
15
|
+ iex> to_map([a: [b: 2], c: [1, 2, 3], d: []])
|
|
16
16
|
%{a: %{b: 2}, c: [1, 2, 3], d: []}
|
|
17
17
|
|
|
18
|
- iex> Benchee.Utility.DeepConvert.to_map(%{a: %{b: 2}, c: %{d: 3, e: 5}})
|
|
18
|
+ iex> to_map(%{a: %{b: 2}, c: %{d: 3, e: 5}})
|
|
19
19
|
%{a: %{b: 2}, c: %{d: 3, e: 5}}
|
|
20
20
|
|
|
21
|
- iex> Benchee.Utility.DeepConvert.to_map([])
|
|
21
|
+ iex> to_map([])
|
|
22
22
|
%{}
|
|
23
23
|
|
|
24
|
- iex> Benchee.Utility.DeepConvert.to_map([a: [b: [f: 5]]], [:a])
|
|
24
|
+ iex> to_map([a: [b: [f: 5]]], [:a])
|
|
25
25
|
%{a: [b: [f: 5]]}
|
|
26
26
|
|
|
27
|
- iex> Benchee.Utility.DeepConvert.to_map([a: [b: [f: 5]], c: [d: 3]], [:b])
|
|
27
|
+ iex> to_map([a: [b: [f: 5]], c: [d: 3]], [:b])
|
|
28
28
|
%{a: %{b: [f: 5]}, c: %{d: 3}}
|
|
29
29
|
"""
|
|
30
30
|
def to_map(structure, exclusions \\ [])
|
changed
lib/benchee/utility/erlang_version.ex
|
|
@@ -8,7 +8,7 @@ defmodule Benchee.Utility.ErlangVersion do
|
|
8
8
|
|
|
9
9
|
Used to check if a bugfix has already landed.
|
|
10
10
|
|
|
11
|
- Applies some manual massaging, as erlang likes to report versios number not compatible with
|
|
11
|
+ Applies some manual massaging, as erlang likes to report versions number not compatible with
|
|
12
12
|
SemVer. If we can't parse the version, to minimize false positives, we assume it's newer.
|
|
13
13
|
|
|
14
14
|
Only the `version_to_check` is treated loosely. `version_to_check` must be SemVer compatible,
|
|
|
@@ -16,68 +16,76 @@ defmodule Benchee.Utility.ErlangVersion do
|
|
16
16
|
|
|
17
17
|
## Examples
|
|
18
18
|
|
|
19
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0.0", "22.0.0")
|
|
19
|
+ iex> includes_fixes_from?("22.0.0", "22.0.0")
|
|
20
20
|
true
|
|
21
21
|
|
|
22
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0.1", "22.0.0")
|
|
22
|
+ iex> includes_fixes_from?("22.0.1", "22.0.0")
|
|
23
23
|
true
|
|
24
24
|
|
|
25
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0.0", "22.0.1")
|
|
25
|
+ iex> includes_fixes_from?("22.0.0", "22.0.1")
|
|
26
26
|
false
|
|
27
27
|
|
|
28
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0.4", "22.0.5")
|
|
28
|
+ iex> includes_fixes_from?("22.0.4", "22.0.5")
|
|
29
29
|
false
|
|
30
30
|
|
|
31
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0.4", "22.0.4")
|
|
31
|
+ iex> includes_fixes_from?("22.0.4", "22.0.4")
|
|
32
32
|
true
|
|
33
33
|
|
|
34
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0.5", "22.0.4")
|
|
34
|
+ iex> includes_fixes_from?("22.0.5", "22.0.4")
|
|
35
35
|
true
|
|
36
36
|
|
|
37
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("21.999.9999", "22.0.0")
|
|
37
|
+ iex> includes_fixes_from?("21.999.9999", "22.0.0")
|
|
38
38
|
false
|
|
39
39
|
|
|
40
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("23.0.0", "22.0.0")
|
|
40
|
+ iex> includes_fixes_from?("23.0.0", "22.0.0")
|
|
41
41
|
true
|
|
42
42
|
|
|
43
43
|
# weird longer version numbers work
|
|
44
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0.0.0", "22.0.0")
|
|
44
|
+ iex> includes_fixes_from?("22.0.0.0", "22.0.0")
|
|
45
45
|
true
|
|
46
46
|
|
|
47
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0.0.14", "22.0.0")
|
|
47
|
+ iex> includes_fixes_from?("22.0.0.14", "22.0.0")
|
|
48
48
|
true
|
|
49
49
|
|
|
50
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("23.3.5.14", "22.0.0")
|
|
50
|
+ iex> includes_fixes_from?("23.3.5.14", "22.0.0")
|
|
51
51
|
true
|
|
52
52
|
|
|
53
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("21.3.5.14", "22.0.0")
|
|
53
|
+ iex> includes_fixes_from?("21.3.5.14", "22.0.0")
|
|
54
54
|
false
|
|
55
55
|
|
|
56
56
|
# weird shorter version numbers work
|
|
57
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0", "22.0.0")
|
|
57
|
+ iex> includes_fixes_from?("22.0", "22.0.0")
|
|
58
58
|
true
|
|
59
59
|
|
|
60
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0", "22.0.1")
|
|
60
|
+ iex> includes_fixes_from?("22.0", "22.0.1")
|
|
61
61
|
false
|
|
62
62
|
|
|
63
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.1", "22.0.0")
|
|
63
|
+ iex> includes_fixes_from?("22.1", "22.0.0")
|
|
64
64
|
true
|
|
65
65
|
|
|
66
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("21.3", "22.0.0")
|
|
66
|
+ iex> includes_fixes_from?("21.3", "22.0.0")
|
|
67
67
|
false
|
|
68
68
|
|
|
69
69
|
# rc version numbers work
|
|
70
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("22.0-rc3", "22.0.0")
|
|
70
|
+ iex> includes_fixes_from?("22.0-rc3", "22.0.0")
|
|
71
71
|
false
|
|
72
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("23.0-rc0", "22.0.0")
|
|
72
|
+ iex> includes_fixes_from?("23.0-rc0", "22.0.0")
|
|
73
|
+ true
|
|
74
|
+
|
|
75
|
+ # since we are falling back to general OTP versions now, test those as well
|
|
76
|
+ iex> includes_fixes_from?("21", "22.0.0")
|
|
77
|
+ false
|
|
78
|
+ iex> includes_fixes_from?("22", "22.0.0")
|
|
79
|
+ true
|
|
80
|
+ iex> includes_fixes_from?("23.0", "22.0.0")
|
|
73
81
|
true
|
|
74
82
|
|
|
75
83
|
# completely broken versions are assumed to be good to avoid false positives
|
|
76
84
|
# as this is not a main functionality but code to potentially work around an older erlang
|
|
77
85
|
# bug.
|
|
78
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("super erlang", "22.0.0")
|
|
86
|
+ iex> includes_fixes_from?("super erlang", "22.0.0")
|
|
79
87
|
true
|
|
80
|
- iex> Benchee.Utility.ErlangVersion.includes_fixes_from?("", "22.0.0")
|
|
88
|
+ iex> includes_fixes_from?("", "22.0.0")
|
|
81
89
|
true
|
|
82
90
|
"""
|
|
83
91
|
def includes_fixes_from?(version_to_check, reference_version) do
|
|
|
@@ -104,6 +112,7 @@ defmodule Benchee.Utility.ErlangVersion do
|
|
104
112
|
case dot_count do
|
|
105
113
|
3 -> Regex.replace(@last_version_segment, erlang_version, "")
|
|
106
114
|
1 -> deal_with_major_minor(erlang_version)
|
|
115
|
+ 0 -> "#{erlang_version}.0.0"
|
|
107
116
|
_ -> erlang_version
|
|
108
117
|
end
|
changed
lib/benchee/utility/file_creation.ex
|
|
@@ -65,46 +65,46 @@ defmodule Benchee.Utility.FileCreation do
|
|
65
65
|
|
|
66
66
|
## Examples
|
|
67
67
|
|
|
68
|
- iex> Benchee.Utility.FileCreation.interleave("abc.csv", "hello")
|
|
68
|
+ iex> interleave("abc.csv", "hello")
|
|
69
69
|
"abc_hello.csv"
|
|
70
70
|
|
|
71
|
- iex> Benchee.Utility.FileCreation.interleave("abc.csv", "Big Input")
|
|
71
|
+ iex> interleave("abc.csv", "Big Input")
|
|
72
72
|
"abc_big_input.csv"
|
|
73
73
|
|
|
74
|
- iex> Benchee.Utility.FileCreation.interleave("abc.csv", "String.length/1")
|
|
74
|
+ iex> interleave("abc.csv", "String.length/1")
|
|
75
75
|
"abc_string_length_1.csv"
|
|
76
76
|
|
|
77
|
- iex> Benchee.Utility.FileCreation.interleave("bench/abc.csv", "Big Input")
|
|
77
|
+ iex> interleave("bench/abc.csv", "Big Input")
|
|
78
78
|
"bench/abc_big_input.csv"
|
|
79
79
|
|
|
80
|
- iex> Benchee.Utility.FileCreation.interleave("bench/abc.csv",
|
|
80
|
+ iex> interleave("bench/abc.csv",
|
|
81
81
|
...> ["Big Input"])
|
|
82
82
|
"bench/abc_big_input.csv"
|
|
83
83
|
|
|
84
|
- iex> Benchee.Utility.FileCreation.interleave("abc.csv", [])
|
|
84
|
+ iex> interleave("abc.csv", [])
|
|
85
85
|
"abc.csv"
|
|
86
86
|
|
|
87
|
- iex> Benchee.Utility.FileCreation.interleave("bench/abc.csv",
|
|
87
|
+ iex> interleave("bench/abc.csv",
|
|
88
88
|
...> ["Big Input", "Comparison"])
|
|
89
89
|
"bench/abc_big_input_comparison.csv"
|
|
90
90
|
|
|
91
|
- iex> Benchee.Utility.FileCreation.interleave("bench/A B C.csv",
|
|
91
|
+ iex> interleave("bench/A B C.csv",
|
|
92
92
|
...> ["Big Input", "Comparison"])
|
|
93
93
|
"bench/A B C_big_input_comparison.csv"
|
|
94
94
|
|
|
95
|
- iex> Benchee.Utility.FileCreation.interleave("bench/abc.csv",
|
|
95
|
+ iex> interleave("bench/abc.csv",
|
|
96
96
|
...> ["Big Input", "Comparison", "great Stuff"])
|
|
97
97
|
"bench/abc_big_input_comparison_great_stuff.csv"
|
|
98
98
|
|
|
99
|
- iex> marker = Benchee.Benchmark.no_input
|
|
100
|
- iex> Benchee.Utility.FileCreation.interleave("abc.csv", marker)
|
|
99
|
+ iex> marker = Benchee.Benchmark.no_input()
|
|
100
|
+ iex> interleave("abc.csv", marker)
|
|
101
101
|
"abc.csv"
|
|
102
|
- iex> Benchee.Utility.FileCreation.interleave("abc.csv", [marker])
|
|
102
|
+ iex> interleave("abc.csv", [marker])
|
|
103
103
|
"abc.csv"
|
|
104
|
- iex> Benchee.Utility.FileCreation.interleave("abc.csv",
|
|
104
|
+ iex> interleave("abc.csv",
|
|
105
105
|
...> [marker, "Comparison"])
|
|
106
106
|
"abc_comparison.csv"
|
|
107
|
- iex> Benchee.Utility.FileCreation.interleave("abc.csv",
|
|
107
|
+ iex> interleave("abc.csv",
|
|
108
108
|
...> ["Something cool", marker, "Comparison"])
|
|
109
109
|
"abc_something_cool_comparison.csv"
|
|
110
110
|
"""
|
changed
mix.exs
|
|
@@ -2,7 +2,7 @@ defmodule Benchee.Mixfile do
|
|
2
2
|
use Mix.Project
|
|
3
3
|
|
|
4
4
|
@source_url "https://github.com/bencheeorg/benchee"
|
|
5
|
- @version "1.1.0"
|
|
5
|
+ @version "1.2.0"
|
|
6
6
|
|
|
7
7
|
def project do
|
|
8
8
|
[
|
|
|
@@ -26,8 +26,9 @@ defmodule Benchee.Mixfile do
|
|
26
26
|
"safe_coveralls.travis": :test
|
|
27
27
|
],
|
|
28
28
|
dialyzer: [
|
|
29
|
- flags: [:unmatched_returns, :error_handling, :race_conditions, :underspecs],
|
|
30
|
- plt_file: {:no_warn, "tools/plts/benchee.plt"}
|
|
29
|
+ flags: [:unmatched_returns, :error_handling, :underspecs],
|
|
30
|
+ plt_file: {:no_warn, "tools/plts/benchee.plt"},
|
|
31
|
+ plt_add_apps: [:table]
|
|
31
32
|
],
|
|
32
33
|
name: "Benchee",
|
|
33
34
|
description: """
|
|
|
@@ -45,16 +46,23 @@ defmodule Benchee.Mixfile do
|
|
45
46
|
end
|
|
46
47
|
|
|
47
48
|
defp deps do
|
|
48
|
- [
|
|
49
|
+ deps = [
|
|
49
50
|
{:deep_merge, "~> 1.0"},
|
|
50
51
|
{:statistex, "~> 1.0"},
|
|
51
52
|
{:ex_guard, "~> 1.3", only: :dev},
|
|
52
53
|
{:credo, "~> 1.0", only: :dev, runtime: false},
|
|
53
54
|
{:ex_doc, ">= 0.0.0", only: :dev, runtime: false},
|
|
54
55
|
{:excoveralls, "~> 0.13", only: :test},
|
|
55
|
- {:inch_ex, "~> 2.0", only: :docs, runtime: false},
|
|
56
56
|
{:dialyxir, "~> 1.0", only: :dev, runtime: false}
|
|
57
57
|
]
|
|
58
|
+
|
|
59
|
+ # table relies on __STACKTRACE__ which was introduced in 1.7, we still support ~>1.6 though
|
|
60
|
+ # as it's optional, this does not affect the function of Benchee
|
|
61
|
+ if Version.compare(System.version(), "1.7.0") == :gt do
|
|
62
|
+ [{:table, "~> 0.1.0", optional: true} | deps]
|
|
63
|
+ else
|
|
64
|
+ deps
|
|
65
|
+ end
|
|
58
66
|
end
|
|
59
67
|
|
|
60
68
|
defp package do
|