chore: Replace benchmark with cronometro for benchmarking.

[ShogunPanda]

This PR closes #715 and #736 by replacing benchmark with cronometro.

I also refactored the benchmark NPM scripts to use wait-on instead of a flat timeout for better reliability.

[ShogunPanda]

@ronag If you run the new benchmark on the reference machine I'll update the README as well.

[ronag]

Excellent work!

[codecov-commenter]

Codecov Report

Merging #751 (ab5e0f0) into main (fb67839) will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##              main      #751   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files           24        24           
  Lines         1904      1904           
=========================================
  Hits          1904      1904           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fb67839...ab5e0f0. Read the comment docs.

[ronag]

It's a bit weird that http - keepalive is slowest?

[ShogunPanda]

To be honest I have no idea. I've checked the code and everything seems to be fine. I wonder if the socket buffers are filled up and thus performance decrease.

[ronag]

What results do you get if you run it locally? Also what order are they run in? Could it be some kind of warm up thing? Should we run each test in a separate node instance?

[ronag]

Also the multiple socket results seem weird. Maybe they always have. Do we need those? @mcollina

I think in this case pipelining is seems to a disadvantage against core http that doesn't use pipelingin.

[ShogunPanda]

Cronometro runs with "warmup" mode by default. For each test, it executes twice, the first time discarding results.
Tests are run in series and each is run in a separate worker thread so no interaction should be possible.

Here's my results:

╔════════════════════════════════════════════╤═════════╤═══════════════╤═══════════╤═════════════════════════╗
║ Slower tests                               │ Samples │        Result │ Tolerance │ Difference with slowest ║
╟────────────────────────────────────────────┼─────────┼───────────────┼───────────┼─────────────────────────╢
║ http - keepalive                           │     100 │  61.35 op/sec │  ± 1.08 % │                         ║
║ http - no agent                            │     100 │ 201.87 op/sec │  ± 2.03 % │ + 229.03 %              ║
║ http - keepalive - multiple sockets        │     100 │ 302.28 op/sec │  ± 1.92 % │ + 392.69 %              ║
║ undici - pipeline                          │     100 │ 377.43 op/sec │  ± 2.33 % │ + 515.18 %              ║
║ undici - pool - request - multiple sockets │     100 │ 392.54 op/sec │  ± 1.99 % │ + 539.80 %              ║
║ undici - request                           │     100 │ 395.92 op/sec │  ± 1.97 % │ + 545.32 %              ║
║ undici - stream                            │     100 │ 421.28 op/sec │  ± 1.91 % │ + 586.65 %              ║
║ undici - dispatch                          │     100 │ 426.49 op/sec │  ± 1.89 % │ + 595.14 %              ║
╟────────────────────────────────────────────┼─────────┼───────────────┼───────────┼─────────────────────────╢
║ Fastest test                               │ Samples │        Result │ Tolerance │ Difference with slowest ║
╟────────────────────────────────────────────┼─────────┼───────────────┼───────────┼─────────────────────────╢
║ undici - noop                              │     100 │ 538.81 op/sec │  ± 1.81 % │ + 778.21 %              ║
╚════════════════════════════════════════════╧═════════╧═══════════════╧═══════════╧═════════════════════════╝

[ronag]

Maybe http - no agent should be called http - no agent - multiple sockets? Or maybe just remove the no agent test as it is a bit confusing. I think it uses unlimited number of sockets by default.

[dnlup]

It's a bit weird that http - keepalive is slowest?

That option uses 1 socket if I am not mistaken, so I suspect that requests are run sequentially in that case.

[ronag]

It's a bit weird that http - keepalive is slowest?

That option uses 1 socket if I am not mistaken, so I suspect that requests are run sequentially in that case.

Yea, I think it's the no agent case which is misleading.

[ronag]

@ShogunPanda can we split the benchmarks into multi socket and single socket ones? That way the relative values make more sense.

[dnlup]

It's a bit weird that http - keepalive is slowest?

That option uses 1 socket if I am not mistaken, so I suspect that requests are run sequentially in that case.

Yea, I think it's the no agent case which is misleading.

I think so too because it's using the default Agent behind the scenes.

[dnlup]

Awesome work

[ronag]

Is it possible to skip the grouping headers? The fastest test is always in the bottom so it's just noise, no?

[ShogunPanda]

Is it possible to skip the grouping headers? The fastest test is always in the bottom so it's just noise, no?

Unfortunately not, that's fixed in cronometro.

I'll update benchmarks soon.

[ShogunPanda]

Benchmarks updated. We have two different suites now. Let me know if I set everything correct.

[ronag]

Not related to this to this Pr per se. It’s interesting that the multi socket version has less diff between core and undici. Maybe the server should run cluster?