Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chore][exporter/splunkhec] reuse metric buffer #27776

Merged
merged 1 commit into from
Oct 16, 2023

Conversation

atoulme
Copy link
Contributor

@atoulme atoulme commented Oct 16, 2023

Description:
Reuse the byte buffer used when encoding metrics to HEC events JSON.

Before:

goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter
Benchmark_pushMetricData_10_10_1024-10                                      7065            160183 ns/op          188811 B/op       1908 allocs/op
Benchmark_pushMetricData_10_10_8K-10                                        4526            261482 ns/op          906868 B/op       1909 allocs/op
Benchmark_pushMetricData_10_10_2M-10                                          91          14968551 ns/op        209815280 B/op      1919 allocs/op
Benchmark_pushMetricData_10_200_2M-10                                          2         659743354 ns/op        4196565376 B/op    38200 allocs/op
Benchmark_pushMetricData_100_200_2M-10                                         1        4331601167 ns/op        41966510248 B/op          380454 allocs/op
Benchmark_pushMetricData_100_200_5M-10                                         1        7943292125 ns/op        104884261848 B/op         380888 allocs/op
Benchmark_pushMetricData_compressed_10_10_1024-10                           3387            327217 ns/op          190565 B/op       1908 allocs/op
Benchmark_pushMetricData_compressed_10_10_8K-10                             2617            472772 ns/op          911666 B/op       1910 allocs/op
Benchmark_pushMetricData_compressed_10_10_2M-10                              100          10008019 ns/op        209853305 B/op      1915 allocs/op
Benchmark_pushMetricData_compressed_10_200_2M-10                               6         195419910 ns/op        4196171762 B/op    38060 allocs/op
Benchmark_pushMetricData_compressed_100_200_2M-10                              1        3452146542 ns/op        41961539632 B/op          380322 allocs/op
Benchmark_pushMetricData_compressed_100_200_5M-10                              1        8482437541 ns/op        104876172816 B/op         380973 allocs/op
Benchmark_pushMetricData_10_10_1024_MultiMetric-10                          3534            342315 ns/op          249470 B/op       3933 allocs/op
Benchmark_pushMetricData_10_10_8K_MultiMetric-10                            3778            341417 ns/op          249446 B/op       3933 allocs/op
Benchmark_pushMetricData_10_10_2M_MultiMetric-10                            3476            349783 ns/op          249450 B/op       3933 allocs/op
Benchmark_pushMetricData_10_200_2M_MultiMetric-10                            160           8013584 ns/op         4999122 B/op      78080 allocs/op
Benchmark_pushMetricData_100_200_2M_MultiMetric-10                            18          66614023 ns/op        49834769 B/op     780374 allocs/op
Benchmark_pushMetricData_100_200_5M_MultiMetric-10                            18          67741248 ns/op        49833329 B/op     780364 allocs/op
Benchmark_pushMetricData_compressed_10_10_1024_MultiMetric-10               2313            518090 ns/op          249862 B/op       3934 allocs/op
Benchmark_pushMetricData_compressed_10_10_8K_MultiMetric-10                 2253            521544 ns/op          250608 B/op       3934 allocs/op
Benchmark_pushMetricData_compressed_10_10_2M_MultiMetric-10                 2320            523312 ns/op          250555 B/op       3934 allocs/op
Benchmark_pushMetricData_compressed_10_200_2M_MultiMetric-10                  96          10784471 ns/op         4994520 B/op      78077 allocs/op
Benchmark_pushMetricData_compressed_100_200_2M_MultiMetric-10                 10         105036258 ns/op        49735663 B/op     780372 allocs/op
Benchmark_pushMetricData_compressed_100_200_5M_MultiMetric-10                 10         104137875 ns/op        49735001 B/op     780366 allocs/op

After:

goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter
Benchmark_pushMetricData_10_10_1024-10                                      8067            137939 ns/op           87229 B/op       1809 allocs/op
Benchmark_pushMetricData_10_10_8K-10                                        8440            147844 ns/op           94443 B/op       1809 allocs/op
Benchmark_pushMetricData_10_10_2M-10                                        3724            278429 ns/op         2184478 B/op       1811 allocs/op
Benchmark_pushMetricData_10_200_2M-10                                        392           3076075 ns/op         3825066 B/op      36012 allocs/op
Benchmark_pushMetricData_100_200_2M-10                                        38          28106033 ns/op        21429249 B/op     360033 allocs/op
Benchmark_pushMetricData_100_200_5M-10                                        37          28211490 ns/op        27723782 B/op     360034 allocs/op
Benchmark_pushMetricData_compressed_10_10_1024-10                           3838            304303 ns/op           87560 B/op       1809 allocs/op
Benchmark_pushMetricData_compressed_10_10_8K-10                             3948            318874 ns/op           95140 B/op       1809 allocs/op
Benchmark_pushMetricData_compressed_10_10_2M-10                             2132            499910 ns/op         2184845 B/op       1811 allocs/op
Benchmark_pushMetricData_compressed_10_200_2M-10                             176           6691759 ns/op         3816367 B/op      36012 allocs/op
Benchmark_pushMetricData_compressed_100_200_2M-10                             18          64509919 ns/op        19294689 B/op     360015 allocs/op
Benchmark_pushMetricData_compressed_100_200_5M-10                             16          64694784 ns/op        22449746 B/op     360015 allocs/op
Benchmark_pushMetricData_10_10_1024_MultiMetric-10                          3382            347123 ns/op          249432 B/op       3933 allocs/op
Benchmark_pushMetricData_10_10_8K_MultiMetric-10                            3392            350381 ns/op          249390 B/op       3933 allocs/op
Benchmark_pushMetricData_10_10_2M_MultiMetric-10                            3488            349640 ns/op          249388 B/op       3933 allocs/op
Benchmark_pushMetricData_10_200_2M_MultiMetric-10                            156           7492550 ns/op         4999467 B/op      78080 allocs/op
Benchmark_pushMetricData_100_200_2M_MultiMetric-10                            15          68214128 ns/op        49880567 B/op     780369 allocs/op
Benchmark_pushMetricData_100_200_5M_MultiMetric-10                            16          68479602 ns/op        49862135 B/op     780361 allocs/op
Benchmark_pushMetricData_compressed_10_10_1024_MultiMetric-10               2286            521065 ns/op          249857 B/op       3934 allocs/op
Benchmark_pushMetricData_compressed_10_10_8K_MultiMetric-10                 2230            533990 ns/op          250965 B/op       3934 allocs/op
Benchmark_pushMetricData_compressed_10_10_2M_MultiMetric-10                 2216            519334 ns/op          251729 B/op       3934 allocs/op
Benchmark_pushMetricData_compressed_10_200_2M_MultiMetric-10                 100          10730782 ns/op         4994400 B/op      78079 allocs/op
Benchmark_pushMetricData_compressed_100_200_2M_MultiMetric-10                 10         102336283 ns/op        49735569 B/op     780370 allocs/op
Benchmark_pushMetricData_compressed_100_200_5M_MultiMetric-10                 10         103345238 ns/op        49736361 B/op     780377 allocs/op

Benchstat results:

goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter
                                                     │   before.txt   │               after.txt               │
                                                     │     sec/op     │    sec/op     vs base                 │
_pushMetricData_10_10_1024-10                            160.2µ ± ∞ ¹   137.9µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K-10                              261.5µ ± ∞ ¹   147.8µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M-10                            14968.6µ ± ∞ ¹   278.4µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M-10                           659.743m ± ∞ ¹   3.076m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M-10                          4331.60m ± ∞ ¹   28.11m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M-10                          7943.29m ± ∞ ¹   28.21m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024-10                 327.2µ ± ∞ ¹   304.3µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K-10                   472.8µ ± ∞ ¹   318.9µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M-10                 10008.0µ ± ∞ ¹   499.9µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M-10                195.420m ± ∞ ¹   6.692m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M-10               3452.15m ± ∞ ¹   64.51m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M-10               8482.44m ± ∞ ¹   64.69m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_1024_MultiMetric-10                342.3µ ± ∞ ¹   347.1µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K_MultiMetric-10                  341.4µ ± ∞ ¹   350.4µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M_MultiMetric-10                  349.8µ ± ∞ ¹   349.6µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M_MultiMetric-10                 8.014m ± ∞ ¹   7.493m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M_MultiMetric-10                66.61m ± ∞ ¹   68.21m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M_MultiMetric-10                67.74m ± ∞ ¹   68.48m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024_MultiMetric-10     518.1µ ± ∞ ¹   521.1µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K_MultiMetric-10       521.5µ ± ∞ ¹   534.0µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M_MultiMetric-10       523.3µ ± ∞ ¹   519.3µ ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M_MultiMetric-10      10.78m ± ∞ ¹   10.73m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M_MultiMetric-10     105.0m ± ∞ ¹   102.3m ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M_MultiMetric-10     104.1m ± ∞ ¹   103.3m ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean                                                  13.98m         3.061m        -78.11%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                                     │    before.txt     │               after.txt                │
                                                     │       B/op        │     B/op       vs base                 │
_pushMetricData_10_10_1024-10                             184.39Ki ± ∞ ¹   85.18Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K-10                               885.61Ki ± ∞ ¹   92.23Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M-10                              200.095Mi ± ∞ ¹   2.083Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M-10                            4002.157Mi ± ∞ ¹   3.648Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M-10                           40022.38Mi ± ∞ ¹   20.44Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M-10                          100025.43Mi ± ∞ ¹   26.44Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024-10                  186.10Ki ± ∞ ¹   85.51Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K-10                    890.30Ki ± ∞ ¹   92.91Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M-10                   200.132Mi ± ∞ ¹   2.084Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M-10                 4001.781Mi ± ∞ ¹   3.640Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M-10                40017.64Mi ± ∞ ¹   18.40Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M-10               100017.71Mi ± ∞ ¹   21.41Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_1024_MultiMetric-10                  243.6Ki ± ∞ ¹   243.6Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K_MultiMetric-10                    243.6Ki ± ∞ ¹   243.5Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M_MultiMetric-10                    243.6Ki ± ∞ ¹   243.5Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M_MultiMetric-10                   4.768Mi ± ∞ ¹   4.768Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M_MultiMetric-10                  47.53Mi ± ∞ ¹   47.57Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M_MultiMetric-10                  47.52Mi ± ∞ ¹   47.55Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024_MultiMetric-10       244.0Ki ± ∞ ¹   244.0Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K_MultiMetric-10         244.7Ki ± ∞ ¹   245.1Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M_MultiMetric-10         244.7Ki ± ∞ ¹   245.8Ki ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M_MultiMetric-10        4.763Mi ± ∞ ¹   4.763Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M_MultiMetric-10       47.43Mi ± ∞ ¹   47.43Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M_MultiMetric-10       47.43Mi ± ∞ ¹   47.43Mi ± ∞ ¹        ~ (p=1.000 n=1) ²
geomean                                                    25.43Mi         1.990Mi        -92.18%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05

                                                     │  before.txt  │              after.txt               │
                                                     │  allocs/op   │  allocs/op    vs base                │
_pushMetricData_10_10_1024-10                          1.908k ± ∞ ¹   1.809k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_8K-10                            1.909k ± ∞ ¹   1.809k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_2M-10                            1.919k ± ∞ ¹   1.811k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_200_2M-10                           38.20k ± ∞ ¹   36.01k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_100_200_2M-10                          380.5k ± ∞ ¹   360.0k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M-10                          380.9k ± ∞ ¹   360.0k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024-10               1.908k ± ∞ ¹   1.809k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_8K-10                 1.910k ± ∞ ¹   1.809k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_2M-10                 1.915k ± ∞ ¹   1.811k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_200_2M-10                38.06k ± ∞ ¹   36.01k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M-10               380.3k ± ∞ ¹   360.0k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M-10               381.0k ± ∞ ¹   360.0k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_10_10_1024_MultiMetric-10              3.933k ± ∞ ¹   3.933k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_10_10_8K_MultiMetric-10                3.933k ± ∞ ¹   3.933k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_10_10_2M_MultiMetric-10                3.933k ± ∞ ¹   3.933k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_10_200_2M_MultiMetric-10               78.08k ± ∞ ¹   78.08k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_100_200_2M_MultiMetric-10              780.4k ± ∞ ¹   780.4k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_100_200_5M_MultiMetric-10              780.4k ± ∞ ¹   780.4k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_10_10_1024_MultiMetric-10   3.934k ± ∞ ¹   3.934k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_10_10_8K_MultiMetric-10     3.934k ± ∞ ¹   3.934k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_10_10_2M_MultiMetric-10     3.934k ± ∞ ¹   3.934k ± ∞ ¹       ~ (p=1.000 n=1) ³
_pushMetricData_compressed_10_200_2M_MultiMetric-10    78.08k ± ∞ ¹   78.08k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_2M_MultiMetric-10   780.4k ± ∞ ¹   780.4k ± ∞ ¹       ~ (p=1.000 n=1) ²
_pushMetricData_compressed_100_200_5M_MultiMetric-10   780.4k ± ∞ ¹   780.4k ± ∞ ¹       ~ (p=1.000 n=1) ²
geomean                                                26.34k         25.62k        -2.74%
¹ need >= 6 samples for confidence interval at level 0.95
² need >= 4 samples to detect a difference at alpha level 0.05
³ all samples are equal

@dmitryax dmitryax merged commit c044b5a into open-telemetry:main Oct 16, 2023
80 checks passed
@github-actions github-actions bot added this to the next release milestone Oct 16, 2023
@atoulme atoulme deleted the reuse_metric_buffer branch October 16, 2023 23:49
JaredTan95 pushed a commit to openinsight-proj/opentelemetry-collector-contrib that referenced this pull request Oct 18, 2023
Reuse the byte buffer used when encoding metrics to HEC events JSON.
jmsnll pushed a commit to jmsnll/opentelemetry-collector-contrib that referenced this pull request Nov 12, 2023
Reuse the byte buffer used when encoding metrics to HEC events JSON.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants