-
Notifications
You must be signed in to change notification settings - Fork 17.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
testing: benchmark cleanup causes inconsistent and slower benchmark time #38314
Comments
cc also @ianlancetaylor @bcmills |
Change https://golang.org/cl/227458 mentions this issue: |
For map with hint larger than BUCKETSIZE, makemap ignore allocated bucket and allocate buckets itself. So do not allocate bucket in this case, save us the cost of zeroing+assignment to the bucket. name old time/op new time/op delta NewEmptyMap-12 3.89ns ± 4% 3.88ns ± 2% ~ (p=0.939 n=19+20) NewSmallMap-12 23.3ns ± 3% 23.1ns ± 2% ~ (p=0.307 n=18+17) NewEmptyMapHintLessThan8-12 6.43ns ± 3% 6.31ns ± 2% -1.72% (p=0.000 n=19+18) NewEmptyMapHintGreaterThan8-12 159ns ± 2% 150ns ± 1% -5.79% (p=0.000 n=20+18) Benchmark run with commit ab7c174 reverted, see #38314. Fixes #20184 Change-Id: Ic021f57454c3a0dd50601d73bbd77b8faf8d93b6 Reviewed-on: https://go-review.googlesource.com/c/go/+/227458 Run-TryBot: Cuong Manh Le <[email protected]> TryBot-Result: Gobot Gobot <[email protected]> Reviewed-by: Keith Randall <[email protected]>
@ianlancetaylor @bcmills any idea on this? |
When we are talking about numbers like 5 nanoseconds, variance is normal and expected. I don't see a problem here. |
I think it's only an issue if we compare result from go1.13 vs higher version, there you can see go1.13 produce consistent result + low variance. Starting from go1.14, the result has higher variance. If you comparing go1.14 result and higher, it makes much more sense, because both of them has higher variance. Since go1.15 is released, I think we can close this, though I'm still curious whether the high variance after go1.13 is intentional. |
The higher variance is not intentional. I just don't think it matters. If you want to investigate what is causing it, by all means go ahead. But I predict that the answer will not lead to doing anything differently. |
What version of Go are you using (
go version
)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
What did you expect to see?
Low variance in benchmark.
What did you see instead?
Benchmark result has high variance, also benchmark time is higher than go1.13:
git bisect points to ab7c174
cc @rogpeppe
The text was updated successfully, but these errors were encountered: