Do not run benchmark prewarm in parallel with the benchmark #1584
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Looks like ProgramTest defaults to enabling t.Parallel(), and this defeats the purpose of prewarm phase in provider benchmarks. The intent of prewarm was to do a no-op to make sure dependencies are downloaded and avoid measuring that overhead as part of the benchmark. This was not the case though because prewarm was running in parallel with the actual benchmark. After the change, each individual benchmark first does unmeasured prewarm, and then does the measurement. In the future we may want to measure prewarm as it is interesting for user experience as well.
VenelinMartinov
approved these changes
Feb 12, 2024
iwahbe
approved these changes
Feb 12, 2024
|
🍹 The Update for pulumi/k8s-ci-cluster/dcb98b0ba0ac5801fe1b47555bcddeadccc49832-1687 was successful. Resource Changes Name Type Operation
+ multicloud pulumi-kubernetes:ci:GkeCluster create
+ password random:index/randomPassword:RandomPassword create
+ ephemeral-ci-cluster gcp:container/cluster:Cluster create
+ primary-node-pool gcp:container/nodePool:NodePool create
+ gke pulumi:providers:kubernetes create
+ k8s-ci-cluster-dcb98b0ba0ac5801fe1b47555bcddeadccc49832-1687 pulumi:pulumi:Stack create
|
|
Looks like the test failure is an instance of pulumi/pulumi-gcp#1387. I closed the original issue due to a missing repro, but I am re-opening. |
|
🍹 The Destroy for pulumi/k8s-ci-cluster/dcb98b0ba0ac5801fe1b47555bcddeadccc49832-1687 was successful. Resource Changes Name Type Operation
- gke pulumi:providers:kubernetes delete
- primary-node-pool gcp:container/nodePool:NodePool delete
- ephemeral-ci-cluster gcp:container/cluster:Cluster delete
- multicloud pulumi-kubernetes:ci:GkeCluster delete
- password random:index/randomPassword:RandomPassword delete
- k8s-ci-cluster-dcb98b0ba0ac5801fe1b47555bcddeadccc49832-1687 pulumi:pulumi:Stack delete
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Looks like ProgramTest defaults to enabling t.Parallel(), and this defeats the purpose of prewarm phase in provider benchmarks. The intent of prewarm was to do a no-op to make sure dependencies are downloaded and avoid measuring that overhead as part of the benchmark. This was not the case though because prewarm was running in parallel with the actual benchmark.
After the change, each individual benchmark first does unmeasured prewarm, and then does the measurement. In the future we may want to measure prewarm as it is interesting for user experience as well.