Do not run benchmark prewarm in parallel with the benchmark (#1584)

Looks like ProgramTest defaults to enabling t.Parallel(), and this
defeats the purpose of prewarm phase in provider benchmarks. The intent
of prewarm was to do a no-op to make sure dependencies are downloaded
and avoid measuring that overhead as part of the benchmark. This was not
the case though because prewarm was running in parallel with the actual
benchmark.

After the change, each individual benchmark first does unmeasured
prewarm, and then does the measurement. In the future we may want to
measure prewarm as it is interesting for user experience as well.

misc/test/performance_test.go

@@ -230,14 +230,17 @@ func programTestAsBenchmark(
  SkipUpdate: true,
  AllowEmptyPreviewChanges: true,
  AllowEmptyUpdateChanges: true,
+ NoParallel: true,
  })
  prewarmOptions.ExtraRuntimeValidation = nil
  integration.ProgramTest(t, &prewarmOptions)
  })
 
  // Run with --tracing to record measured data.
  t.Run("benchmark", func(t *testing.T) {
- finalOptions := test.With(bench.ProgramTestOptions())
+ finalOptions := test.With(bench.ProgramTestOptions()).With(integration.ProgramTestOptions{
+ NoParallel: true,
+ })
  integration.ProgramTest(t, &finalOptions)
  })
 }