NoSuchElementException when changing parameters

[iphydf]

Changing the values in a dimension crashes the regression reporter. It seems to expect all current values to exist in the previous runs. This limits the usefulness of regression testing, since there is no way to modify any parameters ever without losing all the history.

Instead of rejecting any histories without a certain value, it may be better to assume the value doesn't exist (and do no comparison), or assume the value was exactly the same as the new one (making comparison a trivial no-op).

Reproduce by running this benchmark twice, changing the value 1 to 2 the second time:

  performance of "test" in {
    using(Gen.single("values")(1)) in { value =>
      val _ = value // Irrelevant, just making sure no linters complain.
    }
  }

Results in an exception similar to the following:

[info] - test.Test-0 measurements:
Test threw exception: java.util.NoSuchElementException: key not found: Parameters(values -> 2)
java.util.NoSuchElementException: key not found: Parameters(values -> 2)
        at scala.collection.MapLike$class.default(MapLike.scala:228)
        at scala.collection.AbstractMap.default(Map.scala:59)
        at scala.collection.MapLike$class.apply(MapLike.scala:141)
        at scala.collection.AbstractMap.apply(Map.scala:59)
        at org.scalameter.reporting.RegressionReporter$Tester$OverlapIntervals$$anonfun$34.apply(RegressionReporter.scala:360)
        at org.scalameter.reporting.RegressionReporter$Tester$OverlapIntervals$$anonfun$34.apply(RegressionReporter.scala:358)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at org.scalameter.reporting.RegressionReporter$Tester$OverlapIntervals.apply(RegressionReporter.scala:358)
        at org.scalameter.reporting.RegressionReporter.report(RegressionReporter.scala:30)
        at org.scalameter.Reporter$Composite$$anonfun$report$1.apply(Reporter.scala:23)
        at org.scalameter.Reporter$Composite$$anonfun$report$1.apply(Reporter.scala:23)
        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
        at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
        at org.scalameter.Reporter$Composite.report(Reporter.scala:23)
        at org.scalameter.Executor$$anonfun$run$1.apply(Executor.scala:16)
        at org.scalameter.Executor$$anonfun$run$1.apply(Executor.scala:13)
        at org.scalameter.utils.Tree$$anonfun$1.apply(Tree.scala:17)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
        at scala.collection.immutable.List.map(List.scala:285)
        at org.scalameter.utils.Tree.map(Tree.scala:17)
        at org.scalameter.utils.Tree$$anonfun$2.apply(Tree.scala:18)
        at org.scalameter.utils.Tree$$anonfun$2.apply(Tree.scala:18)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
        at scala.collection.immutable.List.map(List.scala:285)
        at org.scalameter.utils.Tree.map(Tree.scala:18)
        at org.scalameter.Executor$class.run(Executor.scala:13)
        at org.scalameter.execution.SeparateJvmsExecutor.run(SeparateJvmsExecutor.scala:19)
        at org.scalameter.PerformanceTest$Initialization$class.executeTests(PerformanceTest.scala:45)
        at org.scalameter.PerformanceTest.executeTests(PerformanceTest.scala:11)
        at org.scalameter.ScalaMeterFramework$$anon$1$$anonfun$run$1.apply$mcZ$sp(ScalaMeterFramework.scala:57)
        at org.scalameter.ScalaMeterFramework$$anon$1$$anonfun$run$1.apply(ScalaMeterFramework.scala:53)
        at org.scalameter.ScalaMeterFramework$$anon$1$$anonfun$run$1.apply(ScalaMeterFramework.scala:53)
        at org.scalameter.package$$anonfun$withTestContext$1$$anonfun$apply$1$$anonfun$apply$2.apply(package.scala:60)
        at org.scalameter.package$$anonfun$withTestContext$1$$anonfun$apply$1$$anonfun$apply$2.apply(package.scala:59)
        at org.scalameter.MonadicDynVar$$anon$1$$anonfun$foreach$1.apply(MonadicDynVar.scala:7)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
        at org.scalameter.MonadicDynVar$$anon$1.foreach(MonadicDynVar.scala:7)
        at org.scalameter.package$$anonfun$withTestContext$1$$anonfun$apply$1.apply(package.scala:59)
        at org.scalameter.package$$anonfun$withTestContext$1$$anonfun$apply$1.apply(package.scala:58)
        at org.scalameter.MonadicDynVar$$anon$1$$anonfun$foreach$1.apply(MonadicDynVar.scala:7)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
        at org.scalameter.MonadicDynVar$$anon$1.foreach(MonadicDynVar.scala:7)
        at org.scalameter.package$$anonfun$withTestContext$1.apply(package.scala:58)
        at org.scalameter.package$$anonfun$withTestContext$1.apply(package.scala:57)
        at org.scalameter.MonadicDynVar$$anon$1$$anonfun$foreach$1.apply(MonadicDynVar.scala:7)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)
        at org.scalameter.MonadicDynVar$$anon$1.foreach(MonadicDynVar.scala:7)
        at org.scalameter.package$.withTestContext(package.scala:57)
        at org.scalameter.ScalaMeterFramework$$anon$1.run(ScalaMeterFramework.scala:52)
        at sbt.RunnerWrapper$1.runRunner2(FrameworkWrapper.java:223)
        at sbt.RunnerWrapper$1.execute(FrameworkWrapper.java:236)
        at sbt.ForkMain$Run$2.call(ForkMain.java:294)
        at sbt.ForkMain$Run$2.call(ForkMain.java:284)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[error] Uncaught exception when running im.tox.tox4j.core.bench.BenchWip: java.util.NoSuchElementException: key not found: Parameters(values -> 2)

[jilen]

I also have this problem.

[jilen]

@iphydf I found this happens when you re-run the bench. So just clean evertything before re-run may help

[iphydf]

Yes, totally :) the reason is that in the previous run the parameters had different values, and it wants to compare the current run with the previous one(s) as part of the overlap test. I would prefer it to assume there was no previous run if the parameter set is different, instead of crashing.