Stuck in GC loop even with high xmx

[maxhillaert]

Hi,

I've added cover with scalameter to an algorithm. I then intentionally make it worse by doing a bad concat of lists in a tight loop.

Problem is that its so bad it never seems to fail and scalameter output keeps saying "GC detected " and sometimes "Reinstating BenchMark Value"

The memory will have exploded to 4GB, but just sit there,, even though I set XMX 10000m.

This is how the JVM is loaded:

Starting new JVM: java -server -Xmx10048m -Xms4048m -cp C:\Users\hillmax\AppData\Local\Temp\classpath1003926506.jar;C:\Users\hillmax\AppData\Roaming\JetBrains\IntelliJ IDEA Community Edition 2018.1.1\lib\idea_rt.jar org.scalameter.execution.Main C:\Users\hillmax\AppData\Local\Temp\newjvm-5758005586671152155-io

Is XMX really working in this case? Why am I hitting a GC wall at nowhere near the heap size?

[ghost]

Did you try to decrease the default warmup iteration count, the iteration count, and the fork count? Maybe it's just that the algorithm is taking a long time.

  override def defaultConfig = Context(
    exec.minWarmupRuns -> 1,
    exec.maxWarmupRuns -> 1,
    exec.benchRuns -> 4,
    exec.independentSamples -> 1,
    verbose -> true
  )

This is taken from this example:

https://github.com/reactors-io/reactors/blob/master/reactors-common/jvm/src/bench/scala/io/reactors/common/concurrent/cache-trie-benches.scala

Another example if you're using it from Java (at the bottom of the page):

https://scalameter.github.io/home/gettingstarted/0.7/javausage/index.html

Also, if you add verbose -> true, you will see more detailed output about what ScalaMeter is doing.

[ghost]

The default ScalaMeter benchmark configuration is re-creating the dataset every once in a while, and then doing a GC, in order to reduce the probability that the GC cycle disturbs the measurement. This is done by the Measurer.PeriodicReinstantiation trait. So, the GC's that you see are possibly manually requested.

https://github.com/scalameter/scalameter/blob/master/scalameter-core/src/main/scala/org/scalameter/Measurer.scala

https://github.com/scalameter/scalameter/blob/master/scalameter-core/src/main/scala/org/scalameter/Measurer.scala#L209

You can, of course, disable this by implementing your own Measurer class, or instantiating another Measurer implementation in ScalaMeter (see source code, you should get rid of PeriodicReinstantiation).
You should, in that case, specify the measurer in your concrete test (i.e. override the measurer method). This is relatively easy to do, but it sounds that in your case the test is taking long because the default warmup iteration count might have to be reduced.

[maxhillaert]

Yeah I test a table aggregation function on a million rows and introduce bad concatenation of lists. Memory blowing up and slow just like it was in production. When I tone it down to ten thousand rows maximum it will complete the test run. Obviously when I fix the bug it can finish the million rows fine It’s just that I want to make sure if someone reintroduces the concatenation bug , it fails the test rather than get stuck in a loop trying to measure it. I say ‘loop’, but of course it could just take a very long time. My concern is the masses of GC detections in the verbose log, without it appearing to be able to move on. It sure feels like it’s stuck, but I could be wrong. I’ll try your suggestions and get back to you with some logs. I’m keen on using this framework.

…

On Fri, 29 Mar 2019 at 09:50, Cantor Chron ***@***.***> wrote: Did you try to decrease the default warmup iteration count, the iteration count, and the fork count? Maybe it's just that the algorithm is taking a long time. override def defaultConfig = Context( exec.minWarmupRuns -> 1, exec.maxWarmupRuns -> 1, exec.benchRuns -> 4, exec.independentSamples -> 1, verbose -> true ) This is taken from this example: https://github.com/reactors-io/reactors/blob/master/reactors-common/jvm/src/bench/scala/io/reactors/common/concurrent/cache-trie-benches.scala Another example if you're using it from Java (at the bottom of the page): https://scalameter.github.io/home/gettingstarted/0.7/javausage/index.html Also, if you add verbose -> true, you will see more detailed output about what ScalaMeter is doing. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#209 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwNdGQJLu5Q4vWsTcQfRM9oC6Tsqe_Iks5vbdsEgaJpZM4cO9Bi> .

[ghost]

It’s just that I want to make sure if someone reintroduces the
concatenation bug , it fails the test rather than get stuck in a loop
trying to measure it.

One approach could be to implement a custom Measurer that starts a new thread to run the workload, and then stops the thread after a timeout or exits the process.

[maxhillaert]

Yeah that would be good approach. Like a maximum test run time and fail. This way we don’t hog our CI machines unnecessarily.

…

On Fri, 29 Mar 2019 at 10:10, Cantor Chron ***@***.***> wrote: It’s just that I want to make sure if someone reintroduces the concatenation bug , it fails the test rather than get stuck in a loop trying to measure it. One approach could be to implement a custom Measurer that starts a new thread to run the workload, and then stops the thread after a timeout or exits the process. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#209 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAwNdLy1HED1DaDuu7VZcwJ7Rk3uvdVFks5vbeakgaJpZM4cO9Bi> .

[maxhillaert]

So I've tried creating a warmer with a timeout:

var i = 0
    while (i < maxwarmups) {

      setup()
      nogc = true
      val start = System.nanoTime

      if(maxRunningTime.nonEmpty){
        val ftr = Future(f(i))
        Await.result(ftr, maxRunningTime.get )
      }
      else{
        f(i)
      }

It blows up the whole test suite of course, so a bit blunt. Any idea's on how to let it end the test run more gracefully and continue with next test?

[ghost]

I would use Await.ready here, as it does not throw an exception.

But, I would not use futures at all, since Await does not cancel the computation in the future - i.e. your job will still be running. Starting a new thread, and force-stopping it with Thread.stop would be the best way to go if you want to kill the algorithm.

I think that you would then have to implement a custom Reporter that checks if a timeout occurred, and then override the reporter method in your test to decide whether the test failed or passed.