Ensure index is up to date when querying

[rnewson]

Ensure index is up to date when querying

the IndexHealthCheckTest exposed a sequence of events that lead to
test failure, but also point to an undesirable state during production.

Namely;

1. an index is opened
2. an index is updated
3. the IndexManager closes the index due to LRU pressure, and rolls back
   the changes in step 2
4. an index is queried, which returns results that don't reflect the
   changes made in step 2.

This change modifies the SearchRequest so that the caller can ensure
the index is at least up to a certain sequence.

Some attributes are added to IndexManager member variables to clarify
their valid ranges.

add --rerun to the test goal so we can run the tests again even if
the code under test has not changed.

[nickva]

It did show one failure:

024-06-06T22:09:44.508Z] Caching disabled for task ':test' because:

[2024-06-06T22:09:44.508Z]   Build cache is disabled

[2024-06-06T22:09:44.508Z] Task ':test' is not up-to-date because:

[2024-06-06T22:09:44.508Z]   Task.upToDateWhen is false.

[2024-06-06T22:09:44.508Z] Starting process 'Gradle Test Executor 1'. Working directory: /home/jenkins/workspace/jenkins-cm1_PullRequests_PR-5081/25.3.2.12/build/apache-couchdb-3.3.3-89a5769/nouveau Command: /opt/java/openjdk/bin/java -Dorg.gradle.internal.worker.tmpdir=/home/jenkins/workspace/jenkins-cm1_PullRequests_PR-5081/25.3.2.12/build/apache-couchdb-3.3.3-89a5769/nouveau/build/tmp/test/work -Dorg.gradle.native=false -javaagent:/home/jenkins/workspace/jenkins-cm1_PullRequests_PR-5081/25.3.2.12/build/apache-couchdb-3.3.3-89a5769/nouveau/build/tmp/.cache/expanded/zip_cde35f471dab581134460fc9a50e2c59/jacocoagent.jar=destfile=build/jacoco/test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false @/home/jenkins/.gradle/.tmp/gradle-worker-classpath17310477907838578221txt -Xmx512m -Dfile.encoding=US-ASCII -Duser.country=US -Duser.language=en -Duser.variant -ea worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1'

[2024-06-06T22:09:44.508Z] Successfully started process 'Gradle Test Executor 1'
[2024-06-06T22:09:45.706Z] 
[2024-06-06T22:09:45.706Z] IndexHealthCheckTest > testIndexHealthCheck(Path) STANDARD_OUT
[2024-06-06T22:09:45.706Z]     22:09:45.555 [Test worker] INFO org.apache.couchdb.nouveau.core.IndexManager -- deleting indexes below ___test9 (excluding nothing)
[2024-06-06T22:09:45.706Z]     22:09:45.586 [ForkJoinPool.commonPool-worker-3] INFO org.apache.couchdb.nouveau.core.IndexManager -- opening ___test9
[2024-06-06T22:09:46.052Z]     22:09:45.985 [Test worker] INFO org.apache.couchdb.nouveau.core.IndexManager -- closing ___test9 for cause EXPIRED
[2024-06-06T22:09:46.398Z]     22:09:46.167 [Test worker] DEBUG org.apache.couchdb.nouveau.core.IndexManager -- committed ___test9 before close
[2024-06-06T22:09:46.398Z]     22:09:46.173 [ForkJoinPool.commonPool-worker-3] INFO org.apache.couchdb.nouveau.core.IndexManager -- opening ___test9
[2024-06-06T22:09:46.743Z]     22:09:46.366 [Test worker] INFO org.apache.couchdb.nouveau.core.IndexManager -- deleting indexes below ___test9 (excluding nothing)
[2024-06-06T22:09:47.940Z] 
[2024-06-06T22:09:47.940Z] Gradle Test Executor 1 finished executing tests.
[2024-06-06T22:09:47.940Z] 
[2024-06-06T22:09:47.940Z] > Task :test

[nickva]

Interestingly, the failure is not always the same. In a previous run on main the reason for closure was SIZE while in the case above it's EXPIRED

[2024-06-06T20:54:41.651Z] IndexHealthCheckTest > testIndexHealthCheck(Path) STANDARD_OUT
[2024-06-06T20:54:41.651Z]     20:54:41.136 [Test worker] INFO org.apache.couchdb.nouveau.core.IndexManager -- deleting indexes below ___test9 (excluding nothing)
[2024-06-06T20:54:41.651Z]     20:54:41.168 [ForkJoinPool.commonPool-worker-3] INFO org.apache.couchdb.nouveau.core.IndexManager -- opening ___test9
[2024-06-06T20:54:41.996Z]     20:54:41.554 [ForkJoinPool.commonPool-worker-5] INFO org.apache.couchdb.nouveau.core.IndexManager -- closing ___test9 for cause SIZE
[2024-06-06T20:54:41.996Z]     20:54:41.596 [ForkJoinPool.commonPool-worker-5] DEBUG org.apache.couchdb.nouveau.core.IndexManager -- committed ___test9 before close
[2024-06-06T20:54:41.996Z]     20:54:41.706 [ForkJoinPool.commonPool-worker-3] INFO org.apache.couchdb.nouveau.core.IndexManager -- opening ___test9
[2024-06-06T20:54:41.996Z]     20:54:41.723 [ForkJoinPool.commonPool-worker-7] INFO org.apache.couchdb.nouveau.core.IndexManager -- closing ___test9 for cause SIZE
[2024-06-06T20:54:41.996Z]     20:54:41.890 [Test worker] INFO org.apache.couchdb.nouveau.core.IndexManager -- deleting indexes below ___test9 (excluding nothing)
[2024-06-06T20:54:42.464Z] 
[2024-06-06T20:54:42.464Z] IndexHealthCheckTest > testIndexHealthCheck(Path) FAILED
[2024-06-06T20:54:42.464Z]     org.opentest4j.AssertionFailedError: Result{isHealthy=false, message=Wrong number of search results, expected 1, got 0, duration=0, timestamp=2024-06-06T20:54:41.890Z} ==> expected: <true> but was: <false>
[2024-06-06T20:54:42.464Z]         at app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
[2024-06-06T20:54:42.464Z]         at app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
[2024-06-06T20:54:42.464Z]         at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
[2024-06-06T20:54:42.464Z]         at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
[2024-06-06T20:54:42.464Z]         at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
[2024-06-06T20:54:42.464Z]         at app//org.apache.couchdb.nouveau.health.IndexHealthCheckTest.testIndexHealthCheck(IndexHealthCheckTest.java:45)

[rnewson]

@nickva good catch, that was also because idleseconds defaults to zero.

[big-r81]

Nice 👍