Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] FsDirectoryFactoryTests testStoreDirectory and testPreload failing #109681

Closed
piergm opened this issue Jun 13, 2024 · 2 comments · Fixed by #110224
Closed

[CI] FsDirectoryFactoryTests testStoreDirectory and testPreload failing #109681

piergm opened this issue Jun 13, 2024 · 2 comments · Fixed by #110224
Assignees
@arteam
Labels
:Distributed/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@piergm
Copy link
Member

piergm commented Jun 13, 2024

Build scan:
https://gradle-enterprise.elastic.co/s/7ffiubbjjlv4c/tests/:server:test/org.elasticsearch.index.store.FsDirectoryFactoryTests/testStoreDirectory

Reproduction line:

./gradlew ':server:test' --tests "org.elasticsearch.index.store.FsDirectoryFactoryTests.testStoreDirectory" -Dtests.seed=C0F3024401134C2E -Dbuild.snapshot=false -Dtests.jvm.argline="-Dbuild.snapshot=false" -Dtests.locale=es-GT -Dtests.timezone=America/Grenada -Druntime.java=22

Applicable branches:
main

Reproduces locally?:
Yes

Failure history:
Failure dashboard for org.elasticsearch.index.store.FsDirectoryFactoryTests#testStoreDirectory

Failure excerpt:

java.lang.AssertionError: MMAPFS (MMapDirectory@/opt/buildkite-agent/builds/bk-agent-prod-gcp-1718281231524798029/elastic/elasticsearch-periodic/server/build/testrun/test/temp/org.elasticsearch.index.store.FsDirectoryFactoryTests_C0F3024401134C2E-001/tempDir-002/fooUUID/0 lockFactory=org.apache.lucene.store.NoLockFactory@5d42e600)

  at __randomizedtesting.SeedInfo.seed([C0F3024401134C2E:36E17F36229F960F]:0)
  at org.junit.Assert.fail(Assert.java:89)
  at org.junit.Assert.assertTrue(Assert.java:42)
  at org.elasticsearch.index.store.FsDirectoryFactoryTests.doTestStoreDirectory(FsDirectoryFactoryTests.java:169)
  at org.elasticsearch.index.store.FsDirectoryFactoryTests.testStoreDirectory(FsDirectoryFactoryTests.java:147)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1570)
@piergm piergm added :Distributed/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. >test-failure Triaged test failures from CI Team:Distributed Meta label for distributed team labels Jun 13, 2024
piergm added a commit that referenced this issue Jun 13, 2024
@piergm
AwaitsFix: #109681
0869f42
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@elasticsearchmachine elasticsearchmachine added the needs:risk Requires assignment of a risk label (low, medium, blocker) label Jun 13, 2024
@arteam arteam self-assigned this Jun 19, 2024
@arteam arteam added the low-risk An open issue or test failure that is a low risk to future releases label Jun 19, 2024
@elasticsearchmachine elasticsearchmachine removed the needs:risk Requires assignment of a risk label (low, medium, blocker) label Jun 19, 2024
arteam added a commit that referenced this issue Jun 23, 2024
@arteam
Add more logging for FsDirectoryFactoryTests#testStoreDirectory
6660a50 
Test failed on verifying the type of the Lucene directory for the type `MMAPFS`. But in the logs you can see that the
instance being checked is a `MMapDirectory`. It seems that impossible to fail the `instanceof` check.

Unnmute the test (which doesn't seem muted, because I can test runs on CI) and log the full class name of the directory
class being checked.

See #109681
@arteam arteam mentioned this issue Jun 23, 2024
Merged
arteam added a commit that referenced this issue Jun 24, 2024
@arteam
Add more logging for FsDirectoryFactoryTests#testStoreDirectory (#110078
95c9aad 
)

Test failed on verifying the type of the Lucene directory for the type `MMAPFS`. But in the logs you can see that the
instance being checked is a `MMapDirectory`. It seems that impossible to fail the `instanceof` check.

Unnmute the test (which doesn't seem muted, because I can test runs on CI) and log the full class name of the directory
class being checked.

See #109681
@ywangd
Copy link
Member

ywangd commented Jun 27, 2024

New failure with enhanced log message https://gradle-enterprise.elastic.co/s/p5utmwk7k2mqc/tests/task/:server:test/details/org.elasticsearch.index.store.FsDirectoryFactoryTests/testStoreDirectory?top-execution=1

The failures are all for release-tests where the madv_random feature flag is disabled which leads to wrap the MMapDirectory with a FilterDirectory.

See also #110210 and #110211

arteam added a commit that referenced this issue Jun 27, 2024
@arteam
Fix FsDirectoryFactoryTests
2d80dac 
Unwrap the created directory to get access to original directory doing the work,
so we are not affected by directory wrappers enabled by feature flags

Resolves #109681, #110210, #110211
@arteam arteam mentioned this issue Jun 27, 2024
Merged
@arteam arteam closed this as completed in 2f3f09f Jun 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@arteam arteam

Labels
:Distributed/Store Issues around managing unopened Lucene indices. If it touches Store.java, this is a likely label. low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix FsDirectoryFactoryTests elastic/elasticsearch
4 participants
@arteam @ywangd @elasticsearchmachine @piergm