-
Notifications
You must be signed in to change notification settings - Fork 639
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ISSUE #4711] Properly close resources used by WatchFileManagerTest #4716
[ISSUE #4711] Properly close resources used by WatchFileManagerTest #4716
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Welcome to the Apache EventMesh community!!
This is your first PR in our project. We're very excited to have you onboard contributing. Your contributions are greatly appreciated!
Please make sure that the changes are covered by tests.
We will be here shortly.
Let us know if you need any help!
Want to get closer to the community?
WeChat Assistant | WeChat Public Account | Slack |
---|---|---|
Join Slack Chat |
Mailing Lists:
Name | Description | Subscribe | Unsubscribe | Archive |
---|---|---|---|---|
Users | User support and questions mailing list | Subscribe | Unsubscribe | Mail Archives |
Development | Development related discussions | Subscribe | Unsubscribe | Mail Archives |
Commits | All commits to repositories | Subscribe | Unsubscribe | Mail Archives |
Issues | Issues or PRs comments and reviews | Subscribe | Unsubscribe | Mail Archives |
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #4716 +/- ##
============================================
+ Coverage 17.40% 17.54% +0.14%
- Complexity 1759 1778 +19
============================================
Files 797 797
Lines 29856 29872 +16
Branches 2579 2580 +1
============================================
+ Hits 5195 5240 +45
+ Misses 24180 24150 -30
- Partials 481 482 +1 ☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@tylerbertrand Am I encountering these exceptions because the cache hasn't been updated? |
@Pil0tXia I'm not sure what is causing these exceptions, but it shouldn't have anything to do with caching. Caching would only affect whether or not the tests are executed, not their behavior. I executed the tests on a commit before we enabled caching and before we made any changes to this test, and the exceptions are still being logged there. |
I have two questions.
|
Same on my end. This exception has gone unnoticed by both the CI and non-Windows platform developers. I think we need to invoke the But I'm not sure why only the Windows platform is throwing this exception. |
@Pil0tXia it's happening on my Mac as well, but it doesn't fail the test. |
@Pil0tXia Is that something you wanted to address in this PR or is this one good to go? |
@tylerbertrand I would appreciate your assistance in resolving this thrown exception. |
@Pil0tXia Sure, I'll see what I can do. |
The WatchService was being prematurely auto-closed, resulting in ClosedWatchService exceptions
7bea551
@Pil0tXia @pandaapo So the exception is caused by the fact that the I don't have so much domain knowledge here, but would this latest commit be a reasonable fix? If the watch service fails to initialize, there is nothing to close, so a |
Hmm...it looks like the assertions in the test are still not being called |
…nge event is detected
Ok, I realized the problem is that the assertions were actually being executed, but they were failing in another thread, so they did not fail the test. To get around this, I pushed a commit that restructures the test to use Mockito to verify that the The one quirk is that it seems to take a bit for the Let me know what you think. |
The meaning I expressed in point 2 of #4716 (comment) is consistent with the modification you made here.
I agree with the validation approach you modified. |
The 7bea551 passed the test however the 2772731 failed on my IDEA (win11). org.mockito.exceptions.verification.TooManyActualInvocations:
fileChangeListener.onChanged(
<custom argument matcher>
);
Wanted 1 time:
-> at org.apache.eventmesh.common.file.WatchFileManagerTest.testWatchFile(WatchFileManagerTest.java:63)
But was 2 times:
-> at org.apache.eventmesh.common.file.WatchFileTask.precessWatchEvent(WatchFileTask.java:127)
-> at org.apache.eventmesh.common.file.WatchFileTask.precessWatchEvent(WatchFileTask.java:127)
at org.apache.eventmesh.common.file.WatchFileManagerTest.testWatchFile(WatchFileManagerTest.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:110)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:90)
at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:85)
at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.stop(Unknown Source)
at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74) |
FileWriter fw = new FileWriter(tempConfigFile); | ||
properties.store(fw, "newAdd"); | ||
try ( | ||
BufferedReader bufferedReader = new BufferedReader(new FileReader(configFile)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should tempConfigFile
be used to create FileReader
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Because creating a new FileWriter
actually clears the file, the temp config file was empty when Properties.load()
was called, so I instead created the reader on the original config file to ensure that all properties were read. I'll just split this into two try-with-resources blocks instead - I think that is a better way to handle it. The first block will create the buffered reader and load the properties from the temp config file, the second will create the file writer and write the updated properties.
properties.store(fw, "newAdd"); | ||
} | ||
|
||
Mockito.verify(mockFileChangeListener, Mockito.timeout(15_000)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps you can use Mockito.timeout(15_000).atLeast(1)
to address TooManyActualInvocations
error in certain environments. But I don't know why the error occurs in some environments and not in others.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I bet it is a timing thing. Creating the FileWriter
actually clears the file, so depending on when the WatchService
begins monitoring the file, it may pick that up as well, resulting in two calls to onChanged()
. I think it makes sense to verify it's called at least once, like you suggest.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pil0tXia see above for the likely reason why the test failed
…Test This change ensures the properties in tempConfigFile are loaded into the properties object before the tempConfigFile is cleared by creating the new FileWriter
Depending on when the WatchService begins monitoring for file changes, it is possible that creating the new FileWriter, which clears the provided file, will trigger an onChange call, in addition to the onChange call triggered by properties.store().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test passed on my Win11 IDEA. You've done a good job! Thanks a lot.
} catch (IOException ex) { | ||
try { | ||
this.watchService.close(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The watchService
created in WatchFileTask
will not close until the EventMesh server shutdown now. The WatchFileTask
instances are few and I think it's fine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you saying this does not to be closed here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tylerbertrand What @pandaapo mentioned should be here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pil0tXia @pandaapo Is this commit what you had in mind?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see my latest #4716 (review)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tylerbertrand If you are also concerned about the release of WatchService
mentioned in #4716 (comment), perhaps you can close WatchService
in WatchFileTask#shutdown()
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pil0tXia @pandaapo Is this commit what you had in mind?
No, the design of the WatchFileTask
's lifecycle is application-level, while the original lifecycle of the watchService
was terminating as soon as the class was constructed, which seemed to be terminating too early. Therefore, I pointed out the change you made.
try { | ||
this.watchService.close(); | ||
} catch (IOException e) { | ||
throw new RuntimeException("Unable to close WatchService", e); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In your latest commit, you added the shutdown handling for the watchService
in the WatchFileTask#shutdown()
method. However, this doesn't actually change anything because the shutdown()
method is only called before the application is closed. Nevertheless, I think the added code can still be kept.
try { | ||
this.watchService.close(); | ||
} catch (IOException e) { | ||
ex.addSuppressed(e); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
However, you removed the part where the watchService
is closed in the exception handling, and I think that this part should be retained.
989c962
to
23b9ee4
Compare
@Pil0tXia Ah, my misunderstanding, thanks for clarifying. I re-added closing |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't worry, it's fine😊
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this commit what you had in mind?
Yes. WatchFileTask
primarily relies on WatchService
to function, so it is natural and correct to terminate WatchService
in the shutdown()
method of WatchFileTask
. In addition to this, with such a modification, WatchService
can be terminated not only when Java application executes the shutdown hook, but also when WatchFileManager#deregisterFileChangeListener()
is called.
Fixes #4711
Motivation
The
@TempDir
added in the Build caching optimizations PR caused tests to fail when run on a Windows machine. The failure was due to the fact that the directory could not be deleted because it was in use by other processes.Modifications
The resources holding on to the temporary config file that were preventing the temp directory from being deleted are now properly closed.
Documentation