YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled #3802
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e when ATSv2 is enabled
|
Thanks @virajjasani for the quick fix. |
|
Thank you for confirming on Windows OS @minni31 FYI @jojochuang @aajisaka |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Javadoc warnings are not relevant to the PR changes and can be taken up in a separate Jira. |
jojochuang
approved these changes
Dec 17, 2021
There was a problem hiding this comment.
LGTM I believe it was broken by HADOOP-17452 and so my only question is why didn't our UT catch the issue and if we need extra tests.
| <dependency> | ||
| <groupId>com.google.inject</groupId> | ||
| <artifactId>guice</artifactId> | ||
| <version>${hbase-compatible-guice.version}</version> |
There was a problem hiding this comment.
prior to this change, the module depends on guice version 4.2.3.
Guice was updated to 4.2.3 by HADOOP-17452.
There was a problem hiding this comment.
Correct, NodeManager and ResourceManager need to use guice that is compatible with hbase-compatible guava, hence this is one heck of a complex dependency chain.
I will also post my testing results for ATSv2 in 3.3.2 RC as well in a couple of days.
I agree, we need more tests in yarn-nodemanager and yarn-resourcemanager modules. For ATSv2, timelineservice-tests module has some tests but the above two modules should also have some tests. On the other hand, hbase 1 profile will most likely not require any more version upgrade and when hbase 3 is rolled out, we can focus on providing better compatibilities and good no of tests to cover new profile (perhaps make it default profile). |
ashutoshcipher
pushed a commit
to ashutoshcipher/hadoop
that referenced
this pull request
Dec 22, 2021
…e when ATSv2 is enabled (apache#3802)
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…e when ATSv2 is enabled (apache#3802)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Yarn ResourceManager and NodeManager in the current state are not able to interact with HBase cluster because of the incompatible guava and guice dependencies being used in the classpath.
Once we allow RM and NM to use Timeline service specific guava version, it fails due to incompatibilities b/ guava and guice versions because hbase compatible guava is not guice compatible guava for Yarn:
Hence, we need to keep both guice and guava compatible with each other and ensure guava is also compatible with HBase.
Changes on this PR would allow Yarn to use guice and guava versions compatible with HBase and not the ones being derived from hadoop-project.
How was this patch tested?
Tested locally. Triggered MapReduce sample WordCount job. No issues encountered with RM and NM.
Data is also flowing from Yarn to HBase tables:
For code changes: