YARN-11045. ATSv2 storage monitor fails to read from hbase cluster #3796

virajjasani · 2021-12-13T19:42:58Z

Description of PR

HBase compatible guava dependency is bit messed up i.e. timelineservice-hbase modules are still being built with Hadoop's guava version (defined in hadoop-project) and this creates issues with HBaseStorageMonitor reading records from hbase cluster:

java.lang.RuntimeException: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.getHostText()Ljava/lang/String;
        at org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:95)
        at org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.readEntities(TimelineEntityReader.java:283)
        at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseStorageMonitor.healthCheck(HBaseStorageMonitor.java:77)
        at org.apache.hadoop.yarn.server.timelineservice.storage.TimelineStorageMonitor$MonitorThread.run(TimelineStorageMonitor.java:89)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.getHostText()Ljava/lang/String;
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:260)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:233)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:394)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:368)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:143)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
        ... 3 more
Caused by: java.lang.NoSuchMethodError: com.google.common.net.HostAndPort.getHostText()Ljava/lang/String;
        at org.apache.hadoop.hbase.net.Address.getHostName(Address.java:72)
        at org.apache.hadoop.hbase.net.Address.toSocketAddress(Address.java:57)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:576)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:37250)
        at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:405)
        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:274)
        at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:62)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:231)

TimelineReaderServer logs after fixing the issue:

How was this patch tested?

Tested locally
Verified guava jar present in timelineservice/lib
Run all hbase UTs
Checked timelinereader logs to confirm the storage monitor is not failing

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?

virajjasani · 2021-12-13T19:53:02Z

@aajisaka @jojochuang @tasanuma Could you please take a look?

hadoop-yetus · 2021-12-14T02:11:42Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 20s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 34s		trunk passed
+1 💚	compile	9m 41s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	8m 24s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	8m 1s		trunk passed
+1 💚	javadoc	4m 58s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌	javadoc	0m 16s	/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in trunk failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javadoc	0m 17s	/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-2-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-2 in trunk failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	shadedclient	28m 42s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	6m 23s		the patch passed
+1 💚	compile	9m 4s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	9m 4s		the patch passed
+1 💚	compile	8m 14s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	8m 14s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	6m 26s		the patch passed
+1 💚	shellcheck	0m 23s		No new issues.
+1 💚	xml	0m 6s		The patch has no ill-formed XML file.
+1 💚	javadoc	4m 29s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌	javadoc	0m 11s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javadoc	0m 12s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-2-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-2 in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	shadedclient	28m 3s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	230m 39s	/patch-unit-hadoop-yarn-project_hadoop-yarn.txt	hadoop-yarn in the patch passed.
+1 💚	unit	1m 5s		hadoop-yarn-server-timelineservice-hbase-common in the patch passed.
+1 💚	unit	1m 0s		hadoop-yarn-server-timelineservice-hbase-client in the patch passed.
+1 💚	unit	0m 44s		hadoop-yarn-server-timelineservice-hbase-server-1 in the patch passed.
+1 💚	unit	0m 52s		hadoop-yarn-server-timelineservice-hbase-server-2 in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		386m 57s

Reason	Tests
Failed junit tests	hadoop.yarn.csi.client.TestCsiClient

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/1/artifact/out/Dockerfile
GITHUB PR	#3796
Optional Tests	dupname asflicense codespell shellcheck shelldocs compile javac javadoc mvninstall mvnsite unit shadedclient xml
uname	Linux d9af839a777b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `1c6c36e`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/1/testReport/
Max. process+thread count	952 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2 U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/1/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

virajjasani · 2021-12-14T06:38:12Z

Javadoc warnings are not relevant to this PR. They can be taken up in a separate task to fix existing checkstyle issues.

jojochuang

LGTM. Thanks a lot for reporting the bug and providing the fix!

jojochuang · 2021-12-14T08:46:52Z

Will merge later today

minni31 · 2021-12-14T09:36:56Z

@virajjasani , I tried porting your changes and I can still see yarn resourcemanager having the same issue for me.

virajjasani · 2021-12-14T09:39:59Z

@minni31 have you also done changes in bin/yarn as per this PR?

After porting the changes, building the tarball (mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true) adds guava-12.0 jar under share/hadoop/yarn/timelineservice/lib.
And as per bin/yarn change, starting timeline reader server will always add timelineservice/lib/* contents first, overriding hadoop's own guava-27.0 jar version in the classpath. Can you please confirm if this works for you?

minni31 · 2021-12-14T10:17:34Z

Yes @virajjasani, it fails, even after changes in bin/yarn

virajjasani · 2021-12-14T10:24:41Z

With the changes on this PR, it works for me.
From the timeline reader logs, can you provide classpath value? It must be at the beginning (3rd or 4th line from the beginning). Does the classpath have guava 12.0 jar? (guava 12.0 jar should be present before guava 27.0 jar)

For me, timeline reader works well with this patch. Without this patch, the Exception is thrown as mentioned on the PR description.
Let's try to understand what might be missing in your test. Which platform are you using and which command?
I tried on Mac and Linux both with command bin/yarn --daemon start timelinereader

virajjasani · 2021-12-14T13:21:15Z

Just pushed an addendum to make the same Yarn classpath change for Windows OS.

virajjasani · 2021-12-14T17:25:03Z

I had an offline sync with @minni31
The recent commit on this PR helps for TimelineReaderServer on Windows platform. However, there is an additional issue related to RM and NM (YARN-11047). I will create PR for the issue once this is merged since both changes are inter-related.
Thanks @minni31

FYI @jojochuang

hadoop-yetus · 2021-12-14T19:48:22Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 42s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	13m 5s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 39s		trunk passed
+1 💚	compile	9m 42s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	8m 25s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	8m 2s		trunk passed
+1 💚	javadoc	5m 0s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌	javadoc	0m 17s	/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in trunk failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javadoc	0m 17s	/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-2-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-2 in trunk failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	shadedclient	28m 17s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 27s		Maven dependency ordering for patch
+1 💚	mvninstall	6m 31s		the patch passed
+1 💚	compile	9m 0s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	9m 0s		the patch passed
+1 💚	compile	8m 20s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	8m 20s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	6m 25s		the patch passed
+1 💚	shellcheck	0m 23s		No new issues.
+1 💚	xml	0m 5s		The patch has no ill-formed XML file.
+1 💚	javadoc	4m 24s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌	javadoc	0m 11s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javadoc	0m 10s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-2-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-2 in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	shadedclient	28m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	230m 40s	/patch-unit-hadoop-yarn-project_hadoop-yarn.txt	hadoop-yarn in the patch passed.
+1 💚	unit	0m 52s		hadoop-yarn-server-timelineservice-hbase-common in the patch passed.
+1 💚	unit	0m 52s		hadoop-yarn-server-timelineservice-hbase-client in the patch passed.
+1 💚	unit	0m 42s		hadoop-yarn-server-timelineservice-hbase-server-1 in the patch passed.
+1 💚	unit	0m 44s		hadoop-yarn-server-timelineservice-hbase-server-2 in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		386m 24s

Reason	Tests
Failed junit tests	hadoop.yarn.csi.client.TestCsiClient

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/2/artifact/out/Dockerfile
GITHUB PR	#3796
Optional Tests	dupname asflicense codespell shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall shadedclient xml
uname	Linux e31ff50d5f08 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `86b25f3`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/2/testReport/
Max. process+thread count	949 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2 U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/2/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2021-12-15T00:34:56Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 43s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	shelldocs	0m 1s		Shelldocs was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+0 🆗	mvndep	12m 53s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 43s		trunk passed
+1 💚	compile	9m 39s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	compile	8m 21s		trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	mvnsite	8m 0s		trunk passed
+1 💚	javadoc	4m 58s		trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌	javadoc	0m 16s	/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in trunk failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javadoc	0m 17s	/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-2-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-2 in trunk failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	shadedclient	28m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 28s		Maven dependency ordering for patch
+1 💚	mvninstall	6m 25s		the patch passed
+1 💚	compile	8m 58s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
+1 💚	javac	8m 58s		the patch passed
+1 💚	compile	8m 16s		the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
+1 💚	javac	8m 16s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	mvnsite	6m 22s		the patch passed
+1 💚	shellcheck	0m 24s		No new issues.
+1 💚	xml	0m 5s		The patch has no ill-formed XML file.
+1 💚	javadoc	4m 29s		the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04
-1 ❌	javadoc	0m 12s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-1-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-1 in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
-1 ❌	javadoc	0m 11s	/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase_hadoop-yarn-server-timelineservice-hbase-server_hadoop-yarn-server-timelineservice-hbase-server-2-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt	hadoop-yarn-server-timelineservice-hbase-server-2 in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.
+1 💚	shadedclient	29m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	231m 55s	/patch-unit-hadoop-yarn-project_hadoop-yarn.txt	hadoop-yarn in the patch passed.
+1 💚	unit	1m 6s		hadoop-yarn-server-timelineservice-hbase-common in the patch passed.
+1 💚	unit	1m 1s		hadoop-yarn-server-timelineservice-hbase-client in the patch passed.
+1 💚	unit	0m 43s		hadoop-yarn-server-timelineservice-hbase-server-1 in the patch passed.
+1 💚	unit	0m 52s		hadoop-yarn-server-timelineservice-hbase-server-2 in the patch passed.
+1 💚	asflicense	1m 2s		The patch does not generate ASF License warnings.
		388m 25s

Reason	Tests
Failed junit tests	hadoop.yarn.csi.client.TestCsiClient

Subsystem	Report/Notes
Docker	ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/3/artifact/out/Dockerfile
GITHUB PR	#3796
Optional Tests	dupname asflicense codespell shellcheck shelldocs mvnsite unit compile javac javadoc mvninstall shadedclient xml
uname	Linux b8a6b50ac3e7 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19 23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `86b25f3`
Default Java	Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/3/testReport/
Max. process+thread count	946 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-1 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-yarn-server-timelineservice-hbase-server/hadoop-yarn-server-timelineservice-hbase-server-2 U: hadoop-yarn-project/hadoop-yarn
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3796/3/console
versions	git=2.25.1 maven=3.6.3 shellcheck=0.7.0
Powered by	Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org

This message was automatically generated.

minni31 · 2021-12-15T07:01:08Z

Thanks @virajjasani
This PR is fixing the issue in yarn timelinereader and issues in NM and RM can be handled as part of YARN-11047.

jojochuang · 2021-12-15T08:47:34Z

+1 based on the good faith of Viraj. Merging it now.

…pache#3796)

YARN-11045. ATSv2 storage monitor fails to read from hbase cluster

1c6c36e

jojochuang approved these changes Dec 14, 2021

View reviewed changes

change classpath for Windows

86b25f3

jojochuang merged commit f43ac31 into apache:trunk Dec 15, 2021

ashutoshcipher pushed a commit to ashutoshcipher/hadoop that referenced this pull request Dec 22, 2021

YARN-11045. ATSv2 storage monitor fails to read from hbase cluster (a…

2d1142a

…pache#3796)

HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022

YARN-11045. ATSv2 storage monitor fails to read from hbase cluster (a…

c373a7f

…pache#3796)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11045. ATSv2 storage monitor fails to read from hbase cluster #3796

YARN-11045. ATSv2 storage monitor fails to read from hbase cluster #3796

virajjasani commented Dec 13, 2021 •

edited

Loading

virajjasani commented Dec 13, 2021 •

edited

Loading

hadoop-yetus commented Dec 14, 2021

virajjasani commented Dec 14, 2021

jojochuang left a comment

jojochuang commented Dec 14, 2021

minni31 commented Dec 14, 2021

virajjasani commented Dec 14, 2021 •

edited

Loading

minni31 commented Dec 14, 2021

virajjasani commented Dec 14, 2021

virajjasani commented Dec 14, 2021

virajjasani commented Dec 14, 2021

hadoop-yetus commented Dec 14, 2021

hadoop-yetus commented Dec 15, 2021

minni31 commented Dec 15, 2021

jojochuang commented Dec 15, 2021

YARN-11045. ATSv2 storage monitor fails to read from hbase cluster #3796

YARN-11045. ATSv2 storage monitor fails to read from hbase cluster #3796

Conversation

virajjasani commented Dec 13, 2021 • edited Loading

Description of PR

How was this patch tested?

For code changes:

virajjasani commented Dec 13, 2021 • edited Loading

hadoop-yetus commented Dec 14, 2021

virajjasani commented Dec 14, 2021

jojochuang left a comment

Choose a reason for hiding this comment

jojochuang commented Dec 14, 2021

minni31 commented Dec 14, 2021

virajjasani commented Dec 14, 2021 • edited Loading

minni31 commented Dec 14, 2021

virajjasani commented Dec 14, 2021

virajjasani commented Dec 14, 2021

virajjasani commented Dec 14, 2021

hadoop-yetus commented Dec 14, 2021

hadoop-yetus commented Dec 15, 2021

minni31 commented Dec 15, 2021

jojochuang commented Dec 15, 2021

virajjasani commented Dec 13, 2021 •

edited

Loading

virajjasani commented Dec 13, 2021 •

edited

Loading

virajjasani commented Dec 14, 2021 •

edited

Loading