-
Notifications
You must be signed in to change notification settings - Fork 13.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FLINK-17819][yarn] Fix error msg for yarn deployments when hadoop not in classpath #12317
Conversation
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit 284e50d (Mon May 25 08:32:42 UTC 2020) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
|
||
throw new IllegalStateException( | ||
"No valid command-line found. If you were targeting a Yarn cluster, please make sure to have " + | ||
"hadoop in your classpath. For more information refer to the \"Deployment & Operations\" section " + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"hadoop in your classpath. For more information refer to the \"Deployment & Operations\" section " + | |
"Hadoop in your classpath. For more information refer to the \"Deployment & Operations\" section " + |
maybe also say sth aboutHADOOP_CLASSPATH
here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually this is wrong now because we load the FallbackYarnSessionCLI
so if we reach here, it is not simply because of the classpath.
throw new IllegalStateException( | ||
"No ClusterClientFactory found to execute the application." + | ||
" If you were targeting a Yarn cluster, please make sure to have " + | ||
"hadoop in your classpath. For more information refer to the " + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"hadoop in your classpath. For more information refer to the " + | |
"Hadoop in your classpath. For more information refer to the " + |
same as above, also mention HADOOP_CLASSPATH
Thanks for the comments @aljoscha. I updated the PR accordingly. Let me know what you think. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot for this PR. It will help our users a lot when the hadoop is not correctly configured.
For yarn-session.sh
, it has the same problem. But the exception is easier to understand. Maybe it is already enough.
wangyang-pc:build-target danrtsey.wy$ ./bin/yarn-session.sh
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
flink-yarn/src/main/java/org/apache/flink/yarn/executors/YarnJobClusterExecutorFactory.java
Outdated
Show resolved
Hide resolved
return false; | ||
} | ||
|
||
private boolean originalIsActive(CommandLine commandLine) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So we are intentional to ignore the Yarn properties file here. Right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, the reason is that:
- we would have to copy more code
- I would assume that if we have the properties (hidden) file, it means that we created the Yarn cluster using the session CLI, so we already have hadoop in the classpath.
(In fact the solution here is already fragile, as whoever touches the isActive()
should know to keep both methods in sync)
Do you agree @wangyang0918 ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, i agree with you that we do not need to add the logics for Yarn magic properties file here.
The changes looks really good to me. +1 for merging. |
…t in classpath This closes #12317.
…t in classpath This closes apache#12317.
…t in classpath This closes apache#12317.
What is the purpose of the change
This PR simply changes the error message presented to the user when he/she is trying to submit to Yarn but hadoop is not in the classpath. The new message will be:
Brief change log
The changes are:
FallbackYarnSessionCli
that will be loaded by theCliFrontend
in case there is a loading error when the frontend tries to load the actualFlinkYarnSessionCli
. This cli will simply throw an exception with the above message when it detects that theFlinkYarnSessionCli
would be used but it failed to be loaded (theisActive
method).YarnJobClusterExecutorFactory
and theYarnSessionClusterExecutorFactory
when the executor cannot be instantiated. This is to cover for the case where the user tries to access a Yarn cluster through theExecutorCLI
.Verifying this change
Was verified manually and also added the
FallbackYarnSessionCliTest
.Does this pull request potentially affect one of the following parts:
@Public(Evolving)
: (yes / no)Documentation