-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-37445][BUILD] Rename the maven hadoop profile to hadoop-3 and hadoop-2 #34715
Conversation
Kubernetes integration test starting |
Should we need to clarify this change in https://spark.apache.org/docs/latest/api/python/getting_started/install.html#manually-downloading ? cc @HyukjinKwon |
Test build #145648 has finished for PR 34715 at commit
|
Kubernetes integration test status failure |
Kubernetes integration test starting |
Test build #145653 has finished for PR 34715 at commit
|
Kubernetes integration test status failure |
Test build #145705 has finished for PR 34715 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks pretty good @AngersZhuuuu ! will take another pass later.
From a cursory look, seems fine to me too but I think it's best to defer to @sunchao |
+CC @xkrogen |
resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/ClientSuite.scala
Outdated
Show resolved
Hide resolved
sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala
Outdated
Show resolved
Hide resolved
@@ -34,8 +34,8 @@ HADOOP_MODULE_PROFILES="-Phive-thriftserver -Pmesos -Pkubernetes -Pyarn -Phive \ | |||
-Pspark-ganglia-lgpl -Pkinesis-asl -Phadoop-cloud" | |||
MVN="build/mvn" | |||
HADOOP_HIVE_PROFILES=( | |||
hadoop-2.7-hive-2.3 | |||
hadoop-3.2-hive-2.3 | |||
hadoop-2-hive-2.3 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unrelated but the hive-2.3
seems redundant, since we don't support alternative Hive profile right now
Kubernetes integration test starting |
Kubernetes integration test status failure |
Test build #145925 has finished for PR 34715 at commit
|
Kubernetes integration test starting |
Kubernetes integration test status failure |
Test build #145969 has finished for PR 34715 at commit
|
Kubernetes integration test starting |
Kubernetes integration test status failure |
Test build #145975 has finished for PR 34715 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, thanks @AngersZhuuuu !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks pretty good. After this gets merged, let's also send an email to dev mailing list about the renamed profiles to prevent other external build trains to be updated too in other forked repos.
return spark_version, convert_old_hadoop_version(spark_version, hadoop_version), hive_version | ||
|
||
|
||
def convert_old_hadoop_version(spark_version, hadoop_version): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing this together guys.
@HyukjinKwon This GA failed caused by the same SQLMetricsSuite mentioned by you in my PR.
@sunchao I think we can merge this if there are no more question.
and try to make it stable. |
Thanks, merged! |
Awesome. @AngersZhuuuu, mind sending an email to dev mailing list articulating what's the diff and impact? We would have to list up e.g.) profile name, released tar file name, pip installation option, etc. |
e.g.) if people are using hadoop-2.7 in their build, now their build will use hadoop 3 by default with warning that hadoop 2.7 profile does not exist. |
Yea. Will organize an email later |
…p3 for the scheduled GA job for branch-3.2 ### What changes were proposed in this pull request? This PR fixes an issue that the scheduled GA job for `branch-3.2` fails. SPARK-37445 (#34715), renamed the profile name `hadoop3.2` to `hadoop3` but It should be `hadoop3.2` for the scheduled build. https://github.com/apache/spark/runs/4453894964?check_suite_focus=true ### Why are the changes needed? To recover the job. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The scheduled job itself. Closes #34835 from sarutak/followup-SPARK-37445. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
What changes were proposed in this pull request?
Rename the maven hadoop profile to hadoop-3 and hadoop-2
Why are the changes needed?
Upgrade hadoop profile
Does this PR introduce any user-facing change?
n this pr, we change hadoop profile form hadoop-3 and hadoop-2, and hadoop-3 is default hadoop profile.
User can build other lower version by -Dhadoop.version=3.2.1 such as
If user want to built hadoop 2 related version. you can run
How was this patch tested?
Not need