Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Fix for centos9 build of Gluten #6183

Open
wants to merge 33 commits into
base: main
Choose a base branch
from

Conversation

deepashreeraghu
Copy link
Contributor

What changes were proposed in this pull request?

TO support building Gluten on Centos9

How was this patch tested?

I am trying to build Gluten on Centos9. Will update here once I am able to build successfully.

Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Deepa8 and others added 25 commits June 24, 2024 14:43
…pache#6130

What changes were proposed in this pull request?
Fix crash when driver send shutdown command to executor
(Fixes: apache#6122)
…ache#6192

What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)

(Fixes: apache#6178)

How was this patch tested?
Test by ut
[VL] Support KnownNullable for Spark 3.5.
f45966f17 by Deepak Majeti, Use separate headers for DWRF Reader Writer registration API (10132)
00485536f by Zac Wen, Switch to storage read if SSD cache load fails (10256)
3c2cc4b26 by Bikramjeet Vig, Fix NaN handling for in-predicate (10115)
18c4d5e2b by Kevin Wilfong, Capture MemoryArbitrationContext and ThreadDebugInfo in AsyncSource and restore them when invoking make (10186)
171174833 by Jimmy Lu, Count IO execution time in ExponentialBackoff retry policy (10286)
24f5aed63 by zhli1142015, Add support for DECIMAL input to greatest and least Spark functions (10195)
8faac7bf2 by zhli1142015, Add log Spark function (10243)
c97e7fcc8 by Kevin Wilfong, Fix parallel spills lead to crashes in approx_percentile (10268)
54b2ce9a5 by Reetika Agrawal, Add benchmark for IcebergSplitReader (9849)
dcd49ca38 by Krishna Pai, Restrict CAST of string to boolean (9833)
ca5e409aa by xiaoxmeng, Only load stripe footer in buffer input support sync load (10276)
652cf372e by Zac Wen, Fix memory cache hit underreporting in ioStats (10272)
a2366523d by yanngyoung, Add order by plan for memory arbitration fuzzer (10255)
a5b443a70 by Wei He, Update header guards in files in velox/external/date to avoid collision (10269)
… for supporting sink transform (apache#6197)

[CH] [Part 2] Support CH backend with Spark3.5 - Prepare for supporting sink transform

* [Refactor] remove duplicate codes

* Add NativeWriteChecker

* [Prepare to commit] getExtendedColumnarPostRules from Spark shim
* Support aggreate avg return decimal

* update version

* fix rebase

* add ut
Add more configs for AWS s3

spark.gluten.velox.fs.s3a.retry.mode
spark.gluten.velox.fs.s3a.connect.timeout
spark.hadoop.fs.s3a.retry.limit
spark.hadoop.fs.s3a.connection.maximum
[CH] Support flatten

Co-authored-by: liuneng1994 <[email protected]>
[VL] Fix greatest and least function tests.
[VL] Daily Update Velox Version (2024_06_25) (apache#6204)

Velox main changes:
```
1225f773f by joey.ljy, Add session timezone to Parquet PageReader (#9781)
33cdf0a97 by Wei He, Add custom input generator for lead, lag, nth_value, and ntile in WindowFuzzerTest (#8360)
82a12e165 by Deepak Majeti, Remove setup-centos8.sh (#10249)
7be328cac by Zhenyuan Zhao, Make dwrf support taking custom column reader factory (#10267)
1f981ae8f by Orri Erling, Add more size classes (#10139)
dc533655f by Masha Basmanova, Add from_unixtime(epoch, hours, minutes) Presto function (#10215)
7f547dbca by Wei He, Add custom result verifiers for min_by and max_by (#9070)
9974a3339 by Wei He, Allow logging input vectors in aggregation fuzzer (#10229)
```
What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)

(Fixes: apache#6124)

How was this patch tested?
TEST BY UT
What changes were proposed in this pull request?
(Please fill in changes proposed in this fix)

(Fixes: apache#6156)

How was this patch tested?
test by ut
* fix udf library path failed to get resolved on yarn-cluster

* fix signature

* Revert "fix udf library path failed to get resolved on yarn-cluster"

This reverts commit 11f774a.
Copy link

Run Gluten Clickhouse CI

@deepashreeraghu deepashreeraghu marked this pull request as ready for review June 26, 2024 18:24
@deepashreeraghu
Copy link
Contributor Author

@FelixYBW - I am able to build Gluten on Centos9 with these changes. Have got the jar generated, now have to consume the jar and test it.

[INFO] Reactor Summary for Gluten Parent Pom 1.2.0-SNAPSHOT:
[INFO] 
[INFO] Gluten Parent Pom .................................. SUCCESS [  0.216 s]
[INFO] Gluten UI .......................................... SUCCESS [  9.042 s]
[INFO] Gluten Shims ....................................... SUCCESS [  0.160 s]
[INFO] Gluten Shims Common ................................ SUCCESS [ 10.171 s]
[INFO] Gluten Shims for Spark 3.4 ......................... SUCCESS [  9.944 s]
[INFO] Gluten Ras ......................................... SUCCESS [  0.150 s]
[INFO] Gluten Ras Common .................................. SUCCESS [ 23.889 s]
[INFO] Gluten Core ........................................ SUCCESS [ 53.233 s]
[INFO] Gluten Data ........................................ SUCCESS [ 10.314 s]
[INFO] Gluten Backends Velox .............................. SUCCESS [ 47.250 s]
[INFO] Gluten Celeborn .................................... SUCCESS [  0.024 s]
[INFO] Gluten Celeborn Common ............................. SUCCESS [  4.708 s]
[INFO] Gluten Celeborn Velox .............................. SUCCESS [  4.012 s]
[INFO] Gluten Celeborn Package ............................ SUCCESS [  0.041 s]
[INFO] Gluten Uniffle ..................................... SUCCESS [  0.153 s]
[INFO] Gluten Uniffle Velox ............................... SUCCESS [  4.328 s]
[INFO] Gluten Uniffle Package ............................. SUCCESS [  0.212 s]
[INFO] Gluten Package ..................................... SUCCESS [01:10 min]
[INFO] Gluten Ras Planner ................................. SUCCESS [  0.158 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:09 min
[INFO] Finished at: 2024-06-26T19:58:05Z
[INFO] ------------------------------------------------------------------------
+ mkdir -p /incubator-gluten/dev/../package/target/thirdparty-lib
+ [[ centos == \u\b\u\n\t\u ]]
+ [[ centos == \p\o\p ]]
+ '[' centos == centos ']'
+ '[' 9 == 9 ']'
+ process_setup_centos_9
+ cp /lib64/libre2.so.9 /lib64/libdouble-conversion.so.3 /lib64/libevent-2.1.so.7 /lib64/libdwarf.so.0 /lib64/libgsasl.so.7 /lib64/libicudata.so.67 /lib64/libicui18n.so.67 /lib64/libicuuc.so.67 /lib64/libidn.so.12 /lib64/libntlm.so.0 /lib64/libsodium.so.23 /incubator-gluten/dev/../package/target/thirdparty-lib/
+ cp /usr/local/lib/libhdfs3.so.1 /usr/local/lib/libboost_context.so.1.84.0 /usr/local/lib/libboost_filesystem.so.1.84.0 /usr/local/lib/libboost_program_options.so.1.84.0 /usr/local/lib/libboost_regex.so.1.84.0 /usr/local/lib/libboost_system.so.1.84.0 /usr/local/lib/libboost_thread.so.1.84.0 /usr/local/lib/libboost_atomic.so.1.84.0 /usr/local/lib/libprotobuf.so.32 /incubator-gluten/dev/../package/target/thirdparty-lib/
+ cp /usr/local/lib64/libgflags.so.2.2 /usr/local/lib64/libglog.so.1 /incubator-gluten/dev/../package/target/thirdparty-lib/
+ cd /incubator-gluten/dev/../package/target/thirdparty-lib/
+ jar cvf gluten-thirdparty-lib-centos-9-x86_64.jar ./
added manifest
adding: libre2.so.9(in = 478136) (out= 209171)(deflated 56%)
adding: libdouble-conversion.so.3(in = 86680) (out= 38620)(deflated 55%)
adding: libevent-2.1.so.7(in = 366424) (out= 161644)(deflated 55%)
adding: libdwarf.so.0(in = 497200) (out= 197190)(deflated 60%)
adding: libgsasl.so.7(in = 139832) (out= 61127)(deflated 56%)
adding: libicudata.so.67(in = 28408488) (out= 11295428)(deflated 60%)
adding: libicui18n.so.67(in = 3201200) (out= 1278222)(deflated 60%)
adding: libicuuc.so.67(in = 2024088) (out= 847392)(deflated 58%)
adding: libidn.so.12(in = 213288) (out= 71597)(deflated 66%)
adding: libntlm.so.0(in = 28264) (out= 10484)(deflated 62%)
adding: libsodium.so.23(in = 362928) (out= 171315)(deflated 52%)
adding: libhdfs3.so.1(in = 40390056) (out= 13882117)(deflated 65%)
adding: libboost_context.so.1.84.0(in = 16848) (out= 2921)(deflated 82%)
adding: libboost_filesystem.so.1.84.0(in = 180784) (out= 64226)(deflated 64%)
adding: libboost_program_options.so.1.84.0(in = 534136) (out= 168539)(deflated 68%)
adding: libboost_regex.so.1.84.0(in = 397512) (out= 155365)(deflated 60%)
adding: libboost_system.so.1.84.0(in = 15648) (out= 1854)(deflated 88%)
adding: libboost_thread.so.1.84.0(in = 156392) (out= 48431)(deflated 69%)
adding: libboost_atomic.so.1.84.0(in = 18296) (out= 5742)(deflated 68%)
adding: libprotobuf.so.32(in = 10773312) (out= 2351675)(deflated 78%)
adding: libgflags.so.2.2(in = 424336) (out= 104170)(deflated 75%)
adding: libglog.so.1(in = 268064) (out= 80727)(deflated 69%)

@deepashreeraghu
Copy link
Contributor Author

@zhztheplayer - Could you please check on this PR if there is some pending action from me ?

@wanglinsong
Copy link

centos 9 or centos stream 9?
https://www.redhat.com/en/blog/faq-centos-stream-updates#Q3

@deepashreeraghu
Copy link
Contributor Author

@wanglinsong - Yes, I am using FROM quay.io/centos/centos:stream9 and building on top of it.

@FelixYBW FelixYBW changed the title Fix for centos9 build of Gluten [VL] Fix for centos9 build of Gluten Jul 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet