Upgrade to Spark 3.3.0

[eycho-am]

Issue #, if available:
#437

Description of changes:
Update to use Spark 3.3.0

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[eycho-am]

Output from running mvn clean install:

Run completed in 5 minutes, 41 seconds.
Total number of tests run: 453
Suites: completed 53, aborted 0
Tests: succeeded 453, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
22/12/07 10:11:31 INFO ShutdownHookManager: Shutdown hook called
22/12/07 10:11:31 INFO ShutdownHookManager: Deleting directory /private/var/folders/df/80nzcb4d0f3gn753lffh0fnw0000gr/T/spark-49d311ec-c505-4d5f-9867-98fc8d75a759
[INFO] 
[INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ deequ ---
[INFO] Building jar: /Users/eycho/Documents/deequ/target/deequ_2.12-2.0.2-spark-3.3.jar
[INFO] 
[INFO] --- maven-jar-plugin:3.2.0:test-jar (default) @ deequ ---
[INFO] Building jar: /Users/eycho/Documents/deequ/target/deequ_2.12-2.0.2-spark-3.3-tests.jar
[INFO] 
[INFO] --- scalastyle-maven-plugin:1.0.0:check (default) @ deequ ---
Saving to outputFile=/Users/eycho/Documents/deequ/target/scalastyle-output.xml
Processed 167 file(s)
Found 0 errors
Found 0 warnings
Found 0 infos
Finished in 5187 ms
[INFO] 
[INFO] --- maven-install-plugin:2.4:install (default-install) @ deequ ---
[INFO] Installing /Users/eycho/Documents/deequ/target/deequ_2.12-2.0.2-spark-3.3.jar to /Users/eycho/.m2/repository/com/amazon/deequ/deequ/2.0.2-spark-3.3/deequ-2.0.2-spark-3.3.jar
[INFO] Installing /Users/eycho/Documents/deequ/pom.xml to /Users/eycho/.m2/repository/com/amazon/deequ/deequ/2.0.2-spark-3.3/deequ-2.0.2-spark-3.3.pom
[INFO] Installing /Users/eycho/Documents/deequ/target/deequ_2.12-2.0.2-spark-3.3-tests.jar to /Users/eycho/.m2/repository/com/amazon/deequ/deequ/2.0.2-spark-3.3/deequ-2.0.2-spark-3.3-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:58 min
[INFO] Finished at: 2022-12-07T10:11:38-05:00
[INFO] ------------------------------------------------------------------------

[shehzad-qureshi]

Need to also update the docs and maven build files.

[raghachellappa]

@shehzad-qureshi @eycho-am - Any idea when is the next official release planned (to support Spark 3.3)? I see the merge has been done, hence asking.

[cmachgodaddy]

@eycho-am @shehzad-qureshi , what is your java version? I couldn't get it to build with this spark 3.3.0 . I am running into this error:

MetricsRepositoryMultipleResultsLoaderTest:
RepositoryMultipleResultsLoader
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
23/01/06 14:27:20 INFO SparkContext: Running Spark version 3.3.0
23/01/06 14:27:20 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
23/01/06 14:27:21 INFO ResourceUtils: ==============================================================
23/01/06 14:27:21 INFO ResourceUtils: No custom resources configured for spark.driver.
23/01/06 14:27:21 INFO ResourceUtils: ==============================================================
23/01/06 14:27:21 INFO SparkContext: Submitted application: test
23/01/06 14:27:21 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
23/01/06 14:27:21 INFO ResourceProfile: Limiting resource is cpu
23/01/06 14:27:21 INFO ResourceProfileManager: Added ResourceProfile id: 0
23/01/06 14:27:21 INFO SecurityManager: Changing view acls to: cmach
23/01/06 14:27:21 INFO SecurityManager: Changing modify acls to: cmach
23/01/06 14:27:21 INFO SecurityManager: Changing view acls groups to:
23/01/06 14:27:21 INFO SecurityManager: Changing modify acls groups to:
23/01/06 14:27:21 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(cmach); groups with view permissions: Set(); users with modify permissions: Set(cmach); groups with modify permissions: Set()
23/01/06 14:27:21 INFO Utils: Successfully started service 'sparkDriver' on port 61507.
23/01/06 14:27:21 INFO SparkEnv: Registering MapOutputTracker
23/01/06 14:27:21 INFO SparkEnv: Registering BlockManagerMaster
23/01/06 14:27:21 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/01/06 14:27:21 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
*** RUN ABORTED ***
java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x54307ae0) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x54307ae0
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213)
at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala)
at org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:114)
at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:353)
at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:290)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:339)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:194)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:279)
at org.apache.spark.SparkContext.(SparkContext.scala:464)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)

[lsgrep]

I want to use Deequ with Spark 3.3.0 as well. Is there an official date for the 3.3.0 release? Thank you

[shehzad-qureshi]

released to maven and tagged in this repo.