Skip to content

Commit

Permalink
[SPARK-35672][FOLLOWUP][TESTS] Add an exclusion rule to MimaExcludes.…
Browse files Browse the repository at this point in the history
…scala for Scala 2.13

### What changes were proposed in this pull request?

This PR mitigate an issue that MiMa fails for Scala 2.13 after SPARK-35672 (apache#34120).
```
$ dev/change-scala-version.sh 2.13
$ dev/mima
...
[error] spark-core: Failed binary compatibility check against org.apache.spark:spark-core_2.13:3.2.0! Found 8 potential problems (filtered 905)
[error]  * method userClassPath()scala.collection.mutable.ListBuffer in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.userClassPath")
[error]  * method copy(java.lang.String,java.lang.String,java.lang.String,java.lang.String,Int,java.lang.String,scala.Option,scala.collection.mutable.ListBuffer,scala.Option,Int)org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy")
[error]  * synthetic method copy$default$10()Int in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy$default$10")
[error]  * synthetic method copy$default$8()scala.collection.mutable.ListBuffer in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments has a different result type in current version, where it is scala.Option rather than scala.collection.mutable.ListBuffer
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy$default$8")
[error]  * synthetic method copy$default$9()scala.Option in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments has a different result type in current version, where it is Int rather than scala.Option
[error]    filter with: ProblemFilters.exclude[IncompatibleResultTypeProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.copy$default$9")
[error]  * method this(java.lang.String,java.lang.String,java.lang.String,java.lang.String,Int,java.lang.String,scala.Option,scala.collection.mutable.ListBuffer,scala.Option,Int)Unit in class org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.this")
[error]  * the type hierarchy of object org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments is different in current version. Missing types {scala.runtime.AbstractFunction10}
[error]    filter with: ProblemFilters.exclude[MissingTypesProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend$Arguments$")
[error]  * method apply(java.lang.String,java.lang.String,java.lang.String,java.lang.String,Int,java.lang.String,scala.Option,scala.collection.mutable.ListBuffer,scala.Option,Int)org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments in object org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments does not have a correspondent in current version
[error]    filter with: ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.apply")
...
```

It's funny that the class `Arguments` is `public` but it's a member class of `CoarseGrainedExecutorBackend` which is `package private` and MiMa doesn't raise error for Scala 2.12, but adding an exclusion rule is one workaround.

### Why are the changes needed?

To keep the build stable.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Confirmed MiMa passed.
```
$ dev/change-scala-version.sh 2.13
$ dev/mima
```

Closes apache#34649 from sarutak/followup-SPARK-35672-mima.

Authored-by: Kousuke Saruta <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
sarutak authored and dongjoon-hyun committed Nov 19, 2021
1 parent 43b05e7 commit 145436c
Showing 1 changed file with 3 additions and 0 deletions.
3 changes: 3 additions & 0 deletions project/MimaExcludes.scala
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ object MimaExcludes {

// Exclude rules for 3.3.x from 3.2.0
lazy val v33excludes = v32excludes ++ Seq(
// [SPARK-35672][CORE][YARN] Pass user classpath entries to executors using config instead of command line
// This is necessary for Scala 2.13.
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.executor.CoarseGrainedExecutorBackend#Arguments.*"),
)

// Exclude rules for 3.2.x from 3.1.1
Expand Down

0 comments on commit 145436c

Please sign in to comment.