[BEAM-2377] Allow usage of scala 2.11 dependencies as well as cross compilation for flink runner

[neurolabs]

Flink allows being built against scala 2.11. But the Flink Runner did
not.

This commit alleviates that, as well as allowing for ensuring that
builds work against scala 2.11 dependencies for flink. It introduces a
flink.scala.version mvn property that is set to 2.10 as a default, as well as
a mvn profile that overrides the flink scala version to 2.11.

It would be nice if you could cross publish the flink runner for
scala-2.11 based on this pull request.

[coveralls]

Coverage increased (+0.01%) to 70.7% when pulling 3e2564d on neurolabs:master into 27dd7a9 on apache:master.

[asfbot]

Build finished.
--none--

[neurolabs]

scala-2.11 build fails in sdks/java/io/hadoop/jdk1.8-tests due to dependency issue. Investigating.

[neurolabs]

Cleaned up the pull request, there were unexplainable dependency problems with enabling cross compilation for spark/scala-2.11, so I removed that. My itch is scratched with support for flink-runner/scala-2.11.

[asfbot]

Build finished.
--none--

[davorbonaci]

R: @davorbonaci @aljoscha

[aljoscha]

@davorbonaci I commented on https://issues.apache.org/jira/browse/BEAM-2377. I'm wondering how we should integrate this with the snapshot release process and the actual release process.

[aljoscha]

@davorbonaci What do you think we should do for integrating different Scala versions into the release process? Create a new nightly snapshot Jenkins job that also deploys Scala 2.11 builds? And for actual releases we probably have to update the release instructions or do you think it's possible to automate this?

[kennknowles]

If Flink users choose a Scala version first and then a corresponding Flink version, I would very much like to support the Beam FlinkRunner working simply for these users. How is this done for Flink?

[aljoscha]

With Flink you have to use the right dependencies for your Scala version, i.e. with Scala 2.10 you would have flink-streaming-java_2.10 and with Scala 2.11 you would have flink-streaming-java_2.11. It is very unfortunate that our Scala dependency is shining through here and at some point we might be able to resolve that. But not now.

[aljoscha]

What I see as an open question is how releases would work. We use maven-release-plugin which does the publishing to maven, creating the release tag and updating the branch. If we want to release for several Scala versions we would have to do this several times, maybe? But only for the publishing-maven-artifacts part.

[aljoscha]

@neurolabs I'd like to finally merge this, however, I switched the roles of 2.10 and 2.11 in my local branch and I'd like it to merge like this. I amended your commit. Would that be ok for you?

[neurolabs]

Sure, go ahead.

[aljoscha]

@neurolabs Thanks for working on this! 😃

[davorbonaci]

(And thanks @aljoscha for helping and pushing it across the finish line, despite the lack of response/action from me -- much appreciated.)

[charlesccychen]

Jenkins presubmit runs are failing with the following error:

[EnvInject] - Variables injected successfully.
Parsing POMs
Downloaded artifact https://repo.maven.apache.org/maven2/org/apache/apache/18/apache-18.pom
ERROR: Failed to parse POMs
java.io.IOException: remote file operation failed: /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2 at hudson.remoting.Channel@7b1b4155:beam7: hudson.remoting.ProxyException: hudson.maven.MavenModuleSetBuild$MavenExecutionException: org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[ERROR] 'dependencies.dependency.version' for org.apache.beam:beam-runners-flink_2.10:jar is missing. @ line 68, column 21
[WARNING] 'artifactId' contains an expression but should be a constant. @ org.apache.beam:beam-runners-flink_${flink.scala.version}:2.2.0-SNAPSHOT, /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/runners/flink/pom.xml, line 29, column 15

	at hudson.FilePath.act(FilePath.java:993)
	at hudson.FilePath.act(FilePath.java:975)
	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.parsePoms(MavenModuleSetBuild.java:985)
	at hudson.maven.MavenModuleSetBuild$MavenModuleSetBuildExecution.doRun(MavenModuleSetBuild.java:690)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:490)
	at hudson.model.Run.execute(Run.java:1735)
	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:542)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:405)
Caused by: hudson.remoting.ProxyException: hudson.maven.MavenModuleSetBuild$MavenExecutionException: org.apache.maven.project.ProjectBuildingException: Some problems were encountered while processing the POMs:
[...]

https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/14503/console

I would like to propose reverting this change to unblock presubmits and development at head.