[BEAM-6630] upgrade to gradle 5.2

[adude3141]

Upgrade to the latest gradle 5.2.1

This will

• reenable the build cache, i.e. test results will be cacheable again.
• unlock java11 support

---

Follow this checklist to help us incorporate your contribution quickly and easily:

• Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.

Post-Commit Tests Status (on master branch)

Lang	SDK	Apex	Dataflow	Flink	Gearpump	Samza	Spark
Go		---	---	---	---	---	---
Java
Python		---			---	---	---

[adude3141]

R: @swegner

[adude3141]

We need to fix

• https://issues.apache.org/jira/browse/BEAM-6634
• https://issues.apache.org/jira/browse/BEAM-6635

[swegner]

Exciting! Some other features that 5.0 brings which we might find useful:

• "Production ready" Kotlin DSL". According to Google trends, Kotlin is now more popular than Groovy with much greater momentum. We haven't had great contributor adoption working on the Gradle build; perhaps moving to a more palatable language would help. /cc @kennknowles
• Dependency-version alignment This allows using consistent dependency versions defined in a Maven BOM file /cc @garrettjonesgoogle
• Jacoco parallel test execution support This should make it easier to collect code coverage across our many test suites. /cc @Ardagan

[kennknowles]

Definitely excited to try the Kotlin DSL. Even better than just getting types, it is getting one of the most modern type systems available in a mainstream language.

[adude3141]

Run Java PostCommit

[swegner]

LGTM. All tests are green, is this ready to go in?

[adude3141]

@swegner
Unless u have more test in mind we should run before merging, I d say we should merge.

[adude3141]

Thx, @swegner

Will monitor next few Jenkins build to see which impact this has.

[swegner]

Will monitor next few Jenkins build to see which impact this has.

I skimmed through the Jenkins jobs that run continuously; I don't see anything broken, and the only significant execution time diff I noticed is beam_PostCommit_Java_ValidatesRunner_Flink:

Before (#3174): 16min, after (#3175): 8min.

Comparing the Gradle scans (before, after), the :beam-runners-flink_2.11:validatesRunnerBatch task is dramatically faster (12min -> 2.5 min). The scans show we're still running the same number of tests (475 in both cases), so I believe this is actual improvement.

I wonder what the cause of the improvement is; maybe the differences in how Gradle allocated memory to the test tasks? /cc @mxm @tweise

[adude3141]

Hmm... nice.

Unfortunately, I have now idea how this is (or even could be) related to the gradle upgrade. We probably reduced memory available to the test jvm (was: 1/4th of physically available mem, is: 2G). I assume, we do have more than 8G on the Jenkins agents, but I did not find yet any documentation on that.

Even if mem would be increased now - and test jvm was gc'ing all the time, I have no idea, why batch should require significantly more memory compared to streaming?

[mxm]

Thanks for sharing @swegner. That's indeed an interesting change. It could come from the batch algorithms spilling to disk which is terribly slow compared to processing everything in memory. I've seen deletion of spilling directories while running tests.

Looking at the tests, it is surprising some of them take so long. There is room for improvement because every test brings up a mini Flink cluster at the moment. Some form of caching would be desirable.

[swegner]

One hypothesis: the overall reduction in Gradle task memory usage has left more available memory for the local Flink cluster. It's to say for sure, the Gradle scan doesn't give a whole lot of context.

At any rate, the improvement has stuck over the last 15 runs, and the first run showing improvement only has the Gradle 5 upgrade in the delta.

[angoenka]

The apex validate runner test has started to fail since this change https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/buildTimeTrend
The gradle scans and the logs are not very useful.

Jira: https://issues.apache.org/jira/browse/BEAM-6681

[adude3141]

I ll have a look. Thank you very much for reporting.

[adude3141]

Opened #7861 to fix Apex Runner failing with OutOfMemoryError.

[adude3141]

Yet another one:
https://issues.apache.org/jira/browse/BEAM-6698

Any hint whom to ask about that portable runner flink stuff?

[mxm]

Please ask me :) Also @angoenka, @tweise, @ryan-williams, @robertwb (quoted to avoid pinging them here).

[adude3141]

thx @mxm... will try to understand what's going on there and come back to ask questions

[aaltay]

Release publish fails with this change. Reverting one commit (cadb6f7) to downgrade gradle resolves the issue: https://issues.apache.org/jira/browse/BEAM-6726