Update to Gradle 8 #3966

qwwdfsad · 2023-12-01T17:36:59Z

Update to Gradle 8.

Each commit should be more or less self-describing and [potentially] strightforward.

Note that this shouldn't be merged until all the required checks are performed:

Classfile versions manual checks
Consumption of publications from frameworks with Java version in [8; 11; 17]
module-info manual checks
Consumption from multiplatform project with the similar source-sets structure

…ther changes

* Use Java 17 as the main toolchain * Use Java 8 target for the compatibility

* Dokka implicitly uses tasks outputs without dependency * Since Gradle 8, it's an error because implicit dependency execution heavily depends on the tasks order * Make an implicit dependency explicit

* Restore System.out eagerly to avoid chaining failures * Add CountDownLatch to the list of excluded Java classes

…cident

If the thread is in 'NEW' state, then 'isAlive: false' and 'join' returns immediately, meaning that there is a race between thread start and shutdown sequence that makes post-join assertion sporadically fail

In order to properly add jvm argument, we have to check jdk toolchain version (where actual tasks are executed) instead of current JDK version (where Gradle is configured)

…sign tasks

...where our hero slowly descends into madness

* We are never using it ourselves, yet it is *some* compatibility burden * It is also excluded from aggregate builds thus does not test anything

…to the dedicated convention * Document where appropriate * Verify that knitCheck passes * Also, move dokka.gradle.kts into convention to have a proper IDE support

They are no longer needed after migration to JdkToolchain

* Use Gradle 8.5 assignment operator instead of by infix operator * Get rid of Properties.kt * Add some commentary here and there * Cleanups and more Kotlin-ish code * Reduce the number of warnings

whyoleg · 2023-12-07T18:05:03Z

buildSrc/src/main/kotlin/configure-compilation-conventions.gradle.kts

@@ -16,17 +17,26 @@ configure(subprojects) {
 apiVersion = getOverriddenKotlinApiVersion(project)
 if (isMainTaskName && versionsAreNotOverridden) {
 allWarningsAsErrors = true
+ freeCompilerArgs = freeCompilerArgs + "-Xexplicit-api=strict"


BTW, as we are here, AFAIK kotlinOptions will be deprecated soon-ish and can be replaced already with compilerOptions which has API using Gradle providers instead of just getters/setters
Maybe it make sense to replace those usages in this PR? (maybe also in other places, if they exist)
Or even move this configuration from tasks to compilations?

I changed this in some of the more stable places, but kept it as is in the code that will yet go through rewrites.

qwwdfsad · 2023-12-08T11:15:07Z

ui/kotlinx-coroutines-android/build.gradle.kts

@@ -28,24 +28,24 @@ dependencies {


 testImplementation("org.smali:baksmali:${version("baksmali")}")
- "r8"("com.android.tools.build:builder:7.1.0-alpha01")
+ "r8"("com.android.tools.build:builder:8.1.0")


FTR this change reduced the dex size from 79824 to 75120.

It will affect our stats, it is expected

https://kotlinlang.org/docs/multiplatform-hierarchy.html#manual-configuration

dkhalanskyjb

It's far from done, but there doesn't seem to be too much low-hanging fruit left, and in any case, I think this is already a significant improvement.

dkhalanskyjb · 2023-12-08T10:20:04Z

ui/kotlinx-coroutines-android/build.gradle.kts

-val optimizedDexFile = File(optimizedDexDir, "classes.dex")
-val unOptimizedDexFile = File(unOptimizedDexDir, "classes.dex")
+val optimizedDexFile = optimizedDexDir.map { it.dir("classes.dex") } .get().asFile
+val unOptimizedDexFile = unOptimizedDexDir.map { it.dir("classes.dex") }.get().asFile


Doesn't get here hinder configuration avoidance?

dkhalanskyjb · 2023-12-08T14:18:51Z

buildSrc/src/main/kotlin/configure-compilation-conventions.gradle.kts

@@ -16,17 +17,26 @@ configure(subprojects) {
 apiVersion = getOverriddenKotlinApiVersion(project)
 if (isMainTaskName && versionsAreNotOverridden) {
 allWarningsAsErrors = true
+ freeCompilerArgs = freeCompilerArgs + "-Xexplicit-api=strict"


I changed this in some of the more stable places, but kept it as is in the code that will yet go through rewrites.

dkhalanskyjb · 2023-12-15T15:33:43Z

I don't know where to put it appropriately, so I'll write it here for now:

# Usage:
# rm -rf ~/.m2
# ./gradlew clean publishToMavenLocal
# python check_module_info.py
from pathlib import Path
from shutil import rmtree
import subprocess
import os

version_property_name="version"
version = None
for line in open('gradle.properties'):
    if line.startswith(f"{version_property_name}="):
        version = line.removeprefix(f"{version_property_name}=").rstrip()
        break

if version is None:
    raise AssertionError(
        f"Didn't find '{version_property_name}=' in gradle.properties"
    )

modules_with_module_info = [
  ("", "kotlinx-coroutines-core", "jvm"),
  ("ui", "kotlinx-coroutines-javafx", ""),
  ("ui", "kotlinx-coroutines-android", ""),
  ("ui", "kotlinx-coroutines-swing", ""),
  ("", "kotlinx-coroutines-test", "jvm"),
  ("", "kotlinx-coroutines-debug", ""),
  ("reactive", "kotlinx-coroutines-reactive", ""),
  ("reactive", "kotlinx-coroutines-jdk9", ""),
  ("reactive", "kotlinx-coroutines-reactor", ""),
  ("reactive", "kotlinx-coroutines-rx3", ""),
  ("reactive", "kotlinx-coroutines-rx2", ""),
]

m2_path=str(Path.home().joinpath(Path('.m2')))

class ModuleInfo:
    def __init__(
        self,
        name: bytes,
        exports: set[bytes],
        requires: set[(bytes, bytes)],
        provides: set[(bytes, bytes)],
        uses: set[bytes],
        contains: set[bytes],
    ):
        self.name = name
        self.exports = exports
        self.requires = requires
        self.provides = provides
        self.uses = uses

    def __eq__(self, other):
        return (self.name == other.name and
            self.exports == other.exports and
            self.requires == other.requires and
            self.provides == other.provides and
            self.uses == other.uses)

    def __str__(self):
        return f'{self.name}[exports {self.exports}, requires {self.requires}, provides {self.provides}, uses {self.uses}]'

    def __repr__(self):
        return f'ModuleInfo({self.name}, {self.exports}, {self.requires}, {self.provides}, {self.uses})'

def jar_describe(module: str, artifact: str) -> [ModuleInfo]:
    description = subprocess.run(
        ["jar", "--describe-module", "--file", artifact, "--release", "9"],
        stdout = subprocess.PIPE
    )
    lines = description.stdout.split(b'\n')
    if lines[0] != b'releases: 9': raise AssertionError(f'Expected the dump to start with b"releases: 9", got {lines[0]}')
    if lines[1] != b'': raise AssertionError(f'Expected an empty line, got {lines[1]}')
    jvmModule = module.replace('-', '.')
    i = 2
    module_info = bytes(f'jar:file:https://{artifact}/!META-INF/versions/9/module-info.class', 'utf-8')
    infos = []
    def read_header(line: bytes) -> bytes:
        name, current_module_info = line.split(b' ')
        if current_module_info != module_info: raise AssertionError(f'Expected a header with the module-info {module_info}, got {line}')
        return name
    while i < len(lines) - 1:
        module_name = read_header(lines[i])
        expected_third_line=bytes(f'{jvmModule} jar:file:https://{artifact}/!META-INF/versions/9/module-info.class', 'utf-8')
        if lines[i] != expected_third_line: raise AssertionError(f'Expected the header {expected_third_line}, got {lines[i]}')
        current_line = None
        i += 1
        exports = set()
        requires = set()
        provides = set()
        uses = set()
        contains = set()
        while lines[i] != b'':
            line = lines[i]
            operation, *rest = line.split()
            try:
                if operation == b'exports':
                    [package] = rest
                    exports.add(package)
                elif operation == b'requires':
                    if len(rest) == 1:
                        [package] = rest
                        modifier = None
                    else:
                        package, modifier = rest
                    requires.add((package, modifier))
                elif operation == b'provides':
                    [service, wth, cls] = rest
                    if wth != b'with': raise AssertionError(f'Unexpected syntax for "provides"')
                    provides.add((service, cls))
                elif operation == b'uses':
                    [package] = rest
                    uses.add(package)
                elif operation == b'contains':
                    [package] = rest
                    contains.add(package)
                else:
                    raise AssertionError(f'Unknown operation')
            except:
                print(f'An error on line {line}')
                raise
            i += 1
        infos.append(
            ModuleInfo(module_name, exports, requires, provides, uses, contains)
        )
        i += 1
    return infos

def read_module_info(module_info: str) -> ModuleInfo:
    infos = []
    mapping = dict()
    with open(module_info, 'rb') as fl:
        content = fl.read().replace(b'@SuppressWarnings("JavaModuleNaming")', b'')
        new_lines = []
        for line in content.split(b'\n'):
            before_comment, *rest = line.split(b'//')
            new_lines.append(before_comment)
        content = b''.join(new_lines)
        lines = content.replace(b'{', b';{;').replace(b'}', b';};').split(b';')
        new_lines = []
        for line in lines:
            line = line.strip()
            if line == b'': continue
            new_lines.append(line)
        lines = new_lines
        i = 0
        while i < len(lines):
            line = lines[i]
            i += 1
            try:
                operation, argument = line.split()
                if operation == b'import':
                    name = argument.split(b'.')[-1]
                    mapping[name] = argument
                elif operation == b'module':
                    module_name = argument
                    exports = set()
                    requires = set([(b'java.base', b'mandated')])
                    provides = set()
                    uses = set()
                    contains = set()
                    if lines[i] != b'{': raise AssertionError('Expected an "{"')
                    i += 1
                    while lines[i] != b'}':
                        line = lines[i]
                        operation, *rest = line.split()
                        if operation == b'exports':
                            [package] = rest
                            exports.add(package)
                        elif operation == b'requires':
                            if len(rest) == 1:
                                [package] = rest
                                modifier = None
                            else:
                                modifier, package = rest
                            if package != b'kotlinx.atomicfu' or modifier != None:
                                requires.add((package, modifier))
                        elif operation == b'provides':
                            [cls, wth, service] = rest
                            if wth != b'with': raise AssertionError(f'Unexpected syntax for "provides"')
                            provides.add((mapping.get(cls, cls), mapping.get(service, service)))
                        elif operation == b'uses':
                            [cls] = rest
                            uses.add(mapping.get(cls, cls))
                        elif operation == b'contains':
                            [package] = rest
                            contains.add(package)
                        else:
                            raise AssertionError(f'Unknown operation')
                        i += 1
                    i += 1
                    infos.append(
                        ModuleInfo(module_name, exports, requires, provides, uses, contains)
                    )
                else:
                    raise AssertionError('Unknown operation')
            except:
                print(f'An error on line {i} of {module_info}: {line}')
                raise
        return infos

for (prefix, module, platform) in modules_with_module_info:
    artifact_name=f'{module}{"-" + platform if platform else ""}'
    print(f'Checking {artifact_name}...')
    from_jar_describe = jar_describe(module, f'{m2_path}/repository/org/jetbrains/kotlinx/{artifact_name}/{version}/{artifact_name}-{version}.jar')
    from_file = read_module_info(f'{prefix or "."}/{module}{"/" + platform if platform else ""}/src/module-info.java')
    if from_file != from_jar_describe: raise AssertionError(f'Expected {from_file} and {from_jar_describe} to be the same')
    print("OK")

The manual check of module-info is done:

Checking kotlinx-coroutines-core-jvm...
OK
Checking kotlinx-coroutines-javafx...
OK
Checking kotlinx-coroutines-android...
OK
Checking kotlinx-coroutines-swing...
OK
Checking kotlinx-coroutines-test-jvm...
OK
Checking kotlinx-coroutines-debug...
OK
Checking kotlinx-coroutines-reactive...
OK
Checking kotlinx-coroutines-jdk9...
OK
Checking kotlinx-coroutines-reactor...
OK
Checking kotlinx-coroutines-rx3...
OK
Checking kotlinx-coroutines-rx2...
OK

dkhalanskyjb · 2023-12-20T09:53:02Z

The version files are also all there:

$ find ~/.m2 -name 'kotlinx-coroutines-*SNAPSHOT.jar' | xargs -n1 unzip -l | grep -F .version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_reactive.version
       17  1980-02-01 00:00   META-INF/kotlinx_coroutines_core.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_jdk8.version
       17  1980-02-01 00:00   META-INF/kotlinx_coroutines_test.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_javafx.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_jdk9.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_reactor.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_rx3.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_play_services.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_slf4j.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_rx2.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_debug.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_guava.version
       17  2023-12-15 14:19   META-INF/kotlinx_coroutines_android.version
       17  2023-12-15 14:21   META-INF/kotlinx_coroutines_swing.version

kotlinx-coroutines-debug/build.gradle.kts

* We have no Java 17 on our agents * Downloading toolchain can only be configured with a custom third-party plugin that we don't want to depend on

dkhalanskyjb · 2023-12-20T14:43:57Z

Checked the classfile versions. All of the classes that we publish have major version 52, except the following ones:

./kotlinx/coroutines/jdk9/PublishKt.class:   major version: 53
./kotlinx/coroutines/jdk9/ReactiveFlowKt.class:   major version: 53
./kotlinx/coroutines/jdk9/ReactiveFlowKt$collect$1.class:   major version: 53
./kotlinx/coroutines/jdk9/AwaitKt.class:   major version: 53
./kotlinx/coroutines/repackaged/net/bytebuddy/...:   major version: 49
./META-INF/versions/9/module-info.class:   major version: 53

echo 'Extracting...'
for dir in ~/.m2/repository/org/jetbrains/kotlinx/kotlinx-coroutines*; do
    find "$dir" -name '*.jar' -exec unzip -o -q '{}' ';'
done
echo 'Checking every classfile...'
find -name '*.class' -print0 |
xargs -0 -n1 sh -c 'printf "%s: %s\n" "$0" "$(javap -verbose "$0" | grep "major version:")"' |
grep -v ': 52'

Broken in #3966

* Remove the workarounds for IDEA import from gradle.kts Without the workarounds, everything still seems to be working fine, even after invalidating the caches. * Clarify the opt-ins. Unify the handling of all opt-ins in the project, and also add opt-ins for Native that would be widespread otherwise. Unfortunately, even after that, the IDE still complains about not having the right opt-in in kotlinx-coroutines-test/common/test/Helpers.kt. * Extract the logic of grouping several source sets into a single shared one. * Update the kotlinx-coroutines-core build script: - Utilize the multiplatform hierarchy to avoid defining extra source sets. - Reorder the code for readability. * Fix the IDE failing to find `actual` implementations for jvmCore by making sure `jvmCoreMain` is only defined when the IDE is not active. Without a dependency from jvmCoreMain to jvmMain, `expect` declarations in the IDE complained that they couldn't find `actual` in the `jvmCore` source-set. For example, this could be seen in kotlinx-coroutines-core/common/src/internal/Concurrent.common.kt. * Fix passing the stressTest system property to tests Broken in #3966 * kotlinOptions -> compilerOptions

qwwdfsad force-pushed the update-gradle branch 7 times, most recently from 5eed2ca to 612fe4e Compare December 5, 2023 12:39

qwwdfsad requested a review from dkhalanskyjb December 5, 2023 15:49

qwwdfsad marked this pull request as ready for review December 5, 2023 15:49

qwwdfsad added 17 commits December 6, 2023 12:11

Update deprecated API to seamlessly migrate to Gradle 8

066883c

Update animal sniffer

690ec00

Migrate compile-jvm-multiplatform.gradle to .gradle.kts without any o…

94b5ec4

…ther changes

Update to Gradle 8.1

ee3a1cf

Migrate to jvmToolchain [forced by KGP update]

1d62a9d

* Use Java 17 as the main toolchain * Use Java 8 target for the compatibility

Workaround for Dokka issue

88a3438

* Dokka implicitly uses tasks outputs without dependency * Since Gradle 8, it's an error because implicit dependency execution heavily depends on the tasks order * Make an implicit dependency explicit

Workaround for tests:

610c6a1

* Restore System.out eagerly to avoid chaining failures * Add CountDownLatch to the list of excluded Java classes

Use dedicated module name for Wasm JS so outputs are not reused by ac…

73d81df

…cident

Fix a race in CoroutineScheduler.shutdown

cb48ac0

If the thread is in 'NEW' state, then 'isAlive: false' and 'join' returns immediately, meaning that there is a race between thread start and shutdown sequence that makes post-join assertion sporadically fail

jdkToolchain update

e31a8ca

In order to properly add jvm argument, we have to check jdk toolchain version (where actual tasks are executed) instead of current JDK version (where Gradle is configured)

Gradle publishing: establish explicit dependency between publish and …

88a297b

…sign tasks

Workaround for KT-58303

196501c

The second part of Sign plugin workarounds

27c7875

...where our hero slowly descends into madness

Update to Gradle 8.5

aeb0ca7

Encapsulate JS-specific repo logic in CacheRedirector

956b081

Introduce Project.configureCommunityBuildTweaks()

874928a

Remove "example-frontend-js" module

ca9e962

* We are never using it ourselves, yet it is *some* compatibility burden * It is also excluded from aggregate builds thus does not test anything

qwwdfsad force-pushed the update-gradle branch from 93fcadf to ca9e962 Compare December 6, 2023 11:12

Move all [stretched] Dokka configuration from top-level build.gradle …

3e8e751

…to the dedicated convention * Document where appropriate * Verify that knitCheck passes * Also, move dokka.gradle.kts into convention to have a proper IDE support

qwwdfsad force-pushed the update-gradle branch from 308cc15 to 3e8e751 Compare December 6, 2023 12:32

Remove jdk-conventions

a973035

They are no longer needed after migration to JdkToolchain

qwwdfsad added 3 commits December 7, 2023 18:20

Add explanation to CoroutineScheduler check

1fa3397

Build cleanup:

f514e7a

* Use Gradle 8.5 assignment operator instead of by infix operator * Get rid of Properties.kt * Add some commentary here and there * Cleanups and more Kotlin-ish code * Reduce the number of warnings

Update R8, get rid of deprecated API in kotlinx-coroutines-android

5aafa62

whyoleg reviewed Dec 7, 2023

View reviewed changes

qwwdfsad requested a review from dkhalanskyjb December 7, 2023 18:18

qwwdfsad commented Dec 8, 2023

View reviewed changes

dkhalanskyjb added 5 commits December 8, 2023 14:35

Better isolate the community projects configuration

934a0f8

Use the default source-set hierarchy

a5d356f

https://kotlinlang.org/docs/multiplatform-hierarchy.html#manual-configuration

Small cleanup

4d815f3

set() -> =

28ce533

Drop a comment

cf2e1e4

dkhalanskyjb approved these changes Dec 8, 2023

View reviewed changes

Fix trying to create a top-level directory from Gradle on *nix

5786f3d

Upstream Space packages publication

583f10a

qwwdfsad force-pushed the update-gradle branch from 65c1667 to 583f10a Compare December 18, 2023 13:38

dkhalanskyjb reviewed Dec 20, 2023

View reviewed changes

kotlinx-coroutines-debug/build.gradle.kts Show resolved Hide resolved

Lower JDK toolchain version to 11

4509e20

* We have no Java 17 on our agents * Downloading toolchain can only be configured with a custom third-party plugin that we don't want to depend on

qwwdfsad force-pushed the update-gradle branch from de3cdcb to 4509e20 Compare December 20, 2023 11:12

qwwdfsad added 2 commits December 20, 2023 14:07

Add to build configuration to ensure Gradle variant compatibility

e0ce55c

Remove line about Gradle 5.3 from our README

bfa22e9

qwwdfsad force-pushed the update-gradle branch from 6f032a3 to bfa22e9 Compare December 20, 2023 13:07

qwwdfsad merged commit d12eb45 into develop Dec 21, 2023
1 check passed

dkhalanskyjb added a commit that referenced this pull request Jan 4, 2024

Fix passing the stressTest system property to tests

ad8ee39

Broken in #3966

dkhalanskyjb added a commit that referenced this pull request Jan 5, 2024

Fix passing the stressTest system property to tests

9205e01

Broken in #3966

dkhalanskyjb added a commit that referenced this pull request Jan 25, 2024

Fix passing the stressTest system property to tests

ecefdef

Broken in #3966

qwwdfsad mentioned this pull request Feb 19, 2024

Tidy up our build setup Kotlin/kotlinx.serialization#2574

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to Gradle 8 #3966

Update to Gradle 8 #3966

qwwdfsad commented Dec 1, 2023 •

edited

Loading

whyoleg Dec 7, 2023

dkhalanskyjb Dec 8, 2023

qwwdfsad Dec 8, 2023

dkhalanskyjb left a comment

dkhalanskyjb Dec 8, 2023

dkhalanskyjb Dec 8, 2023

dkhalanskyjb commented Dec 15, 2023

dkhalanskyjb commented Dec 20, 2023

dkhalanskyjb commented Dec 20, 2023

Update to Gradle 8 #3966

Update to Gradle 8 #3966

Conversation

qwwdfsad commented Dec 1, 2023 • edited Loading

whyoleg Dec 7, 2023

Choose a reason for hiding this comment

dkhalanskyjb Dec 8, 2023

Choose a reason for hiding this comment

qwwdfsad Dec 8, 2023

Choose a reason for hiding this comment

dkhalanskyjb left a comment

Choose a reason for hiding this comment

dkhalanskyjb Dec 8, 2023

Choose a reason for hiding this comment

dkhalanskyjb Dec 8, 2023

Choose a reason for hiding this comment

dkhalanskyjb commented Dec 15, 2023

dkhalanskyjb commented Dec 20, 2023

dkhalanskyjb commented Dec 20, 2023

qwwdfsad commented Dec 1, 2023 •

edited

Loading