Migrate Dokka's analysis to the K2 compiler API

[IgnatBeresnev]

Background

Dokka depends on the Kotlin compiler to analyze source code and KDoc comments.

Kotlin's K2 compiler has been in the works for some time, and it is going stable in Kotlin 2.0.

That means that the K1 compiler API that Dokka uses will be phased out over time, and Dokka should migrate its analysis to use the new K2 compiler API.

Steps

There are 4 major steps to make this migration happen:

1. Extract K1 analysis from dokka-base (✅)

The current K1 analysis is deeply ingrained in dokka-base, with internal compiler types being exposed in public API and used by downstream plugins. This makes the migration to K2 very painful both for maintainers and for downstream users.

K1 analysis should be extracted from dokka-base, and a common analysis-api abstraction should be introduced, making it possible for K1 and K2 implementations to coexist at the same time and be used interchangeably.

• Decompose Kotlin/Java analysis #3034

Update: This step has been done, see #3099 for more details.

2. Implement a prototype to identify blockers (✅)

K2 already provides some semi-stable API that Dokka can use. We should migrate as much of the analysis as possible to identify the missing API and other blocking issues that would need to be addressed by the compiler team.

• Dokka K2 analysis #3094

3. Stabilize JVM-only projects (In Progress)

Kotlin/JVM projects should be easier to support and debug on, so this is the first step in migrating users to the new K2 analysis.

Once generating documentation for Kotlin/JVM projects is stable enough, we will enable K2 analysis by default for such projects.

• [K2] Stabilize pure Kotlin/JVM and mixed Kotlin/Java projects #3140

4. Stabilize KMP projects (semi-blocked)

Once Kotlin/JVM projects are stabilized, the only thing left will be to support Kotlin Multiplatform projects. This is a separate step as it will likely require more time to test it and support less trivial parts of KMP, such as native libraries.

• [K2] Support KMP projects #3137

While some KMP projects do already work, there are many corner cases and issues that need to be fixed, both on Dokka's side and in Analysis API.

Blocking issues:

• KT-62696 - Analysis API: KMP support for Standalone Mode

Technical debt

A collection of issues and TODOs that are not necessarily part of any particular step, but that need to be addressed either during the migration or after it:

• Develop integration tests that compare the output of different Dokka versions #3142
• [K2] Use only stable compiler API #3141
• Once the API is more or less stable and no significant changes are expected:
  • Revisit the architecture / do the final code review.
  • Write internal documentation and KDocs to make it easier to onboard new contributors.

Related issues

Have a look at the issues marked with the topic: K2 tag to see what's been done and what still needs to be addressed in relation to this issue.

[IgnatBeresnev]

The K2 analysis prototype has been imlpemented in #3094 🎉

Currently, the following projects are supported:

• All Kotlin/JVM projects that only use Kotlin's Standard Library (so no Java SE API or any 3rd party libraries). On any JVM version.
• Kotlin/JVM project that use Java SE API directly (for example, import java.util.HashMap) only work if it's Java >= 9, because non-modularized Java (JCL) is not supported.

However, we are unable to proceed further and stabilize Kotlin/JVM projects until K2 introduces some additional API. A separate issue has been created to keep track of this:

• [K2] Stabilize pure Kotlin/JVM and mixed Kotlin/Java projects #3140

---

This issue's description has been updated, and additional issues have been created to reflect the current status.

[IgnatBeresnev]

Current status

The team has made significant progress in stabilizing the K2 analysis implementation. Many of our efforts also help find and fix bugs in the K2 IDE plugin, since we share a lot of the underlying analysis logic (same low-level API).

Since the last update, we've been unblocked with stabilizing Kotlin/JVM projects, have fixed the known major issues, and have completed the first iteration of testing Dokka manually on a variety of user projects (#3328).

The subtasks have been updated accordingly, they now include the newly discovered bugs and other issues.

• [K2] Stabilize pure Kotlin/JVM and mixed Kotlin/Java projects #3140
• [K2] Support KMP projects #3137

We will proceed with stabilizing Kotlin/JVM projects in the near future, aiming to enable K2 for them by default in Dokka 2.0.0, possibly making it available for public testing before the release. Once Kotlin/JVM projects are stable, we'll switch to KMP projects, aiming to enable K2 for them by default in Dokka 2.1.0, but we'll do it sooner if possible (Dokka depends a lot on the upstream API issues).

---

The API that Dokka uses for K2 analysis is not yet stable, so we encounter various issues and bugs, which we report to Kotlin's YouTrack. Analysis API issues that affect Dokka in one way or another are marked with the kt-analysis-dokka tag in YouTrack.

I'll leave a list of issues below for future reference.

Bugs affecting Kotlin/JVM projects:

• KT-62695 - K2 IDE: Unresolved extension functions in KDoc
• KT-64051 - K2 IDE: Analysis API: Unresolved links to typealias in KDoc
• KT-64190 - K2 IDE: Analysis API: KDoc link leads to a function instead of interface
• KT-62880 - K2 IDE: Unresolved java annotation methods in KDoc
• KT-62405 - Analysis API: Symbols SUBSTITUTION_OVERRIDE have no overridden symbols
• KT-64067 - Analysis API: Member scope has an inherited nested class from Java

KMP-specific issues:

• KT-62696 - Analysis API: KMP support for Standalone Mode
  • (child) KT-64159 - Analysis API: JS klib do not work for Kotlin <1.9.0
  • (child) KT-64155 - Analysis API: KotlinIllegalArgumentExceptionWithAttachments: Unexpected FirClassLikeSymbol null for class org.jetbrains.kotlin.fir.symbols.impl.ConeClassLikeLookupTagImpl

API stabilization issues:

• KT-61387 - Analysis API: stabilize Analysis API
  • (child) KT-61404 - Analysis API: implement proper library publishing structure
  • (child) KT-61403 - Analysis API: Stabilize API Surface
• KT-63339 - Analysis API: Provide a way to extract KDoc for symbols
• KT-53669 - Analysis API: redesign KtSymbolOrigin to distinguish kotlin/java source/library declarations

Noticeable differences in Dokka's behaviour that likely come from the upstream Analysis API, but it's undecided if they are bugs, so no YouTrack tickets yet:

• [K2] Reference to a class with quoted name is not resolved #3356
• [K2] Unresolved reference to a class property declared in constructor from kdoc #3334
• [K2] Unstable documentable order #3250

[vmishenev]

Current status

We are fixing some issues discovered after the second iteration of testing. TBD...

User-affective issues (28):

• Java-Kotlin interop — synthetic Java properties (7 issues):

  • [K2] Incorrect modifier for property from getter/setter #3339
  • [K2] Inconsistent member representation for Java/Kotlin files #3338
  • [K1] Incorrect Java synthetic properties #3128
  • [K2] Incorrect signature in synthetic property for Java field with a private setter  #3672
  • [K2] Java synthetic properties in Kotlin inheritor class shown with open modifier #3669
  • [K2] KMP: Multiple pages associated with key reported for an actual class overriding method from java parent #3685
  • [K2] Properties inherited from Java fields/synthetic properties with types that can be nullable shown as nullable  #3675

• KDoc spec (11 issues):

  • Design doc is done
    • [K2] Ambiguous KDoc link with the same name #3451
    • Ambiguous link for properties from a primary constructor #3179
    • Links to extensions with type parameters #3555 ;
  • Requires some clarifications on the specification.
    • KDoc link resolving
      • [K2] Inconsistent KDoc link into inhertited documentation #3639
      • [K1] Resolved references to synthetic java properties #3586
      • [K2] Inconsistent reference from constructor kdoc to class page for K1/K2 #3604
      • [K2] Inconsistent resolution of KDoc links to extensions #3632
      • [K2] KDoc references that goes after markdown blocks don't have links #3385
      • [K2] Reference to a class with quoted name is not resolved #3356
    • Others:
      • [K2] Non-existent parameter from KDoc doesn't appear on the implicit primary constructor page #3348
      • [K2] Modules are inconsistent reported on using samples from different source root #3373

• Currently blocked by Analysis API (6 issues):

  • [K2] Unresolved declarations are resolved in KDoc #3681
  • [K2] Unresolved KDoc link to two extensions with the same name #3631
  • [K2] KMP: Unresolved declarations from another module in Android + JVM project #3702
  • [K2] KMP: JS legacy-compiled libraries do not work for Kotlin <1.9.0 #3751
  • KMP: Support intersecting source roots #3701 and #3373

• Order [K2] Unstable documentable order #3250 and [K2] Inconsistent documentable rendering order #3590

- Inconsistent resolution of KDoc links to extensions #3632

• Link to an inherited declaration in @see block replace class container to parent one in link label
  #3680
• KMP: Expect/actual enums have expect modifiers but don't have actual for builtin functions
  #3704
• Overriden methods in enum entries are not rendered #3682

Small impact on a user (5):

• [K2] Duplication in logging for unresolved samples reference #3383
• [K2] Too verbose messaging about Dokka K2 in preview #3699
• Plugin Kotlin-as-Java (3 issues)

Non-user-affective (5):

• KMP: Generating StdLib [K2] Support generating documentation for stdlib #3354 (blocked)
• [K2] Use only stable compiler API #3141 (blocked)
• Integration testing
  • Develop integration tests that compare the output of different Dokka versions #3142
  • [K2] Add an integration test for jvm-only project #3136
  • [K2] Enable the K2 integration test for coroutines #3618