Skip to content

ProjectMapK/kogera-benchmark

Repository files navigation

This project is a benchmark to compare the performance of jackson-module-kogera and jackson-module-kotlin.

Instructions for running the benchmarks are provided at the bottom of the README.

About basic trends

Because Kogera offers more features, it often performs worse than Original in SingleShot mode.
This difference is expected to decrease as more features are incorporated into Original.
I have tried to improve it with Kogera, but now the majority of the initialization cost is spent on analysis with Jackson and kotlinx-metadata-jvm, so further improvement is difficult.

In Throughput mode, the biggest difference is in the deserialization process. By eliminating the overhead of function calls with kotlin-reflect, a significant improvement is achieved. On the other hand, there is almost no difference in serialization, because there is no particular point that can be improved.

About benchmark results

The currently published scores compare jackson-module-kogera 2.15.2-beta4 with jackson-module-kotlin 2.15.2.

The letters A and E in the benchmark names are values to adjust the sort order of the results and represent the number of properties (1 = A, 5 = E, ...).

The deserialization benchmarks show four patterns of values per number of properties, depending on the creator invoked and the use of default arguments.
The Ctor and Func in the title indicate that the deserialization was done by a constructor or factory function.
The present and missing indicate that all arguments were read from the input or all were not read (= default arguments were used).

Raw data is stored in ci-reports.
The spreadsheet used to generate the graphs can be found here.

A note on the results on README

Please note that these scores are measured by GitHub Actions and are not always the same.
Also, the cache may stop updating the images for up to 5 minutes, and the README may not have been updated after the benchmark was run, so the description and images may not always match.

Results

Basic use cases

Throughput mode

Deserialize

Kogera is particularly good at deserialization.

Serialize

As mentioned above, there is little difference between Kogera and Original as no particular improvements have been made.

SingleShot mode

Due to the performance degradation associated with the additional features, Kogera scores are inferior in the comparison at this time.

Deserialize

Serialize

Comparison of normal class and value class

This section compares the serialization and deserialization performance using Kogera for the value class and the data class (wrapper).
Only present is compared, since the use of default arguments makes almost no difference in principle.

Overall, the value class scores are significantly inferior.
The reason for this is that the value class requires more reflection processing.

In Kogera, the JsonKUnbox annotation has been added to improve the performance of serialization, which requires only unbox. When used, the throughput will outperform the data class.

Throughput mode

Deserialize

Serialize

Normal

With JsonKUnbox

SingleShot mode

Although the difference is smaller in SingleShot mode, the tendency for inferior scores in the value class is still present.

Deserialize

Serialize

Normal

With JsonKUnbox

Comparison of strictNullChecks

If the StrictNullChecks option is enabled, the deserialization score associated with the Collection is reduced because of element checking.
Also, Original will theoretically lower the score for all deserialization operations because of the null checks required for all arguments on every call.
On the other hand, Kogera has its own improvements to reduce these score reductions.

Here are the benchmark scores defined in extra.deser.Collections and main.deser.

Throughput mode

For the Collection, there is almost no difference for Kogera, whereas for Original there is a relatively large difference.
There is not much difference except for the Collection.

Original

Kogera

SingleShot mode

The trend is the same as for the throughput mode, but the difference is smaller.

Original

Kogera

How to run benchmarks

./gradlew jmh to run the benchmark.
The benchmark results are output to jmh-reports/reports/jmh/.

The main benchmark is to serialize and deserialize for classes with 1, 5, and 20 properties.

The extra benchmark relate to options that affect performance.
Currently, benchmarks for deserialization of Collection (mainly to check the impact of strictNullChecks) and benchmarks for value class have been implemented.
Also, these benchmarks are not run by default, and benchmarks related to deserialization of value class are only run when using Kogera.

A simple configuration is provided in arguments.
By rewriting the isSingleShot property, you can benchmark the performance on the first run.
By rewriting the mapper property, you can benchmark using the ObjectMapper configuration defined in org.wrongwrong.Mapper.
By rewriting the benchmarkSet property, you can choose which benchmarks to run(Locally, it is recommended to run OnlyMain or Full).

Available mapper types are as follows

  • Kogera(default): Not customized KogeraModule
  • Original: Not customized KotlinModule
  • KogeraStrictNullCheck: KogeraModule with enabled strictNullChecks
  • OriginalStrictNullCheck: OriginalModule with enabled strictNullChecks

The following is an example of running all four benchmarks described in the README in succession.

./gradlew jmh -Pmapper=Original -PisSingleShot=true -PbenchmarkSet=Full;
./gradlew jmh -Pmapper=Original -PisSingleShot=false -PbenchmarkSet=Full;
./gradlew jmh -PisSingleShot=true -PbenchmarkSet=Full;
./gradlew jmh -PisSingleShot=false -PbenchmarkSet=Full

If you want to specify more detailed options, edit build.gradle.kts.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages