CRD generation fails with StackOverflow in case of cyclic references #3651

andreaTP · 2021-12-15T11:12:07Z

Describe the bug

Using the CRD generation preview module and having cyclic references in the POJOs breaks the generator.

Discovered trying to produce a CRD from the Keycloak Real Representation:
https://github.com/andreaTP/repro_crd_gen_stackoverflow/blob/e398e95c265c9bd2fe992e517b6b554382ae863b/src/main/java/org/keycloak/operator/crds/KeycloakRealmSpec.java#L23

Fabric8 Kubernetes Client version

5.10.1@latest

Steps to reproduce

git clone https://github.com/andreaTP/repro_crd_gen_stackoverflow
cd repro_crd_gen_stackoverflow
mvn clean compile -e

Expected behavior

In order of preference:

the cyclic reference is correctly handled
the cyclic reference can be somehow excluded (e.g. with an additional annotation? passing settings to the generator?)
in presence of a cyclic reference an appropriate error is thrown to the user

Runtime

other (please specify in additional context)

Kubernetes API Server version

other (please specify in additional context)

Environment

Linux, macOS

Fabric8 Kubernetes Client Logs

ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project foo-sample: Fatal error compiling: java.lang.StackOverflowError -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project foo-sample: Fatal error compiling
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoExecutionException: Fatal error compiling
    at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute (AbstractCompilerMojo.java:796)
    at org.apache.maven.plugin.compiler.CompilerMojo.execute (CompilerMojo.java:129)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.codehaus.plexus.compiler.CompilerException: java.lang.StackOverflowError
    at org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess (JavaxToolsCompiler.java:191)
    at org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile (JavacCompiler.java:169)
    at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute (AbstractCompilerMojo.java:785)
    at org.apache.maven.plugin.compiler.CompilerMojo.execute (CompilerMojo.java:129)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.lang.RuntimeException: java.lang.StackOverflowError
    at com.sun.tools.javac.api.JavacTaskImpl.invocationHelper (JavacTaskImpl.java:168)
    at com.sun.tools.javac.api.JavacTaskImpl.doCall (JavacTaskImpl.java:100)
    at com.sun.tools.javac.api.JavacTaskImpl.call (JavacTaskImpl.java:94)
    at org.codehaus.plexus.compiler.javac.JavaxToolsCompiler.compileInProcess (JavaxToolsCompiler.java:126)
    at org.codehaus.plexus.compiler.javac.JavacCompiler.performCompile (JavacCompiler.java:169)
    at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute (AbstractCompilerMojo.java:785)
    at org.apache.maven.plugin.compiler.CompilerMojo.execute (CompilerMojo.java:129)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.lang.StackOverflowError
    at io.sundr.model.functions.GetDefinition.apply (GetDefinition.java:44)
    at io.sundr.model.functions.GetDefinition.of (GetDefinition.java:38)
    at io.fabric8.crd.generator.utils.Types.projectDefinition (Types.java:75)
    at io.fabric8.crd.generator.utils.Types.lambda$projectSuperClasses$0 (Types.java:148)
    at java.util.stream.ReferencePipeline$7$1.accept (ReferencePipeline.java:273)
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining (ArrayList.java:1625)
    at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:509)
    at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:499)
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential (ReduceOps.java:921)
    at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234)
    at java.util.stream.ReferencePipeline.collect (ReferencePipeline.java:682)
    at io.fabric8.crd.generator.utils.Types.projectSuperClasses (Types.java:149)
    at io.fabric8.crd.generator.utils.Types.unshallow (Types.java:58)
    at io.fabric8.crd.generator.utils.Types.typeDefFrom (Types.java:70)
    at io.fabric8.crd.generator.AbstractJsonSchema.internalFrom (AbstractJsonSchema.java:405)
    at io.fabric8.crd.generator.AbstractJsonSchema.internalFrom (AbstractJsonSchema.java:143)
    at io.fabric8.crd.generator.AbstractJsonSchema.internalFrom (AbstractJsonSchema.java:416)

and keeps bouncing in the last two lines:

    at io.fabric8.crd.generator.AbstractJsonSchema.internalFrom (AbstractJsonSchema.java:143)
    at io.fabric8.crd.generator.AbstractJsonSchema.internalFrom (AbstractJsonSchema.java:416)

full reference: https://github.com/andreaTP/repro_crd_gen_stackoverflow/blob/main/out.log

The text was updated successfully, but these errors were encountered:

andreaTP · 2021-12-15T11:35:31Z

Additional considerations:

$ref is not supported Allow use of $ref in in CRD validation schema kubernetes/kubernetes#62872 yet
a possible "workaround" for the encoding might be to produce "untyped" objects when a cycle is found, e.g. (based on the provided repro):

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: keycloakrealms.keycloak.org
spec:
  group: keycloak.org
  names:
    kind: KeycloakRealm
    plural: keycloakrealms
    shortNames:
    - kc
    singular: keycloakrealm
  scope: Namespaced
  versions:
  - name: v1alpha1
    schema:
      openAPIV3Schema:
        properties:
          spec:
            properties:
              realm:
                properties:
                  example:
                    properties:
                      reference:
                        x-kubernetes-preserve-unknown-fields: true
                        type: object
                    type: object
                  reference:
                    properties:
                      example:
                        x-kubernetes-preserve-unknown-fields: true
                        type: object
                    type: object
                type: object
            type: object
          status:
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}

…finition fabric8io#3651

…efinition fabric8io#3651

andreaTP · 2021-12-20T15:34:43Z

another alternative might be to create "virtual" references:

as soon as we detect a cycle we convert the property to a "Ref" type (e.g. backed by a plain string?) and we create an additional synthetic field at the top level of the Spec that will be a Map<String, Object> where the structure of the values is checked against the property involved in the cycle.
This way we break the cycles and we can express arbitrary graphs.

This is a technical proposal for implementing, in this context, a standard pattern for breaking cycles ( example ).

Despite this being a technically appealing solution, I do see a few downsides on this proposal:

if $ref gets (ever) supported by Kubernetes this workaround should be removed entirely
The Kubernetes API is not going to validate that a reference is actually available (including typos etc.) and this check should be implemented manually in the operator handling the CR
Creating synthetic fields based on POJOs structure introduces an additional(and possibly subtle) way to break CRD versioning
The loading of the CR should be handled properly and might be "unsound" in other programming languages

andreaTP · 2021-12-21T17:15:52Z

The current path forward is to explore adding an annotation that enable users to specify an alternative CRD implementation instead of using the one automatically derived from the Java object.

cc. @metacosm

…efinition #3651

andreaTP added a commit to andreaTP/kubernetes-client that referenced this issue Dec 15, 2021

Avoid a StackOverflow and properly fail on cyclic refernces in CRD de…

c2f0036

…finition fabric8io#3651

andreaTP mentioned this issue Dec 15, 2021

Avoid a StackOverflow and properly fail on cyclic references in CRD generation #3652

Merged

11 tasks

andreaTP added a commit to andreaTP/kubernetes-client that referenced this issue Dec 15, 2021

Avoid a StackOverflow and properly fail on cyclic references in CRD d…

fdd5c66

…efinition fabric8io#3651

andreaTP added a commit to andreaTP/kubernetes-client that referenced this issue Dec 16, 2021

Avoid a StackOverflow and properly fail on cyclic references in CRD d…

d35933e

…efinition fabric8io#3651

This was referenced Dec 23, 2021

Introduce SchemaFrom annotation as escape hatch #3689

Merged

Realm CRD keycloak/keycloak#9172

Closed

manusa assigned andreaTP Dec 24, 2021

manusa added this to the 5.11.1 milestone Dec 24, 2021

manusa added the bug label Dec 24, 2021

manusa removed this from the 5.11.1 milestone Dec 24, 2021

manusa pushed a commit that referenced this issue Dec 24, 2021

Avoid a StackOverflow and properly fail on cyclic references in CRD d…

67544fc

…efinition #3651

manusa closed this as completed in #3689 Jan 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CRD generation fails with StackOverflow in case of cyclic references #3651

CRD generation fails with StackOverflow in case of cyclic references #3651

andreaTP commented Dec 15, 2021 •

edited

Loading

andreaTP commented Dec 15, 2021

andreaTP commented Dec 20, 2021

andreaTP commented Dec 21, 2021

CRD generation fails with StackOverflow in case of cyclic references #3651

CRD generation fails with StackOverflow in case of cyclic references #3651

Comments

andreaTP commented Dec 15, 2021 • edited Loading

Describe the bug

Fabric8 Kubernetes Client version

Steps to reproduce

Expected behavior

Runtime

Kubernetes API Server version

Environment

Fabric8 Kubernetes Client Logs

andreaTP commented Dec 15, 2021

andreaTP commented Dec 20, 2021

andreaTP commented Dec 21, 2021

andreaTP commented Dec 15, 2021 •

edited

Loading