Java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance"

[zifeihan]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• Question or discussion
• Bug
• Requirement
• Feature or performance improvement

---

Question

• What do you want to know?

---

Bug

• Which version of SkyWalking, OS, and JRE?
  SkyWalking version:8.4.0-SNAPSHOT,
  jdk version:jdk1.8
  os version: mac 10.14.2

• Which company or project?

• What happened?
  If possible, provide a way to reproduce the error. e.g. demo application, component version.
  When there is an app to use skywalking, application report this error, and crash.

2021-01-11 11:17:53.580|[main]|ERROR|org.springframework.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'reCreateChatRoomJob' defined in URL [jar:file:/Users/echo/Downloads/target/target.jar!/BOOT-INF/classes!/xxx.xxx.xxx/bg/job/ReCreateChatRoomJob.class]: Initialization of bean failed; nested exception is org.springframework.aop.framework.AopConfigException: Could not generate CGLIB subclass of class com.zhangmen.yun.csw.marketlesson.bg.job.ReCreateChatRoomJob$$EnhancerBySpringCGLIB$$5a04f5ab: Common causes of this problem include using a final class or a non-visible class; nested exception is org.springframework.cglib.core.CodeGenerationException: java.lang.ClassFormatError-->Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file xxx.xxx.xxx/bg/job/ReCreateChatRoomJob$$EnhancerBySpringCGLIB$$f3689c3c
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:601)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:140)
	at com.zhangmen.yun.csw.core.launch.ZmApplication.run(ZmApplication.java:34)
	at com.zhangmen.yun.csw.marketlesson.bg.ZmyunMarketLessonBgApplication.main(ZmyunMarketLessonBgApplication.java:36)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:107)
	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.springframework.aop.framework.AopConfigException: Could not generate CGLIB subclass of class com.zhangmen.yun.csw.marketlesson.bg.job.ReCreateChatRoomJob$$EnhancerBySpringCGLIB$$5a04f5ab: Common causes of this problem include using a final class or a non-visible class; nested exception is org.springframework.cglib.core.CodeGenerationException: java.lang.ClassFormatError-->Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file xxx.xxx.xxx/bg/job/ReCreateChatRoomJob$$EnhancerBySpringCGLIB$$f3689c3c
	at org.springframework.aop.framework.CglibAopProxy.getProxy(CglibAopProxy.java:208)
	at org.springframework.aop.framework.ProxyFactory.getProxy(ProxyFactory.java:110)
	at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.createProxy(AbstractAutoProxyCreator.java:471)
	at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.wrapIfNecessary(AbstractAutoProxyCreator.java:350)
	at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.postProcessAfterInitialization(AbstractAutoProxyCreator.java:299)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:429)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1782)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
	... 23 common frames omitted
Caused by: org.springframework.cglib.core.CodeGenerationException: java.lang.ClassFormatError-->Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file xxx.xxx.xxx/bg/job/ReCreateChatRoomJob$$EnhancerBySpringCGLIB$$f3689c3c
	at org.springframework.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:530)
	at org.springframework.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:363)
	at org.springframework.cglib.proxy.Enhancer.generate(Enhancer.java:582)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:110)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:108)
	at org.springframework.cglib.core.internal.LoadingCache$2.call(LoadingCache.java:54)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:61)
	at org.springframework.cglib.core.internal.LoadingCache.get(LoadingCache.java:34)
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData.get(AbstractClassGenerator.java:134)
	at org.springframework.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:319)
	at org.springframework.cglib.proxy.Enhancer.createHelper(Enhancer.java:569)
	at org.springframework.cglib.proxy.Enhancer.createClass(Enhancer.java:416)
	at org.springframework.aop.framework.ObjenesisCglibAopProxy.createProxyClassAndInstance(ObjenesisCglibAopProxy.java:57)
	at org.springframework.aop.framework.CglibAopProxy.getProxy(CglibAopProxy.java:205)
	... 30 common frames omitted
Caused by: java.lang.ClassFormatError: Duplicate interface name "org/apache/skywalking/apm/agent/core/plugin/interceptor/enhance/EnhancedInstance" in class file xxx.xxx.xxx/bg/job/ReCreateChatRoomJob$$EnhancerBySpringCGLIB$$f3689c3c
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.cglib.core.ReflectUtils.defineClass(ReflectUtils.java:527)
	... 44 common frames omitted

link issue,#2647, #4949, #5077, #5673

---

Requirement or improvement

• Please describe your requirements or improvement suggestions.

Troubleshooting:

1. When no agent is added, cglib will only generate a cglib dynamic agent for ReCreateChatRoomJob.
   
2. When adding the agent, remotely debug the agent. At this time, it was found that the dynamic agent class of ReCreateChatRoomJob entered skywalking twice to enhance the bytecode.

Therefore, I dumped the bytecode of the two cglib dynamic proxy classes enhanced by skywalking and decompiled before the exception caused the application to exit.
the first time:

the second time:

From the above, we can know that cglib will nest in the process of doing dynamic proxy. When the first level of nesting is enhanced by skywalking, this class has implemented the EnhancedInstance class of skywalking, so it enters:
org.springframework.aop.framework.CglibAopProxy#getProxy(java.lang.ClassLoader)

if (ClassUtils.isCglibProxyClass(rootClass)) {
   proxySuperClass = rootClass.getSuperclass();
   Class<?>[] additionalInterfaces = rootClass.getInterfaces();
   for (Class<?> additionalInterface : additionalInterfaces) {
      this.advised.addInterface(additionalInterface);
   }
}

Since rootClass is already an EnhancedInstance class that implements skywalking, cglib is used to generate a dynamic proxy class again. When the generated dynamic proxy class triggers the skywalking bytecode interception process, it implements the EnhancedInstance interface, so the verification of this class fails and an exception occurs. App exit

/**
 * Manipulate class source code.<br/>
 *
 * new class need:<br/>
 * 1.Add field, name {@link #CONTEXT_ATTR_NAME}.
 * 2.Add a field accessor for this field.
 *
 * And make sure the source codes manipulation only occurs once.
 *
 */
if (!context.isObjectExtended()) {
    newClassBuilder = newClassBuilder.defineField(CONTEXT_ATTR_NAME, Object.class, ACC_PRIVATE | ACC_VOLATILE)
                                     .implement(EnhancedInstance.class)
                                     .intercept(FieldAccessor.ofField(CONTEXT_ATTR_NAME));
    context.extendObjectCompleted();
}

Solve the problem:

1. It can be solved by setting the agent.is_open_debugging_class parameter of skywalking.config to true. This parameter sets whether bytebuddy performs bytecode verification.

2. Before the original isObjectExtended judgment, add another isAssignableTo to judge whether the current class is a subclass of EnhancedInstance

/**
 * Manipulate class source code.<br/>
 *
 * new class need:<br/>
 * 1.Add field, name {@link #CONTEXT_ATTR_NAME}.
 * 2.Add a field accessor for this field.
 *
 * And make sure the source codes manipulation only occurs once.
 *
 */
if (!typeDescription.isAssignableTo(EnhancedInstance.class)) {
    if (!context.isObjectExtended()) {
        newClassBuilder = newClassBuilder.defineField(
            CONTEXT_ATTR_NAME, Object.class, ACC_PRIVATE | ACC_VOLATILE)
                                         .implement(EnhancedInstance.class)
                                         .intercept(FieldAccessor.ofField(CONTEXT_ATTR_NAME));
        context.extendObjectCompleted();
    }
}

[wu-sheng]

@zifeihan I unpinned this issue, we usually don't do this.