Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue]: Avro schemas with references #397

Open
2 tasks done
CROprogrammer opened this issue Jul 7, 2023 · 6 comments
Open
2 tasks done

[Issue]: Avro schemas with references #397

CROprogrammer opened this issue Jul 7, 2023 · 6 comments
Assignees
Labels
enhancement New feature or request

Comments

@CROprogrammer
Copy link

What happened?

Oirignal discussion: #396

Im using Apicurio schema registry 2.2.3.Final

Hi,

I was trying to test if Avro schema with a reference to other schema can be loaded to KLoadGen from schema registry, but unfortunately it cannot.

I was using schema Student:

{
  "type": "record",
  "namespace": "com.example",
  "name": "Student",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "age",
      "type": "int",
      "default": -1
    }
  ]
}

and schema Address that has reference to schema Student:

{
  "type": "record",
  "namespace": "com.example",
  "name": "Address",
  "fields": [
    {
      "name": "name",
      "type": "string"
    },
    {
      "name": "student",
      "type": "com.example.Student"
    }
  ]
}

I kept getting error message:
org.apache.avro.SchemaParseException: "com.example.Student" is not a defined name. The type of the "student" field must be a defined name or a {"type": ...} expression.

JMX testplan:
KLoadGen Test Plan.zip

KloadGen Version

KLoadGen 5.6.5

Relevant log output

2023-07-07 10:44:36,165 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[#39,AWT-EventQueue-0,6,main]
org.apache.avro.SchemaParseException: "com.example.Student" is not a defined name. The type of the "student" field must be a defined name or a {"type": ...} expression.
	at org.apache.avro.Schema.parse(Schema.java:1676) ~[kloadgen-5.6.5.jar:?]
	at org.apache.avro.Schema$Parser.parse(Schema.java:1433) ~[kloadgen-5.6.5.jar:?]
	at org.apache.avro.Schema$Parser.parse(Schema.java:1421) ~[kloadgen-5.6.5.jar:?]
	at io.apicurio.registry.serde.avro.AvroSchemaUtils.parse(AvroSchemaUtils.java:74) ~[kloadgen-5.6.5.jar:?]
	at io.apicurio.registry.serde.avro.AvroSchemaParser.parseSchema(AvroSchemaParser.java:59) ~[kloadgen-5.6.5.jar:?]
	at io.apicurio.registry.serde.avro.AvroSchemaParser.parseSchema(AvroSchemaParser.java:38) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.schemaregistry.impl.ApicurioSchemaRegistry.setSchemaBySchemaType(ApicurioSchemaRegistry.java:110) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.schemaregistry.impl.ApicurioSchemaRegistry.getSchemaBySubject(ApicurioSchemaRegistry.java:97) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.util.JMeterHelper.getParsedSchema(JMeterHelper.java:30) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.extractor.extractors.ExtractorFactory.flatPropertiesList(ExtractorFactory.java:68) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.extractor.SchemaExtractor.flatPropertiesList(SchemaExtractor.java:33) ~[kloadgen-5.6.5.jar:?]
	at com.sngular.kloadgen.property.editor.SerialisedSubjectPropertyEditor.actionPerformed(SerialisedSubjectPropertyEditor.java:79) ~[kloadgen-5.6.5.jar:?]
	at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1972) ~[?:?]
	at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2313) ~[?:?]
	at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:405) ~[?:?]
	at javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:262) ~[?:?]
	at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:279) ~[?:?]
	at com.github.weisj.darklaf.ui.button.DarkButtonListener.mouseReleased(DarkButtonListener.java:72) ~[darklaf-core-2.7.3.jar:2.7.3]
	at java.awt.Component.processMouseEvent(Component.java:6620) ~[?:?]
	at javax.swing.JComponent.processMouseEvent(JComponent.java:3398) ~[?:?]
	at java.awt.Component.processEvent(Component.java:6385) ~[?:?]
	at java.awt.Container.processEvent(Container.java:2266) ~[?:?]
	at java.awt.Component.dispatchEventImpl(Component.java:4995) ~[?:?]
	at java.awt.Container.dispatchEventImpl(Container.java:2324) ~[?:?]
	at java.awt.Component.dispatchEvent(Component.java:4827) ~[?:?]
	at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4948) ~[?:?]
	at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4575) ~[?:?]
	at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4516) ~[?:?]
	at java.awt.Container.dispatchEventImpl(Container.java:2310) ~[?:?]
	at java.awt.Window.dispatchEventImpl(Window.java:2780) ~[?:?]
	at java.awt.Component.dispatchEvent(Component.java:4827) ~[?:?]
	at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:775) ~[?:?]
	at java.awt.EventQueue$4.run(EventQueue.java:720) ~[?:?]
	at java.awt.EventQueue$4.run(EventQueue.java:714) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:399) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:97) ~[?:?]
	at java.awt.EventQueue$5.run(EventQueue.java:747) ~[?:?]
	at java.awt.EventQueue$5.run(EventQueue.java:745) ~[?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:399) ~[?:?]
	at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86) ~[?:?]
	at java.awt.EventQueue.dispatchEvent(EventQueue.java:744) ~[?:?]
	at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:203) ~[?:?]
	at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:124) ~[?:?]
	at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:113) ~[?:?]
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:109) ~[?:?]
	at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101) ~[?:?]
	at java.awt.EventDispatchThread.run(EventDispatchThread.java:90) ~[?:?]

Have you added your JMX Testplan or a replica?

  • I have added a JMX testplan

Have you added your Schema or a replica?

  • I have added the Schema
@AdrianLagartera AdrianLagartera added the enhancement New feature or request label Jul 7, 2023
@AdrianLagartera AdrianLagartera self-assigned this Jul 11, 2023
@AdrianLagartera
Copy link
Member

Thank you for collaborate again with us, we will work on it, but will take some time, so as we said in the discussion attached, we recommend you use protobuf for this purpose meanwhile you wait for this feature.

@CROprogrammer
Copy link
Author

Hi,
what's the current status of this issue?

@jemacineiras
Copy link
Contributor

Hi @CROprogrammer,

we are looking into it. Sorry for the delay, holidays and so..

@jemacineiras
Copy link
Contributor

Hi @CROprogrammer,

as far as we know, and concern, when we request the schema from the server that should be send with fully resolved, I am still not sure that we should do this task from kloadgen.
Can you provide a working example about how you uploaded your schemas into ApiCurio?

Cheers

@CROprogrammer
Copy link
Author

CROprogrammer commented Jul 27, 2023

I've tried using Apicurio Maven plugin to register schema with reference to other schema, and it works fine. When I try to get https://localhost:8080/apis/registry/v2/ids/globalIds/{globalId}/references I indeed can confirm that my schema has reference to other schema.

My pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="https://maven.apache.org/POM/4.0.0" xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="https://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.0.6</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>avro2java_maven</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>avro2java_maven</name>
    <description>avro2java_maven</description>
    <properties>
        <java.version>17</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>1.11.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.avro</groupId>
                <artifactId>avro-maven-plugin</artifactId>
                <version>1.11.0</version>
                <configuration>
                    <sourceDirectory>${project.build.directory}/src/main/avro/</sourceDirectory>
                    <outputDirectory>${project.build.directory}/src/main/generated-sources/java/</outputDirectory>
                    <stringType>String</stringType>
                    <enableDecimalLogicalType>true</enableDecimalLogicalType>
                    <fieldVisibility>PRIVATE</fieldVisibility>
                </configuration>
            </plugin>
            <plugin>
                <groupId>io.apicurio</groupId>
                <artifactId>apicurio-registry-maven-plugin</artifactId>
                <version>2.3.1.Final</version>
                    <configuration>
                        <registryUrl>https://localhost:8080/apis/registry/v2</registryUrl>
                        <artifacts>
                            <artifact>
                                <groupId>test-group</groupId>
                                <artifactId>Address</artifactId>
                                <version>1.0</version>
                                <type>AVRO</type>
                                <file>
                                    ${project.basedir}/src/main/resources/schemas/Address.avsc
                                </file>
                                <ifExists>RETURN_OR_UPDATE</ifExists>
                                <canonicalize>true</canonicalize>
                                <references>
                                    <reference>
                                        <name>com.example.Student</name>
                                        <groupId>test-group</groupId>
                                        <artifactId>Student</artifactId>
                                        <version>1.0</version>
                                        <type>AVRO</type>
                                        <file>
                                            ${project.basedir}/src/main/resources/schemas/Student.avsc
                                        </file>
                                        <ifExists>RETURN_OR_UPDATE</ifExists>
                                        <canonicalize>true</canonicalize>
                                    </reference>
                                </references>
                            </artifact>
                        </artifacts>
                        <output />
                    </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Schemas I've used:

{
  "type": "record",
  "namespace": "com.example",
  "name": "Student",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "age", "type": "int", "default" : -1 }
  ]
}
{
  "type": "record",
  "namespace": "com.example",
  "name": "Address",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "student", "type": "com.example.Student"}
  ]
}

@jemacineiras
Copy link
Contributor

Aye, what I´m trying to say, we delegate in apicurio client to download the schema, so it should came complete. I will try your pom and build a full path for testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

3 participants