Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

avro schema generator #174

Merged
merged 9 commits into from
Jun 25, 2024
Merged

avro schema generator #174

merged 9 commits into from
Jun 25, 2024

Conversation

lyuboxa
Copy link
Contributor

@lyuboxa lyuboxa commented Jun 22, 2024

Description

This PR adds the functionality to generate an Avro schema for each snapshot, create and update operations.
The avro schema is rendered in JSON and added to each record under the postgres.avro.schema metadata key.

The schema may be used to hint about types where precision or other may be lost as they are converted to go types, e.g. dates, numeric, etc.

Usage of the schema is a temporary measure until Conduit schema management is released.

In order for the schema to be emitted, the following config var will need to be set logrepl.withAvroSchema

Quick checks:

  • I have followed the Code Guidelines.
  • There is no other pull request for the same update/change.
  • I have written unit tests.
  • I have made sure that the PR is of reasonable size and can be easily reviewed.

Copy link
Contributor

@hariso hariso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lyuboxa @samirketema I left a few comments, I'm mostly interested in nullability of columns and that we're fine with the list of supported types for now. Awesome work!

@lyuboxa lyuboxa marked this pull request as ready for review June 24, 2024 19:31
@lyuboxa lyuboxa requested a review from a team as a code owner June 24, 2024 19:31
@lyuboxa lyuboxa merged commit 5ecd48e into main Jun 25, 2024
9 checks passed
@lyuboxa lyuboxa deleted the lk/emit-avro-schema branch June 25, 2024 11:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants