Graphql schema storage as dockerized on-premise service for federated graphql gateway server (based on apollo server) as alternative to Apollo studio and The Guild's Hive

• Stores versioned schema for graphql-federated services
• Serves supergraph schema for graphql gateway based on provided services & their versions
• Validates new schema to be compatible with other running services
• Provides UI for developers to see stored schema & its history diff
• Stores service urls emulating managed federation: you no longer need to hardcode the services in your gateway's constructor, or rely on an additonal service (etcd, consul) for service discovery
• Does async schema usage analysis to minimize breaking changes from reaching production
• Publishes schema registration events to minimize supergraph update time
• Stores & shows in UI automatic persisted queries (passed by the gateway) for better visibility

With default settings, UI should be accessible at https://localhost:6001

git clone https://github.com/pipedrive/graphql-schema-registry.git && cd graphql-schema-registry
cp example.env .env && nano .env
npm install && npm run build
node app/schema-registry.js

We have docker image published for main node service. It assumes you have mysql/redis running separately. Use exact IP instead of localhost. Use exact docker image tag to avoid breaking changes.

docker pull pipedrive/graphql-schema-registry:5.0.0
docker run -e DB_HOST=localhost -e DB_USERNAME=root -e DB_PORT=6000 -p 6001:3000 pipedrive/graphql-schema-registry

git clone https://github.com/pipedrive/graphql-schema-registry.git && cd graphql-schema-registry
docker-compose -f docker-compose.base.yml -f docker-compose.prod.yml up

flowchart LR
    GW[federated-gateway] == poll schema every 10 sec\n POST /schema/compose ==> SR["schema registry\n(gql-schema-registry)"] -- store schemas --> DB[("mysql 8\n(gql-schema-registry-db)")]
    SR -- cache persisted queries\nstore & query last logs --> R[("redis 6\n(gql-schema-registry-redis)")]
    SR -- publish schema change --> KF1("kafka\n(gql-schema-registry-kafka)\ngraphql-schema-updates topic") -- listen schema updates --> GW
    GW -- publish queries --> KF2("kafka\n(gql-schema-registry-kafka)\ngraphql-queries topic")
    KF2 --> QA["query analyzer\n(gql-schema-registry-worker)"]
    QA --update schema hits --> DB
    GW -- query A --> S1["service A"] -- register schema in runtime --> SR
    GW -- query B --> S2["service B"] -- register schema in runtime --> SR
    S2["service B"] -. validate schema \n on commit/cli/CI .-> SR

    style KF1 fill:#0672e6,color:white
    style KF2 fill:#0672e6,color:white
    style DB fill:#0672e6,color:white
    style R fill:#0672e6,color:white
    style SR fill:#ffe43e
    style QA fill:#ffe43e
    style GW fill:#c5f7c9
    style S1 fill:#c5f7c9
    style S2 fill:#c5f7c9

Migrations are done using knex

erDiagram
    services {
        id int PK
        name varchar
        is_active int
        updated_time datetime
        added_time datetime
        url varchar
    }
    clients {
        id int PK
        name varchar
        version varchar
        calls bigint
        updated_time datetime
        added_time datetime
    }
    schema {
      id int PK
      service_id int FK
      is_active tinyint
      type_defs mediumtext
      updated_time datetime
      added_time datetime
    }
    container_schema{
      id int PK
      service_id int FK
      schema_id int FK
      version varchar
      added_time datetime
    }
    persisted_queries{
      key varchar PK
      query text
      is_active int
      updated_time datetime
      added_time datetime
    }
    clients_persisted_queris_rel{
      version_id int
      pq_key varchar
    }
    schema_hit{
      client_id int
      entity varchar
      property varchar
      day date
      hits bigint
      updated_time bigint
    }

    services ||--o{ schema : defines
    schema ||--|{ container_schema : "is registered by"
    clients_persisted_queris_rel ||--|{ persisted_queries : use
    clients_persisted_queris_rel ||--|{ clients : have
    clients ||--|{ schema_hit : "use schema properties"
    schema ||..|{ schema_hit : "relates to"

We use environment variables for configuration. You can:

• pass them directly
• add .env file and dotenv will pick them up
• add them to docker-compose.yml or Dockerfile

The following are the different environment variables that are looked up that allow configuring the schema registry in different ways.

For development we rely on docker network and use hostnames from docker-compose.yml. Node service uses to connect to mysql & redis and change it if you install it with own setup. For dynamic service discovery (if you need multiple hosts for scaling), override app/config.js and diplomat.js

On service start-up (runtime), make POST to /schema/push to register schema (see API reference for details). Make sure to handle failure.

See example for nodejs/ESM.

Usually in production, POST /schema/push requires unique version that should be unique git or docker hash. But, if you are developing a service and you run schema-registry locally, you can set version: "latest" to skip this version check.

• if your gateway uses /schema/compose then no, schema is composed based on services you see as healthy
• if your gateway uses /schema/latest then yes, service has is_active flag in DB that you can manually toggle (no API yet)

On pre-commit / deploy make a POST /schema/validate to see if its compatible with current schema. If you have multiple regions or environments (test), makes sense to check all.

If service A contains schema that needs to be migrated to service B with close to 0 downtime, you will need to orchestrate schema & traffic change. Instead of juggling with schema status flags, we suggest the following scenario:

sequenceDiagram
    participant service_A
    participant service_B
    participant schemaRegistry

    loop Every 5 sec
       service_B->>+schemaRegistry: register schema B + A1
       schemaRegistry->>-service_B: validation error, A1 is registered to service A
    end

    service_A->>+schemaRegistry: remove schema A1
    schemaRegistry->>-service_A: ok

    service_B->>+schemaRegistry: register schema B + A1
    schemaRegistry->>-service_B: ok, A1 is registered to service B, stop retries

nvm use
npm install
npm run build
docker-compose  -f docker-compose.base.yml  -f docker-compose.dev.yml up

To have fast iteration of working on UI changes, you can avoid running node service in docker, and run only mysql & redis

docker-compose -f docker-compose.base.yml up -d
npm run develop
npm run develop-worker

To create new DB migration, use:

npm run new-db-migration

If not using the default configuration of executing DB migrations on service startup, you can run the following npm command prior to starting the registry:

npm run migrate-db

The command can be prefixed with any environment variable necessary to configure DB connection (in case you ALTER DB with another user), such as:

DB_HOST=my-db-host DB_PORT=6000 npm run migrate-db

use jest, coverage is quite low as most logic is in db or libraries.

npm run test

require docker, mostly blackbox type - real http requests are done against containers. DB tables are truncated after every test from within test/functional/bootstrap.js Jest runs in single worker mode to avoid tests from affecting each other due to same state.

#docker-compose -f docker-compose.light.yml up -d
#npm run develop

npm run test-functional

use k6 + dockerized setup similar to functional tests above + grafana and influxdb for reporting the load these tests are intended just to show/detect avg latencies of most important endpoints

docker-compose -f docker-compose.perf-tests.yml up
open https://localhost:8087/dashboard/import
// Add "2587" to ID, pick influxdb datasource, import dashboard and observe it when you run tests

docker-compose -f docker-compose.perf-tests.yml run --rm k6 run /scripts/schema-latest.test.js

If you change build process in Dockerfile or Dockerfile.CI, consider checking also testing it

# run db
docker-compose -f docker-compose.light.yml up

#build local image
docker build -t local/graphql-schema-registry .

# try to run it
docker run -e DB_HOST=$(ipconfig getifaddr en0) -e DB_USERNAME=root -e DB_PORT=6000 -p 6001:3000 local/graphql-schema-registry

• There is not strict process on finding or updating vulnerabilitites. The license also states there there is no Liability or Warranty, so be aware of that
• You can use Slack or Github discussions / issues as a communication channel to notify about vulnerabilities.
• We use snyk in PR checks. We try to look at npm audit reports manually on PR creation to minimize issues.
• We intentionally use strict versioning of nodejs dependencies which prevents automatic dependabot PRs. Thus version upgrades are manual. Why? Because sometimes we saw external dependencies rolling out breaking changes in minor/patch versions which broke our master. Same thing with hacked libraries.

• Commit often (instead of making huge commits)
• Add verb at the beginning of commit message
• Add why you're doing something in commit message
• Reference issues
• When making a pull request, be sure to follow the format of what is the problem you're fixing, what was changed & how to test it. Screenshots/videos are a welcome
• Fill CHANGELOG
• To avoid vulnerabilities, please use fixed versions in package.json

Current maintainer - @tot-ra. Mention in PR, if it is stuck

Original internal mission that resulted in this project going live:

• aleksandergasna
• ErikSchults
• LexSwed

See main blog post

{
  "services": [
    { "name": "service_a", "version": "ke9j34fuuei" },
    { "name": "service_b", "version": "e302fj38fj3" }
  ]
}

{
  "success": true,
  "data": [
    {
      "id": 2,
      "service_id": 3,
      "version": "ke9j34fuuei",
      "name": "service_a",
      "url": "https://localhost:6111",
      "added_time": "2020-12-11T11:59:40.000Z",
      "type_defs": "\n\ttype Query {\n\t\thello: String\n\t}\n",
      "is_active": 1
    },
    {
      "id": 3,
      "service_id": 4,
      "version": "v1",
      "name": "service_b",
      "url": "https://localhost:6112",
      "added_time": "2020-12-14T18:51:04.000Z",
      "type_defs": "type Query {\n  world: String\n}\n",
      "is_active": 1
    }
  ]
}

{
  "name": "service_a",
  "version": "ke9j34fuuei",
  "type_defs": "\n\ttype Query {\n\t\thello: String\n\t}\n"
}

{
  "name": "service_b",
  "version": "jiaj51fu91k",
  "type_defs": "\n\ttype Query {\n\t\tworld: String\n\t}\n",
  "url": "https://service-b.develop.svc.cluster.local"
}