$ref resolution process change in v7 (bug in v6)

[benjamin-mogensen]

Apologies for the fairly long JSON schema. It is a stripped down version of a quite large schema which is being bundled using json-schema-ref-parser. The problem occurs with the $ref values. The ajv-cli version 3.3.0 compiles the schema and claims it is valid. Newer versions of AJV can't compile as it cannot resolve the reference.

The problem seems to be that the $ref for property ContainedWithin is pointing to ShipperBookingLineParent which in turn points to ShipperBookingLine.

• I know it's a bit convoluted, but is AJV supposed to be able to resolve that kind of reference (apparently in older version it was ok) or is it not supported anymore?

It is possible to update the $ref by removing parts of it (ShipperBookingLineParent/allOf/5/properties/) however the non-stripped down schema has many $ref which are created by json-schema-ref-parser so I am not able to post process them in a meaningful manner as some would require removal and others replacement of paths. Perhaps also worth mentioning that the tool Stoplight Studio has no problem resolving the reference.

What version of Ajv are you using? Does the issue happen if you use the latest version?
ajv-cli 5.0.0
ajv 8.0.5 (Have a Javascript running with this version, same result)

Ajv options object
Not used - running with defaults

JSON Schema

{
  "$schema": "http:https://json-schema.org/draft-07/schema#",
  "$id": "/models/01-MaerskInformationModel-Bundled/Booking/ShipperBooking/ShipperBooking-bundled.v1-stripped.json",
  "type": "object",
  "properties": {
    "ShipperBookingLines": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "DangerousCargoInformation": {
            "type": "object",
            "properties": {
              "HazardousClassification": {
                "type": "object",
                "properties": {
                  "RequiredDangerousCommodityPackageType": {
                    "type": "object",
                    "properties": {
                      "AwesomeProperty": {
                        "type": "string"
                      },
                      "ContainedWithin": {
                        "$ref": "#/properties/ShipperBookingLines/items/properties/ShipperBookingLineParent/properties/DangerousCargoInformation/properties/HazardousClassification/properties/RequiredDangerousCommodityPackageType"
                      }
                    }
                  }
                }
              }
            }
          },
          "ShipperBookingLineParent": {
            "$ref": "#/properties/ShipperBookingLines/items"
          }
        }
      }
    }
  }
}

Sample data

Not applicable, schema compilation fails

Your code

npx ajv-cli compile -s ShipperBooking-bundled.v1-stripped.json

Validation result, data AFTER validation, error messages
ajv-cli:

schema ShipperBooking-bundled.v1-stripped.json is invalid
error: can't resolve reference #/properties/ShipperBookingLines/items/properties/ShipperBookingLineParent/properties/DangerousCargoInformation/properties/HazardousClassification/properties/RequiredDangerousCommodityPackageType from id /models/01-MaerskInformationModel-Bundled/Booking/ShipperBooking/ShipperBooking-bundled.v1-stripped.json

ajv:

MissingRefError: can't resolve reference #/properties/ShipperBookingLines/items/properties/ShipperBookingLineParent/properties/DangerousCargoInformation/properties/HazardousClassification/properties/RequiredDangerousCommodityPackageType from id /models/01-MaerskInformationModel-Bundled/Booking/ShipperBooking/ShipperBooking-bundled.v1-stripped.json
    at Object.code (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\vocabularies\core\ref.js:21:19)
    at keywordCode (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\validate\index.js:451:13)
    at C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\validate\index.js:185:25
    at CodeGen.code (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\codegen\index.js:439:13)
    at CodeGen.block (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\codegen\index.js:568:18)
    at schemaKeywords (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\validate\index.js:185:13)
    at typeAndKeywords (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\validate\index.js:129:5)
    at subSchemaObjCode (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\validate\index.js:116:5)
    at subschemaCode (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\validate\index.js:92:13)
    at KeywordCxt.subschema (C:\dev\workspace2\JSON-schema-test\node_modules\ajv\dist\compile\validate\index.js:425:9) {
  missingRef: '/models/01-MaerskInformationModel-Bundled/Booking/ShipperBooking/ShipperBooking-bundled.v1-stripped.json#/properties/ShipperBookingLines/items/properties/ShipperBookingLineParent/properties/DangerousCargoInformation/properties/HazardousClassification/properties/RequiredDangerousCommodityPackageType',
  missingSchema: '/models/01-MaerskInformationModel-Bundled/Booking/ShipperBooking/ShipperBooking-bundled.v1-stripped.json'
}

What results did you expect?
That the schema compiles as it does with ajv-cli 3.3.0

Are you going to resolve the issue?
I am not familiar with the AJV source code, so I can't.

[epoberezkin]

The reference is wrong - the resolution is not supposed to go via $ref.

In this ref #/properties/ShipperBookingLines/items/properties/ShipperBookingLineParent/properties/DangerousCargoInformation/properties/HazardousClassification/properties/RequiredDangerousCommodityPackageType the resolution would expect to have properties inside ShipperBookingLineParent, but there is no properties, only $ref.

I can see that it is indeed "working" in 3.3.0 of ajv-cli / and in ajv 6.12.6 (https://runkit.com/esp/606e16df5822b50013e83ac9) - that seems like a bug to me, tbh, that was somehow fixed with the new versions (it is correctly failing in v7 as well).

It may be better not to bundle multiple schemas in a single one, and instead use one of the approaches to manage multiple schemas suggested here: https://ajv.js.org/guide/managing-schemas.html

[benjamin-mogensen]

Thanks for taking a look at it. The unbundled schemas do compile with AJV when I load them to AJV and their dependant schemas as resources. As I want to make sure my bundled schemas are still valid after using json-schema-ref-parser I run AJV on the bundled ones as well. So I guess the conclusion is that json-schema-ref-parser creates a $ref that is not compatible with AJV - the funny thing is that a tool like Stoplight Studio has no problem resolving this reference so I am still thinking if AJV should not be able to as well - or are we saying that the $ref to $ref above is illegal in JSON schema definitions and that json-schema-ref-parser is doing things wrong?

[epoberezkin]

They are invalid according to the JSON Schema spec, not just incompatible.

I’m not sure if Stoplight Studio uses Ajv - Stoplight 100% does - they may be on v6 still - best to ask them?

[epoberezkin]

or are we saying that the $ref to $ref above is illegal in JSON schema definitions and that json-schema-ref-parser is doing things wrong?

exactly

[epoberezkin]

(Unless I’m completely wrong)

cc @Relequestual - ref resolution should not traverse via another ref like in the example above, am I right?

[benjamin-mogensen]

Once again thanks for the quick answer @epoberezkin looking forward to hear what @Relequestual comes back with. I will try and get hold of the Stoplight Studio team and see how they manage to resolve this kind $ref.

Just FYI I have raised a ticket with Stoplight team - it seems we cannot open issues on their Github site.

[benjamin-mogensen]

Just to share with community - When importing the schema in to the Confluent platform it will fail validation just as AJV will fail compilation. So it seems that at least other products agree that this kind of $ref to $ref resolution should not be possible. Unfortunate as I am now stuck with hard code fixing the paths in my schemas as no bundling tool I have tried is able actually bundle correctly.

There was a package of json-schema-ref-parser that was suggested to be able to fix this issue but unfortunately it turned out not to fix it. It will be difficult for me to reproduce the bundling in to a small example so I don’t have much hope for having this fixed in json—schema-ref-parser 😕

[philsturgeon]

I see, looks like the trouble is coming from:

                    "ContainedWithin": {
                        "$ref": "#/properties/ShipperBookingLines/items/properties/ShipperBookingLineParent/properties/Dangerous...

When ShipperBookingLineParent only has a $ref and no properties.

      "ShipperBookingLineParent": {
            "$ref": "#/properties/ShipperBookingLines/items"
      }

This is likely a problem with json-schema-ref-parser then. The Stoplight crew help maintain that package so we can try to take a look, but if anyone else has time feel free to take a look.

[Relequestual]

Once again thanks for the quick answer @epoberezkin looking forward to hear what @Relequestual comes back with. I will try and get hold of the Stoplight Studio team and see how they manage to resolve this kind $ref.

Late to reply, sorry!

Reference resolution happens "all at once".

The reference you had points to a location that does not exist. @epoberezkin is totally correct here.

I know json-schema-ref-parser has a number of issues, and I think ultimatly it's going to be replaced with simpler libraries that follow the (now defined) bundling process.

It used to be $ref could be, in some situations, replaced with the referenced schema. That caused all sorts of headaches and is no longer the case, but it's still often what people want or expect. We're trying hard to break that expectation.

I think you can go ahead and close this issue, because it is as intended.

[epoberezkin]

@Relequestual thank you!

[benjamin-mogensen]

Thanks @Relequestual for your input, much appreciated. Could you let me know about this "now defined" bundling process? I am interested in what is changing. Finally I ended up coding a post processor that fixes the invalid $ref's generated by json-schema-ref-parser. I don't think it 100% works for all situations but it works for what we need at the moment.

[Relequestual]

The part of the specification where this is defined is found at https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-00#section-9.3

I've written 99% of a blog post which explains it a little, and it's pending publication.

Part of the problem with how json-schema-ref-parser works, and what changed in 2019-09 made things even worse (for the library). You can read about key comments here: APIDevTools/json-schema-ref-parser#145

If you're allergic to reading RFC spec documents, you could join the JSON Schema slack and watch out for updates on the announcements channel for the blog posting on bundling.

There wasn't a defined bundling process before, and it's somewhat easier to define with the changes we made in 2019-09 (outlined in that link above).

[benjamin-mogensen]

@Relequestual thanks for sharing that - well written! Are there any libraries that currently follows this proposed bundling/compound specification? Thinking how I can get a head start on this and sunset use of json-schema-ref-parser. I tried bundling some of our schemas using Stoplight Studio which is able to put referenced schemas in to a definition section however the $refs inside the output file are not valid according AJV so I am hesitant to use whatever is be used under the hood of Stoplight at the moment.

[Relequestual]

@benjamin-mogensen late reply, but there wasn't, so it probably wouldn't have helped.
I just posted a thread about this: https://twitter.com/relequestual/status/1460968218319806464?s=20

There is an implementation, but it's pre-release: https://github.com/hyperjump-io/json-schema-bundler