[Apollo-Compiler][Feature Request] Expose validation data access

[Xinkai]

Description

I am trying to build a custom GraphQL validation tool using apollo-compiler v1.x beta.

I like the Apollo implementation's soundness thanks to the tight Rust type system.

However, the type system is restrictive for my use case, that is I can't programmatically walk over WithErrors and read its underlying data. It only outputs a very nicely formatted text.

Is apollo-compiler supposed to be a library or an end-user facing application? If the Apollo team intends to build it as a library, I think it's best to fully expose the data structure of WithErrors and DiagnosticList.

The user-friendly error reporting can be moved to an end-user facing application, or via a separate API.

Version:
apollo-compiler = { version = "1.0.0-beta.12" }

[SimonSapin]

apollo-compiler is a library: users write code against its public API. To make upgrading painless we like to avoid breaking changes to public API where possible, or batch them so breaking upgrades are infrequent. So there’s always a tension between making the public API more powerful to enable more use cases, and not exposing too many internal details so that we can evolve them (for example) without breaking changes.

Currently you can iterate a DiagnosticList and convert each diagnostic either to pretty formatting for CLI-like output, or to JSON as found in the errors field of a GraphQL response.

Additionally, CliReport can do pretty formatting for your custom errors or diagnostics.

Could you say more about what you’re trying to do, and what’s missing for you in the current API?

[Xinkai]

Thanks for the quick response!

I am exploring the idea where an LLM reads in GQL schema as a prompt, and writes out a query/mutation to fulfill users' requests.

This ticket is about dealing with LLM hallucination. Although LLMs hallucinate from time to time, I found that the hallucinations almost always "make sense", and can be used to build a roadmap for future GQL API evolution if they can be gathered and classified.

I tried .to_json() method on one of my test cases, and I got the following, similar to what graphql-js gives.

GraphQLError {
    message: "type `Query` does not have a field `viewer`",
    locations: [
        GraphQLLocation {
            line: 2,
            column: 5,
        },
    ],
    path: [],
    extensions: {},
}

What I would rather have is something like: { kind: UndefinedField, type: Query, field: viewer }. A kind that clearly signals this is a hallucination I care about.

I would be totally fine with building a regex based solution around the current implementation if there is some guarentee on the error message text format. Is there? I didn't find it here

Currently, I am using such a regex based solution with a unit test to prevent future breakage. So it's not a really deal breaker for me.

But it might be for some other use cases in the future. That's why I brought this up hoping to sand off rough edges before 1.0 finally drops.