Discussion: EIP 1901 Add OpenRPC Service Discovery To JSON-RPC Services

[shanejonas]

Add OpenRPC Service Discovery To JSON-RPC Services

EIP: 1901
Title: Add OpenRPC Service Discovery To JSON-RPC Services
Author: Shane Jonas, Zachary Belford
Status: Draft
Type: Interface
Created: 2019-02-25

Abstract

What is this?

This is a proposal to add OpenRPC support to existing and future JSON-RPC services by adding the method rpc.discover to the projects JSON-RPC APIs, enabling automation and tooling.

The OpenRPC Document that specifies all the methods an EVM-based blockchain should implement can be found here.

Motivation

This was first proposed here as an ECIP, but the benefits of this kind of tooling is apparent across Bitcoin, Ethereum Classic, Ethereum and other JSON-RPC accessible blockchains.

What is the problem?

• Maintenence time and accuracy on:
  • documentation
  • APIs
  • clients
• Not all JSON-RPC servers return the same methods or return the same version of the API.
• There is nothing that provides both human and machine-readable interfaces to be able to interact with JSON-RPC.
• Communication between services in different programming languages is not easy.
• Implementation usually comes at the cost of user experience consuming the API.

Implementation

How do I Solve the problem?

JSON-RPC APIs can support the OpenRPC specification by implementing a service discovery method that will return the OpenRPC document for the JSON-RPC API. The method MUST be named rpc.discover. The rpc. prefix is a reserved method prefix for JSON-RPC 2.0 Specification system extensions.

Use Case

This is the vision for the use case of OpenRPC and how it would relate to multi-geth:

Specification

What is OpenRPC?

The OpenRPC Specification defines a standard, programming language-agnostic interface description for JSON-RPC 2.0 APIs, which allows both humans and computers to discover and understand the capabilities of a service without requiring access to source code, additional documentation, or inspection of network traffic. When properly defined via OpenRPC, a consumer can understand and interact with the remote service with a minimal amount of implementation logic, and share these logic patterns across use cases. Similar to what interface descriptions have done for lower-level programming, the OpenRPC Specification removes guesswork in calling a service.

Structure

This is the structure of an OpenRPC Document:

Rationale

Why would we do this?

Services need to figure out how to talk to each other. If we really want to build the next generation of automation, then having up to date libraries, documented APIs, and modern tools are going to provide easy discovery, on-boarding, and enable end user and developer interaction.

Use cases for machine-readable JSON-RPC 2.0 API definition documents include, but are not limited to:

• A common vocabulary and document will keep developers, testers, architects, and technical writers all in sync.
• Server stubs/skeletons generated in many languages
• Clients generated in many languages
• Mock Server generated in many languages
• Tests generated in many languages
• Documentation Generation

Alternative

OpenRPC documents just describe JSON-RPC APIs services, and are represented in JSON format. These documents may be produced and served statically OR generated dynamically from an application and returned via the rpc.discover method. This gives projects and communities the flexibility to adopt tools before the rpc.discover method is implemented.

Resources

• EIP: Remote procedure call specification #1474
• Document JSON-RPC interface in an EIP #1442
• Universal specs for JSON-RPC API methods #217
• EIP RPC errors #136
• https://ethereum-magicians.org/t/eip-remote-procedure-call-specification/1537/20
• Discussion: ECIP 1053 Add OpenRPC Service Discovery To JSON-RPC Services etclabscore/ECIPs#16
• EIP-1901: Add OpenRPC Service Discovery To JSON-RPC Services #1901

[shemnon]

Can an OpenRPC document be composed of multiple other OpenRPC documents?

IMHO this EIP should be one or more OpenRPC specs reflecting the already standard JSON-RPC calls for Ethereum. Client Implementors then would then use those specific files and aggregate them with their own Admin RPCs. The example linked externally should then include an example of including those standard OpenRPC documents.

If OpenRPC is not that flexible, and if the existing RPCs are not specced in some form in this EIP, then I don't think this EIP adds any value other than saying "please use OpenRPC"

[shanejonas]

@shemnon you can use $ref to reference local or remote documents the same way JSON Schema allows.

As we're writing the openrpc.json for multi-geth its definitely getting pretty hefty, lots of undocumented debug_ methods for example, and like you said, admin_.

You can check that out here: https://raw.githubusercontent.com/etclabscore/multi-geth/feat/openrpc-discover/internal/openrpc/openrpc.json

or you can check it out in the playground: https://playground.open-rpc.org/?schemaUrl=https://raw.githubusercontent.com/etclabscore/multi-geth/feat/openrpc-discover/internal/openrpc/openrpc.json

We could narrow it to the methods outlined in: #1474 within this EIP?

Also, I gave a talk at the recent EDCON on OpenRPC and how it can help our ecosystem, here's the slides: https://docs.google.com/presentation/d/13cz42ONpzbTucNGMlQOQ46Rfu-DZWon9mvlSIXuhgy8

[shanejonas]

@shemnon I took some time and narrowed it down to all the methods from #1474 and #1436 and getRawTransaction*, added the openrpc.json to the EIP, can be found in assets/eip-1901/openrpc.json on the PR.

heres a link to the playground:
https://playground.open-rpc.org/?schemaUrl=https://raw.githubusercontent.com/ethereum/EIPs/dbf64faff4dfe2f832d752292a39b8d7b9b46d2a/assets/eip-1901/openrpc.json

[shemnon]

Those two eips are a great place to start.

• Can the schema file name have some reference to the EIPs they are implementing, such as openrpc_eip_1474.json? If they are intended for composition then the name of the file should provide clarity as to what is being included. opejrpc.json could be, literally, anything.
• Do we need the server section? Again, if this file is intended for composition then a partial file should be sufficient, providing only the standardized parts of the spec. Opening on local host on 8545 is a standard port, but not a requirement for the service.
• Can we get an example of how multigeth would compose the standardized files?

[shanejonas]

@shemnon

here is an example of composition with a JSON Schema tool json-schema-ref-parser:

here is a local reference

methods: [
  {
      "$ref": "schemas/methods/eth_getProof.json"
  }
]

and a remote reference

methods: [
  {
      "$ref": "http:https://wayne-enterprises.com/methods/get_batmobile.json"

  }
]

Although I'm not a fan of this approach if it includes relative paths, if geth returns rpc.discover with a relative $ref path, that will need to be served up to actually be resolved later. Or geth (or any client) needs to dereference relative paths before served.

Maybe we can limit it to absolute urls/remote and link to EIPs with .json files?

My other thought is that this EIP implements #1474 and #1436 as of now, and it could be the canonical place to update methods, (like 1474 is now).

[shanejonas]

Took the time to put this repo together to house the specification we've started to outline here: https://github.com/etclabscore/ethereum-json-rpc-specification

[BelfordZ]

@shemnon Just wondering if you have any time to review this?

As per your question about composition -- All OpenRPC tooling demands an entire OpenRPC document to be passed in. How you assemble the document is up to the user, and therefore is as composible as any other JSON document. For most cases, composition probably means splitting method definitions out into their own file, which is really as simple as concatenating the set of methods into the methods array.

for the server section question -- it is not a required field and I think you are right there, for geth/multi-geth/ethereum-json-rpc-specification, we should not use the servers section ATM. That being said, it allows for many interesting applications, such as treating the servers array returned by a services rpc.discover as a DHT of available peers in a L2 p2p system.

[BelfordZ]

Forgot to mention - Multi-geth is now implementing OpenRPC along with its service discovery method, so you can have a look at how it was implemented if you wish
https://github.com/multi-geth/multi-geth#openrpc-discovery

[shemnon]

If current OpenRPC tooling requires a singular text document to serve as a schema, then I feel that the OpenRPC tooling should be improved. One doesn't put all their Go interfaces in one file after all.

[BelfordZ]

OpenRPC is language agnostic - where/how you implement your methods, where you store the OpenRPC Documents JSON, or how you construct it is entirely up to you. This is not a concern of the tooling that we have thus far, which handles generating clients, documentation, mock servers, etc etc.

What I think you are talking about (please correct me if I'm wrong) is a tool which would convert go interfaces into JSON schemas + MethodObjects. I'd love to chat about making such a tool for OpenRPC, as you are now the 2nd Go dev to ask for such).

[shemnon]

Actually I'm a Java dev and our platform is Java, that wasn't a request for Go tooling.

What I want the OpenRPC spec to be able to handle is that I want to say my schema is made up of multiple pieces, evaluated in aggregate. And I don't want to have to go into the files and do any JSON querying or assembly. For example, I want to include the byte identical definition of the core Ehtereum JSON-RPCs, the byte identical definition of the EEA extension RPCs, and a set of PegaSys specific admin RPCs and say that those three files, in aggregate, form the schema of the endpoint. I'm OK making a fourth document pointing to the other three, but I don't want to have to cut and paste the three documents in any way, shape, or form into one mega-document. If my tool has to do any manipulation of the spec documents and those byte identical documents need to be modified in any form I consider it a problem with the OpenRPC spec, not the tooling.

[BelfordZ]

@shemnon Understood, and I see your point about having to muck around merging JSON files together is not ideal when what you are looking for is a pattern to compose several documents into one.

give me some time to think of other ways we could achieve this.

[BelfordZ]

merging/composing the methods is really simple. What is hard to understand is how composing OpenRPC documents would handle merging fields such as info

[BelfordZ]

perhaps we could try a Specification Extension to start with: x-extends which would allow you to specify a reference to another OpenRPC document from which to extend from.

OpenRPC tooling such as parse could then return a single document that has been composed as you describe.

[shanejonas]

🔔 just wanted to update this thread and show progress:

Implementation

Multi-geth has full OpenRPC support, for Ethereum, Ethereum classic, kotti, goerli, and many other chains:
https://github.com/etclabscore/multi-geth#openrpc-discovery

The Spec:

The repo itself is getting some contributors, added missing methods like eth_pendingTransactions:
https://github.com/etclabscore/ethereum-json-rpc-specification

The Tooling

The playground has gotten some upgrades to be able to make JSON-RPC calls from within the docs:

You can play with that here:
https://playground.open-rpc.org/?schemaUrl=https://raw.githubusercontent.com/etclabscore/ethereum-json-rpc-specification/master/openrpc.json

[github-actions]

There has been no activity on this issue for two months. It will be closed in a week if no further activity occurs. If you would like to move this EIP forward, please respond to any outstanding feedback or add a comment indicating that you have addressed all required feedback and are ready for a review.

[shanejonas]

Would like to see this moved through the process as OpenRPC is now being used here: https://github.com/ethereum/execution-apis

[MicahZoltu]

@shanejonas I recommend checking out EIP-1 to understand the EIP process and how to get this pushed through. Make sure you are prepared to do the leg work championing it, or you know someone who is, as this will require discussion with client dev teams.

[shanejonas]

I believe the client team devs are already involved with https://github.com/ethereum/execution-apis as it was used for EIP 1559 to ensure compatibility, and now for the ETH2 merge.

I've also had a chat with client devs on All Core Devs previously which sparked some interest, but now that its being used it would be great to chat again to see where client teams are in regards to the current execution-apis in terms of compatibility.

[github-actions]

There has been no activity on this issue for two months. It will be closed in a week if no further activity occurs. If you would like to move this EIP forward, please respond to any outstanding feedback or add a comment indicating that you have addressed all required feedback and are ready for a review.

[github-actions]

This issue was closed due to inactivity. If you are still pursuing it, feel free to reopen it and respond to any feedback or request a review in a comment.