HTTP3 based SCION control plane

[oncilla]

HTTP3 based SCION control plane

Next to SVC resolution (as discussed in #4388), our SCION control plane RPCs are another messy leftover
from long gone times of experimentation and making it work. Currently, we are
hacking gRPC (which is using HTTP/2) on top of a single QUIC stream, which
emulates a TCP connection. To spell it out, our SCION control plane RPC stack
is: gRPC/HTTP2/QUIC/SCION. We are abusing QUIC streams and not leveraging QUIC
effectively. During standardization, it will be very hard to argue that this
is a good way of doing things (because it is not).

We originally decided to use QUIC because it had good user space
implementations. Now, we can profit from this decision. QUIC has become the new
transport of choice. With it, HTTP/3 has become the application transport layer
of choice too. Support for it is growing, due to the synergies with QUIC. If we can
leverage these two technologies for our SCION control plane RPCs, it will be
very easy to argue why we use them, given they are basically the standard for
current networking stacks.

While gRPC has allowed us to have a reliable SCION control plane RPC stack so
far, it does not look like they
will adopt HTTP/3 any time soon. Even more worrying, The grpc-go implementation
is tightly coupled to HTTP/2
(which they have re-implemented the transport themselves). Custom transports
have been proposed 5 years ago
already. Moving forward, I think we might want to adapt a different RPC stack
which builds on top of HTTP/3 and QUIC.

Connect is an RPC framework that promises the same
benefits as gRPC (API specified in ProtoBuf, boilerplate generation, etc.).
However, it is a very focused project with a small code base. They even offer
gRPC compatible servers (which we do not need though). Connect also defines
their own protocol for RPC over HTTP.
Note that they do not depend on framing details of specific HTTP versions, thus
this protocol is compatible with HTTP/1.1 (except bi-directional streaming),
HTTP/2, and HTTP/3. The protocol is also so simple that it can be easily
implemented in any language that offers an HTTP and Protobuf library.

I propose that we change to connect rpc, which will allow us to do away with
our gRPC-over-QUIC-stream hack, as we can just use regular HTTP/3 over QUIC.

Keeping Backwards Compatibility - ALPN

Same as for SVC resolution, changing the RPC protocol is not an AS local change
and needs to be rolled out in a backwards compatible way. Here we need the same.

We can distinguish the target RPC protocol based on the application protocol
that is negotiated via ALPN. In our current gRPC based deployments, we specify
"SCION" as the next protocol in the ALPN extension. Given connect rpc is based
on HTTP/3, the value in the ALPN extension will be "h3". A backwards compatible
server would advertise ["h3", "SCION"]. Based on the negotiated application
protocol, the server will then either serve the old gRPC based RPCs, or the
new connect based RPCs.

We can access the negotiated application protocol via the
quic.ConnectionState
that contains the
tls.ConnectionState which lists
the NegotiatedProtocol. The quic-go http3.Server has support for serving
individual QUIC connections with
http3.Server.ServeQUICConn.
This makes it easy to write a backwards compatible server.

On the client side, there are two different solutions.

1. Happy eyeballs
2. Decide on RPC protocol after dialing QUIC connection

Alternatives Considered

TODO

• twirp
• REST

[matzf]

Just for the record. At the IETF 118 Hackathon in Prague, @oncilla, @FR4NK-W and I have worked on a prototype for transitioning from current gRPC/HTTP2/QUIC to connect/HTTP3. The code is in the branch scionproto/scion:connectrpc.
We haven't converted all the RPCs, but a good fraction and so far, everything went smoothly. A cool (even if perhaps not very useful) feature is that we were able to trigger RPCs with curl (for AS-local requests) or scion-bat (from the scion-apps repo).

Notes:

• should we combine this with other changes to the RPCs, notably fixing the namespace? -> Perhaps not; changing the namespace can be done smoothly later. On the server side, we can very easily support this in a backwards compatible way by rewriting the request path.
• the "stack" with connect and quic-go/http3 make it trickier to access the address-object related to the connection. This is especially tricky for beaconing, where the path information (specifically the local interface ID that the beacon is received on) is needed to process the request. Dominik worked around this by minimally patching quic-go to expose this address in the context. This patching using gazelle will not be effective for go build, so we'll need to find a different long-term solution.
• of the two client side options mentioned above, 1. (happy eyeballs) is implemented in the prototype.
• For the intra-AS RPCs that use TCP, we can get rid of the grpc-go server entirely, as this is also handled by connect-go. For the inter-AS RPCs, we kept the grpc-go server and handle requests based on the negotiated TLS protocol either passing it into http3 and connect, or use the http2-grpc-shim.
  • Just a thought while writing this; if we rebuild the http2/quic-shim based on the standard http library, we might be able to rid of grpc-go entirely and use connect even to handle the gRPC "legacy" requests in the inter-domain. This might make it easier to organize the code, as we wouldn't need wrappers to make things work for grpc-go and connect-go.

[oncilla]

This patching using gazelle will not be effective for go build, so we'll need to find a different long-term solution.

I think the only long-term approach is to create the patch upstream and convince them to merge it. Otherwise, we will have to fork the project to please to go toolchain.

If we use a Go mod replace approach, that will not translate for dependents of scionproto/scion (which might be fine)

Just a thought while writing this; if we rebuild the http2/quic-shim based on the standard http library, we might be able to rid of grpc-go entirely and use connect even to handle the gRPC "legacy" requests in the inter-domain. This might make it easier to organize the code, as we wouldn't need wrappers to make things work for grpc-go and connect-go.

Interesting thought. It might just work. But it depends on the effort we need to put in. The current approach works, and given we want to nuke the old way, it might not be worth investing more time into it. It could also introduce potential new edge cases for a thing that is being ripped out anyway, and introduces additional compatibility testing burden.

[oncilla]

connect has joined the CNCF (see: https://buf.build/blog/connect-rpc-joins-cncf).

[nicorusti]

JFYI we documented this in the control plane IETF draft, see scionassociation/scion-cp_I-D#15

[matzf]

Going forward, we're likely going accept this proposal. The final comment period starts now, and unless there are objections during this period, the proposal will be marked as accepted.

[marcfrei]

Looking at the branch connectrpc, the intra AS control plane will use connect/HTTP2/TCP, right? Will we support both application/proto and application/json for intra-AS?

[oncilla]

Will we support both application/proto and application/json for intra-AS?

I think so, yes. Connect gives us this for free.
It can be convenient for some RPCs, but for most the data is base64 encoded anyway due to signed blobs.

[marcfrei]

Thanks! I'll also need to take a closer look at the actual Connect protocol spec to fully understand how exactly RPCs can be used via binary protobufs from languages without a code generator, e.g., from the new Java implementation or the C version for embedded systems we are currently working on.

[oncilla]

For intra AS communication, I think we will not drop gRPC support anytime soon, given it is there for free.
As long as the client language supports gRPC generation, we are good.

The proposal is only concerned with inter AS communication fwiw

[marcfrei]

Ah, very good. gRPCs are already kind of working in our client.

[matzf]

No objections where raised during the final comment period. Accepted.