http3: implement HTTP Datagrams and Capsules (RFC 9297)

[marten-seemann]

https://www.rfc-editor.org/rfc/rfc9297.html

[marten-seemann]

Writing and parsing of CAPSULEs was implement by #3607.

[marten-seemann]

The API will be a bit tricky. Here's a proposal.

On the client side

RoundTripper.EnableDatagrams enables both QUIC and HTTP datagrams.

HTTP Datagrams are always associated with an HTTP Request (section 2 of the RFC):

HTTP Datagrams MUST only be sent with an association to an HTTP request that explicitly supports them. For example, existing HTTP methods GET and POST do not define semantics for associated HTTP Datagrams; therefore, HTTP Datagrams associated with GET or POST request streams cannot be sent.

It seems like the http.RoundTripper interface isn't suitable for that use case, since we need to be able to send datagrams right after sending the HTTP request (without having to wait for the response). However, RoundTrip blocks until the HTTP response has been received from the server. This means that we need to introduce a new interface:

type RoundTripperWithDatagrams interface {
     RoundTripWithDatagrams(*http.Request) (Datagrammer, <-chan *http.Response, error)
}

Calling RoundTripWithDatagrams immediately returns to allow sending datagrams. The HTTP response is sent on the channel once it is received. Datagrammer is an interface that allows sending of HTTP datagrams:

type Datagrammer interface {
     // SendMessage sends an HTTP Datagram associated with an HTTP request.
     // It must only be called while the send side of the stream is still open, i.e.
     // * on the client side: before calling Close on the request body
     // * on the server side: before calling Close on the response body
     SendMessage([]byte) error
     // SendMessage receives an HTTP Datagram associated with an HTTP request:
     // * on the server side: datagrams can be received while the request handler hasn't returned, AND 
     //      the client hasn't close the request stream yet
     // * on the client side: datagrams can be received with the server hasn't close the response stream
     ReceiveMessage(context.Context) ([]byte, error)
}

Note that while this interface looks like the functions exposed the quic.Connection, it does prepend the Quarter Stream ID on SendMessage as defined in section 2.1 of the RFC, and demultiplexes (and removes) it on ReceiveMessage.

On the server side

Server.EnableDatagrams enables both QUIC and HTTP datagrams.

On the server side, we need to be able to associate datagrams with an HTTP handler. This is easier to implement, since http.ResponseWriter is an interface, and can therefore be interface-asserted to a Datagrammer.

[tanghaowillow]

Would like to take a stab.

[marten-seemann]

Go for it! Happy to review a PR. Let me know if you have any questions!

[marten-seemann]

I'll be working on this issue in the coming weeks. Thank you @tanghaowillow for your PR, there are a lot of helpful things in there.

API-wise, the client side is the tricky part: The client is allowed to send HTTP datagrams as soon as it has opened the stream (or better, to reduce reordering, after the request headers have been sent out). However, RoundTrip blocks until the HTTP response has been received, i.e. for 1 RTT commonly.

#4141 suggests the following API:

// RoundTripResult is the result of a HTTP RoundTrip execution
type RoundTripResult struct {
	Resp *http.Response
	Err  error
}

datagrammer, respChan, err := client.Transport.(*http3.RoundTripper).RoundTripWithDatagrams(req)

This function would return immediately, or probably better, after the server's SETTINGS were received and the check for HTTP DATAGRAM support was performed. Note that there are now two errors to check: the one returned from RoundTripWithDatagrams and the one in the RoundTripResult.

I'm currently leaning towards a slightly different API that doesn't change the RoundTrip semantics. Instead, it adds another field to the RoundTripOpts:

// RoundTripOpt are options for the Transport.RoundTripOpt method.
type RoundTripOpt struct {
	// DatagrammerReady is called as soon as the server's SETTINGS frame is received,
	// and HTTP DATAGRAM support is enabled.
	// If HTTP DATAGRAMs are not supported, the request won't be sent to the server
	// and a ErrNoDatagrams error is returned.
	DatagrammerReady func(Datagrammer)
}

I don't particularly like this API either, but we're kind of limited with what we can do if we don't want to stray away too far from the standard library's net/http API pattern.

[taoso]

Hi @marten-seemann sorry for missing this issue. And let move our discussion here.

In my previous proposal #4372 I suggest this following APIs.

For the client side

req, err := http.NewRequest(http.MethodConnect, addr, nil)
req.Proto = "connect-ip"

rsp, err := client.Do(req)
s := rsp.Body.(http3.HTTPStreamer).HTTPStream()
defer s.Close()
d, ok := s.Datagrammer()

data, _ := d.ReceiveMessage(context.Background())
d.SendMessage(data)

For the server side

server.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.(http.Flusher).Flush()

    s := r.Body.(http3.HTTPStreamer).HTTPStream()
    defer s.Close()
    d, ok := s.Datagrammer()

    data, _ := d.ReceiveMessage(context.Background())
    d.SendMessage(data)
})

You does not agree this proposal:

Thanks for the PR, but I don’t think this is right way to go. It has to be possible to send datagrams right away,
not only after receiving the HTTP response.

Please allow me to explain my design.

According to RFC9297:

If an HTTP/3 Datagram is received and its Quarter Stream ID field maps to a stream that has not yet been created,
the receiver SHALL either drop that datagram silently or buffer it temporarily (on the order of a round trip)
while awaiting the creation of the corresponding stream.

In theory, the datagram can be transmitted with known stream. However, in reality, it may not. Because if the server
is not ready or the client is not authorized, buffering these unknown may be harmful or wasteful. In scenario like
RFC 9484 (Proxying IP in HTTP), it make no sense to send datagram early because the client need the server's
response to finish setup its network.

The server's SETTINGS frame is received does not necessarily means that the client should send the datagam now.

But allowing datagram sending as early as possible is also of performance benefit. But this will complicate the API design
and implementation.

Just FYI.

[taoso]

@marten-seemann Anyway, wish this issue being fixed as soon as possible!

[tanghaowillow]

@marten-seemann Sorry I should have updated my status on #4141 earlier. The implementation was more complex than I thought and yes the API did make me more hesitate to go on. Must say that the newly proposed API is much cleaner since it can get rid of checking peer's setting in every call of SendDatagram and returnning a strange error like ErrDatagramNegotiationNotFinished to users.

[taoso]

Hello @marten-seemann , would you like to disclose any progress or schedule for you work on this issue?

Thanks~

I'll be working on this issue in the coming weeks.

[marten-seemann]

It's part of the milestone for the next release.

[marten-seemann]

HTTP Datagram support was just merged in #4452! Took us a while, but we're finally there :)
This will be released in v0.43 very soon.

Thanks for your patience everyone!