HTTP2: Android gRPC h2c calls fail with PROTOCOL_ERROR

[JamesNK]

Describe the bug

Calling gRPC hosted on Kestrel from an Android device over h2c will fail. The client on Android reports a protocol error.

The original reporter of the issue tested Android calling C-core and it succeeded. Android calling Kestrel with TLS succeeded.

Combination of these three will fail:

• gRPC on Kestrel
• Android client
• h2c

Originally reported:

• .NET Core 3 fails accepting Java client calls grpc/grpc-dotnet#564 (comment)
• Repro: https://github.com/ivan-penchev/gRPC-Java-dotnetcore3/tree/master

To Reproduce

Steps to reproduce the behavior:

1. Run Kestrel with a gRPC service
2. Run Android emulator and call gRPC service
3. Call fails

Expected behavior

Call succeeds.

Screenshots

Additional context

.NET Core 3.0 GA

Wireshark log: android-protocol-error-nodocker.zip

[JamesNK]

@Tratcher @halter73 @anurse

[analogrelay]

We definitely haven't invested a lot in h2c scenarios (since mainstream browsers don't support it). With gRPC it's probably a good idea to dig a little deeper into it.

[JamesNK]

FYI, the original issue and sample uses Docker. You don't need to run Kestrel inside Docker to reproduce this.

Android calling Kestrel on a local machine will reproduce this.

[Tratcher]

The error doesn't look related to h2c, it should behave the same with and without TLS.

There should be server logs detailing what we thought the problem was.

First impression: The client is starting with stream id 3 instead of 1. That should be valid but it's unusual and may trip some of the validation logic.

[JamesNK]

I'll re-test with verbose logs.

[JamesNK]

dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
      Hosting starting
dbug: Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder[2]
      Discovering gRPC methods for GrpcService10.GreeterService.
dbug: Grpc.AspNetCore.Server.Model.Internal.ServiceRouteBuilder[1]
      Added gRPC method 'SayHello' to service 'Greet.Greeter'. Method type: 'Unary', route pattern: '/Greet.Greeter/SayHello'.
dbug: Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServer[0]
      Using development certificate: CN=localhost (Thumbprint: 2FE5E630929AB7660BC1C995E3F989C91B6CDE2F)
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5001
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://localhost:5000
dbug: Microsoft.AspNetCore.Hosting.Diagnostics[0]
      Loaded hosting startup assembly GrpcService10
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Development
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\Users\James\source\repos\GrpcService10\GrpcService10
dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
      Hosting started
dbug: Microsoft.AspNetCore.Server.Kestrel[39]
      Connection id "0HLQ948TUS99R" accepted.
dbug: Microsoft.AspNetCore.Server.Kestrel[1]
      Connection id "0HLQ948TUS99R" started.
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" sending SETTINGS frame for stream ID 0 with length 18 and flags NONE
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" sending WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" received SETTINGS frame for stream ID 0 with length 0 and flags NONE
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" received SETTINGS frame for stream ID 0 with length 0 and flags ACK
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" received HEADERS frame for stream ID 3 with length 138 and flags END_HEADERS
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" received DATA frame for stream ID 3 with length 20 and flags END_STREAM
dbug: Microsoft.AspNetCore.Server.Kestrel[35]
      Trace id "0HLQ948TUS99R:00000003": HTTP/2 stream error "PROTOCOL_ERROR". A Reset is being sent to the stream.
Microsoft.AspNetCore.Connections.ConnectionAbortedException: The request :scheme header 'https' does not match the transport scheme 'http'.
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" sending RST_STREAM frame for stream ID 3 with length 4 and flags 0x0
trce: Microsoft.AspNetCore.Server.Kestrel[37]
      Connection id "0HLQ948TUS99R" received GOAWAY frame for stream ID 0 with length 8 and flags 0x0
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[6]
      Connection id "0HLQ948TUS99R" received FIN.
dbug: Microsoft.AspNetCore.Server.Kestrel[36]
      Connection id "0HLQ948TUS99R" is closing.
dbug: Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets[7]
      Connection id "0HLQ948TUS99R" sending FIN because: "The client closed the connection."
dbug: Microsoft.AspNetCore.Server.Kestrel[36]
      Connection id "0HLQ948TUS99R" is closed. The last processed stream ID was 3.
dbug: Microsoft.AspNetCore.Server.Kestrel[2]
      Connection id "0HLQ948TUS99R" stopped.

[Tratcher]

The request :scheme header 'https' does not match the transport scheme 'http'.
Ah, that's a legit bug in the client, and does explain why it only breaks with h2c.

[JamesNK]

It seems rather suspicious. Before we point fingers, we should double check the HTTP/2 spec. Every other gRPC server works with the Android gRPC client so we might be doing the wrong thing here.

There is this section on :scheme: https://http2.github.io/http2-spec/#HttpRequest
It doesn't go into much detail.

[jkotalik]

For my knowledge, is it common to use h2c with android for this scenario? Also, I'm not sure if this is a 3.1 candidate if it's only h2c?

[JamesNK]

In development, yes. All gRPC guides and samples use h2c. That's common across all gRPC stacks apart from .NET.

[Tratcher]

The client is clearly using the wrong value, they're connecting over http, not https. The real question here is how strict the server should be about that.

While we work that out a bug should also be filed against the client.

[JamesNK]

That's here: grpc/grpc-java#6202 (comment)

[analogrelay]

Given that the scheme is supposed to represent the URL, I lean towards thinking that validating it is the correct behavior. The spec seems unclear on if the :scheme has to be validated. It certainly would be an HTTP/1.1 "protocol error" to try and speak https when the URL that the user specified was https://....

Can we see what other servers do in this situation? IIS and maybe one like NGINX? Should be possible to craft the HTTP/2 packets correctly such that we are using h2c but sending :scheme https, right?

[Tratcher]

In HTTP/1.1 the scheme is never presented in the request, it's assumed based on the transport. Crossing your transport results in pretty epic failures though.

IIS/Http.Sys doesn't support h2c. We could try the reverse (sending http to https).

[analogrelay]

It really feels like this is correct behavior on Kestrel's side. The scheme is incorrect, thus the protocol has been violated.

In HTTP/1.1 the scheme is never presented in the request, it's assumed based on the transport. Crossing your transport results in pretty epic failures though.

That's basically what I mean. I know the scheme isn't actually in the protocol, but if an HTTP/1.1 client made a plaintext request when the user requested an https:// URL, bad things would happen. In HTTP/2, the URL is now fully encoded into the pseudo-headers, so it doesn't seem unreasonable to validate it against the transport.

[analogrelay]

Our initial triage sense is that we don't plan to take action here. At best we'd be ignoring a malformed HTTP/2 request, which seems bad.

[analogrelay]

If there isn't traction on the client bug, we can revisit this decision. It is true that some other servers are not as rigid about the :scheme psuedo-header.

[ran-su]

The spec seems unclear on if the :scheme has to be validated.

Hello from gRPC. We think the scheme should not be validated. As the spec is unclear, validating it may be too risky. Assuming a client talk to a server through a proxy, the client talk to the proxy using https, and the proxy talk to the server using plaintext. In this case the scheme can be https, and it is not matching the transport.

[halter73]

In the case of the proxy, we tell app developers to configure middleware to set the appropriate scheme. If you trust the request is coming from a proxy server, you can pull the original scheme from X-Forwarded-Proto or something like that.

If we decide not validate the scheme, we would report the real scheme instead of the invalid one provided by the psuedo-header.

[halter73]

Is anyone aware of any proxies that use the :scheme instead of something else like X-Forwarded-Proto to communicate the scheme of the request?

[Tratcher]

@halter73 I haven't even seen signs of a proxy that supports HTTP/2 yet. Most still seem to tunnel it using the standard HTTPS tunneling.

[analogrelay]

I'll re-open to continue considering but it still feels wrong to allow an invalid request like this. The spec is pretty clear that "The ":scheme" pseudo-header field includes the scheme portion of the target URI ([RFC3986], Section 3.1)" and the scheme for an h2c connection should already be http... Sending https is incorrect.

As for proxies, it would also be invalid for a proxy to simply pass through the https value. Generally proxies do not attempt to masquerade as the client, the X-Forwarded- headers are there to make clear when a request is proxies and what the original values are.

[ran-su]

I agree with you that :scheme is part of the target URI, but I think the URI should be the client's URI.

To your response, I have two main arguments for not checking the scheme. First, the X-Forwarded- headers are a de facto standard, but not an official standard. Use something less standardized to replace a standard is risky. Second, a server can just get the type of connection without the need to check the :scheme. Validating it is not helpful.

We went through some old http work group threads, and found some messages clearly arguing for our side (1 2). I also have to said there are people against our option too.

If these are not enough to convince you, I think bring this to ietf for a clarification is a good idea.

[Tratcher]

I agree with you that :scheme is part of the target URI, but I think the URI should be the client's URI.

The client URI for h2c is "http", not "https".

The proxy discussion is hypothetical at this point, let's focus on the Java library.

[ran-su]

There are real people using gRPC through TCP proxies right now, why do you think it is hypothetical?
I think the focus should be interpretation of the HTTP/2 spec.

[halter73]

Interesting. When you first mentioned proxies, I was thinking about a layer 7 proxy. I hadn't considered how validation might play with a simple TLS termination proxy. Definitely something to consider.

How do these proxies communicate the result of the ALPN handshake? Are they configured to route the connection to a different endpoint based on the negotiated protocol?

[Tratcher]

Can anybody give names of specific HTTP/2 proxies they're using that would be worth doing interop testing with?

[analogrelay]

At this point, given that the gRPC Java client has been updated to fix this, we're going to close this. If there are use cases that come up for which this validation is a problem, we can reconsider. Right now, this is the only client we've encountered in the wild with this mismatch.

[Tratcher]

https://tools.ietf.org/html/rfc8164 outlines a use case where sending an http scheme over a TLS HTTP/2 connection has specific semantics and is disallowed unless the server has indicated support for it.