Content codings (Content-Encoding)

[annevk]

I looked into #710 (comment) a bit. Browser interoperability here is far from good. Interesting scenarios:

• gzip
• unknown
• gzip,unknown
• unknown,gzip
• deflate
• gzip,deflate
• gzip,gzip
• gzip listed a large number of times with the content encoded with gzip as many times (see https://bugzilla.mozilla.org/show_bug.cgi?id=1248577)
• having a response body of which only the first half is encoded (see https://bugs.chromium.org/p/chromium/issues/detail?id=654889)

Somewhat related:

• https://bugs.chromium.org/p/chromium/issues/detail?id=567163
• Content codings contract httpwg/http-core#58

It does not seem worth blocking #710 on this as it's an improvement over the status quo, but this could use more work.

cc @MoritzKn @TimothyGu @MattMenke2 @mnot

[annevk]

The other thing that's wrong about the current setup is that Brotli requires HTTPS so that should also be passed through somehow. Cannot just have a codings list and bytes.

Edit: Brotli should require a secure context so it works for localhost: https://bugzilla.mozilla.org/show_bug.cgi?id=1675054. So fun that none of this was standardized...

[annevk]

Changes to "handling content codings":

1. Iterate over codings and do decoding for each if them, if supported, and retain input as-is otherwise. If decoding fails, return failure.
2. Pass in the response or some such so you can determine the HTTPS bit. (Might as well move parsing of Content-Encoding there.)

Questions:

1. Should we define a limit? Maybe at least a lower bound.
2. Can we make unsupported values an error? Probably not...
3. ...

[annevk]

From @MattMenke2 at #710 (comment):

That there's also another, related issue here: What if we support an encoding, but didn't advertise it?

Chrome used to let such requests through, applying decoding, but that led to interop issues with brotli (servers were sniffing the user agent instead of looking at the Accept-Encoding header, and sending Brotli responses over HTTP), so we changed, and now reject unadvertised encodings, at least if we recognize them. That change led to issues with other servers that unconditionally send gzip - Chrome doesn't advertise gzip support when requesting media files, so it can use range requests.

[annevk]

Another thing that's wrong here is that the handlers need to keep around some internal state for compression tables, offsets, etc. So it's effectively a sequence of specialized transform streams.

[annevk]

Again from @MattMenke2 now at web-platform-tests/wpt#10548 (comment):

[W]e should ideally have tests with a compressed body and a Content-Length that matches the uncompressed body length (both when the compressed body is larger and smaller than the Content-Length). Chrome considers the latter case not an error in the connection-close case, I believe, but we should probably decide on a common set of behaviors there.

[yoavweiss]

^^ @pmeenan