Cowboy (actually Phoenix) and broken HTTP/1.0 client: Upload without 'transfer-encoding: chunked' and 'Content-Length' headers

[niko]

I'm writing a phoenix app and I'm forced to support an HTTP/1.0 client which streams data to the app. As it's HTTP/1.0 it doesn't support transfer-encoding: chunked and as it doesn't know the size of the upload beforehand it doesn't set a Content-Length: … header.

I don't control the client and it's ubiquitous.

I found out cowboy doesn't support reading the body of requests where both these headers are missing.

Is there any way to make cowboy support this? Perhaps via a custom transport similar to websockets? Ideally I just want to wrap the default http handler and add the transfer-encoding: chunked to the incoming request if certain conditions are met (certain verb and/or certain path).

(I'm sorry should this be the wrong channel for this question; where else can I search for help?)

[essen]

Which client?

And how is Cowboy supposed to know when the body was sent fully?

[niko]

It's an icecast source client. The protocol is described here: https://gist.github.com/ePirat/adc3b8ba00d85b7e3870.

How the server should know when the body is fully sent? Good question. The more I poke into the problem the more I realize how much I don't know about the HTTP wire protocol. In my actual case I think the server knows the client is finished when the client closes the tcp connection or doesn't send any data for some time (5 seconds).

[essen]

Yeah that's not valid HTTP. See https://tools.ietf.org/html/rfc7230#section-3.3.3

In particular:

   6.  If this is a request message and none of the above are true, then
       the message body length is zero (no message body is present).

Afraid there's no way to deal with this without patching Cowboy. I would be OK with having an option to allow a body when there shouldn't be any, of course disabled by default. It sounds like when using the PUT method it does use transfer-encoding, but when it uses SOURCE it doesn't, am I correct? If this SOURCE method is specified somewhere better than a gist, then we can probably do something special about it.

[niko]

PUT seems to add transfer-encoding: chunked, but I have no idea what version of libshout implements that. I've yet to see it in the wild. So I guess time will solve this problem. A lot of time.

I was able to hack a forced_body_length option into read_body/2 of the cowboy 1.1.x branch (default with elixir): niko@5240e30

It's doing what I want and I think it's better as a read_body/2 option than as a global cowboy option. It's probably missing any other quality (as it's the first Erlang code I've ever written). I'm happy to get hints on how to improve the implementation if it's even close to meet your idea of a solution.

I'm certainly able to add documentation. I could give a shot at adding a test (although all I could make the tests do so far is a rebar_abort error).

[essen]

Right this should work for Cowboy 1.0. No further releases are planned for this though.

This won't work for Cowboy 2.0 and I have little idea what would as user code no longer reads directly. I suppose the only thing that'd work is having a special return value from stream handlers init/3 that tells Cowboy to override what it thinks about the body. I don't see me adding something like this anytime soon though considering Cowboy 2.0 is quite new and there are bigger fishes to fry.

Let's keep this open for now.