(Question) How to view contents of request

[niko-dunixi]

First off, thanks for your time. I'm sure you're busy.

I'm trying to debug some restful applications speaking to each other, and I've been trying to setup your mitm service to do so.

Reading your documentation I can get to the point where I dump the headers of a given request, but I can't figure out how to view the body of the request.

Here's what I have so far:

    @FunctionalInterface
    private interface MyFilter extends HttpFiltersSource {
        default int getMaximumRequestBufferSizeInBytes() {
            return 0;
        }
        default int getMaximumResponseBufferSizeInBytes() {
            return 0;
        }
    }

and a simple main method to run the application:

    private static final AttributeKey<String> CONNECTED_URL = AttributeKey.valueOf("connected_url");

    public static void main(String[] args) throws RootCertificateException {
        HttpProxyServer server =
                DefaultHttpProxyServer.bootstrap()
                        .withPort(3455)
                        .withManInTheMiddle(new CertificateSniffingMitmManager())
                        .withFiltersSource((MyFilter) (originalRequest, context) -> {

                            System.out.println("-----------------------------------------------------");
                            HttpMethod method = originalRequest.getMethod();
                            String originalUri = originalRequest.getUri();
                            System.out.println(method + " -> " + originalUri);
                            originalRequest.headers().forEach(entry -> {
                                System.out.println("-- " + entry.getKey() + ":" + entry.getValue());
                            });

                            if (HttpMethod.CONNECT.equals(method) && Objects.nonNull(context) && originalUri.endsWith(":443")) {
                                String url = "https://" + originalUri.replaceFirst(":443$", "");
                                context.channel().attr(CONNECTED_URL).set(url);
                                System.out.println("(Manipulating connection request for successful HTTPS: " + originalUri + " -> " + url + ")");
                            }
                            return new HttpFiltersAdapter(originalRequest, context);
                        })
                        .start();

        System.out.println("Running mitm server");
    }

I've been digging into the ChannelHandlerContext via debugging and some of the javadoc, and my initial assumption about the body being located here appears to be false. So really my question, I suppose, is how do I access the request body given that it doesn't appear to be present in either the request or context object?

[niko-dunixi]

I have found a (seemingly) working solution using a little more experimentation and reading. I'm sure I won't be the only one that has this question, so I'll include what I came up with here. I'm not sure what the implications of HttpFilters#clientToProxyRequest returning null are (I can't find any documentation at the moment to explain what the expected behavior is) but it isn't breaking (I'm going by the assumption that because the static NOOP_FILTER field doesn't override this method that it's standard and okay behavior).

.withFiltersSource(new HttpFiltersSourceAdapter() {
    public HttpFilters filterRequest(HttpRequest originalRequest, ChannelHandlerContext context) {
        System.out.println("-----------------------------------------------------");
        HttpMethod method = originalRequest.getMethod();
        String originalUri = originalRequest.getUri();
        System.out.println("CLIENT " + method + " -> " + originalUri);
        originalRequest.headers().forEach(entry -> {
            System.out.println("-- " + entry.getKey() + ":" + entry.getValue());
        });

        if (HttpMethod.CONNECT.equals(method) && Objects.nonNull(context) && originalUri.endsWith(":443")) {
            String url = "https://" + originalUri.replaceFirst(":443$", "");
            context.channel().attr(CONNECTED_URL).set(url);
            System.out.println("(Manipulating connection request for successful HTTPS: " + originalUri + " -> " + url + ")");
            return new HttpFiltersAdapter(originalRequest, context);
        }

        return new HttpFiltersAdapter(originalRequest) {

            @Override
            public HttpResponse clientToProxyRequest(HttpObject httpObject) {
                if (httpObject instanceof LastHttpContent) {
                    LastHttpContent copy = ((LastHttpContent) httpObject).copy();
                    String body = copy.content().toString(StandardCharsets.UTF_8);
                    System.out.println("-- Content:" + body);
                }
                return null;
            }

            @Override
            public HttpObject serverToProxyResponse(HttpObject httpObject) {
                if (httpObject instanceof HttpResponse) {
                    HttpResponse response = (HttpResponse) httpObject;
                    System.out.println("SERVER RESPONSE");
                    response.headers().forEach(entry -> {
                        System.out.println("-- " + entry.getKey() + ":" + entry.getValue());
                    });
                }
                return httpObject;
            }
        };
    }
})

(Just as a side note, the reason I am having to check for LastHttpContent might be because I set the buffer sizes to 0 in my default interface, this will require further digging when I have more time: stackoverflow citation)

[ganskef]

In clientToProxyRequest you shouln't find an instance of LastHttpContent. A last content of a chunked response should be always empty in HTTP/1.1, but FullHttpResponse extends LastHttpContent given to you by io.netty.handler.codec.http.HttpObjectAggregator.

It is important to set org.littleshoot.proxy.HttpFiltersSource.getMaximumResponseBufferSizeInBytes() to a size like 10 MB depending your devices memory and network transfer rate. This causes LittleProxy to add an inflater and aggregator the pipeline which avoids compressed and chunked responses. But if you detect a huge download you have to remove both from the pipeline.

Javadoc: Indicate how many (if any) bytes to buffer for incoming HttpResponses. A value of 0 or less indicates that no buffering should happen and that messages will be passed to the HttpFilters response filtering methods chunk by chunk. A positive value will cause LittleProxy to try an create a FullHttpResponse using the data received from the server, with its content already decompressed (in case the server was compressing it). If the response size exceeds the maximum buffer size, the response will fail.

I hope this helps. See the description of filters source and filters in #32 too.