HTTPClient POST larger files doesn't work anymore.

[comino]

Basic Infos
ESP8266HTTPClient: TCP socket hangs up at sending larger files with.
With stable version is was working just fine even with very large files so it was introduced lately.

Hardware
Hardware: ESP-WROOM2
Core Version: latest github

Description
Im using HTTPClient.sendRequest("POST", file).
With smaller files <1kB it works just fine. With larger files ~50kB it failes with HTTPC_ERROR_SEND_PAYLOAD_FAILED after a long timeout.

The error occurs in this part of the code:

L492-499 ESP8266HTTPClient.cpp

bytesWrite = _tcp->write((const uint8_t *) (buff + bytesWrite), leftBytes);
bytesWritten += bytesWrite;

if(bytesWrite != leftBytes) {
DEBUG_HTTPCLIENT("[HTTP-Client][sendRequest] short write, asked for %d but got %d failed.\n", leftBytes, bytesWrite);
free(buff);
return returnError(HTTPC_ERROR_SEND_PAYLOAD_FAILED);
}

Output
"short write, asked for 1460 but got 0 failed"

So there is no more data send via "_tcp->write()" after ~4kB

Any major change in the WiFiClient which may cause this?

[igrr]

Yes, lots of major changes in WiFiClient.
Can you please mention how you are sending the file? From file system or SD or some other stream?

[comino]

sure. Im just creating a minimal example. Will come back as I know more!

[comino]

I cant reproduce it currently and assume the problem is outside the arduino code.

[comino]

OK its getting more confusing ! I tried the same code in 4 different networks - in 3 of 4 it works just fine.
In 1 of the 4 networks the tcp connection "hangs up" like described above with increasing file size.

Switched the router and still the same.. hard to track this one down, but with stable version it works in 4 of 4.

Any hints how to debug welcome.. Im a bit lost here.

[comino]

The issue appears in this part of ClientContext.h

    size_t _write_from_source(DataSource* ds)
    {
        assert(_datasource == nullptr);
        _datasource = ds;
        _written = 0;
        _write_some();
        while (_datasource && !_noblock) {
            _send_waiting = true;
            esp_yield();
        }
        _send_waiting = false;
        return _written;
    }

And to be more exact - in the "write_some" function -

    void _write_some()
    {
        if (!_datasource || !_pcb) {
            return;
        }

        size_t left = _datasource->available();
        size_t can_send = tcp_sndbuf(_pcb); // <<<<<<<< THIS RETURNS 0 AFTER SOME PACKETS

        if (_pcb->snd_queuelen >= TCP_SND_QUEUELEN) {
            can_send = 0;
        }
        size_t will_send = (can_send < left) ? can_send : left;
        if (will_send) {
            const uint8_t* buf = _datasource->get_buffer(will_send);
            err_t err = tcp_write(_pcb, buf, will_send, TCP_WRITE_FLAG_COPY);
            _datasource->release_buffer(buf, will_send);
            if (err == ERR_OK) {
                _written += will_send;
                tcp_output(_pcb);
            }
        }

The tcp_sndbuf returns 0 ( size_t can_send = tcp_sndbuf(_pcb);) after some packets and following the connections times out.

This one is going deeper into the rabbit hole... So the tcp_write is appearently filling the buffer until its full... Any ideas ?

[me-no-dev]

I'm having similar reports on the Async side with some networks and over 3G. So far the problem is that the other end does not respond with ACK and lwip is holding the packets.

[comino]

ok glad Im not alone ;)

Link to the according AsyncTCP issue: me-no-dev/ESPAsyncTCP#20

[me-no-dev]

what are our options though? If I figure out how to clear the lwip buffer and let other data be sent, what is the assurance that that will work and what will happen in cases where the other end is actually gone and there is nobody to take the data.

[me-no-dev]

@comino can you take tcpdump with the packets so we can look at such transfer?

[me-no-dev]

@comino is the router that have problems a pppoe router? Can you check it's MTU?

[comino]

Im just trying to convince wireshark to show me any packets. Somehow I cant capture the packets from. I will oost them here as I get it to work.

To be honest I have very little information about the router - its: CH7466CE and Google doenst tell me what conenction method its using.

Edit:
MTU:1500 !

[comino]

It's been some time, but afaik the issue is still alive.
I tried to hunt the issue down in lwip, but it went far down the rabbit hole and I gave up.

The issue has been discussed here:
me-no-dev/ESPAsyncTCP#20

Currently I have this issue in only one network I have access to. Its using a Cisco Router with an MTU of 1450.

Citation of @me-no-dev :
"packet fragmentation is not the issue as neither side can really know what the MSS is through the connection in between. The limiting router can be somewhere externally for both and both to have full 1460 available locally. Issue is that lwip does not even have the code for handling "packet needed fragmentation" ICMP messages. The only code for ICMP is for ping and I am still not that familiar with all the goes inside. I do know that newer versions of LwIP have that taken care of and I know that we will update ESP8266... "

Might be solved with:
#3206

[igrr]

The LwIP version distributed with non-OS SDK 2.1.0 also includes these (possibly relevant) changes:

https://github.com/espressif/ESP8266_NONOS_SDK/blob/master/examples/lwip_open_src_template_proj/lwip/core/ipv4/icmp.c#L109-L122

Which are referred to as "Fix MTU negotiate bug" in the release notes.

[d-a-v]

There's no such thing (icmp 'Destination UnReachable' case for MTU update) in official lwip's master.
Hence #3206 will not fix this.

However, if this espressif patch applied to #3206 works, it would be interesting to submit it to lwip.

[devyte]

@igrr is this part of the 2.4.0 milestones?

[igrr]

Unfortunately i don't see a remedy for this other than updating LwIP, which we have agreed to hold off until next+1.

…

On Sat, Oct 14, 2017, 00:08 Develo ***@***.***> wrote: @igrr <https://github.com/igrr> is this part of the 2.4.0 milestones? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2437 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEJcevdvLdEvruKK8p-_dU4RR54WZGQPks5ssDPSgaJpZM4Js8gn> .

[devyte]

@comino does PR #3362 fix this for you?

[mcanyucel]

I know it has been some time over the last post, but I am trying to post a csv file (which rests on the SD card) to one of our servers and I am kind of stuck on a similar error.

The posting works fine if I generate the POST request string manually and write the file byte by byte into the client using a WiFiClient, as indicated in here by Interein2. But it does not look a streamlined approach to me and I am trying to implement the HttpClient library. In my tests, the post request by HTTPClient::POST(String payload) works perfect, but when I try to post a file using HTTPClient::sendRequest(const char * type, Stream * stream, size_t size), the board is stuck.

The code that I use for testing on a NodeMCU ESP8266 is as follows (I just removed the server address):

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <SD.h>
#include <ESP8266HTTPClient.h>
#define USE_SERIAL Serial
#define MTU_Size 2 * 1460;

const int chipSelect = 4;
bool sdReady;
char filename[] = "Buffer.CSV";

void setup()
{
USE_SERIAL.begin(115200);
USE_SERIAL.setDebugOutput(true);

USE_SERIAL.println();
USE_SERIAL.println();
USE_SERIAL.println();

// Trying to solve pn open, type:2 0 error with this line:
WiFi.mode(WIFI_OFF);

for (uint8_t t = 4; t > 0; t--)
{
    USE_SERIAL.printf("[SETUP] WAIT %d...\n", t);
    USE_SERIAL.flush();
    delay(200);
}

if (!SD.begin(chipSelect))
{
    Serial.print("FAILED TO INITIALIZE SD CARD");
    // don't do anything more
    sdReady = false;
}
else
    sdReady = true;

// Check if we have a dummy payload file
if (sdReady)
{
    File payloadFile = SD.open(filename, FILE_WRITE);
    if (!payloadFile)
    {
        // if the file does not exist create a dummy file
        payloadFile.print(String(millis()) + ",1,2,3,4");
        payloadFile.close();
    }
    else
    {
        payloadFile.close();
    }
}
}

void loop()
{
USE_SERIAL.println("Connecting to WIFI");
// wait for WiFi connection
WiFi.mode(WIFI_STA);
WiFi.begin("YAPIDESTEK", "somePasswordHere");
while (WiFi.status() != WL_CONNECTED)
{
    USE_SERIAL.print(".");
    delay(50);
}

USE_SERIAL.println("\nWIFI CONNECTED");
USE_SERIAL.print("[HTTP] begin...\n");
TestPostFileStream();
delay(10000);
}

/**
 * Fetches dummy data from the jsonplaceholder site using HTTPClient.GET 
 * This method works as intended
 */
void TestGet()
{
HTTPClient http;
http.begin("https://jsonplaceholder.typicode.com/posts/1"); //HTTP
USE_SERIAL.print("[HTTP] GET...\n");
// start connection and send HTTP header
int httpCode = http.GET();

// httpCode will be negative on error
if (httpCode > 0)
{
    // HTTP header has been send and Server response header has been handled
    USE_SERIAL.printf("[HTTP] GET... code: %d\n", httpCode);

    // file found at server
    if (httpCode == HTTP_CODE_OK)
    {
        String payload = http.getString();
        USE_SERIAL.println("SUCCESS! DATA RECEIVED: ");
        USE_SERIAL.println(payload);
    }
}
else
{
    USE_SERIAL.printf("[HTTP] GET... failed, error: %s\n", http.errorToString(httpCode).c_str());
}

http.end();
}

/**
 * Sends dummy data to either jsonplaceholder or our server in the form of string using 
HTTPClient.POST(String) 
 * This method works as intended
 */
void TestPostString()
{
HTTPClient http;
String jsonPlaceholderAddress = "https://jsonplaceholder.typicode.com/posts";
String archDomainAddress = "someDomainAddressComesHere";
http.begin(archDomainAddress);
http.addHeader("Content-Type", "application/x-www-form-urlencoded");
String postString = "date=" + String(millis()) + "&channel1=3&channel2=3&channel3=3&channel4=3&deviceID=espPostClient";
int httpCode = http.POST(postString);
if (httpCode > 0)
{
    // HTTP header has been send and Server response header has been handled
    USE_SERIAL.printf("[HTTP] POST... code: %d\n", httpCode);

    if (httpCode >= 200 && httpCode < 300)
    {
        String payload = http.getString();
        USE_SERIAL.println("SUCCESS! DATA RECEIVED: ");
        USE_SERIAL.println(payload);
    }
}
else
{
    USE_SERIAL.printf("[HTTP] POST... failed, error: %s\n", http.errorToString(httpCode).c_str());
}
http.end();
 }

// NOT WORKING FOR SOME UNKNOWN REASONS
void TestPostFileStream()
{
if (!sdReady)
{
    USE_SERIAL.println("SD Card not initialized properly, quitting...");
    return;
}

HTTPClient http;
String archDomainAddress = "someDomain/uploadFile.php";
http.begin(archDomainAddress);
http.addHeader("Content-Type", "multipart/form-data");

File payloadFile = SD.open(filename, FILE_READ);
if (payloadFile)
{
    USE_SERIAL.println("File exists, starting POST");
    // THE CODE STOPS ON NEXT LINE
    int httpCode = http.sendRequest("POST", &payloadFile, payloadFile.size());
    // THE REST IS NOT EXECUTED AT ALL
    http.writeToStream(&Serial);

    if (httpCode > 0)
    {
        // HTTP header has been send and Server response header has been handled
        USE_SERIAL.printf("[HTTP] POST... code: %d\n", httpCode);

        // file found at server
        if (httpCode >= 200 && httpCode < 300)
        {
            String payload = http.getString();
            USE_SERIAL.println("SUCCESS! DATA RECEIVED: ");
            USE_SERIAL.println(payload);
        }
    }
    else
    {
        USE_SERIAL.printf("[HTTP] POST... failed, error: %s\n", http.errorToString(httpCode).c_str());
    }
    http.end();
}
else
{
    USE_SERIAL.println("File does not exist!");
}
}

You may see the serial output of the code in here:

The board is stuck after pm open,type:2 0 and does not respond to even a reset click. I searched this output (pm open, type:2 0) but could not find any clue on what could it mean.

I am now planning to see if the other library methods int HTTPClient::POST(uint8_t * payload, size_t size) and HTTPClient::sendRequest(const char * type, uint8_t * payload, size_t size) will work or not, but is it possible to solve this issue with any other method, library or workaround?

[d-a-v]

@comino Can you retry with latest git ? It seems to be #1872 duplicate fixed by 199fe0f.

[devyte]

Closing due to age and lack of feedback. Also, lwip2 is now well tested and is the default.
If you think there is still a problem, please retest with latest release, then open a new issue and follow the template instructions.