update SDK to 2.1.0+

[igrr]

#3215

Todo:

• Update libraries
• fix millis/micros redefined in libmain.a (Linking Error on new SDK 2 with time.h #2998, [SDK2.1.0] Colliding time functions #3281)
• Run existing tests
• Test new WiFi reconnect behaviour using WiFiManager library

8MB/16MB support

• Add linker scripts and board options for 8/16 MB flash
• esptool-ck: support generation of 8/16 MB binaries
• esptool-ck: support uploading to 8/16 MB flash chips
• Update esptool release in the package template
• Expose SPIEraseBlock or SPIEraseArea to speed up flash erase
• Optimize SPIFFS parameters (page size, block size) for 8MB and 16 MB?
• Test 8/16 MB flash support

[igrr]

8MB/16MB support appears to be working, needs esptool-ck from this branch: feature/8m_16m.

Formatting SPIFFS on a 16MB flash currently takes ~330 seconds. Need to speed up flash erase by exposing the 64kB block erase function.

SPIFFS uploading over serial doesn't work yet. Needs some additional changes to esptool.

[comino]

I have a comment on "Optimise SPIFFS parameters":

SPIFFS has an issue with write/delete cycles of "big" files creating lots of fragmentation.
A 'bad' implementation can make this problem worse. The garbage collection is simply too slow.

There are open issues for that in the SPIFFS repository:
pellepl/spiffs#142
pellepl/spiffs#139

I wrote a unit test to reproduce the issue I can share. A solution is quite simple:

• SPIFFS_GC_MAX_RUNS is at the moment at 5. Increasing the parameter lead to a much lower fragmentation. Expecting 0.5MB to be the maximum fileSize I increased the SPIFFS_GC_MAX_RUNS to 25. I planned to check the effect on wear level - but could find time for it yet. A slightly increased wear load is expected.

https://github.com/pellepl/spiffs/wiki/Configure-spiffs#spiffs_gc_max_runs for more information.

• I exposed the Garbage collection in the core SPIFFS implementation to manually call a GC.

[lucysrausch]

Hey,
awesome work!
I got I2C, SPI and WS2813 working, can't notice any problems yet. However, when I try to print something on a lcd, the Adafruit GFX library crashes at:

if(i < 5) line = pgm_read_byte(font+(c*5)+i);

This worked perfectly fine with the 1.x.x SDK. Any idea why pgm_read_byte causes:

Fatal exception 3(LoadStoreErrorCause):
epc1=0x40203500, epc2=0x00000000, epc3=0x00000000, excvaddr=0x4023823a, depc=0x00000000

Thanks!

EDIT: This fixed it for me:
#3140 (comment)

[torntrousers]

ESP.deepSleep still only takes uint32_t not uint64_t, but system_deep_sleep does appear to work now with the extended uint64_t sleep times. Terrific, thanks so much for this.
Also the Version and changelog.txt files haven't been updated in hardware\esp8266com\esp8266\tools\sdk

[pieman64]

@torntrousers and @igrr might be a "bad" calculator here but does the uint64_t mean ESP's can now sleep for 585m years?

[igrr]

@pieman64 The hardware RTC timer is still 32-bit, so the limit is now 2^32 / RTC_TIMER_FREQ seconds instead of 2^32 / 1000000 seconds.

@NiklasFauth: that's very interesting, we have a fix for pgm_read_byte merged already... So you have replaced the assembly version of pgm_read_byte with the version mentioned in the issue?

[torntrousers]

Thats interesting about the RTC timer. What is the value of RTC_TIMER_FREQ - is that what system_rtc_clock_cali_proc() returns? Trying system_rtc_clock_cali_proc() it returns 24716, so 2^32 / 24716 = 173772 secs or 48 hours, is that right?

[pieman64]

@torntrousers previously it was around 72 minutes so if your calculation is right then 48 hours is considerably longer, but not the 585m years my "bad" calculator came up with.

[torntrousers]

Just as FYI I've got nowhere trying the > 2^32 deepsleep intervals, as far as I can tell they just don't work and wakeups are often seemingly random short intervals. No answers about what should be possible or what the actual max sleep time should be from the espressif bbs.

[pieman64]

@torntrousers did you apply the Core patch at https://www.esp8266.com/viewtopic.php?p=67729#p67729 before carrying out your tests?

@igrr you stated

2^32 / RTC_TIMER_FREQ seconds

but what does this calculate out to, approximately, as there is some confusion over the calculation?

See also #3302 for reference to the patch by @5chufti to include the missing uint64_t time_us and uint32_t cali_proc() from the Core;

[pieman64]

@igrr I am looking to use the extended deepSleep durations and what I did was look through all the changes in this pull request and manually edited the relevant files that I believe referred to deepSleep. But what I didn't do was select any of the new binaries. Which of binary files are deepSleep related?

[5chufti]

@pieman64 I would suggest to swap everything under tools/sdk (to get sdk 2.1.0)

[pieman64]

@5chufti @igrr one last question (hopefully).
I see the following README.md in the sdk_2.1.0 repository. Does any of this apply to me if I am just trying to paste in the new tools/sdk files?

## Updating SDK libraries

After updating SDK libraries to a new version, do the following changes.


1. Remove mem_manager.o from libmain.a to use custom heap implementation

    ```bash
    xtensa-lx106-elf-ar -d libmain.a mem_manager.o
    ```

## Updating libstdc++

After building gcc using crosstool-NG, get compiled libstdc++ and remove some objects:

    xtensa-lx106-elf-ar d libstdc++.a pure.o
    xtensa-lx106-elf-ar d libstdc++.a vterminate.o
    xtensa-lx106-elf-ar d libstdc++.a guard.o
    xtensa-lx106-elf-ar d libstdc++.a functexcept.o
    xtensa-lx106-elf-ar d libstdc++.a del_op.o
    xtensa-lx106-elf-ar d libstdc++.a del_opv.o
    xtensa-lx106-elf-ar d libstdc++.a new_op.o
    xtensa-lx106-elf-ar d libstdc++.a new_opv.o

[5chufti]

take the tools/sdk part from the 2.1.0 branchs zip and you are ok
https://github.com/esp8266/Arduino/archive/update_sdk_2.1.0.zip

[pieman64]

@5chufti I did that earlier today and I am still getting the old maximum of 72m with anything above this giving modulus 72.

[5chufti]

hmmm, what does sdk/include/user_interface.h say about "system_deep_sleep"?
I just checked with ESP.deepSleep(4440000000); what should give me ds of ~142s according to you but still sleeps after 5mins; but I'm on a full checkout of 2.1.0 branch.
I think this is getting OT in this issue and we should move to issue 3302 or the esp8266.com thread.

[bryceschober]

Has any progress been made on this SDK update work?

[TridentTD]

Wish Arduino support ESP-Mesh last version (version 1.3) too.

Thank you

[devyte]

@igrr For the WifiManager testing, I did some reference testing with the current latest git (basically just tried out all the examples that don't have mqtt), and I think I understand what to look for. I am now trying to build with this update_sdk_2.1.0 branch.

I'm a bit confused. The file tools/sdk/version says 2.0.0_16_08_09. In the same dir, changelog.txt doesn't seem to contain the 2.1.0 chanelog. I started to doubt whether I did the whole git thing correctly (I'm rather new to github), but looking at the changed files here on github, I don't see changes for those files. Also, the files in my local git seem to match with what github says (i.e.: the file headers have the new MIT license thing).

Are the contents for the above version and changelog.txt files expected? Or did I mess something up? Or is this PR incomplete still?

This is what I did, in case I messed something up:
git status -> on branch master ... clean
git fetch origin pull/3278/head:update_sdk_2.1.0
git checkout update_sdk_2.1.0
git status -> on branch update_sdk_2.1.0 ... clean

[5chufti]

Hi,
I noticed this differences too but looking at the lib files, they are definitely from the 2.1.0 sdk.
Pbly Igrr just forgot to replace the version and changelog files.

[igrr]

Indeed, i forgot to update the version and changelog files. Will do.

[igrr]

Updated to the latest non-OS SDK. There's another small but very important update landing this week, so holding off merging this until then.

Will work on the rest of 8M/16M support in a separate PR.

[Juppit]

For distribution the binaries may be compressed.
For example, I did this as follows:

$ find bin -maxdepth 2 -type f -perm /0111 -exec upx -1 "{}" +

with a compress ratio of more than 40 %. For example:

2305587 -> 1141811 49.52% xtensa-lx106-elf-gcc.exe

[igrr]

@Juppit upx is known to trigger some virus scanners, do you think it is really worth it? We don't update the gcc toolchain when updating the SDK, so it doesn't look like an issue related to SDK updates.

[Juppit]

I realize that it has become old fashioned to save resources. :-)
But you're right, not for that price.

[devyte]

@igrr I just got the Wemos D1 pro boards with 16MB, so I can now help with testing that. Is there anything specific I should look at?

[igrr]

@devyte I think basic support for 8/16 is there: one can upload binaries above 4M mark using esptool-ck, and linker scripts for 8/16MB are included. Performance of SPIFFS on 8M and 16M partitions is pretty bad though, in part due to flash erase times. For example, formatting a 16M partition takes enough time for me to grab a coffee in the next room... So there's some optimization work that needs to be done to make such large SPIFFS partitions more usable.