Add FSTools with examples of how to convert between SPIFFS and LITTLEFS. #7696
Conversation
…t address. Took a while to find that. There maybe a better way to store all these configs
fix ESP.h to Esp.h
|
To fix the style errors, please run the For the other build issues, we use |
|
Ah i wondered about the style fail, and there was no comment as to why. |
|
unfortunately if i pull the latest git to work with nothing works in my arduino IDE. I'm now fixing errors that didn't show up using error all in the IDE but relate to FS.h so not my addition... but all passed this time! |
Yeah, it looks like none of our example code (used in CI) was exercising that bit of code in FS.h you had to change. Good catch, and thx for fixing it. Unfortunately, I don't understand the other comment. Is this worth testing, or is is still a WIP ("nothing works")? |
|
It is worth testing. My other comment is just referring to my ability to test source code from github either via PIO or the arduino IDE as for various reasons they all error. not released to this specific pull request. |
|
Awesome, I will try and resolve all these points. |
Run test/restyle.sh Remove commented code Use #ifdef blocks for debugging. `DEBUG_ESP_CORE` and `DEBUG_ESP_PORT` use `static constexpr layout` which saves ROM ~500B for unused vars
|
See how these get on. Its taken more time to fix 'git' issues with merging than it did to fix the issues. I feel like my brain is bleeding. |
There was a problem hiding this comment.
Please think about the yield() comment, but it looks good to me for a first commit. I think docs would be a good follow-on to help understand how to use it in users' own code, but I think for people who care about doing this, they should be able to figure it our from the examples.
Add yield in between file copy
There was a problem hiding this comment.
If I understand this correctly, this works as follows:
- create a secondary FS in the empty flash area
- copy files over from the main FS to the secondary FS
- reformat/mount primary FS
- copy files over from secondary FS to newly created primary FS
In the above there is one very dangerous thing that I don't see considered: reset/power loss. If during steps 3/4 there is a reset or power loss, you lose the primary FS and have no state to remember, so you end up with a dead FS and no realistic way to recover.
Before an attempt is made to address that here with some custom fix, I suggest considering a different approach: create a FS image in the empty area and then figure out a way to trigger the FS update on next boot with the existing update mechanism like Updater does. That would streamline the update process with what we already know works, including existing support for power loss as well as future upcoming changes.
|
@earlephilhower pointed out to me a limitation in implementing the approach I outlined above: the secondary fs size is limited to the size of the empty area, and therefore the new FS is as well, which may not be desirable if the original fs was big but had very low usage. |
|
@devyte Thank you for your feedback, this is a very good point that you raise. I will need to think about this some more. I think the penny just dropped for the ATOMIC_FS flag though. Does this use the OTA area similar to what I am doing and then signal to the bootloader to copy it on reboot to the actual FS area. If this is the case how does it cope with FS that are bigger than the ota area... or does it not and it just fails. or.... can it decompress an image... if so are there compression tools available... ie possible to compress the files when stored in the OTA area... The boot flag bits are set in the RTC no, so an overt power fail would not preserve the state... I seem to remember the first 76 bytes of RTC memory are used for this purpose? |
|
Having dug a little deeper I should be able to use |
Unfortunately, I doubt that there is enough free RAM to actually do gzip compression. Just to decompress requires over 36K free RAM (not a problem in the bootloader, but not likely in a running app). You don't need to set atomic_fs_update if you want the bootloader to copy over the new FS from one spot to another, just set the proper copy command addresses and it will work on reboot using the same method that the ATOMIC_FS_UPDATE flag uses. But I'd say that is the wrong thing to do and won't work. If you rely on eboot to copy, in any way, that means your replacement FS total size would be limited in size to whatever it was generated as (i.e. <400KB). You can't just extend LittleFS or SPIFFS FSes like XFS or EXT4. Plus, the core would blindly use the full 1MB/2MB/3MB FS the code was compiled to expect, but your binary FS image would not be that size leading to very bad things. Instead a sequence like:
If your app reboots/loses power before 3) then no FS changes were done and you can start from scratch or leave the old FS as-is. |
|
thank you @earlephilhower that does make a lot of sense and tbh i've not currently got the time to dive deep into trying to get that to work! The EEPROM idea sounds solid. I will implement something. There is likely to be a bit of a delay, I'm currently studying for some exams which take priority atm. |
|
This is not a breaking change, removing 3.0.0 milestone. |
|
@sticilface do you think you can run the new |
|
I'm very sorry, I just saw this. I'm afraid life got in the way, new job, moved house and have more professional exams so I've not touched programming since November and I'm unlikely to till next year. Thank you for finishing and merging, much appreciated. |
This lib provides a simple way, with some caveats, to 'in the field' conversion of a SPIFFS partition to LITTLEFS of different sizes.
The main caveat is the total size of all the files is limited to the gap between the end of the sketch and the start of the partition you are trying to copy.
I've created layout objects for all the possible layouts, these are in the namespace
FST, however, they are global and are ALL linked with is a bit annoying. If any C++ gurus know how to create globals that can be dropped if they are not used, or has any other bright ideas, then let me know!The example sketch basically tries to mount the new FS and fails, then the lib kicks in does the conversion and leaves the new FS mounted.
There is support for custom partitions as well if you are trying to roll an intermediate sketch and need to define your own rules.