Resolve flash address issues with SDK v3.0.0

[mhightower83]

Fix EEPROM vs RF_CAL flash address conflict. The EEPROM address and RF_CAL address were the same.

Add support for Flash size: "Mapping defined by Hardware and Sketch" aka autoconfig

Change at_partition_table from dynamic to static.

When the Arduino IDE flash size selected is correct, the flash sector positions for EEPROM, RF_CAL, and SYSTEM_PARAMETER are kept constant across SDK selections; however, when you change SDK versions, you should erase WiFi settings. The file system should be portable across SDK selections; however, if you have valued data backup before changing. Data may be lost if the flash size setting and device mismatch.

Status: Ready

[mcspr]

Also missing / may be wrong

• IDE menu Erase Flash > Sketch + WiFi Settings
• EspClass::eraseConfig()
• all non-auto-size .ld files

Embedding anything in .ld might be detrimental with simultaneous support of SDK2 and SDK3

As a hypothetical solution, what about restricting SDK3 to the auto-size flash mode only.
ref. #8595, everything until empty space between the app and fs is handled as pairs of {name, type, size}, allocated backwards from the end of the flash (to reduce boilerplate and avoid us remembering hard addrs)

EEPROM (optional, custom size), FS (optional, custom size) are user modifiable either through an override or something that appends things into the partition table object. Then it is system_partition_get_item or something custom for LittleFS, EEPROM, anything else storing data at fixed addresses

[mhightower83]

I think things are covered with "non-auto-size .ld files"; however, I plan to change RF_CAL and parameters to be the last 4 sectors of the flash. Get rid of the EEPROM relative approach. That will be more consistent with ESP.eraseConfig(). I'll keep the PHY_DATA read spoofing over the EEPROM. Of course, my thinking may change when I go to do it.

Technically, because we have to tell the SDK the position of PHY_DATA, we tell the SDK that PHY_DATA is the address of the EEPROM. To prevent conflict detection, we never tell the SDK the address of the EEPROM. During the short spoofing window, all reads for 128 bytes receive PHY_DATA. This is similar to how it was done for earlier SDKs; however, for those, PHY_DATA and RF_CAL shared the same sector location.

With auto-size flash mode the data in PROGMEM is used before flashinit() has run. For user_pre_init, it runs before any C runtime code runs and before flashinit() in user_init() and other "C++" initializations. I think we will need a version that can be called from user_pre_init and __flashindex will need to move to the .noinit section. Simple "C" runtime init for variables in sections .bss and .data has finished prior to calling of user_pre_init. I think structures are done with "ctors". I am confused about global structures.

And, panic() will need to be dealt with differently.

[mcspr]

Suggested change

	_at_partition_table_sz = sizeof(at_partition_table) / sizeof(at_partition_table[0]);
	_at_partition_table_sz = std::size(at_partition_table);

[mhightower83]

I assume this will happen at C++ compile time and we can count on not needing C++ runtime init.

[mcspr]

It is, number of elements is part of the type info.
template <typename T, size_t Size> constexpr size_t size(const T (&)[Size]) { return Size; }

[mcspr]

Suggested change

	#if FLASH_MAP_SUPPORT & defined(DEBUG_ESP_PORT)
	#if FLASH_MAP_SUPPORT && defined(DEBUG_ESP_PORT)

(would also suggest moving all #... to the beginning of line? things are pretty dense here)

[mhightower83]

I have gone back and forth on this. I kept getting lost with all the #ifs in column one. The density of the #ifs is the reason I started indenting the innermost #ifs to reduce the visual disruption to a block of "C" code. When nested the outer #if are at the beginning of the line. I don't know if doing this is a style violation.

[amrlsayed]

Hello, sorry for commenting on an old thread, but I thought your comment will help me out.
I have a project using the SDK 3.0.1 just before this merge, and I have the problem that the EEPROM is overwritten randomly that I am not able to reproduce.
My question is does this issue relate to that ? as the EEPROM address and RF_CAL address were the same, so may be RF_CAL data overwrites EEPROM data ?
Thanks

[mhightower83]

... EEPROM is overwritten randomly that I am not able to reproduce.

Yes, EEPROM writes may write over RF_CAL data. And SDK RF_CAL writes may write over EEPROM data. I suspect/expect the SDK implements wear leveling on their writes. This means the location of the writing might not be the same each time. This could contribute to the reproducibility problem.