feature request: CRC for program memory

[s0170071]

I do occasionally struggle with failing flash chips and are implementing a CRC32. From 0x400000 to 0x410000-1 which is the SPI flash mapped to the memory map.
That part I got working, however, I need a matching, link-time generated CRC in memory.

Question:

1. is there a mechanism which does that already? (Would be awesome :-)
2. is there a linker option to place a (const) variable (my CRC) at a specific location in the memory, e.g. 0x410000-4 ?

[ondabeach]

@s0170071, have a look here (https://github.com/FrankBoesing/FastCRC), not sure if it works with 8266 but there are examples you can try.

Is your code regularly writing to the flash chip using SPIFFS or EEPROM? If so that may be your problem. Flash chips have a relatively low number of lifetime write cycles. The cheaper chips may only survive a few thousand writes before failing. The better quality ones can survive a few hundred thousand, but still. not a good idea to be writing to them repeatedly, they're really only meant to store your code and things like user settings.

If you want to write to memory repeatedly you should use an SD card instead.

[devyte]

@ondabeach previous comment is correct. I would only add that SPIFFS does wear leveling, so the flash lifetime is much longer vs writing directly to a flash sector like with eeprom.

[igrr]

@ondabeach @devyte as far as i understand, @s0170071 is asking about the memory-mapped part of flash, i.e the program memory (not SPIFFS or EEPROM).
For program memory, unfortunately there isn't a simple LD trick that i know of to generate CRC32 at compile time. Wish there was one, this question comes up pretty often!

What you could do is to calculate CRC32 of the binary using some external tool, and then embed this value into some known location in flash using e.g. dd (for example, at end of the first sector, 0xffc).

[ondabeach]

@devyte good to know :)

@igrr ah... gotcha :p

[s0170071]

That's exactly what I was thinking. The CRC goes to the bin file after linking. Using a py script or so. Although I have found md5 options for another linker, but they are not working here. Could you point out how I would place the CRC into that sector you were talking about?

[igrr]

dd if=crc.bin of=firmware.bin bs=1 count=4 seek=0xffc conv=notrunc
will write the contents of crc.bin (assuming it's a 4 byte long file) into firmware.bin at offset 0xffc.

[s0170071]

@igrr: Thank you. Your solution assumes that the location 0xffc has not previously been used by the linker. I know that there are linker commands to place e.g. jumptables at specific locations in the memory, I just don't know them for the ESP8266 linker.
But why 0xffc ? Isn't SPI flash mapped to 0x40200000 ?

[s0170071]

@ondabeach Thank you for the hint on fastCRC. For now I use https://github.com/bakercp/CRC32/tree/master/examples/CRC32. Does the job and since the CRC is only calculated once on startup I am fine for the moment... Got to admit I have not measured the time yet :-o

[igrr]

The first sector of flash (0x0 - 0x1000) is occupied by the bootloader, which is why i assume that there's nothing in the last word of this sector, at the moment (i.e. bootloader size is less than 4092 bytes).

ESP8266 linker is just your normal GNU LD from binutils, so normal binutils documentation applies.

[s0170071]

I was not aware the bootloader is flashed every time along with the binary.
According to https://linux.die.net/man/1/ld the linker should accept the "--build-id=md5" option but I was gettig an error when trying.

[igrr]

I've just tried that and got a warning:

warning: no memory region specified for loadable section `.note.gnu.build-id'

which makes sense because the linker script does not place this section anywhere in the output file. You can modify the linker script to place this section at some known location.

[s0170071]

Thanks. I'll try on and let you know when I got something usable.

[s0170071]

got something. Short version:

• you cannot read the bootloader section using pgm_read_byte(). Placing CRC in normal flash.
• define a CRC dummy in sketch and a check routine
  volatile char CRC[32] __attribute__ ((section (".irom0.text"))) = "MD5_MD5_MD5_MD5_STRT_END1234567" ;
• compile but do not upload
• use a python script to replace dummy string with real MD5
• now upload

Tested, works.
limitations: so far only checks irom0 section. (which is the largest by far)
very nice to have improvement: include python script functionality in esptool.

[s0170071]

long version:

use this arduino sketch:

#include <MD5.h>

// array that is placed in irom section and is getting replaced by true MD5 using a python script after linking
// it holds the MD5 hash, the memory start address (uint32_t) and the end address 
volatile char CRC[32] __attribute__ ((section (".irom0.text"))) = "MD5_MD5_MD5_MD5_STRT_END1234567" ;

bool checkMD5(){
    uint8_t buf [16];     // buffer to hold the calculated MD5
    uint8_t temp=0;       // 
    boolean crcOK = true; 
    uint32_t* FlashStart = (uint32_t *)&CRC[16]; // pointer to flash section, init with what python writes in the CRC array.
    uint32_t* FlashEnd   = (uint32_t *)&CRC[16+4];
    MD5Builder md5;
 
    md5.begin();
    for (uint32_t i = *FlashStart; i< *FlashEnd; i++){
      temp = pgm_read_byte(i);  // read a single byte
      uint8_t *tt = &temp;      // pointer to that byte
      if (i == (uint32_t)&CRC) 
        i=i+16+8-1; // hit the CRC location. skip it, skip two ints(=8) and account for next loop increment(-1)
      else
        md5.add( tt,1 );        // add byte to md5. 
   }
   md5.calculate();
   md5.getBytes(buf);
   for (uint32_t i = 0; i<16; i++) 
    if (buf[i] !=  pgm_read_byte((uint32_t)&CRC+i)) 
      crcOK=false; 
   return crcOK;
}

void setup() {

  Serial.begin(115200);
  Serial.print("\n\n\nboot..."); 
 }

void loop() {
   
    if (checkMD5()) 
      Serial.println("CRC check OK");  
    else
      Serial.println("CRC check FAIL");  
    
   for (;;) delay(1);
}

It checks the checksum that has been uploaded along with the program code against a calculated checksum (MD5). When you compile, the linker places a dummy string in the memory. There is no adjustment of linker scripts necessary. The dummy string will be replaced by the real checksum and the region of memory which has been taken into consideration for this calculation.
I wrote a python script to do this. The script takes the filename of the binary as command line parameter.

import sys
import binascii
import struct
import md5

PLACEHOLDER = "MD5_MD5_MD5_MD5_STRT_END"
lastSegmentMemoryStart=0
lastSegmentMemoryEnd=0
lastSegmentFileStart=0
lastSegmentFileEnd=0


def showSegments (fileContent,offset):
    global lastSegmentMemoryStart, lastSegmentMemoryEnd, lastSegmentFileStart, lastSegmentFileEnd  
    header = struct.unpack("ii", fileContent[offset:offset+8])
    if  fileContent.find( PLACEHOLDER, offset+8, offset+8+header[1]) >0 :
        herestr= " <-- CRC is here."
        lastSegmentMemoryStart=header[0]
        lastSegmentMemoryEnd=header[0]+header[1]
        lastSegmentFileStart=offset+8
        lastSegmentFileEnd=offset+8+header[1]
    else:
        herestr =""
    print ("SEGMENT: memory position: " + hex(header[0])+ " length: " + hex(header[1])+herestr)
    return (8+offset+ header[1]); # return start of next segment

def showParts(fileContent, offset):
    header = struct.unpack("BBBBi", fileContent[offset:offset+8])
    print ("\n\nBINARY PART")
    print ('Segments: ') + (hex(header[1]))
    nextpos =offset+8
    for x in range (0,header[1]):
        nextpos = showSegments(fileContent,nextpos)

    nextSegmentOffset = (fileContent.find("\xe9", nextpos))
    return nextSegmentOffset

if len(sys.argv) !=2 :
    print ("please give a filename")
    k=input("press close to exit")
    sys.exit(1)
    
FileName = sys.argv[1]#"C:/ArduinoPortable/sketchbooks/build/sketch_jan15a.ino.bin"
with open(FileName, mode='rb') as file: # b is important -> binary
    nextpos =0;
    fileContent = file.read()
    while nextpos >=0:
        nextpos = showParts(fileContent,nextpos)

if (lastSegmentMemoryEnd==0 ):
    print("MD5 placeholder not found in binary")
else:
    print("\nwriting output file:\n" + FileName)
    md5string =(fileContent[lastSegmentFileStart:lastSegmentFileEnd])
    md5string= md5string.replace (PLACEHOLDER,"",1) 
    m = md5.new()
    m.update (md5string)
    md5hash = m.digest()
    print("MD5 hash of irom0_0_seg: "+ m.hexdigest())
    fileWriteContent = fileContent.replace("MD5_MD5_MD5_MD5_",md5hash)
    fileWriteContent = fileWriteContent.replace("STRT",struct.pack("I",lastSegmentMemoryStart))
    fileWriteContent = fileWriteContent.replace("_END",struct.pack("I",lastSegmentMemoryEnd))
    with open(FileName, mode='wb') as file: # b is important -> binary
        file.write(fileWriteContent)
k=input("press close to exit") 

If you use the script with the binary of the sketch provided above, it outputs:

BINARY PART
Segments: 0x1
SEGMENT: memory position: 0x4010f000 length: 0x568


BINARY PART
Segments: 0x4
SEGMENT: memory position: 0x40201010 length: 0x2f454 <-- CRC is here.
SEGMENT: memory position: 0x40100000 length: 0x7000
SEGMENT: memory position: 0x3ffe8000 length: 0x380
SEGMENT: memory position: 0x3ffe8380 length: 0x5a0

writing output file:
C:\ArduinoPortable\sketchbooks\build\sketch_jan15a.ino.bin
MD5 hash of irom0_0_seg: b5d505ac7874a340b44975160b2335a4
press close to exit

You can see a list of the memory segments in the binary. If the crc dummy is found in any of them , this will be indicated.
The first binary part is the bootloader. Once it is written to the ESP, this section of memory is not readable.
The second binary part holds the core and the user program as well as some text sections. I think the first section of the second binary part is the most important to check. And it is the largest.

If all goes well, the Arduino sketch outputs a simple

boot...CRC check OK

What needs some doing is

• integrating the python script into the tool workflow. Anyone knows how to call additional scripts right after linking and before uploading ?
• maybe check the other segments of the second binary part as well ? Its just a matter of doing, the basics are all there...
• (back to the feature request) automatically replacing the crc summy with the checksum using the esptool. Since dummy has a unique signature this should't be hard.

[igrr]

Anyone knows how to call additional scripts right after linking and before uploading?

https://github.com/arduino/arduino/wiki/arduino-ide-1.5-3rd-party-hardware-specification#pre-and-post-build-hooks-since-ide-165

automatically replacing the crc summy with the checksum using the esptool

Why do this with esptool though? It already mixes two entirely unrelated features: generation of binaries and uploading. I don't see a reason to use esptool for this when this can be done using e.g. dd. Especially I wouldn't like adding ad-hoc and fragile functionality like replacing some strings with some other strings.

[s0170071]

@igrr,

I don't see a reason to use esptool for this when this can be done using e.g. dd. Especially I wouldn't like adding ad-hoc and fragile functionality like replacing some strings with some other strings.

I understand what you mean. I thought of the esptool because its already part of the toolchain. This would allow anybody to check the memory just by adding it to the sketch. Defining hooks and and adding python scripts to the toolchain might put off many people. And you do it over and over again if you re-install.

dd alone does not work as you still need to calculate the md5- and just for the part of the memory that you can read at run-time. An md5 for the whole binary would not work. You have to sort of disassemble the .bin to find the segments. Furthermore you need to know where in the memory the segments go. They are not necessarily adjacent. Once you know all that you could use dd - or - stick with python as you probably do all the above with it anyway and dd is Linux only.

Btw, why is it that I cannot read the bootloader section at run time ? Is it a hardware constraint or does pgm_read_byte() filter that ?

[s0170071]

I got it up and running myself. Anyone who wants to secure their flash contents, please have a look here:
https://github.com/s0170071/CRC4ESP