External Loader (QSPI flash acces with the CubeProgrammer_API.dll)

[mihaisvcc]

@maniglia

Do you have experience, or you have tested the functions for the external loader ?
For example the functions setExternalLoaderPath, getExternalLoaderPath

I have an HMI project based on STM32F4 and I have attached also an external QSPI flash and for the moment I'm writing my own custom bootloader. It should produce at the end a custom *.stldr file
I attached a ST training link for reference.

Kind Regards,

[maniglia]

Hello @mihaisvcc,
I haven't had a chance to try it yet, I think I need to review the ExternalLoader data structure.
Thank you for bringing attention to these features.
Yours seems like a very interesting project, how can I help you?
I'm sorry but I don't see the attached link.
Regards

[mihaisvcc]

Hello @maniglia ,

Thank you for the help offer,
Yes, it is an interesting project , a HMI with external QSPI flash. If you are not aware the PROM (internal) STM32 bootloader is not capable to write/read the 90 milion address (QSPI flash). So I need a custom bootloader to accomplish this. Besides I have internal flash sectors with parameters and even a portion with a second firmware element. The link was about how to create a custom *.stldr loader for the external QSPI. In the CubeProgrammer there are already loaders defined for some development boards.
Now I already made a custom bootloader capable of accessing both internal and external flash over USB.
I need to understand an to create a *.stldr capable of accesing internal and external memory , and i general how an *.stldr interacts with the CubeProgrammer_API.dll. Do you have any information in this direction ?. I want to share my experience if I will find a solution maybe it will help other people.

https://www.st.com/content/st_com/en/support/learning/stm32-education/stm32-moocs/external_QSPI_loader.html

My DfuSE custom boadr, with custom boot loader

Regards,

[maniglia]

Hello @mihaisvcc ,
this is all very interesting and completely new to me.
It will take me some time to study the topic.
As soon as I have gathered the information I need, I will get to work.

Regards

[maniglia]

Hello @mihaisvcc ,
there is a new version of the library, 0.2.4.
I revisited and improved the setExternalLoaderPath and getExternalLoaderPath functions, essential to achieve the goal.
Now the GetExternalLoaders function returns the list of all .stldr files it finds in the specified directory, by default the function will search for the .stldr already present in the nuget package in the following path: .nuget\packages\ksociety.sharpcubeprogrammer\0.2.4\contentFiles\any\any\st\Programmer\ExternalLoader
From the list of files it is possible to search for the file we are interested in and use the SetExternalLoaderPath function to set it.
This is an example of use:

var deviceExternalStorageInfo = CubeProgrammerApi.GetExternalLoaders();

if (deviceExternalStorageInfo.HasValue)
{
    var externalStorage = deviceExternalStorageInfo.Value.ExternalLoader.FirstOrDefault();

    var externalLoader = CubeProgrammerApi.SetExternalLoaderPath(externalStorage.filePath);

    if (externalLoader.HasValue)
    {
        CubeProgrammerApi.RemoveExternalLoader(externalLoader.Value.filePath);
    }
}

The FileOpen function has also been changed, it now returns all the segments present in the file.
Personally, I still haven't understood what these segments are for and what they represent.

Obviously the hardest part is creating the .stldr file.

Regards

[mihaisvcc]

Hello @maniglia

Thank you for the update, I want to share my knowledge also.
Regarding to the FileOpen function segmentData_C structure and what are representing the segments :
Talking in general if you have one simple firmware project for STM32Fx after compile and link you will have one flash image segment starting in flash from the address 8 milion and having a certain size.
In this case the FileOpen will return only one segmetsNbr and only one segmentData_C with start address/ size and data byte. Now there are many cases in that we can pack inside the flash more than one firmware segment/program.
The flash memory will be divided according to the compiler/linker scatter file.
I'll give you my exemple here: I have an HMI project which has a firmware zone starting with the 8000000 and another segment starting with 8104000 containing the language texts for the HMI. In this case segmetsNbr is 2 and I have two segmetData_C structures. We can have for example two firmware segments/images one for current use and one for backup etc...
If you look also to the UM0391 DFU file format, in case of a DFU file this can contain more than one DFU image and the DFU image can contain more then one Image Element see 2.3.3 Chapter Image Element. This will be in the case if you convert the hex file to a dfu but this is another story...the CubeProgrammer_API does not have this functionality.

Now for the external loader *.stldr . I have found that the CubeProgrammer_API is transferring the *.stldr program with the help of STLink inside the sram of the microcontroller and it got executed from there. ST has documentation on how to create an external loader and how to map external flash. In my case I need to do do this without the help of the STLink and that is why I have developed a custom bootloader that initialize and map external and internal flash.
With the old DfuSE utility this was working fine. I want to do the same and to fully understand how the CubeProgrammer_API is working.
I have opened a support ticket and I'm in contact with ST to get more information. I'll keep you updated.

Kind Regards,

[maniglia]

Thanks for your great explanation.
Some observations:

1. STM32CubeProgrammer software does not handle DFU files
2. .bin .hex and .srec are the only file types that can be used

At this point the only solution that comes to mind is to somehow create a .hex file that contains all the firmware, both for the internal and external flesh, what do you think?
Why don't you want to use the external loader?

Regards

[mihaisvcc]

Hello @maniglia

Thanks for the appreciation,
Sorry for the delay response I was ill last week,
Answering the observation 1: yes DFU file is not supported it was made in the time of DfuSE utility in order to unify and to have one extension for firmware downloading. If needed an DFU parser can be added.
To the observation 2: I have tested the .axf and woks also. In my opinion is a good parser. I already have a hex parser. Anyway it should decode the format intel hex for ARM.
The final goal is to retrive the data byte array(s), and the start address(s) in the flash. The write function is working with this.
Of core it exist also file download function. It is pretty straightforward when no selective upgrade is needed.

At this point the only solution that comes to mind is to somehow create a .hex file that contains all the firmware, both for the internal and external flesh, what do you think?

I will comment on this: It should be made with another tool...in my opinion this extra step is not necessary and I will go in detail.
I have a project with two chip solution I have an STM32 and an spartan FPGA hanging on one bus together. The STM32 should load the external QSPI memory and the spartan FPGA should boot after that from the QSPI. So I have a .hex file from Vivado formatted for the the QSPI flash. This was the motivation for a custom bootloader that is mapping and accessing over USB the 90 milion external QSPI flash address. In the end I will have two files one for the microcontroler internal flash and one for the FPGA. ST has even template files on how to map the external memory in the DFU mode middlewares. But like in the case of the new API all is poor documented. In my opinion I have no advantage to combine them, maybe I want to update only microcontroller or maybe only FPGA at one time.

Another project is an HMI. This time in the external flash are stored the extra grafic elements.

Why don't you want to use the external loader?

Because in the case of external loader, an STLink is needed. In my case the requirement is to be able to perform firmware upgrade with the only help of a micro USB cable.In the case of the old DfuSe was possible to do this.
I can confirm that I can use the WriteMemory(0x90000000, toWrite); and ReadMemory(0x90000000, 100); for the address 90milion in the external Flash with my custom bootloader. I still have problems with the GetStorageStructure() and GetDeviceGeneralInf(). it seems that the CubeProgrammer_API.dll is using the xml files from the Data_Base folder. And for me is not clear how are working the stldr from Programmer/FlashLoader together with the build in bootloader in case of internal flash ??

Kind Regards,

[maniglia]

Hello @mihaisvcc

I still have problems with the GetStorageStructure() and GetDeviceGeneralInf(). it seems that the CubeProgrammer_API.dll is using the xml files from the Data_Base folder.

I'm pretty sure it's exactly as you say!
I could do a very simple experiment and modify an xml file to confirm this hypothesis.

And for me is not clear how are working the stldr from Programmer/FlashLoader together with the build in bootloader in case of internal flash ??

I have no experience with this, but it might be a good exercise to try creating a stldr file, even if in your case you can't take advantage of external loader. Just to understand what exactly is inside these stldr files.

Regards

[mihaisvcc]

Hello @maniglia

I could do a very simple experiment and modify an xml file to confirm this hypothesis.

Please do it, If you have time, i will try also here. In the meanwhile I 'm in contact with ST and and try to see if they can help me. Unfortunately until now they didn't want to share the source code of the CubeProgrammer nor the source code for the API dll. The old DfuSE had the source code, it was easier to work with. According to the AN3156 USB DFU protocol used in the STM32 bootloader, the CubeProgrammer_API.dll is designed to perform according to the USB1.1a for the STM32 microcontrollers. The USB 1.1a is the ST DFU protocol derived and with some modifications on the communication concept from the more generic USB1.1

I want to mention here as a side note, an interesting project that is allowing to perform firmware upgrade from the USB port without using the CubeProgrammer_API.dll (using LibUSBNotNet to implement the USB DFU protocol)

Getting back to the stldr files what I know for sure, that in the case of external memory they are firmware for ST that is started in SRAM and are initializing the USB and the QSPI in order to perform file transfer in the extern OSPI.
I'm not sure what role they play in the case of STM32 internal bootloader.

Regards,

[mihaisvcc]

https://github.com/IntergatedCircuits/LibUsbDfu

[mihaisvcc]

https://www.st.com/content/st_com/en/support/learning/stm32-education/stm32-moocs/external_QSPI_loader.html

[maniglia]

Unfortunately I'm very busy at the moment. As soon as I'm free I'll study the content of the link you sent me.
Thank you very much.
Regards

[maniglia]

Hello @mihaisvcc
I have a quick question, have you ever used the LibUsbDfu library?
Do you think it can be exploited to achieve our goal?
Regards

[mihaisvcc]

Hello @maniglia,

I have tested the LibUSBDfu and it is working, I was able to do the flashing in case I have only one firmware element and don't want do selective sector erasing.It can be used as an alternative to the CubeProgrammer_API.dll. The source code is available and can be checked. The drawback is the laking of selective sector delete, retrieve the memory model, verify function, option bytes etc. It should be further developed.

In the last week I was busy with other things I was not able to further research on the CubeProgrammer_API.dll. I'm still in contact and pound ST to give me a solution for my case. Until now I got frustrated, because ST was not willing to share any information about their CubeProgrammer_API.dll. I'm still waiting for a solution on how to use the functions GetStorageStructure() and GetDeviceGeneralInf() with my customised bootloader.

It should work somehow because I have build my custom bootloader according to the ST official documentation UM1734

https://www.st.com/resource/en/user_manual/um1734-stm32cube-usb-device-library-stmicroelectronics.pdf

If you look at chapter 6.3.1 and specially on page 43 you will see that is possible to add external memory and there is a template in this direction. I just want to share with you how i've build my customized bootloader.

Regards,

[maniglia]

Hello @mihaisvcc
I have another question:
what happens when you try to convert your .DFU file to .HEX using DFU file manager (DfuSeDemo) software?
If this DFU file manager software works properly, then I can integrate it easily.

Regards

[mihaisvcc]

Hello @maniglia,

I don't need the DfuSEDemo to convert the dfu file in hex. I have my own dfu parser...this part is not so difficult...
The DFU file specification is in the user manual UM0391 https://rc.fdr.hu/UM0391.pdf.
In the end we need the start address(es), the byte array and length in flash in order to feed the Write function.
Those can be obtained either from a dfu parser or a hex parser. A dfu to hex parser can be created but is not necessary in my opinion.
because Keil microvision or IAR usually create a hex file. I have made also an hex2dfu parser.
The CubeProgrammer_API.dll has only a hex, elf parser. It seems they dropped the part with conversion into dfu
and work only with hex , elf files.
If you have interest and you want to integrate in your wrapper a dfu parser, I can gladly share my C# code...in order to save your development time.

Regards,

[maniglia]

Exactly, that's exactly what I was thinking of doing:

1. obtain the addresses and bytes from the .dfu or .hex file after parsing and then write them to the flesh with the WriteMemory function.

Searching online I found this:
Programming an external Flash memory using the UART bootloader

do you think it can be adapted fore dfu instead of uart?
Regards

[mihaisvcc]

My custom C# flashing program is capable now to identify automatically the driver STDFU.dll or CubeProgrammer_API.dll
Then is parsing the firmware file and in the case of hex I have already written before a hex parser, or can be used the parser from the CubeProgrammer_API.dll depending on the driver installed. The same apply, if I check and find a dfu file then I use my dfu parser.
The end result from pasing will be the start address(es) and the byte array(s). This will be feed into the Write function (the old STDFU.dll has a similar write function) . Of course the flash memory should be deleted totally or partially first.
This is already working for the internal flash, and also partially working with my custom bootloader.

Thanks for you UART suggestion...I know the link... I have similar implementation in a older project. UART offers simplicity, but the major drawback is the slow speed.
I had a smaller SPI external memory and in that time it took many minutes to fill it with the image of an small and old FPGA.
In my current situation I have a big QSPI and it will take eons to transfer a big bit file from computer into memory over UART.
Same applies even for the internal flash it is very slow.
This is the main reason why I want to go with the USB.

Kind Regards,

[maniglia]

Small update

I finally found the time to try out the functions GetStorageStructure and GetDeviceGeneralInf by editing .xml files,
and I can confirm that both functions return the information present in this files.

[mihaisvcc]

Hello @maniglia,

This is a big update, 👍 thank you, i see it as a step forward, I didn't had the time to test it, I will try to customise the xml files from Data_Base to adapt them to my custom bootloader flash configuration. Do you know how is the xml file selected , based on the read controller ID, e.g 0x451 for the F7 ? If I can make the two above mentioned functions work, with my custom bootloader then my problem is solved.

[maniglia]

Hello @mihaisvcc
Yes, based on device ID (in your case the 0x451) the 0x451.stldr file and STM32_Prog_DB_0x451.xml file are loadded.
You can edit the xml file and check the effects directly with the CubeProgrammer.