Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Software crashes with an User exception #9029

Open
6 tasks done
hmueller01 opened this issue Nov 12, 2023 · 15 comments
Open
6 tasks done

Software crashes with an User exception #9029

hmueller01 opened this issue Nov 12, 2023 · 15 comments

Comments

@hmueller01
Copy link
Contributor

Basic Infos

  • This issue complies with the issue POLICY doc.
  • I have read the documentation at readthedocs and the issue is not addressed there.
  • I have tested that the issue is present in current master branch (aka latest git).
  • I have searched the issue tracker for a similar issue.
  • If there is a stack dump, I have decoded it.
  • I have filled out all fields below.

Platform

  • Hardware: [ESP-12]
  • Core Version: [framework-arduinoespressif8266 @ 3.2.0-dev+sha.fb8d6d6]
  • Development Env: [Platformio]
  • Operating System: [MacOS]

Settings in IDE

  • Module: [Generic ESP8266 Module]
  • Flash Mode: [qio]
  • Flash Size: [4MB]
  • lwip Variant: [v1.4|v2 Lower Memory|Higher Bandwidth]
  • Reset Method: [ck|nodemcu]
  • Flash Frequency: [40Mhz]
  • CPU Frequency: [80Mhz]
  • Upload Using: [SERIAL]
  • Upload Speed: [115200] (serial upload only)

Problem Description

Software crashes with an User exception after connecting to WiFi and MQTT TLS

Reset reason   : External System
connected with @ XXX, channel 1
dhcp client start...
ip:192.168.X.X,mask:255.255.255.0,gw:192.168.X.X
Station connected: 192.168.X.X
otaSetup: Setting up ArduinoOTA
timeSetup: Setting time using SNTP
timeSetup: Current time: Sun Nov 12 09:47:38 2023
mqttSetup: MQTT client id: ESP-XXX
mqttSetup: Attempting connection to server XXX.XXX:8883

No poison after block at: 0x3fff2680, actual data: 0x65 0x0 0xa5 0xa5

User exception (panic/abort/assert)
--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Panic umm_malloc.cpp:751 umm_malloc_core

>>>stack>>>

ctx: sys
sp: 3fffecf0 end: 3fffffb0 offset: 0010
...

But it is not always at the same point in software. Sometimes it also starts the HTTP server and transmits several packages before the crash, like this:

Reset reason   : Software/System restart
connected with @ XXX, channel 1
dhcp client start...
ip:192.168.X.X,mask:255.255.255.0,gw:192.168.X.X
Station connected: 192.168.X.X
otaSetup: Setting up ArduinoOTA
timeSetup: Setting time using SNTP
timeSetup: Current time: Sun Nov 12 09:47:38 2023
mqttSetup: MQTT client id: ESP-XXX
mqttSetup: Attempting connection to server XXX.XXX:8883
mqttSetup: Sucsessfully connected.
webserverSetup: Setting up webserver http:https://192.168.X.X
mqttConnect: connected using PubSubClient
cisternSwitch: switch cistern OFF
mqttConnect: Current time: Sun Nov 12 09:47:52 2023
mqttCallback: Received cistern_timeout_time 60
URI: /status, Method: GET, #Args: 0
getFreeHeap: 13712, getHeapFragmentation: 3, getMaxFreeBlockSize: 13304
webserverHandleStatus: json size/capacity: 741/1024
...
URI: /status, Method: GET, #Args: 0
getFreeHeap: 11792, getHeapFragmentation: 11, getMaxFreeBlockSize: 10576
webserverHandleStatus: json size/capacity: 741/1024
No poison after block at: 0x3fff3008, actual data: 0x65 0x0 0xa5 0xa5

User exception (panic/abort/assert)
--------------- CUT HERE FOR EXCEPTION DECODER ---------------

Panic umm_malloc.cpp:751 umm_malloc_core

>>>stack>>>

ctx: sys
sp: 3fffe9a0 end: 3fffffb0 offset: 0010
...

I have no idea where to search for the problem. Can someone give me a hint (looking at the dump)? There seems to be enough heap available and fragmentation changes between 3 and 19 in the example above but does not constantly grow.

Another this that's strange to me is that ESP.getResetReason() outputs Software/System restart after that. For me this looks like an Exception and not like a "normal" Software/System restart ...

MCVE Sketch

Will follow, if needed.

Debug Messages

ctx: sys
0x40101160: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:753
0x40100ed8: put_poison at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:41
0x40101064: check_poison_neighbors at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:59
0x401014a2: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:912
0x4022955e: esp8266::MDNSImplementation::MDNSResponder::stcMDNSServiceTxt::allocKey(unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp:116
0x4010150c: umm_poison_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:173
0x401014a8: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:914
0x4022ada7: esp8266::MDNSImplementation::MDNSResponder::_readRRAnswerTXT(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRAnswerTXT&, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:678
0x40100940: malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/heap.cpp:250
0x40224a67: operator new[](unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/abi.cpp:52
0x4022955e: esp8266::MDNSImplementation::MDNSResponder::stcMDNSServiceTxt::allocKey(unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp:116
0x4022958f: esp8266::MDNSImplementation::MDNSResponder::stcMDNSServiceTxt::setKey(char const*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp:131
0x4022adbd: esp8266::MDNSImplementation::MDNSResponder::_readRRAnswerTXT(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRAnswerTXT&, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:679
0x40229e0c: esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRAnswer::stcMDNS_RRAnswer(esp8266::MDNSImplementation::MDNSResponder::_enuAnswerType, esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRHeader const&, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Structs.cpp:914
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x4022b1e9: esp8266::MDNSImplementation::MDNSResponder::_readRRAnswer(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_RRAnswer*&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:519
0x4022007c: Stream::Stream() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/Stream.h:63
(inlined by) HardwareSerial::HardwareSerial(int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/HardwareSerial.cpp:41
0x40228bc4: esp8266::MDNSImplementation::MDNSResponder::_parseResponse(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_MsgHeader const&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:770
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x4022b414: esp8266::MDNSImplementation::MDNSResponder::_readMDNSMsgHeader(esp8266::MDNSImplementation::MDNSResponder::stcMDNS_MsgHeader&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1274 (discriminator 2)
0x40278084: print_number at /home/wchen/gwen/nonos/app/main/eagle_lib.c:99
0x3fff0218: _Pri_3_Stack at nmi-vector.o:?
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x3fff03f8: _Pri_3_Stack at nmi-vector.o:?
0x40228c99: esp8266::MDNSImplementation::MDNSResponder::_parseMessage() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:135
0x40101014: check_poison_block at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:100
0x40101064: check_poison_neighbors at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:59
0x40100040: _KernelExceptionVector at ??:?
0x40228db0: esp8266::MDNSImplementation::MDNSResponder::_process(bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:80
0x40228de8: esp8266::MDNSImplementation::MDNSResponder::_callProcess() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Helpers.cpp:161
0x4022f801: std::_Function_handler<void (), std::_Bind<bool (esp8266::MDNSImplementation::MDNSResponder::*(esp8266::MDNSImplementation::MDNSResponder*))()> >::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:293
0x401000e5: std::function<void ()>::operator()() const at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:623
0x3ffeff70: _Pri_3_Stack at nmi-vector.o:?
0x40219694: UdpContext::_s_recv(void*, udp_pcb*, pbuf*, ip4_addr const*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/include/UdpContext.h:590
0x3ffe8938: ?? at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:123
0x3ffeff70: _Pri_3_Stack at nmi-vector.o:?
0x3fff0208: _Pri_3_Stack at nmi-vector.o:?
0x40244289: ip4_input at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/lwip2-src/src/core/ipv4/ip4.c:1461
0x3ffeff70: _Pri_3_Stack at nmi-vector.o:?
0x40101470: umm_poison_free_fl at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:161
0x4027458c: ppRecycleRxPkt at /home/wchen/gwen/nonos/app/pp/pp.c:768
0x3ffeff70: _Pri_3_Stack at nmi-vector.o:?
0x4023b031: ethernet_input_LWIP2 at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/lwip2-src/src/netif/ethernet.c:188
0x4023ae4c: git2glue_err at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-lwip/lwip-git.c:118
(inlined by) esp2glue_ethernet_input at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-lwip/lwip-git.c:494
0x402779ea: ethernet_input at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c:365
0x402779fc: ethernet_input at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c:374
0x40278737: lwip_if0_task at /home/wchen/gwen/nonos/app/main/eagle_lwip_if.c:56
0x40106c05: call_user_start_local at ??:?
0x40106c0b: call_user_start_local at ??:?
0x40101430: umm_init at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:569
0x4010000d: call_user_start at ??:?
0x401000ab: app_entry_redefinable at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:391
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x40102df1: rcUpdateTxDone at /home/wchen/gwen/nonos/app/pp/trc.c:806
0x40105f2d: ets_timer_handler_dsr at /home/wchen/gwen/nonos/app/main/ets_timer.c:335
0x40105c83: lmacTxDone at /home/wchen/gwen/nonos/app/pp/lmac.c:1730
0x4025866e: br_aes_ct_bitslice_Sbox at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct.c:191
0x4024e643: shift_rows at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:45 (discriminator 3)
(inlined by) br_aes_ct_bitslice_encrypt at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:105 (discriminator 3)
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x4024e814: shift_rows at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:45
(inlined by) br_aes_ct_bitslice_encrypt at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:110
0x4024cda8: br_aes_ct_ctr_run at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_ctr.c:81
0x4024e814: shift_rows at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:45
(inlined by) br_aes_ct_bitslice_encrypt at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_enc.c:110
0x4024cda8: br_aes_ct_ctr_run at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/ssl/bearssl/src/symcipher/aes_ct_ctr.c:81
0x4022b36a: esp8266::MDNSImplementation::MDNSResponder::_udpAppend8(unsigned char) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1178
0x4022b4b9: esp8266::MDNSImplementation::MDNSResponder::_write8(unsigned char, esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1311
0x4022b77d: esp8266::MDNSImplementation::MDNSResponder::_writeMDNSServiceDomain(esp8266::MDNSImplementation::MDNSResponder::stcMDNSService const&, bool, bool, esp8266::MDNSImplementation::MDNSResponder::stcMDNSSendParameter&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1488 (discriminator 3)
0x40101014: check_poison_block at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:100
0x3ffe8938: ?? at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:123
0x40101064: check_poison_neighbors at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:59
0x4010114f: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:751
0x40100ed8: put_poison at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:41
0x40100f0d: get_poisoned at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:121
0x401014a8: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:914
0x40101014: check_poison_block at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:100
0x3ffe8938: ?? at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:123
0x40101064: check_poison_neighbors at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_local.c:59
0x4010114f: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_malloc.cpp:751
0x401009ac: sdk3_pvPortMalloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/heap.cpp:360
0x4022b36a: esp8266::MDNSImplementation::MDNSResponder::_udpAppend8(unsigned char) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1178
0x4010150c: umm_poison_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/umm_malloc/umm_poison.c:173
0x4022b36a: esp8266::MDNSImplementation::MDNSResponder::_udpAppend8(unsigned char) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Transfer.cpp:1178
0x40275317: ppProcessWaitingQueue at /home/wchen/gwen/nonos/app/pp/pp.c:3057
0x40275366: ppMapTxQueue at /home/wchen/gwen/nonos/app/pp/pp.c:3078
0x40275472: ppMapTxQueue at /home/wchen/gwen/nonos/app/pp/pp.c:3146
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x4025cd07: ieee80211_output_pbuf at /home/wchen/gwen/nonos/app/net80211/ieee80211_output.c:313
0x401029c0: pp_post at /home/wchen/gwen/nonos/app/pp/pp.c:3252
0x40105c9b: lmacRxDone at /home/wchen/gwen/nonos/app/pp/lmac.c:1740
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x4010354b: wDev_IndicateFrame at /home/wchen/gwen/nonos/app/pp/wdev.c:324
0x401029c0: pp_post at /home/wchen/gwen/nonos/app/pp/pp.c:3252
0x40103728: wDev_ProcessRxSucData at /home/wchen/gwen/nonos/app/pp/wdev.c:603
0x3ffeff58: _Pri_3_Stack at /home/earle/src/esp-quick-toolchain/arduino/tools/sdk/lwip2/builder/glue-esp/lwip-esp.c:213
0x40103bfa: wDev_ProcessFiq at /home/wchen/gwen/nonos/app/pp/wdev.c:1187
0x40103bfa: wDev_ProcessFiq at /home/wchen/gwen/nonos/app/pp/wdev.c:1187
0x4010390c: wDev_ProcessFiq at /home/wchen/gwen/nonos/app/pp/wdev.c:1072
0x40105f2d: ets_timer_handler_dsr at /home/wchen/gwen/nonos/app/main/ets_timer.c:335
0x401007b8: millis at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_wiring.cpp:165
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x40100795: millis at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_wiring.cpp:158
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x40105f2d: ets_timer_handler_dsr at /home/wchen/gwen/nonos/app/main/ets_timer.c:335
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x40212d63: ESP_WiFiManager_Lite::run() at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/.pio/libdeps/upstream_develop_flash_serial_debug/ESP_WiFiManager_Lite/src/ESP_WiFiManager_Lite.h:863
0x40212fb9: ESP_WiFiManager_Lite::isConfigMode() at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/.pio/libdeps/upstream_develop_flash_serial_debug/ESP_WiFiManager_Lite/src/ESP_WiFiManager_Lite.h:1196
(inlined by) loop at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:953
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:96
0x40215fd1: WiFiClient::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClient.cpp:254
0x40215fd1: WiFiClient::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClient.cpp:254
0x402181c5: BearSSL::WiFiClientSecureCtx::_run_until(unsigned int, bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:598
0x3ffef600: _Pri_3_Stack at nmi-vector.o:?
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffef664: _Pri_3_Stack at nmi-vector.o:?
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffef410: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:99
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x40218760: BearSSL::WiFiClientSecureCtx::_pollRecvBuffer() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:426 (discriminator 2)
0x3ffef410: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:99
0x40218868: BearSSL::WiFiClientSecureCtx::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:443
0x4022dd8a: BearSSL::WiFiClientSecure::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.h:277
0x3ffef410: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:99
0x40213f14: PubSubClient::loop() at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:387
0x40230420: base64_encode_value at ??:?
0x40228dc0: esp8266::MDNSImplementation::MDNSResponder::_process(bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:88 (discriminator 1)
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:96
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:96
0x40212fd1: loop at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:957
0x40208dd0: std::_Function_base::_Function_base() at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:240
(inlined by) std::function<void ()>::function(std::function<void ()>&&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:357
(inlined by) Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}::function({lambda()#1}&&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/Ticker/src/Ticker.h:81
(inlined by) std::_Function_base::_Base_manager<Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}>::_M_init_functor(std::_Any_data&, {lambda()#1}&&, std::integral_constant<bool, false>) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:237
(inlined by) std::_Function_base::_Base_manager<Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}>::_M_init_functor(std::_Any_data&, {lambda()#1}&&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:208
(inlined by) std::function<void ()>::function<Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}, void, void>(Ticker::attach_ms_scheduled(unsigned int, std::function<void ()>)::{lambda()#1}) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:609
0x40105f2d: ets_timer_handler_dsr at /home/wchen/gwen/nonos/app/main/ets_timer.c:335
0x3ffef650: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:95
0x402097c5: esp8266webserver::ESP8266WebServerTemplate<WiFiServer>::begin() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WebServer/src/ESP8266WebServer-impl.h:82
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffefc78: _Pri_3_Stack at nmi-vector.o:?
0x3ffef380: _Pri_3_Stack at nmi-vector.o:?
0x40206818: debouncePinTimer_Cb() at /Users/hmueller/src/PlatformIO/Projects/garage-platformio/src/garage-arduino.cpp:388
0x4022e294: std::_Function_handler<void (), void (*)()>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:271
0x4022e280: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:289
0x4022e28a: std::_Function_handler<void (), void (*)()>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:293
0x401000e5: std::function<void ()>::operator()() const at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:623
0x40221911: run_scheduled_functions() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/Schedule.cpp:173
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:243
0x40100679: esp_schedule at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:136
0x40224e6a: loop_wrapper() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:269

@hmueller01
Copy link
Contributor Author

After building another project with similar code and crashes I found out that this was the problem:

  configTime(TZ_Europe_Berlin, PSTR("de.pool.ntp.org"), PSTR("europe.pool.ntp.org"), PSTR("time.nist.gov"));

Changing this to

  configTime(TZ_Europe_Berlin, "de.pool.ntp.org", "europe.pool.ntp.org", "time.nist.gov");

did the trick.

But it didn't fully help on this project. There I still have a crash after a zalloc/ malloc right here:

void mqttCallback(char *topic, byte *payload, unsigned int length) {
  bool topic_handled = false;
  char topic_pub[100];

  DEBUGF_P("%s" LF, __func__);
  DEBUGF_P("getFreeHeap: %d, getHeapFragmentation: %d, getMaxFreeBlockSize: %d" LF, ESP.getFreeHeap(), ESP.getHeapFragmentation(), ESP.getMaxFreeBlockSize());
  DEBUGF_P("length: %d" LF, length);

  // copy payload to a char string
  //char *data = (char *)zalloc(length + 1);
  char *data = (char *)malloc(length + 1);
  DEBUG_P("after malloc, before memcpy");
  memcpy(data, payload, length);

It doesn't reach the DEBUG_P("after malloc, before memcpy"); any more.
But how is this possible? length is only 2 bytes and the getMaxFreeBlockSize is 12080!

timeSetup: Setting time using SNTP
timeSetup: Current time: Sun Jan 21 20:08:31 2024
mqttSetup: MQTT client id: ESP-0015AADA
mqttSetup: HomA system id: 123456-garage
mqttSetup: Attempting connection to server mqtt:8883
mqttSetup: Sucsessfully connected.
Stop multiResetDetecting
mqttConnect: connected using PubSubClient
cisternSwitch: switch cistern OFF
mqttConnect: Current time: Sun Jan 21 20:08:32 2024
mqttCallback
getFreeHeap: 13712, getHeapFragmentation: 12, getMaxFreeBlockSize: 12080
length: 2
No poison after block at: 0x3fff2838, actual data: 0x65 0x0 0xa5 0xa5
User exception (panic/abort/assert)
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
Panic umm_malloc.cpp:751 umm_malloc_core
sp: 3ffffcc0 end: 3fffffc0 offset: 0010
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr

ctx: cont
0x40101118: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_malloc.cpp:753
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
0x4010145a: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_malloc.cpp:912
0x401014c4: umm_poison_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_poison.c:173
0x40263737: umm_max_block_size at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_info.c:203
0x401008f8: malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/heap.cpp:250
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
0x40208da5: mqttCallback(char*, unsigned char*, unsigned int) at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:455
0x4021244b: BearSSL::WiFiClientSecureCtx::_run_until(unsigned int, bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:497
0x40211d28: BearSSL::WiFiClientSecureCtx::_engineConnected() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:261 (discriminator 5)
0x40212c38: BearSSL::WiFiClientSecureCtx::read(unsigned char*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:383 (discriminator 4)
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x402295e8: BearSSL::WiFiClientSecureCtx::read() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:414
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x40212bbc: BearSSL::WiFiClientSecureCtx::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:443
0x402289d2: BearSSL::WiFiClientSecure::read() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.h:279
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x4020d9c5: PubSubClient::readByte(unsigned char*) at /Users/hmueller/src/PlatformIO/Projects/garage-pio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:297
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x40228ea4: std::_Function_handler<void (char*, unsigned char*, unsigned int), void (*)(char*, unsigned char*, unsigned int)>::_M_invoke(std::_Any_data const&, char*&&, unsigned char*&&, unsigned int&&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:293
0x40100160: std::function<void (char*, unsigned char*, unsigned int)>::operator()(char*, unsigned char*, unsigned int) const at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:623
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x3ffef48c: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:99
0x4020e3f0: PubSubClient::loop() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:416
0x3fff0478: _Pri_3_Stack at nmi-vector.o:?
0x3ffef968: _Pri_3_Stack at nmi-vector.o:?
0x40222f28: esp8266::MDNSImplementation::MDNSResponder::_process(bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:88 (discriminator 1)
0x3ffef968: _Pri_3_Stack at nmi-vector.o:?
0x402213f4: esp8266::MDNSImplementation::MDNSResponder::update() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS/src/LEAmDNS.cpp:1248
0x3ffef508: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:96
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x3ffef508: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:96
0x4020d1d0: loop at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:937
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
0x4021a0c8: HardwareSerial::write(unsigned char const*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/HardwareSerial.h:194
0x4021b454: Print::printf_P(char const*, ...) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/Print.cpp:89
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40210afe: void esp_delay<ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}>(unsigned int, ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}&&, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/coredecls.h:69
(inlined by) void esp_delay<ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}>(unsigned int, ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}&&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/coredecls.h:78
(inlined by) ClientContext::connect(ip4_addr*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/include/ClientContext.h:147
(inlined by) WiFiClient::connect(IPAddress, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClient.cpp:164
0x40281178: etharp_output at ??:?
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40212a40: BearSSL::WiFiClientSecureCtx::connect(char const*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:229
0x4022ac10: base64_encode_value at ??:?
0x4022ac10: base64_encode_value at ??:?
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x40281178: etharp_output at ??:?
0x3ffef4d8: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:98
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40208103: mqttSetup() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:433
0x40228dfc: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Function_handler<void (), otaSetup()::{lambda()#1}> const&, std::_Manager_operation) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:271
0x40206650: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:289
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?
0x40209ca8: setup at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:922
0x40100658: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/core_esp8266_main.cpp:243
0x3ffefcf8: _Pri_3_Stack at nmi-vector.o:?
0x4021efb2: loop_wrapper() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/core_esp8266_main.cpp:263
0x401005a9: cont_wrapper at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/cont.S:81

@mcspr
Copy link
Collaborator

mcspr commented Jan 21, 2024

No poison after block at: 0x3fff2838, actual data: 0x65 0x0 0xa5 0xa5
Panic umm_malloc.cpp:751 umm_malloc_core

Memory is corrupted, allocator is configured to panic() when that happens

@hmueller01
Copy link
Contributor Author

Tx for the quick response. Is there a way to figure out the location in the trace? I currently have no clue where to look. I already removed almost everything except the WiFi configuration and the mqtt stuff (over TLS using BearSSL on top of PubSubClient). This is really a pain ...

@d-a-v
Copy link
Collaborator

d-a-v commented Jan 21, 2024

It doesn't reach the DEBUG_P("after malloc, before memcpy"); any more.

Adding Serial.flush() after every printf might help to narrow the issue.
You can also try to use gdb.

@mcspr
Copy link
Collaborator

mcspr commented Jan 21, 2024

0x3ffefae8: _Pri_3_Stack at nmi-vector.o:?

...or something to do with SDK3? I don't see this one in the default
edit: are you analogWrite'ing or running something in the NMI?

@hmueller01
Copy link
Contributor Author

It doesn't reach the DEBUG_P("after malloc, before memcpy"); any more.

Adding Serial.flush() after every printf might help to narrow the issue. You can also try to use gdb.

Thanks @d-a-v . It crashes exactly at the malloc function (added the flush after the malloc printf / DEBUG_P). Thanks for the gdb hint. Wasn't aware that this is possible with ESP8266 ... but if memory is corrupted I doubt gdb will be of much help. :-(

@hmueller01
Copy link
Contributor Author

hmueller01 commented Jan 22, 2024

nmi-vector.o:?

I use the Ticker, but only once_scheduled() or attach_ms_scheduled() which should execute the callback in the loop context. I will remove all that code and will see if anything changes.

BTW: The crash happens in the PubSubClient MQTT callback message/topic received. But I use this code in other projects in the same way and do not see a crash there.

Edit: And for sure PubSubClient isn't bugfree - and unsupported since a long time. I already fixed a bug in context with PROGMEM data

boolean PubSubClient::publish_P(const char* topic, const char* payload, boolean retained) {
    return publish_P(topic, (const uint8_t*)payload, payload ? strnlen(payload, this->bufferSize) : 0, retained);
}

to (as I have seen crashes at strnlen())

boolean PubSubClient::publish_P(const char* topic, const char* payload, boolean retained) {
    return publish_P(topic, (const uint8_t*)payload, payload ? strnlen_P(payload, this->bufferSize) : 0, retained);
}

@hmueller01
Copy link
Contributor Author

Removing all the Ticker code didn't help at all. Dump output looks almost the same.

--------------- CUT HERE FOR EXCEPTION DECODER ---------------

mqttConnect: connected using PubSubClient
mqttConnect: Current time: Mon Jan 22 20:29:47 2024
mqttCallback
getFreeHeap: 14872, getHeapFragmentation: 2, getMaxFreeBlockSize: 14712
length: 2
No poison after block at: 0x3fff2410, actual data: 0x65 0x0 0xa5 0xa5
User exception (panic/abort/assert)
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
Panic umm_malloc.cpp:751 umm_malloc_core
sp: 3ffffcc0 end: 3fffffc0 offset: 0010
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr

ctx: cont
0x40100ea0: umm_malloc_core at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_malloc.cpp:753
0x401011e2: umm_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_malloc.cpp:912
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x4010124c: umm_poison_malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_poison.c:173
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x40261da3: umm_max_block_size at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/umm_malloc/umm_info.c:203
0x40100680: malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/heap.cpp:250
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x4020c518: mqttCallback(char*, unsigned char*, unsigned int) at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:451
0x40211e6b: BearSSL::WiFiClientSecureCtx::_run_until(unsigned int, bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:497
0x40211748: BearSSL::WiFiClientSecureCtx::_engineConnected() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:261 (discriminator 5)
0x40212658: BearSSL::WiFiClientSecureCtx::read(unsigned char*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:383 (discriminator 4)
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x40227f20: BearSSL::WiFiClientSecureCtx::read() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:414
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x402125dc: BearSSL::WiFiClientSecureCtx::available() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:443
0x402273ce: BearSSL::WiFiClientSecure::read() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.h:279
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x4020d3e1: PubSubClient::readByte(unsigned char*) at /Users/hmueller/src/PlatformIO/Projects/garage-pio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:297
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x4022784c: std::_Function_handler<void (char*, unsigned char*, unsigned int), void (*)(char*, unsigned char*, unsigned int)>::_M_invoke(std::_Any_data const&, char*&&, unsigned char*&&, unsigned int&&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:293
0x40100160: std::function<void (char*, unsigned char*, unsigned int)>::operator()(char*, unsigned char*, unsigned int) const at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:623
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x3ffef244: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:90
0x4020de0c: PubSubClient::loop() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/.pio/libdeps/upstream_develop_flash_serial_debug/PubSubClient/src/PubSubClient.cpp:416
0x3fff0228: _Pri_3_Stack at nmi-vector.o:?
0x3ffef718: _Pri_3_Stack at nmi-vector.o:?
0x402223fc: esp8266::MDNSImplementation::MDNSResponder::_process(bool) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS/src/LEAmDNS_Control.cpp:88 (discriminator 1)
0x3ffef718: _Pri_3_Stack at nmi-vector.o:?
0x402208c8: esp8266::MDNSImplementation::MDNSResponder::update() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266mDNS/src/LEAmDNS.cpp:1248
0x3ffef2c0: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:87
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x3ffef2c0: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:87
0x4020cc18: loop at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:944
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x40219590: HardwareSerial::write(unsigned char const*, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/HardwareSerial.h:194
0x4021a91c: Print::printf_P(char const*, ...) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/Print.cpp:89
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x4021051e: void esp_delay<ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}>(unsigned int, ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}&&, unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/coredecls.h:69
(inlined by) void esp_delay<ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}>(unsigned int, ClientContext::connect(ip4_addr*, unsigned short)::{lambda()#1}&&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/coredecls.h:78
(inlined by) ClientContext::connect(ip4_addr*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/include/ClientContext.h:147
(inlined by) WiFiClient::connect(IPAddress, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClient.cpp:164
0x4027f7c8: etharp_output at ??:?
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40212460: BearSSL::WiFiClientSecureCtx::connect(char const*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:229
0x402294b8: base64_encode_value at ??:?
0x402294b8: base64_encode_value at ??:?
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x4027f7c8: etharp_output at ??:?
0x3ffef290: _Pri_3_Stack at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:89
0x3ffe88e4: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x4020804f: mqttSetup() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:429
0x402277d8: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Function_handler<void (), otaSetup()::{lambda()#1}> const&, std::_Manager_operation) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:271
0x4020664c: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:289
0x3ffef838: _Pri_3_Stack at nmi-vector.o:?
0x3ffef898: _Pri_3_Stack at nmi-vector.o:?
0x40209500: setup at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:927
0x401003e0: ets_post at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/core_esp8266_main.cpp:243
0x3ffefa88: _Pri_3_Stack at nmi-vector.o:?
0x4021e0e2: loop_wrapper() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/core_esp8266_main.cpp:263
0x4010171d: cont_wrapper at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266/cores/esp8266/cont.S:81

garage-arduino.cpp _Pri_3_Stack lines mentioned above:

86: ESP8266WebServer m_webserver(80);
87: ESP_WiFiManager_Lite m_wifi_manager;
88: WiFiEventHandler m_station_got_ip_handler;
89: BearSSL::WiFiClientSecure m_esp_client;
90: PubSubClient m_mqtt_client(m_esp_client);

@mcspr
Copy link
Collaborator

mcspr commented Jan 22, 2024

garage-arduino.cpp _Pri_3_Stack lines mentioned above:

...I meant that this shows up b/c of enabled SDK3, apparently. SDK2 190703 does not have this symbol in .a, SDK305 does. platformio.ini options weren't mentioned above

@hmueller01
Copy link
Contributor Author

hmueller01 commented Jan 22, 2024

@mcspr You should have access to
https://github.com/arduino-hmueller01/garage-pio/blob/main/platformio.ini

Yes, I use SDK305. I'll see if it compiles with SDK2 and report.

Edit: I used [env:upstream_develop_flash_serial_debug] target here ...

@mcspr
Copy link
Collaborator

mcspr commented Jan 26, 2024

https://github.com/arduino-hmueller01/garage-pio/blob/main/platformio.ini

I must be doing something wrong, it is still 404 for me 🤷

@hmueller01
Copy link
Contributor Author

hmueller01 commented Jan 26, 2024

Oh sorry. I added additional access for you. Hope it works now.

In the meantime I tried SDK 2.x and code crashed even earlier. Maybe I have to remove more code to narrow down the issue. It is really strange that some projects of mine work and others don't. Even that the code is almost the same now.

connected with xxx, channel 1
dhcp client start...
wifi evt: 0
ip:192.168.x.x,mask:255.255.255.0,gw:192.168.x.x
wifi evt: 3
Station connected: 192.168.x.x
[WIFIM] Connected:
[WIFIM]   SSID: xxx
[WIFIM]   IP:   192.168.x.x
[WIFIM]   MAC:  48:5D:35:7D:74:41
[WIFIM]   CH:   1
[WIFIM]   RSSI: -66
[WML] WiFi connected after num retries: 0
[WML] SSID=xxx,RSSI=-66
[WML] Channel=1,IP=192.168.x.x
[WML] bg: WiFi OK.
otaSetup: Setting up ArduinoOTA
timeSetup: Setting time using SNTP
timeSetup: Current time: Tue Jan 23 19:22:25 2024
mqttSetup: MQTT client id: ESP-0015AADA
mqttSetup: HomA system id: 123456-garage
mqttSetup: Attempting connection to server y:8883
[hostByName] request IP for: y
No poison after block at: 0x3fff2ef8, actual data: 0x65 0x0 0xa5 0xa5
User exception (panic/abort/assert)
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr
Abort called
sp: 3ffffe60 end: 3fffffc0 offset: 0010
To make this dump useful, DECODE IT - https://tinyurl.com/8266dcdr

ctx: cont
0x4020e4ec: make_unique<(anonymous namespace)::_dns_found_result, (anonymous namespace)::_dns_found_result> at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/unique_ptr.h:962
(inlined by) hostByNameImpl at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp:625
0x401006b0: malloc at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/heap.cpp:246
0x4021e557: operator new(unsigned int) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/abi.cpp:41
0x3ffe876c: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x4020e4ec: make_unique<(anonymous namespace)::_dns_found_result, (anonymous namespace)::_dns_found_result> at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/unique_ptr.h:962
(inlined by) hostByNameImpl at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp:625
0x4022a0fc: base64_encode_value at ??:?
0x40211ddc: BearSSL::WiFiClientSecureCtx::setFingerprint(char const*) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:659
0x3ffe876c: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x4020e7dc: ESP8266WiFiGenericClass::hostByName(char const*, IPAddress&) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/ESP8266WiFiGeneric.cpp:685
0x402129c5: BearSSL::WiFiClientSecureCtx::connect(char const*, unsigned short) at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/libraries/ESP8266WiFi/src/WiFiClientSecureBearSSL.cpp:220
0x4022a0fc: base64_encode_value at ??:?
0x4027f420: etharp_output at ??:?
0x3ffef010: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:89
0x3ffe876c: ?? at /Users/hmueller/src/PlatformIO/Projects/garage-pio/include/dynamicParams.h:52
0x40207f05: mqttSetup() at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:423
0x4022841c: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_manager(std::_Any_data&, std::_Function_handler<void (), otaSetup()::{lambda()#1}> const&, std::_Manager_operation) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:271
0x4020664c: std::_Function_handler<void (), otaSetup()::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /Users/hmueller/.platformio/packages/toolchain-xtensa/xtensa-lx106-elf/include/c++/10.3.0/bits/std_function.h:289
0x40209450: setup at /Users/hmueller/src/PlatformIO/Projects/garage-pio/src/garage-arduino.cpp:927
0x4021e987: loop_wrapper() at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/core_esp8266_main.cpp:260
0x40101701: cont_wrapper at /Users/hmueller/.platformio/packages/framework-arduinoespressif8266@src-31d658a59f41540201fc3726a1394910/cores/esp8266/cont.S:81

(and of course I erased flash before downgrading of the SDK ...)

@mcspr
Copy link
Collaborator

mcspr commented Jan 27, 2024

Watch out for unchecked (s)printf

diff --git a/src/garage-arduino.cpp b/src/garage-arduino.cpp
index 85602ae..3088409 100644
--- a/src/garage-arduino.cpp
+++ b/src/garage-arduino.cpp
@@ -419,7 +420,7 @@ void mqttSetup() {
   INFOF_P("%s: MQTT client id: %s" LF, __func__, host_name);

   SAFE_FREE(m_homa_id);
-  m_homa_id = (char *)malloc(strlen_P(PSTR(HOMA_SYSTEM_ID) + 1));
+  m_homa_id = (char *)malloc(strlen_P(PSTR(HOMA_SYSTEM_ID)) + 1);
   sprintf_P(m_homa_id, PSTR(HOMA_SYSTEM_ID));  //, ESP.getChipId());
   INFOF_P("%s: HomA system id: %s" LF, __func__, m_homa_id);

strlen_P measures -1 byte, then sprintf_P breaks the memory block by putting last char of the string and \0 into the poison block

For example, HOMA_SYSTEM_ID is 123. umm fails here for me as well when allocating tcp connection memory after noticing that neighbouring memory is broken

No poison after block at: 0x3fff2ba6, actual data: 0x33 0x0 0xa5 0xa5

0xa5 is the 'poison' byte which surrounds the allocation block in groups of four, but it is overwritten with 0x33 (char 3) and 0x0 (aka null terminator), b/c allocation size is 2 and not the expected 4.

(gdb) x/16xb 0x3fff2ba6 - 8
0x3fff2b9e:     0x00    0x00    0xa5    0xa5    0xa5    0xa5    0x31    0x32
0x3fff2ba6:     0x33    0x00    0xa5    0xa5    0x00    0x00    0x00    0x00

Looks like 123\0 above. Pointing at the symbol specifically, it is definitely the neighbour and the reason of failure.

(gdb) p m_homa_id
$1 = 0x3fff2ba4 "123"
(gdb) x/16xb (uint8_t*)m_homa_id - 4
0x3fff2ba0:     0xa5    0xa5    0xa5    0xa5    0x31    0x32    0x33    0x00
0x3fff2ba8:     0xa5    0xa5    0x00    0x00    0x00    0x00    0x00    0x00

@hmueller01
Copy link
Contributor Author

@mcspr you are incredible! Thank you sooo much! What a stupid mistake of mine. Shame on me. I owe you something.
From my other project I changed

#define _HOMA_SYSTEM_ID "%06x-rhts"  // numerical part (%06x) gets added during runtime
  m_homa_id = (char *)malloc(8 + strlen_P(PSTR(_HOMA_SYSTEM_ID)));
  sprintf_P(m_homa_id, PSTR(_HOMA_SYSTEM_ID), ESP.getChipId());

to

#define HOMA_SYSTEM_ID	"123456-garage"
  m_homa_id = (char *)malloc(strlen_P(PSTR(HOMA_SYSTEM_ID) + 1));
  sprintf_P(m_homa_id, PSTR(HOMA_SYSTEM_ID));  //, ESP.getChipId());

What a bad idea to write the +1 at the end at the slightly wrong position.

But good to know how to interpret the 'poison' bytes. Is there a documentation somewhere (tried to find something without success)? And good to know how to use the debugger here. Thanks. Thanks. Thanks.

A last question is still there. Is it possible to set another reset reason? This 'panic' resets the ESP and after reboot
ESP.getResetReason() outputs Software/System restart (REASON_SOFT_RESTART) I would expect an Exception and not like a 'normal' restart ...

@mcspr
Copy link
Collaborator

mcspr commented Feb 4, 2024

But good to know how to interpret the 'poison' bytes. Is there a documentation somewhere (tried to find something without success)?

Only through reading the umm code and its comments. I think the example above could be a good addition to the gdb (https://github.com/esp8266/Arduino/blob/master/doc/gdb.rst) or faq (https://github.com/esp8266/Arduino/blob/master/doc/faq/a02-my-esp-crashes.rst , where it would corrupt some memory on purpose with an expected result

A last question is still there. Is it possible to set another reset reason? This 'panic' resets the ESP and after reboot
ESP.getResetReason() outputs Software/System restart (REASON_SOFT_RESTART) I would expect an Exception and not like a 'normal' restart ...

Something overlooked. iirc, reset reason is passed through the 'custom callback' correctly but is not preserved during reboot as we read from SDK and SDK does not know of this special case

'Exception' as the term used here would mean Xtensa hw 'General Exception Causes'
https://github.com/esp8266/Arduino/blob/de1029ffe0f85b40465ecb64218ef7ab3643ffa7/doc/exception_causes.rst

Value is also not preserved during the reboot correctly for this case, as it is only for display and not really an 'Exception' at all

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants