cpu hog on teardown? pthread_join

[karlp]

I'm seeing this in an app I have, and I haven't got a lot of debugging yet, it seems to happen after I'm running for a while, but the CPU usage of the app shoots up, and it's no longer "running" but it hasn't exited. I finally hooked gdb up to it today while it was in this state, and it seems like it's stuck trying to tear down?

I've never seen this with a libmosquitto C app, so even though the stack ends in mosquitto, I kinda feel it's an issue in the lua-mosquitto binding?

Source application is at: https://github.com/remakeelectric/owrt_pub_feeds/blob/output-sparkfun/output-sparkfun/files/usr/bin/output-sparkfun

karlp@teros:~/src/smartgate_firmware/openwrt-bb (rme)$ ./scripts/remote-gdb 192.168.1.34:9999 build_dir/target-mips_mips32_uClibc-0.9.33.2/lua-5.1.5/src/lua
Using target mips_mips32 (uClibc-0.9.33.2)
Python Exception <type 'exceptions.ImportError'> No module named gdb: 

warning: 
Could not load the Python gdb module from `/home/karlp/src/smartgate_firmware/openwrt-bb/staging_dir/toolchain-mips_mips32_gcc-4.8-linaro_uClibc-0.9.33.2/share/gdb/python'.
Limited Python support is available from the _gdb module.
Suggest passing --data-directory=/path/to/gdb/data-directory.

GNU gdb (Linaro GDB) 7.6-2013.05
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-redhat-linux --target=mips-openwrt-linux-uclibc".
For bug reporting instructions, please see:
<https://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /home/karlp/src/smartgate_firmware/openwrt-bb/build_dir/target-mips_mips32_uClibc-0.9.33.2/lua-5.1.5/src/lua...done.
0x76f301e0 in pthread_join (threadid=1994786096, thread_return=0x0)
    at libpthread/nptl/pthread_join.c:89
89      lll_wait_tid (pd->tid);
(gdb) bt
#0  0x76f301e0 in pthread_join (threadid=1994786096, thread_return=0x0)
    at libpthread/nptl/pthread_join.c:89
#1  0x77110978 in _mosquitto_destroy ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/libmosquitto.so.1
#2  0x77110ff8 in mosquitto_destroy ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/libmosquitto.so.1
#3  0x7712da1c in ctx_destroy ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/lua/mosquitto.so
#4  0x7722e9a8 in luaD_precall ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#5  0x7722eb60 in luaD_call ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#6  0x7722fe24 in GCTM ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#7  0x77230fa8 in singlestep ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#8  0x77231290 in luaC_fullgc ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#9  0x772299cc in lua_gc ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#10 0x00401168 in docall ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/bin/lua
#11 0x00401cc8 in pmain ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/bin/lua
#12 0x7722e9a8 in luaD_precall ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#13 0x7722eb60 in luaD_call ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#14 0x7722dc44 in luaD_rawrunprotected ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#15 0x7722edc4 in luaD_pcall ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#16 0x77229850 in lua_cpcall ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/lib/liblua.so.5.1.5
#17 0x00400bc0 in main ()
   from /home/karlp/src/smartgate_firmware/openwrt-bb/scripts/../staging_dir/target-mips_mips32_uClibc-0.9.33.2/root-atheros/usr/bin/lua
(gdb) 

[ncopa]

There are known issues in the uclibc threading implementation. I remember that we had to patch uclibc heavily for uclibc. But we concluded that fixing it 100%. For example, vlc would hang on exit. We ended up switching to musl libc and all those nasty threading problems got solved.

Things you might want check:

• check if lua is linked with -lpthread (readelf -d /usr/bin/lua should show that libpthread is linked in).
• check if you have the needed patches for dlopen/dlclose. (lua does dlopen/dlclose for modules)
• look over the uclibc patches we used in alpine linux. Look specially for the dlopen/dlclose and pthread/cancel patches.

A workaround might be to disable threading in mosquitto. (i would really recommedn this with mosquitto on uclibc)

In any case, I don't think this has anything to do with lua-mosquitto.

I will take a closer look at it if you can reproduce this with musl libc (or maybe even glibc), but not with uclibc since I know for sure that uclibc threading is broken.