dump1090 segfault/coredump

[mikenye]

This issue is open to track the issue with the segfault/coredump of dump1090.

This appears to be limited to x86_64 architecture.

a0241c9#commitcomment-45580597

[mikenye]

@stevenfla, are you able to revert to self-building the container while I look into this?

[mikenye]

@stevenfla can I please trouble you to post the output of the following commands?

• While the “broken” container is running:
  • docker exec piaware ldd /usr/local/bin/dump1090
  • docker exec piaware md5sum /usr/local/bin/dump1090
• uname -a
• lscpu
• lscpu -e

Thanks

[stevenfla]

re-posting here. May have posted in the wrong place !

./run: line 46:   462 Illegal instruction     (core dumped) "${DUMP1090_BIN}" "${DUMP1090_CMD[@]}" 2>&1
       463 Done                    | stdbuf -o0 awk '{print "[dump1090] " strftime("%Y/%m/%d %H:%M:%S", systime()) " " $0}'
./run: line 46:   467 Illegal instruction     (core dumped) "${DUMP1090_BIN}" "${DUMP1090_CMD[@]}" 2>&1
       468 Done                    | stdbuf -o0 awk '{print "[dump1090] " strftime("%Y/%m/%d %H:%M:%S", systime()) " " $0}'

# docker exec piaware ldd /usr/local/bin/dump1090
        linux-vdso.so.1 (0x00007ffcd8f7b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8d6e6c4000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8d6e541000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f8d6e537000)
        librtlsdr.so.0 => /usr/local/lib/librtlsdr.so.0 (0x00007f8d6e523000)
        libbladeRF.so.2 => /usr/local/lib/libbladeRF.so.2 (0x00007f8d6e495000)
        libhackrf.so.0 => /usr/local/lib/libhackrf.so.0 (0x00007f8d6e48c000)
        libLimeSuite.so.20.10-1 => /usr/local/lib/libLimeSuite.so.20.10-1 (0x00007f8d6e3b8000)
        libncurses.so.6 => /lib/x86_64-linux-gnu/libncurses.so.6 (0x00007f8d6e38e000)
        libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f8d6e360000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8d6e19f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8d6e73c000)
        libusb-1.0.so.0 => /lib/x86_64-linux-gnu/libusb-1.0.so.0 (0x00007f8d6df86000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8d6de02000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8d6dde6000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f8d6dde1000)
        libudev.so.1 => /lib/x86_64-linux-gnu/libudev.so.1 (0x00007f8d6ddbb000)

stevel@dockerhost:~/.docker/compose$ docker exec piaware md5sum /usr/local/bin/dump1090
eeb36384c2fc1c4ef02f7d73612bba53  /usr/local/bin/dump1090

stevel@dockerhost:~/.docker/compose$ uname -a

Linux dockerhost 5.8.0-33-generic #36-Ubuntu SMP Wed Dec 9 09:14:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
stevel@dockerhost:~/.docker/compose$ 

# lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   36 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              2
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           26
Model name:                      Genuine Intel(R) CPU             000  @ 3.20GHz
Stepping:                        4
Frequency boost:                 enabled
CPU MHz:                         3740.667
CPU max MHz:                     3193.0000
CPU min MHz:                     1596.0000
BogoMIPS:                        7182.08
Virtualization:                  VT-x
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        1 MiB
L3 cache:                        8 MiB
NUMA node0 CPU(s):               0-7
Vulnerability Itlb multihit:     KVM: Mitigation: VMX disabled
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Vulnerable: Clear CPU buffers attempted, no microcode; SMT vulnerable
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni dtes64 monitor
                                 _cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 sse4_2 popcnt lahf_lm pti tpr_shadow vnmi flexpriority ept vpid dtherm ida


stevel@dockerhost:~/.docker/compose$ lscpu -e
CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ    MINMHZ
  0    0      0    0 0:0:0:0          yes 3193.0000 1596.0000
  1    0      0    1 1:1:1:0          yes 3193.0000 1596.0000
  2    0      0    2 2:2:2:0          yes 3193.0000 1596.0000
  3    0      0    3 3:3:3:0          yes 3193.0000 1596.0000
  4    0      0    1 1:1:1:0          yes 3193.0000 1596.0000
  5    0      0    3 3:3:3:0          yes 3193.0000 1596.0000
  6    0      0    2 2:2:2:0          yes 3193.0000 1596.0000
  7    0      0    0 0:0:0:0          yes 3193.0000 1596.0000

[stevenfla]

Thanks Mike. I can move back and forth to working image easily so this is not a problem.

[mikenye]

@stevenfla can you try the command docker run --rm -it --entrypoint dump1090 mikenye/piaware --help, and paste the output?

[mikenye]

I should explain - I want to see whether there's any additional information if you cause a segfault manually.

[stevenflang]

Am currently running with no problem this container: mikenye/piaware:latest_nohealthcheck

Quick note: sum ran on dump1090 executable in for both containers latest and latest_nohealthcheck match.

docker run --rm -it --entrypoint dump1090 mikenye/piaware --help
Unable to find image 'mikenye/piaware:latest' locally
latest: Pulling from mikenye/piaware
53d9ee195005: Already exists
56158c388f18: Already exists
50b68e52cbaa: Pull complete
Digest: sha256:5ec76d8bef80ef20335754ab4ddb6a9c2c7a351ff70184143564c84db8e3543e
Status: Downloaded newer image for mikenye/piaware:latest
-----------------------------------------------------------------------------
| dump1090 ModeS Receiver                               dump1090-fa unknown |
| build options: ENABLE_RTLSDR ENABLE_BLADERF ENABLE_HACKRF ENABLE_LIMESDR  |
-----------------------------------------------------------------------------

--device-type <type>     Select SDR type (default: rtlsdr)

      rtlsdr-specific options (use with --device-type rtlsdr)

--device <index|serial>  select device by index or serial number
--enable-agc             enable digital AGC (not tuner AGC!)
--ppm <correction>       set oscillator frequency correction in PPM
--direct <0|1|2>         set direct sampling mode

      bladeRF-specific options (use with --device-type bladerf)

--device <ident>         select device by bladeRF 'device identifier'
--bladerf-fpga <path>    use alternative FPGA bitstream ('' to disable FPGA load)
--bladerf-decimation <N> assume FPGA decimates by a factor of N
--bladerf-bandwidth <hz> set LPF bandwidth ('bypass' to bypass the LPF)

      HackRF-specific options (use with --device-type hackrf)

--enable-amp             enable amplifier)
--lna-gain               set LNA gain (Range 0-40 in 8dB steps))
--vga-gain               set VGA gain (Range 0-62 in 2dB steps))
--samplerate             set sample rate)
--ppm                    ppm correction)

      limesdr-specific options (use with --device-type limesdr)

--limesdr-verbosity      set verbosity level for LimeSDR messages
--limesdr-serial         serial number of desired device
--limesdr-channel        set number of an RX channel
--limesdr-oversample     set RF oversampling ratio
--limesdr-gain           set normalized gain
--limesdr-lpfbw          set LPF bandwidth
--limesdr-bw             set bandwidth

      ifile-specific options (use with --ifile)

--ifile <path>           read samples from given file ('-' for stdin)
--iformat <type>         set sample format (UC8, SC16, SC16Q11)
--throttle               process samples at the original capture speed

      Common options

--gain <db>              Set gain (default: max gain. Use -10 for auto-gain)
--freq <hz>              Set frequency (default: 1090 Mhz)
--interactive            Interactive mode refreshing data on screen. Implies --throttle
--interactive-ttl <sec>  Remove from list if idle for <sec> (default: 60)
--raw                    Show only messages hex values
--net                    Enable networking with default ports unless overridden
--modeac                 Enable decoding of SSR Modes 3/A & 3/C
--no-modeac-auto         Don't enable Mode A/C if requested by a Beast connection
--net-only               Enable just networking, no RTL device or file used
--net-bind-address <ip>  IP address to bind to (default: Any; Use 127.0.0.1 for private)
--net-ri-port <ports>    TCP raw input listen ports  (default: 30001)
--net-ro-port <ports>    TCP raw output listen ports (default: 30002)
--net-sbs-port <ports>   TCP BaseStation output listen ports (default: 30003)
--net-bi-port <ports>    TCP Beast input listen ports  (default: 30004,30104)
--net-bo-port <ports>    TCP Beast output listen ports (default: 30005)
--net-stratux-port <ports>   TCP Stratux output listen ports (default: disabled)
--net-ro-size <size>     TCP output minimum size (default: 0)
--net-ro-interval <rate> TCP output memory flush rate in seconds (default: 0)
--net-heartbeat <rate>   TCP heartbeat rate in seconds (default: 60 sec; 0 to disable)
--net-buffer <n>         TCP buffer size 64Kb * (2^n) (default: n=0, 64Kb)
--net-verbatim           Make Beast-format output connections default to verbatim mode
                         (forward all messages, without applying CRC corrections)
--forward-mlat           Allow forwarding of received mlat results to output ports
--lat <latitude>         Reference/receiver latitude for surface posn (opt)
--lon <longitude>        Reference/receiver longitude for surface posn (opt)
--max-range <distance>   Absolute maximum range for position decoding (in nm, default: 300)
--fix                    Enable single-bit error correction using CRC
--fix-2bit               Enable two-bit error correction using CRC (use with caution)
--no-fix                 Disable error correction using CRC
--no-crc-check           Disable messages with broken CRC (discouraged)
--mlat                   display raw messages in Beast ascii mode
--stats                  With --ifile print stats at exit. No other output
--stats-range            Collect/show range histogram
--stats-every <seconds>  Show and reset stats every <seconds> seconds
--onlyaddr               Show only ICAO addresses (testing purposes)
--metric                 Use metric units (meters, km/h, ...)
--gnss                   Show altitudes as HAE/GNSS (with H suffix) when available
--snip <level>           Strip IQ file removing samples < level
--quiet                  Disable output to stdout. Use for daemon applications
--show-only <addr>       Show only messages from the given ICAO on stdout
--write-json <dir>       Periodically write json output to <dir> (for serving by a separate webserver)
--write-json-every <t>   Write json output every t seconds (default 1)
--json-location-accuracy <n>  Accuracy of receiver location in json metadata: 0=no location, 1=approximate, 2=exact
--dcfilter               Apply a 1Hz DC filter to input data (requires more CPU)
--version                Show version and build options
--help                   Show this help

[mikenye]

The fact it downloaded the image and then ran fine makes me wonder if something was wrong with the image you were previously running.

If you change over to mikenye/piaware:latest, does that now work ok?

[stevenflang]

Good call on that one Mike and thank you. Problem is resolved.

Removed all containers this time. Check for any remaining images and pulled new ones.
My theory is the images or a portion of those were not being removed.
I suspect this may be due to how I initially set up docker using GhostWriters/DockSTARTer
and maybe also not using the correct command to manage containers in the above environment and or a combination of the above.

Latest works now.
Closing this issue.

[mikenye]

That’s great to hear! Let me know if you have any more issues. :-)

[stevenflang]

Wanted to make a final comment on this issue in that maybe it might be helpful to others.
Before this started happening, due to cpu utilization nearing 100 %, moved off older core 2 cpu and over to I7. I just had too many containers running. To save time simply moved the boot disk to I7 and booted it up. I didn't check log on any containers after doing this where they all seemed started. I did check a few days later so suspect it may have been been in this state since the drive swap. Most likely what happened is when docker compiled dump 1090 it was optimized for core 2 intel cpu and since I didn't remove and pull new image this older cpu arch optimization did not work for I7 causing the core dump. Exactly what you suspected Mike but happened in a different way !

Summary: when moving hard drives between different hardware that use docker images be sure to remove and pull new docker images.