Docker container does not start dump1090 after upgrade to 7.2 #123

jheinitz · 2022-03-26T11:23:45Z

Hello Team!

first of all, I would like to thank you for your efforts on this project. I'm running your piaware image on an old Dell Laptop for quite a while. I do regular updates of the image. The latest happed today. I upgraded from 7.1 to 7.2 and suddenly I could not send message to FlightAware. I logged into the container and figured out that there is no dump1090 process running. I performed a downgrade to 7.1 and everything works as expected. Any ideas?

Kind regards

Jens

mikenye · 2022-03-26T13:37:36Z

Hi @jheinitz, sorry you’re having trouble.

Are you using BEASTHOST or is the piaware container talking to the SDR directly?

Could you please post the container logs (removing any sensitive info)?

Thanks.

jheinitz · 2022-03-27T19:45:02Z

Hello @mikenye ,

my container is talking directly to the USB device. Please find my logs below:

Attaching to piaware
piaware    | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
piaware    | [s6-init] ensuring user provided files have correct perms...exited 0.
piaware    | [fix-attrs.d] applying ownership & permissions fixes...
piaware    | [fix-attrs.d] done.
piaware    | [cont-init.d] executing container initialization scripts...
piaware    | [cont-init.d] 00-libsecomp2: executing...
piaware    | [cont-init.d] 00-libsecomp2: exited 0.
piaware    | [cont-init.d] 01-piaware: executing...
piaware    | Set feeder-id to <my-feeder-id> in /etc/piaware.conf:1
piaware    | Set allow-auto-updates to no in /etc/piaware.conf:2
piaware    | Set allow-manual-updates to no in /etc/piaware.conf:3
piaware    | Set allow-mlat to yes in /etc/piaware.conf:4
piaware    | Set mlat-results to yes in /etc/piaware.conf:5
piaware    | Set receiver-type to rtlsdr in /etc/piaware.conf:6
piaware    | [cont-init.d] 01-piaware: exited 0.
piaware    | [cont-init.d] done.
piaware    | [services.d] starting services
piaware    | [skyaware] 2022/03/27 21:35:13 2022-03-27 21:35:13: server.c.1513) server started (lighttpd/1.4.59)
piaware    | [services.d] done.
piaware    | [piaware] 2022/03/27 21:35:13 ****************************************************
piaware    | [piaware] 2022/03/27 21:35:13 piaware version 7.2 is running, process ID 331
piaware    | [piaware] 2022/03/27 21:35:13 your system info is: Linux piaware 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux
piaware    | [piaware] 2022/03/27 21:35:15 Connecting to FlightAware adept server at piaware.flightaware.com/1200
piaware    | [piaware] 2022/03/27 21:35:19 no ADS-B data program seen listening on port 30005 for 6 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:35:19 UAT support disabled by local configuration setting: uat-receiver-type
piaware    | [piaware] 2022/03/27 21:35:19 Connection with adept server at piaware.flightaware.com/1200 established
piaware    | [piaware] 2022/03/27 21:35:19 TLS handshake with adept server at piaware.flightaware.com/1200 completed
piaware    | [piaware] 2022/03/27 21:35:19 FlightAware server certificate validated
piaware    | [piaware] 2022/03/27 21:35:19 encrypted session established with FlightAware
piaware    | [piaware] 2022/03/27 21:35:19 adept reported location: 53.xxxxx, 9.xxxxx, 150ft AMSL
piaware    | [piaware] 2022/03/27 21:35:19 Receiver location changed, restarting dump1090 and skyaware978
piaware    | [piaware] 2022/03/27 21:35:19 attempting to restart dump1090..
piaware    | [piaware] 2022/03/27 21:35:19 has_invoke_rcd: Running on docker ignoring invoke-rc.d
piaware    | [piaware] 2022/03/27 21:35:19 has_invoke_rcd: Running on docker ignoring invoke-rc.d
piaware    | [piaware] 2022/03/27 21:35:19 attempting to restart dump1090 using '/etc/init.d/dump1090 restart < /dev/null &'...
piaware    | [piaware] 2022/03/27 21:35:19 dump1090 restart appears to have been successful
piaware    | [piaware] 2022/03/27 21:35:19 attempting to restart skyaware978..
piaware    | [piaware] 2022/03/27 21:35:19 can't restart skyaware978, no services that look like skyaware978 found
piaware    | [piaware] 2022/03/27 21:35:19 logged in to FlightAware as user <my-user>
piaware    | [piaware] 2022/03/27 21:35:19 my feeder ID is <my-feeder-id>
piaware    | [piaware] 2022/03/27 21:35:19 site statistics URL: https://flightaware.com/adsb/stats/user/<myuser>#stats-<id>
piaware    | [piaware] 2022/03/27 21:35:20 multilateration data requested
piaware    | [piaware] 2022/03/27 21:35:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:35:49 0 msgs recv'd from dump1090; 0 msgs sent to FlightAware
piaware    | [piaware] 2022/03/27 21:36:19 no ADS-B data program seen listening on port 30005 for 66 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:36:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:37:19 no ADS-B data program seen listening on port 30005 for 126 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:37:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:38:19 no ADS-B data program seen listening on port 30005 for 186 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:38:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:39:19 no ADS-B data program seen listening on port 30005 for 246 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:39:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:40:19 no ADS-B data program seen listening on port 30005 for 306 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:40:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:40:49 0 msgs recv'd from dump1090 (0 in last 5m); 0 msgs sent to FlightAware
piaware    | [piaware] 2022/03/27 21:41:19 no ADS-B data program seen listening on port 30005 for 366 seconds, trying to start it...
piaware    | [piaware] 2022/03/27 21:41:19 attempting to start dump1090..
piaware    | [piaware] 2022/03/27 21:41:19 has_invoke_rcd: Running on docker ignoring invoke-rc.d
piaware    | [piaware] 2022/03/27 21:41:19 has_invoke_rcd: Running on docker ignoring invoke-rc.d
piaware    | [piaware] 2022/03/27 21:41:19 attempting to start dump1090 using '/etc/init.d/dump1090 start < /dev/null &'...
piaware    | [piaware] 2022/03/27 21:41:19 dump1090 start appears to have been successful
piaware    | [piaware] 2022/03/27 21:41:19 Usage: /etc/init.d/dump1090 {start|stop|restart|status}
piaware    | [piaware] 2022/03/27 21:41:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:41:24 child pid 375 exited with status 1
piaware    | [piaware] 2022/03/27 21:41:29 no ADS-B data program seen listening on port 30005 for 10 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:42:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:42:29 no ADS-B data program seen listening on port 30005 for 70 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:43:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:43:29 no ADS-B data program seen listening on port 30005 for 130 seconds, next check in 60s
piaware    | [piaware] 2022/03/27 21:44:20 no ADS-B data program is serving on port 30005, not starting multilateration client yet
piaware    | [piaware] 2022/03/27 21:44:29 no ADS-B data program seen listening on port 30005 for 190 seconds, next check in 60s

Here is my docker-compose.yml:

version: '2.1'

services:
  piaware:
    restart: always
    #image: ghcr.io/sdr-enthusiasts/docker-piaware:v7.1_nohealthcheck
    image: ghcr.io/sdr-enthusiasts/docker-piaware:latest
    tty: true
    container_name: piaware
    devices:
       - /dev/bus/usb:/dev/bus/usb
    ports:
      - 30003:30003
      - 30005:30005
    hostname: piaware
    networks:
      traefik-net:
        aliases:
          - piaware
    environment:
      - TZ=Europe/Berlin
      - LAT=53.xxxxx
      - LONG=9.xxxx
      - FEEDER_ID=<my-feeder-id>
      - RECEIVER_TYPE=rtlsdr
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.piaware.entrypoints=http"
      - "traefik.http.routers.piaware.rule=Host(`<external hostname>`)"
      - "traefik.http.middlewares.piaware-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.routers.piaware.middlewares=piaware-https-redirect"
      - "traefik.http.routers.piaware-secure.entrypoints=https"
      - "traefik.http.routers.piaware-secure.rule=Host(`<external hostname>`)"
      - "traefik.http.routers.piaware-secure.tls=true"
      - "traefik.http.routers.piaware-secure.tls.certresolver=http"
      - "traefik.http.routers.piaware-secure.service=piaware"
      - "traefik.http.services.piaware.loadbalancer.server.port=80"
      - "traefik.docker.network=traefik_proxy"

networks:
  traefik-net:
    external: true
    name: traefik_proxy

When I use the tag v7.1_nohealthcheck instead of latest, everything works fine.

I hope that helps. Please let me know if you want me to do some extra tests.

Kind regards

Jens

jheinitz · 2022-03-27T19:47:28Z

Hi Mike,

it's me again. When I log into the container, there is no 1090 process running:

jens@server1:/opt/piaware$ docker-compose exec piaware bash
root@piaware:/# ps -ef | grep 1090
root       293     1  0 21:35 pts/0    00:00:00 s6-supervise dump1090
root       452   444  0 21:46 pts/1    00:00:00 grep 1090
root@piaware:/# exit
exit
jens@server1:/opt/piaware$

Best regards

Jens

mikenye · 2022-03-28T01:45:25Z

Hi Jens,

You'll need to set the environment variable DUMP1090_DEVICE to the serial or device ID of your RTL-SDR.

You can list your RTL-SDR devices with docker exec -it piaware rtl_test.

Let me know if that fixes your problem.

Kind regards,
Mike

mikenye · 2022-03-28T03:04:44Z

Hi Jens,

I've also updated the container image to prevent this from occurring.

Once PR #124 is merged, this issue will automatically be closed. After this, the new image will build (will take about 1-2 hours). Once complete, please pull the image and update your container.

Please feel free to re-open the issue if you're still having problems after that.

Thanks.

jheinitz · 2022-03-28T08:19:21Z

Hi Mike!

That did the trick. Thanks a lot.

Bye

Jens

joshsmithy · 2022-03-31T11:09:37Z

Hi Mike,

I am facing a smiliar issue with the dump 1090 output.

I use readsb to redistribute receiver data to piaware, hence there is no receiver directly attached to the container...

docker exec -it flightaware_piaware rtl_test
Found 1 device(s):
0: ▒TB, , SN:
Using device 0: Generic RTL2832U OEM

My compose file is as follows:

flightaware_piaware:
image: ghcr.io/sdr-enthusiasts/docker-piaware:latest
tty: true
container_name: flightaware_piaware
restart: always
depends_on:
- readsb
ports:
- XXXX:8080
environment:
- BEASTHOST=readsb
- LAT=
- LONG=-
- TZ="Europe/London"
- FEEDER_ID=
tmpfs:
- /run:exec,size=64M
- /var/log

Reverting to v7.1_nohealthcheck works resolves everything.

Have you any suggestions to fix this?

I assume I need to add the DUMP1090_DEVICE environmental variable, however, without an SN I am unsure what to use?

Thanks,

Josh

mikenye · 2022-03-31T15:29:23Z

Hi @joshsmithy, could you please pull the latest version of the piaware image and recreate the container? Once done, please post the output of docker logs piaware being careful to remove your LAT/LONG & feeder-id.

Thanks.

courtarro · 2022-03-31T20:31:02Z

I pulled the latest image yesterday (and again this morning with no change) and now I'm running into this issue, with dump1090 segfaulting. Here's the error in the Docker container logs:

[piaware] 2022/03/31 20:19:24 ****************************************************
[piaware] 2022/03/31 20:19:24 piaware version 7.2 is running, process ID 342
[piaware] 2022/03/31 20:19:24 your system info is: Linux 335e61975b8d 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 GNU/Linux
./run: line 87: 346 Illegal instruction (core dumped) "${DUMP1090_BIN}" "${DUMP1090_CMD[@]}" 2>&1
347 Done | stdbuf -o0 sed '/^$/d'
348 Done | stdbuf -o0 awk '{print "[dump1090] " strftime("%Y/%m/%d %H:%M:%S", systime()) " " $0}'

This is running against mikenye/piaware:latest, a.k.a. mikenye/piaware:v7.2. I was able to workaround by downgrading to mikenye/piaware:v7.1.

joshsmithy · 2022-03-31T20:55:32Z

I pulled the latest image yesterday (and again this morning with no change) and now I'm running into this issue, with dump1090 segfaulting. Here's the error in the Docker container logs:

[piaware] 2022/03/31 20:19:24 ****************************************************
[piaware] 2022/03/31 20:19:24 piaware version 7.2 is running, process ID 342
[piaware] 2022/03/31 20:19:24 your system info is: Linux 335e61975b8d 5.4.0-104-generic #118-Ubuntu SMP Wed Mar 2 19:02:41 UTC 2022 x86_64 GNU/Linux
./run: line 87: 346 Illegal instruction (core dumped) "${DUMP1090_BIN}" "${DUMP1090_CMD[@]}" 2>&1
347 Done | stdbuf -o0 sed '/^$/d'
348 Done | stdbuf -o0 awk '{print "[dump1090] " strftime("%Y/%m/%d %H:%M:%S", systime()) " " $0}'

This is running against mikenye/piaware:latest, a.k.a. mikenye/piaware:v7.2. I was able to workaround by downgrading to mikenye/piaware:v7.1.

My logs echo the same:

[beast-splitter] 2022/03/31 20:53:41 net(readsb:30005): connected to a Beast-style receiver
[piaware] 2022/03/31 20:53:41 no ADS-B data program seen listening on port 30005 for 3 seconds, next check in 60s
[piaware] 2022/03/31 20:53:41 UAT support disabled by local configuration setting: uat-receiver-type
./run: line 87: 393 Illegal instruction (core dumped) "${DUMP1090_BIN}" "${DUMP1090_CMD[@]}" 2>&1
394 Done | stdbuf -o0 sed '/^$/d'
395 Done | stdbuf -o0 awk '{print "[dump1090] " strftime("%Y/%m/%d %H:%M:%S", systime()) " " $0}'
./run: line 87: 400 Illegal instruction (core dumped) "${DUMP1090_BIN}" "${DUMP1090_CMD[@]}" 2>&1

mikenye · 2022-04-01T02:34:23Z

Hi @joshsmithy & @courtarro,

I'm sorry you're having problems. In order to troubleshoot this, I need some information from your hosts. Could you please both paste the output of lscpu?

Could I please ask that you also try the following:

In your docker-compose.yml file, you can add the following directive to your piaware container configuration (I usually place this directly below image:):

build: https://github.com/sdr-enthusiasts/docker-piaware.git#main

You can then run docker-compose build piaware, which will take a while.

Once built, you can run docker-compose up -d, and the piaware container should be recreated with your newly built image.

The reasoning behind this, is that I think the dump1090 binary built in GitHub's runners is likely using CPU optimisations that are not compatible with all CPUs. If I'm able to confirm this, I need to open a case with the developers of dump1090 (FlightAware) and let them know so they're able to fix this (or if they can't/won't, I will need to disable these optimisations). Building the binary yourself (by building the container yourself) should remove these optimisations at build time. So if the container you build yourself works, then I know this will be the case...

Hopefully this makes sense. Thanks for your patience!

courtarro · 2022-04-01T03:17:39Z

I'm not in a place where I'm ready to run the test build at the moment, but I'm running on an AMD Epyc (Milan generation) inside a Qemu/KVM VM. Here is the output of lscpu:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   40 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      15
Model:                           6
Model name:                      Common KVM processor
Stepping:                        1
CPU MHz:                         2649.998
BogoMIPS:                        5299.99
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       256 KiB
L1i cache:                       256 KiB
L2 cache:                        2 MiB
L3 cache:                        16 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; LFENCE, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm rep_good nopl cpuid extd_apicid 
                                 tsc_known_freq pni cx16 x2apic hypervisor cmp_legacy 3dnowprefetch vmmcall

mikenye · 2022-04-01T03:18:17Z

I'm not in a place where I'm ready to run the test build at the moment, but I'm running on an AMD Epyc (Milan generation) inside a Qemu/KVM VM. Here is the output of lscpu:

Thanks @courtarro

mikenye · 2022-04-01T03:55:36Z

Possible continuation of #79

joshsmithy · 2022-04-10T16:53:28Z

Hi @joshsmithy & @courtarro,

I'm sorry you're having problems. In order to troubleshoot this, I need some information from your hosts. Could you please both paste the output of lscpu?

Could I please ask that you also try the following:

In your docker-compose.yml file, you can add the following directive to your piaware container configuration (I usually place this directly below image:):
build: https://github.com/sdr-enthusiasts/docker-piaware.git#main
You can then run docker-compose build piaware, which will take a while.

Once built, you can run docker-compose up -d, and the piaware container should be recreated with your newly built image.

The reasoning behind this, is that I think the dump1090 binary built in GitHub's runners is likely using CPU optimisations that are not compatible with all CPUs. If I'm able to confirm this, I need to open a case with the developers of dump1090 (FlightAware) and let them know so they're able to fix this (or if they can't/won't, I will need to disable these optimisations). Building the binary yourself (by building the container yourself) should remove these optimisations at build time. So if the container you build yourself works, then I know this will be the case...

Hopefully this makes sense. Thanks for your patience!

Hi Mike,

Apologies for my delay in responding.

I was in a position to test this out this afternoon and I can confirm that after following your instructions, everyting is working well. Hence, it would seem that your hunch regarding CPU optimisation may be correct.

I am running a very similar environment to @courtarro, using an intel processor and Qemu/KVM Environment.

mikenye · 2022-04-11T01:31:22Z

Hi All,

With @wiedehopf's help, we managed to trace this back to compiler optimisations in limesdr.

I've hopefully fixed this issue now, and would appreciate if you could test once the deploy finishes: https://github.com/sdr-enthusiasts/docker-piaware/actions/runs/2145774266

Thanks for your patience.

joshsmithy · 2022-04-11T10:00:06Z

Hi All,

With @wiedehopf's help, we managed to trace this back to compiler optimisations in limesdr.

I've hopefully fixed this issue now, and would appreciate if you could test once the deploy finishes: https://github.com/sdr-enthusiasts/docker-piaware/actions/runs/2145774266

Thanks for your patience.

Hi Mike,

I have just rebuilt using the ghcr.io/sdr-enthusiasts/docker-piaware:latest image and all is working successfully.

Thanks for resolving this.

Josh

mikenye · 2022-04-11T21:47:25Z

Fantastic! Thanks for letting me know.

mikenye self-assigned this Mar 26, 2022

mikenye mentioned this issue Mar 28, 2022

Fixes for 7.2 #124

Merged

mikenye closed this as completed in #124 Mar 28, 2022

mikenye reopened this Mar 31, 2022

mikenye mentioned this issue Apr 11, 2022

Fix dump1090 segfaults #128

Merged

mikenye closed this as completed in #128 Apr 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker container does not start dump1090 after upgrade to 7.2 #123

Docker container does not start dump1090 after upgrade to 7.2 #123

jheinitz commented Mar 26, 2022

mikenye commented Mar 26, 2022

jheinitz commented Mar 27, 2022

jheinitz commented Mar 27, 2022

mikenye commented Mar 28, 2022

mikenye commented Mar 28, 2022

jheinitz commented Mar 28, 2022

joshsmithy commented Mar 31, 2022 •

edited

Loading

mikenye commented Mar 31, 2022

courtarro commented Mar 31, 2022 •

edited

Loading

joshsmithy commented Mar 31, 2022

mikenye commented Apr 1, 2022 •

edited

Loading

courtarro commented Apr 1, 2022 •

edited

Loading

mikenye commented Apr 1, 2022

mikenye commented Apr 1, 2022

joshsmithy commented Apr 10, 2022

mikenye commented Apr 11, 2022

joshsmithy commented Apr 11, 2022

mikenye commented Apr 11, 2022

Docker container does not start dump1090 after upgrade to 7.2 #123

Docker container does not start dump1090 after upgrade to 7.2 #123

Comments

jheinitz commented Mar 26, 2022

mikenye commented Mar 26, 2022

jheinitz commented Mar 27, 2022

jheinitz commented Mar 27, 2022

mikenye commented Mar 28, 2022

mikenye commented Mar 28, 2022

jheinitz commented Mar 28, 2022

joshsmithy commented Mar 31, 2022 • edited Loading

mikenye commented Mar 31, 2022

courtarro commented Mar 31, 2022 • edited Loading

joshsmithy commented Mar 31, 2022

mikenye commented Apr 1, 2022 • edited Loading

courtarro commented Apr 1, 2022 • edited Loading

mikenye commented Apr 1, 2022

mikenye commented Apr 1, 2022

joshsmithy commented Apr 10, 2022

mikenye commented Apr 11, 2022

joshsmithy commented Apr 11, 2022

mikenye commented Apr 11, 2022

joshsmithy commented Mar 31, 2022 •

edited

Loading

courtarro commented Mar 31, 2022 •

edited

Loading

mikenye commented Apr 1, 2022 •

edited

Loading

courtarro commented Apr 1, 2022 •

edited

Loading