container unhealthy possible issue with beast-splitter

[billygr]

Hi,

Setup: dump1090 beast on a different host

pi4: running mikenye/adsbexchange mikenye/planefinder and docker-piware

mikenye/adsbexchange and mikenye/planefinder reports aircrafts with no issues (which means no connection issues etc)

docker-piaware enters unhealthy status almost everyday last messages are like the following:

[piaware] 2022/01/03 23:20:15 no new messages received in 3888 seconds, it might just be that there haven't been any aircraft nearby but I'm going to try to restart everything, just in case...
[piaware] 2022/01/03 23:20:16 faup1090 exited with SIG SIGHUP
[piaware] 2022/01/03 23:20:16 attempting to restart beast-splitter..
[piaware] 2022/01/03 23:20:16 can't restart beast-splitter, no services that look like beast-splitter found

Following the last message you need to restart the connection and data continues to be send

[billygr]

just pulled the latest container just in case

[billygr]

Created the following:

/etc/init.d/beast-splitter

#!/usr/bin/env sh

# "dummy" SysV-style init script, to allow piaware to restart beast-splitter.
# piaware calls SysV init script to restart beast-splitter (package/fa_services.tcl invoke_service_action).
# this script allows that, while still using s6-supervise.
# pkilling beast-splitter causes s6-supervise to restart it.


case "$1" in
    restart)
    pkill /usr/local/bin/beast-splitter
    ;;
    *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

exit 0

If it will work as expected i will fire a pull request

[billygr]

Unfortunately it tries to restart it using
invoke-rc.d beast-splitter restart
invoke-rc.d: could not determine current runlevel
invoke-rc.d: policy-rc.d denied execution of restart.

2022-01-08 19:32:01.600822930 [piaware] 2022/01/08 19:32:01 no new messages received in 3889 seconds, it might just be that there haven't been any aircraft nearby but I'm going to try to restart everything, just in case...
2022-01-08 19:32:02.603323782 [piaware] 2022/01/08 19:32:02 faup1090 exited with SIG SIGHUP
2022-01-08 19:32:02.604424458 [piaware] 2022/01/08 19:32:02 attempting to restart beast-splitter..
2022-01-08 19:32:02.626829558 [piaware] 2022/01/08 19:32:02 invoke-rc.d: could not determine current runlevel
2022-01-08 19:32:02.646426533 [piaware] 2022/01/08 19:32:02 invoke-rc.d: policy-rc.d denied execution of restart.
2022-01-08 19:32:02.646831807 [piaware] 2022/01/08 19:32:02 can't restart beast-splitter, no services that look like beast-splitter found
2022-01-08 19:32:12.669863659 [piaware] 2022/01/08 19:32:12 ADS-B data program 'dump1090' is listening on port 30005, so far so good
2022-01-08 19:32:12.670966224 [piaware] 2022/01/08 19:32:12 Starting faup1090: /usr/lib/piaware/helpers/faup1090 --net-bo-ipaddr localhost --net-bo-port 30005 --stdout --lat 38.028 --lon 23.858
2022-01-08 19:32:12.676913270 [piaware] 2022/01/08 19:32:12 Started faup1090 (pid 4524) to connect to dump1090
2022-01-08 19:32:32.648537083 [piaware] 2022/01/08 19:32:32 76261 msgs recv'd from dump1090 (0 in last 5m); 76248 msgs sent to FlightAware
2022-01-08 19:35:27.579919197 [piaware] 2022/01/08 19:35:27 mlat-client(23696): Receiver status: connected
2022-01-08 19:35:27.581058650 [piaware] 2022/01/08 19:35:27 mlat-client(23696): Server status: not synchronized with any nearby receivers
2022-01-08 19:35:27.582002013 [piaware] 2022/01/08 19:35:27 mlat-client(23696): Receiver: 0.0 msg/s received 0.0 msg/s processed (0%)
2022-01-08 19:35:27.583077209 [piaware] 2022/01/08 19:35:27 mlat-client(23696): Server: 0.0 kB/s from server 0.0kB/s TCP to server 0.0kB/s UDP to server
2022-01-08 19:35:27.584107923 [piaware] 2022/01/08 19:35:27 mlat-client(23696): Aircraft: 0 of 0 Mode S, 0 of 0 ADS-B used
2022-01-08 19:37:31.601968755 [piaware] 2022/01/08 19:37:31 76261 msgs recv'd from dump1090 (0 in last 5m); 76248 msgs sent to FlightAware

[mikenye]

It looks like the pkill syntax in my other "fake" init scripts was wrong.

I've fixed them, and added your script with the correct syntax.

Can you please re-test?

To do this, in your docker-compose.yml file, under the piaware service, can you please:

• comment out the existing image: or build: directives
• add build: https://github.com/mikenye/docker-piaware.git#issue_95
• run docker-compose build to build the image (this will take a while)
• on successful build, run docker-compose up -d to re-create the piaware container using the new image

Monitor, and let me know how you fare.

Thanks.

[billygr]

I have corrected them both on my repo but still not pushed it yet.

There is another change needed on the fa_services.tcl to ignore the invoce-rc.d. Still testing it (and of course on the dockerfile to include it on the build). Renamed it to txt in order to uploaded it
fa_services.tcl.txt

[billygr]

And a log file
2022-01-09 11:16:28.479957353 [piaware] 2022/01/09 11:16:28 Receiver location changed, restarting dump1090 and skyaware978
2022-01-09 11:16:28.480548348 [piaware] 2022/01/09 11:16:28 attempting to restart dump1090..
2022-01-09 11:16:28.482201983 [piaware] 2022/01/09 11:16:28 has_invoke_rcd: Running on docker ignoring invoke-rc.d
2022-01-09 11:16:28.482922070 [piaware] 2022/01/09 11:16:28 has_invoke_rcd: Running on docker ignoring invoke-rc.d
2022-01-09 11:16:28.483649416 [piaware] 2022/01/09 11:16:28 attempting to restart dump1090 using '/etc/init.d/dump1090 restart < /dev/null &'...
2022-01-09 11:16:28.489037853 [piaware] 2022/01/09 11:16:28 dump1090 restart appears to have been successful

Which means it looks like it worked yesterday but i will give it some more testing

[mikenye]

That's promising - thanks for the update. Please keep me updated.

[billygr]

You have a pull request, works for me without issues