dump1090-mutability fails to startup fully #268

[Romeo-Golf]

Cross posting from: https://github.com/jprochazka/adsb-receiver/issues/268

[Romeo-Golf]

I did several fresh installs using the v2.4.0 dump1090-mutability image on Raspberry Pi 3. dump1090-mutability would run, but fail to load the config file and thus never listen on any port or any IP.

My Setup:
Device/system: Raspberry Pi 3
Wait for Network to boot: Enabled
Image/Distro: adsbreceiver image v2.4.0 dump1090-mutability
Static IP: Configured and enabled.
Packages installed: dump1090-mutability, dump978, ADS-B Portal, Flightradar24
Number of Dongles: 2
dump1090 device assignment: 0
dump978 device assignment: 1
dump1090 Listen on any IP: Enabled

RTL Test Output:

Found 2 device(s):
0: Realtek, RTL2838UHIDIR, SN: 00000004
1: Realtek, RTL2838UHIDIR, SN: 00000005

I had to constantly restart the dump1090-mutability service after every reboot to make it work.

I discovered an issue at: /etc/init.d/dump1090-mutability

Steps to fix:

Issue following command:
sudo nano /etc/init.d/dump1090-mutability

Change lines from:
# Required-Start: $remote_fs
# Required-Stop: $remote_fs

Change lines to:
# Required-Start: $local_fs $remote_fs $network
# Required-Stop: $local_fs $remote_fs $network

Save and Exit.

Reboot:
sudo reboot

Verify dump1090-mutability is listening on network:
sudo netstat --program --protocol=inet --tcp --wide --all

[mutability]

$remote_fs implies $local_fs, and $network is irrelevant to config files, so can you explain why this change is needed?

[Romeo-Golf]

Sorry just passing on the issue report; however I dont imagine $remote_fs is relevant in this use case so it may have been the addition of $network that resolved the users issue... will dump1090 only bind to loopback if started before the host gets a dhcp lease?

[mutability]

The report says it is a problem with the config file, not a problem with binding.

I think a second-hand crosspost is fairly pointless unless you can also involve the original reporter.

[ForestFeather]

If you're looking for an original reporter, I currently run on the CHIP platform - I had this same problem and it was fixed by adding $network to the INIT. On the Chip, dump1090 was trying to start before the network went active. By adding $network to the init file, I was able to reliably get dump1090 to restart after the device's restart multiple times in a row.

If you'd like someone who can reproduce this to do other testing, please let me know!

[mutability]

What did the failure look like when dump1090 started before $network?

[ForestFeather]

It originally manifested as being unable to access the USB device, even with the rules file installed. That, after hunting, turned out to be a dbus issue because it hadn't fully initialized and set up hardware access. When I added dbus as a dependency, it was then complaining about binding to network interface - and as two other services on the box were also complaining about starting before the wireless networking chip had fired up, I added $network, and then all seemed well. As a curiosity, I removed $dbus and it still worked just as well once the NetworkManager service resumed. My guess is the NetworkManager is dependent upon the dbus as well, hence $network behaving at that point.

[Romeo-Golf]

So it appears there are situations, such as host relying on DHCP to obtain their IP, where it cannot be guaranteed that the host will know all of its IP address's at least until networking has been started...

Thus if the startup service is not told to wait until networking has been established before starting dump1090 then there seems to be a risk that, even with the bind to all ip's option enabled, that at best it will not be listening on all of the IP's and at worst will not start at all.

So to put it another way, is there any need for dump1090 to start before networking? :)

[mutability]

I don't understand why a wildcard bind would fail in this case; can you explain?

[Romeo-Golf]

Sorry I may not have explained very well...

It seems that that on a device which has started networking, but is awaiting a DHCP lease, then the dump1090 daemon should be able to start with a wildcard bind regardless of what IP the device does or doesn't yet have...

But as reported by two separate users there are cases where, due to dump1090 only being instructed to wait for the local file systems to be available and thus launches before networking, the host network functionality is not yet in a state where it can even support a wildcard bind from dump1090.

But back to my question, is there any need or reason for dump1090 to start before networking? :)

[mutability]

I need an actual specific bug report: what fails? What is the error message reported by dump1090? I am not going to mess around randomly with dependencies in the hope that it fixes a bug that I do not understand the cause of. dump1090 can run just fine without networking (eg only outputting via json)

[Romeo-Golf]

Since @ForestFeather is the second user to report similar issues I asked him to report these directly to you and hope he will be able to provide you with the error message; however from the original thread he did feed back the follow:

"Okay, so here's the update - I set '$network' in dump1090-mutability's init file. That got him working,"

So while I full understand the desire not to make any changes until the cause of the issue with well understood, it would be useful to at least accept that attempting to start dump1090 before networking could lead to situations where at best the networking functionality of dump1090 will not work as expected and at worst will refuse to start.

So back to my question, which rather than "can dump1090 run without networking?" was actually "is there any reason for dump1090 to start before networking?" ;)

[mutability]

So while I full understand the desire not to make any changes until the cause of the issue with well understood

This is exactly it and so I need a specific bug report before I make changes.

[mutability]

FYI, it works fine with no network here:

$ sudo ifconfig eth0 down
$ sudo ifconfig wlan0 down
$ sudo ifconfig lo down
$ sudo ifconfig
  (no output)
$ ./dump1090 --net-only
Wed Feb  8 13:45:20 2017 GMT  dump1090-mutability v1.15-dev-308-gccd8fd5 starting up.
Net-only mode, no RTL device or file open.

(and if I bring lo back up then I can connect to port 30005 etc)

[mutability]

Reviewing http:https://refspecs.linuxfoundation.org/LSB_3.1.0/LSB-Core-generic/LSB-Core-generic/facilname.html gives a different interpretation to $network than the Debian docs, which is a bit unfortunate.

[ForestFeather]

All right, I'll reiterate what happened here and hopefully I can provide more information:

I am running the software on a CHIP - http:https://getchip.com/ namely, one from the alpha varieties/Kickstarter.

After reboots, dump1090-mutability was not restarting automatically. Upon investigation of the logs, it was an issue about being able to access the USB device for the RTL-SDR installed (In my case, a FlightAware ADSB stick). This was confusing because the rtl-sdr.rules file was installed under the necessary directory. However, I had noticed in the logs that several other services were on delay due to networking, one of which was dbus - which was a prerequisite for the networking service.

Now, the CHIP does ethernet/serial/etc-over-usb. It seems as though until the dbus process is fully active, the USB connection to the rtl-sdr is not quite available. I thought, hey, let's add dbus then as a requirement to the init file, but it still had issues complaining about binding while the network interface wasn't ready.

Finally seeing that the network system was causing issue not just to dump1090-mutability but two other services on my device (namely, two tracker reporting clients) I on a whim set $network as a requirement. Since this change, dump1090-mutability has started up without fail every restart.

Now, I'm not sure this is a wide bug, or one that's likely to be seen often - but as the prolifigation of these cheap SoC systems come about, it may come up more. I know that when I ran dump1090 on a raspberry pi and on a desktop system it had no issues, so it's likely something tailored to how linux is set up on these small SoC with local flash memory systems. I do know that $network fixed the problem in the init file, but it may be a symptom, rather than the disease here.

I can readily get more information, though I'd have to undo my fixes and take my tracker down for the process. But if it will help, please let me know what I can do to further investigate if you desire more concrete information.

[mutability]

@ForestFeather: what did dump1090 log when it failed to start when there was no $network dependency?

[ForestFeather]

Here was the original error related to being unable to start up:

Mon Feb 6 05:13:32 2017 CST dump1090-mutability v1.15~dev starting up. Using sample converter: UC8, integer/table path Found 1 device(s): 0: unable to read device details usb_open error -3 Please fix the device permissions, e.g. by installing the udev rules file rtl-sdr.rules Error opening the RTLSDR device: Permission denied
When I had $dbus as a requirement I get this, sometimes, and other times it'd start but with this:

r82xx_write: i2c wr failed=-9 reg=0f len=1 r82xx_init: failed=-9
and I'd seem not to see anything, though it'd be running. But when I set $network, it seems like it's started up pretty normally.

I'll grant that this may be a symptom and not the disease, but it's what I've found so far!

[mutability]

That sounds more like a problem with udev not having settled yet. What is the distro?

[ForestFeather]

I'm not sure it even has a name, but it is a customized version of Debian jessie.

[Romeo-Golf]

Hmm, so based on the LSB definitions below does it seem reasonable to say that in the majority of use cases dump1090 should consider $network to be a requirement?

$network
basic networking support is available. Example: a server program could listen on a socket.

[mutability]

Yeah, I'd take a pull request which did specifically that.

Doesn't help with the underlying udev problem though.

@ForestFeather: is the chip distro using systemd?

[ForestFeather]

It is using systemd, yes. In particular, it is using systemd version 230-7~bpo8+2.

Another thought, could this be related to the fact this is an arm system, maybe? If it's a udev problem though it should also be looked into as, again, I've a sneaking feeling systems like this will slowly become more popular.

[Romeo-Golf]

cheers and agree its more of a band-aide than a solution, but hopefully preparing for the 'wst case' implementations should allow the code to run on as many hardware platforms as possible :)

[mutability]

systemd has a systemd-udev-settle service which waits for all the udev stuff to finish (nothing left in the queue). I'm not sure if you can have a cross-dependency from an init.d script to a systemd service, but adding a Should-Start: systemd-udev-settle dependency might possibly help here.

Really, though, dump1090 should just provide a systemd service file. I have one in dump1090-fa that needs to get backported (the main stumbling block is where to do the config file parsing)

[ForestFeather]

If there's anything I can do to assist, please let me know.

[Romeo-Golf]

@ForestFeather this change has graciously been accepted so if you wouldn't mind testing the latest code on your C.H.I.P. hardware to confirm that that addition of $network, but not $local_fs, resolves the issues you observed that would be great :)