Add remote configurable output port

[nunojpg]

Hey! Not so small pull request here.

The idea is to add one more output mode but in a way that does no increase complexity.

So I added this dynamic port (default 30006), that when you connect gives you a hello JSON with the variant and version of dump1090, and then you send a string to select the stream you want, for example "s\n" for SBS.

The previous switchable mechanism of COOKED vs VERBATIM didn't allow to know safely which stream we were actually getting because there was no ACK and the default depended on dump1090 general config, which we might not control.

And this is to get a reliable way to obtain a verbatim stream, but also without any eventual remote results. We called it BEAST_ANTENNA.

I'm sure this can be improved and I wait for your comments.

[mutability]

I haven't looked at the changes yet (on mobile) but I'm a little reluctant to add yet another port and negotiation thing .. If I understand what you're after there are three things here : 1) select different output formats on a single port 2) have a positive acknowledgment that the settings you requested (eg verbatim mode) have been applied 3) get some metadata about the thing you have connected to I wonder if we can support 2 in a simpler way by having some settings request within the existing port 30005 framework that makes the receiver respond with its current settings. For 1 and 3 I would almost want to start looking at a whole new protocol (I know, adding a new protocol is less than ideal!) and try to fix a whole lot of the issues with the current protocols in one go. This has been on my list for a while - would you be interested in working together on something like that? Specifically for SBS / port 30003 style output, are you really using that for new stuff? What's the use case for it -- that protocol is extremely legacy and fragile and I'm reluctant to make any changes subs the current users are mostly legacy apps that are unlikely to get much in the way of updates and are easily confused by any change ..

[nunojpg]

Hi Oliver, I am only interested in Beast verbatim without remote data, ideally with a hello string to give you the version of the decoder.

AVR and SBS are there just to make it seem complete, but I have no use case for it.

This remote configuration logic is only to try to fit this in a scalable way. We can from this point add more data formats ("profiles") without depending on command line options.

A obvious future addition is IQ samples output, so that you can chain several demodulators on a single SDR. I would also be open to dump1090 just publishing the samples into a shm or fifo, while we have to see it that does not bring problems with custom kernels.

Let me know what requirements you have in mind and we can find a common solution.

[mutability]

For IQ we're probably better off implementing rtl_tcp (or SoapySDRRemote et al) support, since dump1090 is more naturally a consumer of that data, not a producer.

[mutability]

A possible counter-proposal (in the absence of a whole new protocol, which is too much to bite off right now)

• Add a new beast-protocol option l/L:
  <1A> 1 l - disable local-only option; forward remote messages normally (the default setting)
  <1A> 1 L - enable local-only option: only send messages that originated locally

• Add a new beast-protocol command, S:
  <1A> 2 S - request current connection status/metadata

Currently all beast-protocol commands are option-related and start with <1A> 1; the status request is a little different so maybe putting it in a different namespace like this helps? I'll need to check that existing beast-splitter and dump1090 don't choke on such a command, but from a quick look I think it's OK.

When (new) dump1090 hears a 'S' command it emits a status message, something like this perhaps:

<1A> S [NUL-terminated structured string that we can argue about the contents of]

where the string would contain something like: receiver name, receiver version, list of current option states:

<1A> S dump1090-fa;8.0~dev;CdfgijkVL <NUL>

(it could be json if you really wanted, but it's maybe too big an ask to expect clients to include a json parser)

Then you can do your thing by sending:

<1A> 1 V
<1A> 1 L
<1A> 2 S

and if/when you hear a <1A> S response you can verify what settings actually got set.

I think I can make this work with beast-splitter too without a ton of effort (here "local" means "came from a local serial device"). It already knows about verbatim mode.

Initially it might be simpler to have dump1090 implicitly turn on V if L is requested (much like your current implementation, where you can't request local-only corrected messages) but I'd request both because of beast-splitter and maybe future versions that don't combine the two.

(paging @wiedehopf for a readsb opinion)

[nunojpg]

Thanks. Not sure if you remember that you keep individual net_writer for the several times types of messages, for blocking of messages (by size and time). So my decoupling of net_service and net_writer was trying to achieve the option to dynamically instantiate a net_writer for a specific config.

For example, currently you have beast verbatim and cooked. If you can configure for remote on/off then now we have 4 net_writers for the 4 combinations.

So I think it's reasonable to either only add any configuration that we think is reasonable (like no remote), or to try to evolve this for a generic implementation that allows for arbitrary streams.

Also, I mentioned IQ samples directly out of dump1090 because I don't want to increase the complexity of the hosts. A microservice architecture (by adding rtl_tcp on top of everything else) handled by hosts is unfortunately very fragile, unlike for our automatically configured systems. So if dump1090 is there, and we can easily piggyback on it, it seems more robust.

[mutability]

Yeah, the network layer is pretty horrible and has needed a rewrite for years now -- switching to something more conventional where each connection has independent state is the way to go I think -- eventually.

How about we split this up into a few things:

1. adding the l/L option (if you're OK with that approach), implicitly also turning on verbatim, using the existing net_writer approach. We're mid-release-prep right now but this is straightforward enough that I can try to get it into this release.

2. adding the metadata-reporting thing. This is larger and will probably need to wait. I think we can probably squeeze it into the existing infrastructure with a little work though (directly write the response when the request comes in, rather than passing it through the net_writer buffer)

3. reworking all the network code to handle connections with independent state

[nunojpg]

1. adding the l/L option (if you're OK with that approach), implicitly also turning on verbatim, using the existing net_writer approach. We're mid-release-prep right now but this is straightforward enough that I can try to get it into this release.

So we continue with 2 streams only for Beast: cooked and verbatim.
On receiving a Local command we switch that connection to verbatim and;
Stop sending remote data to all clients on verbatim stream.

This?

[mutability]

No, I mean adding a 3rd stream type that is local+verbatim