Reolink doorbell two-way talk via neolink (MUCH lower delay)

[bezmi]

Extending my comment which was originally posted on the go2rtc issue

Here are some detailed instructions to get reolink doorbell two-way talk working via QuantumEntangledAndy/neolink which offers much lower latency than the rtsp backchannel. It's pretty much the same latency as the reolink app. Obviously, this is highly experimental and might break things so use at your own risk!

First, some commands to install neolink and deps in the frigate docker container. see footnote

Make sure you enter the frigate container first: docker exec -it frigate bash

# run these in the frigate container

cd /config

# remove this line if you use frigate 0.14 beta, as it includes a go2rtc version that already works
# get the current latest go2rtc since backchannel audio with the
# exec source isn't supported in the version included with frigate
wget https://github.com/AlexxIT/go2rtc/releases/download/v1.9.3/go2rtc_linux_amd64

mv go2rtc_linux_amd64 go2rtc

# get the "latest" neolink, change the file according to you cpu arch
wget https://github.com/QuantumEntangledAndy/neolink/releases/download/v0.6.2/neolink_linux_x86_64_ubuntu.zip 

unzip neolink_linux_x86_64_ubuntu.zip 

mv neolink_linux_x86_64_ubuntu neolink

# download deps
apt update
apt remove -y libcairo2
apt install -y libcairo2=1.16.0-5
apt install -y \
  libgstrtspserver-1.0-0 \
  libgstreamer1.0-0 \
  libgstreamer-plugins-bad1.0-0 \
  gstreamer1.0-x \
  gstreamer1.0-plugins-base \
  gstreamer1.0-plugins-good \
  gstreamer1.0-plugins-bad

Now create the file /config/neolink/config.toml:

bind = "0.0.0.0"

[[cameras]]
name = "Doorbell"
username = "admin"
password = "password"
address = "doorbell_ip:9000"

and the file /config/neolink/neolink_talk.sh

#!/usr/bin/env bash

ffmpeg -fflags nobuffer -f alaw -ar 8000 -i - -f wav - | /config/neolink/neolink talk Doorbell -c config.toml --volume=1.0 -m -i "fdsrc fd=0"

Make it executable

chmod +x /config/neolink/neolink_talk.sh

Finally, edit the go2rtc streams section in your frigate config file, /config/config.yml:

streams:
  doorbell_sub:
  - ffmpeg:http:https://doorbell_ip/flv?port=1935&app=bcs&stream=channel0_sub.bcs&user=admin&password=PASS#video=copy#audio=opus
  - exec:/config/neolink/neolink_talk.sh#backchannel=1

The first entry is the regular stream, second entry exec:... is the backchannel magic.

note: I remove libcairo2 and install a different version to make gstreamer happy. I think this might be the cause of some errors in the frigate log, but I haven't noticed anything actually going wrong. Use at your own risk.

edit: use the full path to neolink executable in neolink_talk.sh

[reg98643t]

Your go2rtc release is amd64 but your neolink release is arm64, is that correct?

[bezmi]

nope, that's a typo from when I was testing on an arm system. Thanks for pointing it out, I've edited the post.

[x-cimo]

@bezmi I was able to get the above working on my frigate docker compose.

First issue I had was that go2rtc was in '/usr/local/go2rtc/bin/go2rtc' for me, so I have move the newer version there.

The second issue I had was that I needed to add the following to /config/neolink/neolink_talk.sh
cd /config/neolink

Otherwise it would not find the config file

[x-cimo]

also it seem to work better for me by using pcm rather than opus in the frigate config.

[bezmi]

Interesting - I followed the frigate docs which say that any go2rtc executable included in the /config directory will take precedence. Worked fine for me when I was on the frigate stable release. I'm using frigate beta now, so I just use the built-in go2rtc. I'll edit my post to put the full path to neolink to avoid any issues, thanks for the heads up!

[slatec]

Does this overcome the 2way audio not working issue introduced in firmware v3.0.0.3215_2401262240?

[bezmi]

I'm not sure, probably not as it uses the same protocol as reolink stuff. Might be worth a shot. If you're reasonably adept with linux, I can help you test by sending an mp3 to the doorbell from the commandline. This will give confirmation on whether it will work before you go through the hassle of doing the frigate config.

[bezmi]

If you follow the install instructions above, simply execute ffmpeg -i https://rhasspy.github.io/piper-samples/samples/en/en_US/lessac/high/speaker_0.mp3 -f alaw -ar 8000 -f wav - | ./neolink talk Doorbell -c config.toml --volume=1.0 -m -i "fdsrc fd=0" within the frigate container /config/neolink directory and it should play a test mp3 voice clip on the doorbell.

[spikked27]

Attempting to get this to run but receiving nothing but a loading screen when trying to start 2-way talk, if I try to run neolink_talk.sh within the container I get this:

Config and neolink_talk.sh match yours exactly minus the username/password/ip address

[slatec]

Looks like you tried to run a shell script through the python interpreter.

The "| ./neolink ... " part may not work. I used a full path for the executable /config/neolink/neolink and the config file /config/neolink/config.toml. Or do what @x-cimo suggested above and add "cd /config/neolink" near the top of the script.

[spikked27]

Ok changed the script to:

#!/usr/bin/env bash
ffmpeg -fflags nobuffer -f alaw -ar 8000 -i - -f wav - | /config/neolink/neolink talk Doorbell -c /config/neolink/config.toml --volume=1.0 -m -i "fdsrc fd=0"

Config is:

bind = "0.0.0.0"

[[cameras]]
name = "Doorbell"
username = "admin"
password = "redacted"
address = "http:https://192.168.0.112:9000"

(I played around with the address, adding http:https://)
Still just get a loading screen, nothing in the frigate debug logs that would indicate it's doing anything. I'm running 0.14 beta 3, gortc is the same version as you. Anything else I can try?

[x-cimo]

try this inside the neolink folder:
ffmpeg -i https://rhasspy.github.io/piper-samples/samples/en/en_US/lessac/high/speaker_0.mp3 -f alaw -ar 8000 -f wav - | ./neolink talk Doorbell -c config.toml --volume=1.0 -m -i "fdsrc fd=0"

It should play some words (about 10 sec) on the camera speaker.

[bezmi]

I'll try and update to the latest beta and post my setup for that sometime over the weekend. I'm not sure if it'll work with the http. @x-cimo has a good suggestion for how to test the connection between neolink and the doorbell. Also, make sure you're doing all the install and testing stuff from within the frigate container by executing docker exec -it frigate bash. My exact config is below:

bind = "0.0.0.0"

# not sure if this is needed when not using the rtsp in neolink
# but I have it in my config and haven't tested without.
bind_port = 1554

# camera config, obviously change to your values
[[cameras]]
name = "Doorbell"
username = "admin"
password = "password"
address = "192.168.8.61:9000"

[bezmi]

Further: if you're on v3.0.0.3215, it might not work because that broke 2-way audio through the reolink app and neolink uses the same protocol. If that's the case, reverting to the previous firmware should fix it.

[n3xus1]

I tried to get the neolink 2 way audio working on my Reolink doorbell.
I have 2 way audio working with Firmware v3.0.0.3308_2405271823 with frigate:0.14.0-beta4

I'm able to play the audio using the command
ffmpeg -i https://rhasspy.github.io/piper-samples/samples/en/en_US/lessac/high/speaker_0.mp3 -f alaw -ar 8000 -f wav - | ./neolink talk front_door -c config.toml --volume=1.0 -m -i "fdsrc fd=0"

But when i try and use the mic button in home assistant app no 2 way audio come out, also I notice that my stream is not as fast when using neolink.
I'm using Debian 12 with docker compose and a edge tpu (usb), I used the bookworm version of neolink.
I tried both the release you are using and the newer RC with no change.

here is my config

`root@frigate:/config/neolink# cat config.toml
bind = "0.0.0.0"

[[cameras]]
name = "front_door"
username = "admin"
password = "password"
address = "192.168.1.106:9000"`

`root@frigate:/config/neolink# cat neolink_talk.sh
#!/usr/bin/env bash

ffmpeg -fflags nobuffer -f alaw -ar 8000 -i - -f wav - | /config/neolink/neolink talk front_door -c config.toml --volume=1.0 -m -i "fdsrc fd=0"`

[n3xus1]

Okay so I did some more testing and the RC version is very slow and delayed with voice. I got both versions working but 6.2 is faster response time
I had to modify the path in the neolink_talk.sh for the config.toml location. It needed the full path ie /config/neolink/config.toml for it to work.
I have another question for you good folks, how to you get the 2 way audio to work when not on your local network.
I have home assistant setup on a FQDN and SSL setup and working, I can get 2 way working if i VPN in.
Does the frigate server need SSL setup as well?
Also my stream for the camera is on the main stream for it to work well and I think it would be better on the sub stream for less bandwidth. Could someone post up a working config for frigate

Thanks

[n3xus1]

more testing shows that 2 way audio works using the old method when not on the local network but not with neolink. It may be do to not installing ssl package. I'll try testing it out in the near feature