Use EdgeTPU dev board via network

[jpicht]

Describe what you are trying to accomplish and why in non technical terms
Use a Coral dev board with frigate via network.

Describe the solution you'd like
Frigate does not really work on the Coral dev board, but it's the only Coral you can reasonably buy right now. So I'd like to use another machine for everything but the inference and only send the images to the dev board via gigabit networking.

Describe alternatives you've considered
Buying a Coral USB stick for 300€

Additional context
I hacked a bit of networking into frigate as a POC. https://github.com/jpicht/frigate/tree/network

It works for me (surprisingly well), but it's probably not the most elegant solution.

[jpicht]

Would you be interested in merging functionality like this? If so, how you want this to be implemented? I'd be up to rewriting this to your specs.

It currently works like this:

The container is run on both the main machine and the dev-board. The main machine runs everything as normal, just with detector type ip.

Server-side

config.yml:

[...]
detectors:
  edgetpu:
    type: ip
    device: 192.168.2.12:5001
[...]

The IP detector prepares everything for the model, sends the numpy-array via TCP to the EdgeTPU dev-board and receives the resulting array.

EdgeTPU side

On the dev-board following Dockerfile is used to run the inference server only:

ARG arch=aarch64
FROM blakeblackshear/frigate:stable-${arch}

ENTRYPOINT []

CMD [ "python3", "-u", "-m", "frigate", "server" ]

config.yml:

serve:
  port: 5001
  detector:
    type: edgetpu
    device: ''

The ServerApp creates a single LocalObjectDetector, opens a TCP port and listens for connections.

For each connection it creates a thread that calls detect_raw for each numpy array received via the network connection and sends the resulting array back.

[NickM-27]

Similar things have been suggested before. The main concern is latency, what kind of inference times are there? Personally I'd imagine they're similar to local CPU inference times or maybe a bit higher than the performant CPUs

See #2864 (comment)

[jpicht]

I'm getting 55-60 ms. It's not optimized for latency at all - there's probably a lot that can be done.

[NickM-27]

Interesting, that is pretty good then.

I would personally suggest should this be accepted into frigate, probably best to abstract the parts that run on the dev board to its own project / docker container so the image can be smaller and also optimizations could be made without affecting the main application.

[Sarah-Connelli]

Beautiful!

If this could be wrapped-up into a proxy of sorts, it could potentially be deployed to multiple networked units, and then we could define multiple 'netdevice' type tpu's in the detectors section of the Frigate configuration, effectively giving us three types of detectors i.e. 'CPU'/'TPU'/'NET'.

Over time, the right skills could add more advanced technology such as RDMA etc., bringing those latencies down further.

@blakeblackshear - this would bring great flexibility to Frigate, please do consider this.