RockChip 3588(S) Support

[jonadis]

Since Frigate takes advantage of the Google Coral TPU, and those seem to be hard to find, plus for the most part it has to be an add-on to an existing Linux box that runs Frigate (via USB, M.2, etc) ... I submit for consideration adding support for the integrated NPU in the RockChip 3588 family of SoC. Of course I barely understand the difference between a TPU and an NPU but it seems like, at least on the surface, that they could be both employed to serve similar function. In addition the 3588 has support for H.264/H.265 encoding and decoding (though driver support may be lacking at the moment), it seems like SBCs like the Orange Pi 5 that feature the 3588 have the potential to make an efficient all-in-one Frigate solution.

[NickM-27]

I have a draft PR where I am investigating adding support and exploring how viable RockChip will be for detection #3626

[NateMeyer]

I think there may be different flavors of the RKNN detector to implement.

It looks like the RKNN toolkit supports a usb-attached RockChip SoC (RK1808-based Toybrick USB stick) and can use it for inference from a PC, this is only supported on Amd64.

Additionally you should be able to run Frigate locally on a RockChip SoC such as the RK3588 using the RKNPU2 library, which would only be in the arm64 image. Likewise, there are some older parts from RockChip, such as the RK3399, that use the v1 RKNPU library that would likely be a different detector implementation.

I have a Rock 5 with the RK3588 coming soon that I'm hoping to start playing with the RKNPU2 library on.

[NickM-27]

Thanks for the info, makes sense that it's separate based on what I've looked at so far.

[podarok]

Here are packages for rk3588 for orange pi 5
https://github.com/orangepi-xunlong/rk-rootfs-build/tree/rk3588_packages_jammy/ffmpeg

[podarok]

#5733 added armnn detector, please test

[RichardPar]

I have the RK3588 Rock5b preliminary working inside a Frigate docker container - I hope to add it to Frigate/python soon

`root@dceff5be2161:/rknn-toolkit2/rknn_toolkit_lite2/examples/inference_with_lite# python3 test.py
--> Load RKNN model
done
--> Init runtime environment
I RKNN: [12:34:21.289] RKNN Runtime Information: librknnrt version: 1.5.0 (e6fe0c678@2023-05-25T08:09:20)
I RKNN: [12:34:21.289] RKNN Driver Information: version: 0.8.2
I RKNN: [12:34:21.289] RKNN Model Information: version: 4, toolkit version: 1.5.0+1fa95b5c(compiler version: 1.5.0 (e6fe0c678@2023-05-25T16:15:03)), target: RKNPU v2, target platform: rk3588, framework name: PyTorch, framework layout: NCHW, model inference type: static_shape
done
--> Running model
resnet18
-----TOP 5-----
[812]: 0.9996760487556458
[404]: 0.00024927023332566023
[657]: 1.449744013370946e-05
[466 833]: 9.023910024552606e-06
[466 833]: 9.023910024552606e-06

done
root@dceff5be2161:/rknn-toolkit2/rknn_toolkit_lite2/examples/inference_with_lite#`

[wilb]

How is the above getting on? Just ordered myself an Orange Pi 5 with the hope of using it for Frigate - more than happy to do some testing on my setup if I can be of help.

[RichardPar]

I managed to get it working; but there seemed to be a dependency issue between the NPU and the default python Logger. I was slowly removing the logger and had a frame working; but my experiments were just hacks as Python is not my comfort zone.

The other discussion thread
#6650 (reply in thread)