General chat for Rockchip users

[MarcA711]

This is intended as an exchange for Rockchip users. We can talk about problems setting up hardware acceleration, optimizing performance, which distribution works best etc. Moreover, I want to collect some information like which OS + board works or not as well as inference times on different SoCs. I hope that as many users as possible help and submit data. Finally, I would also like to compile a list of FAQ, common pitfalls and known issues.

Latest news

The current beta version of Frigate includes lots of improvements in both hardware video processing and object detection for Rockchip devices. I would appreciate any help testing these changes. However, note that this is a beta version, so use with caution.

In this version, the option core_mask of the rknn detector was replaced by num_cores. You don't pass a complicated bit mask any more but just the number of cores you want to use (e.g. num_cores: 3 for rk3588 SoCs).

Moreover, Frigate Rockchip uses yolo-nas instead of yolov8. Yolo-nas is considered superior to yolov8. It is not suggested to stick to yolov8, since it is not supported anymore. Just changing the model: path: to a yolov8 model will not work because Frigate does not implement post-processing for yolov8. However, keep in mind that yolo-nas is not suited for commercial usage, see this license.

There was a kernel bug in version 6.1.0-1018-rockchip and 6.1.0-1019-rockchip of ubuntu-rockchip's kernel. Upgrade to 6.1.0-1020-rockchip or above. To check your version, type:

$ uname -r
6.1.0-1019-rockchip

Docker image: ghcr.io/blakeblackshear/frigate:0.14.0-rc2-rk
Beta docs: https://deploy-preview-11419--frigate-docs.netlify.app/

Also see these notes for the beta release.

System compatibility

Frigate version	SoC	SBC	OS	VPU	NPU	reported by
0.14.0-058e9a3	rk3588	OPi 5 Plus 16GB	Ubuntu 24.04 by Joshua Riek	✅	✅	@MarcA711
0.14.0-b97e274	rk3588	FriendlyElec NanoPC-T6 8GB	Ubuntu 22.04 by Joshua Riek	✅	✅	@jtut1731 *
0.14.0-d646338	rk3588	OPi 5 Plus 32GB	Ubuntu 24.04 by Joshua Riek	✅	✅	@bugui2007 *
0.14.0-da913d8	rk3566	Orange Pi 3B 8GB V1.1.1	Ubuntu 24.04 by Joshua Riek	✅	✅	@johnp789 *

You can help complete this table by commenting these information:

// can be found under system metrics in the top left corner
Frigate version:
// yes or no
NPU (hardware object detection) working:
VPU (hardware video processing) working: 
// type `uname -a` in terminal
Kernel info:
// type `sudo cat /sys/kernel/debug/rknpu/version` in terminal
RKNPU driver: 
// name of distro and link to it (e.g. Ubuntu 24.04 by Joshua Riek, https://github.com/Joshua-Riek/ubuntu-rockchip)
Distro:
// output of `cat /proc/device-tree/compatible` in terminal
device tree:
// your SoC (e.g. rk3588s)
SoC:
// your SBC (e.g. Orange Pi 5 Plus 16GB)
SBC:

Inference times

All times are in ms.

Frigate version	SoC	SBC	num_cores	deci-fp16-yolonas_s	deci-fp16-yolonas_m	deci-fp16-yolonas_l	reported by
0.14.0-058e9a3	rk3588	OPi 5 Plus 16GB	3	25-30	35-40	45-50	@MarcA711
0.14.0-b97e274	rk3588	FriendlyElec NanoPC-T6 8GB	3	17-26	25-36	31-34	@jtut1731 *
0.14.0-d646338	rk3588	OPi 5 Plus 32GB	3	/	36	/	@bugui2007 *
0.14.0-da913d8	rk3566	Orange Pi 3B 8GB V1.1.1	1	96	190	/	@johnp789 *

You can help complete this table by commenting these information:

// can be found under system metrics in the top left corner
Frigate version:
// type `sudo cat /sys/kernel/debug/rknpu/version` in terminal
RKNPU driver: 
// your SoC (e.g. rk3588s)
SoC:
// your SBC (e.g. Orange Pi 5 Plus 16GB)
SBC:
// num_cores used for this test (see your config.yml, type 0 if not set)
num_cores:
// e.g. deci-fp16-yolonas_s: 25-35 ms
name of model: inference time

FAQ

Is there a Frigate add-on for Home Assistant with support for Rockchip hardware?

No. There are (at least) 3 ways to install Home Assistant: their OS (HAOS), their supervised script (HA Supervised) and their docker image (HA Container). Only HAOS and HA Supervised support add-ons (see this comparison for more details).

HAOS uses the mainline Linux kernel that currently lacks the necessary drivers for Rockchip hardware acceleration (see this table to see the progress of mainlining the rk3588). HA Supervised currently lacks the option to unmask paths (--security-opt systempaths=unconfined, see home-assistant/supervisor#4863). So there will be no add-on for Rockchip hardware until either HAOS or HA Supervised works.

The only way to install Home Assistant and Frigate with Rockchip support is to use HA Container. For details see the next FAQ "How do I install Frigate and Home Assistant?". Also note that you don't need Home Assistant to use Frigate. Frigate already has a UI that received a major overhaul in v0.14. Using Home Assistant with Frigate makes only sense if you already use HA for other services and prefer an all-in-one solution or if you want to trigger other actions for certain events.

How do I install Frigate and Home Assistant?

This is currently only possible using HA Container (see previous FAQ "Is there a Frigate add-on for Home Assistant with support for Rockchip hardware?" for details). Note that you can't install add-ons in HA Container (see this comparison for more details). However, most add-ons are also available as docker images, you just need to configure them manually.

Please note:

• You can use Frigate without Home Assistant. Frigate has a powerful modern UI that received a major overhaul in v0.14. However, possible reasons to use Frigate through HA are that you already use HA for other services and want to access everything in one place or that you want to trigger actions based on certain events.
• I try to keep the instructions below up to date. If some parts do not work or are outdated, please contact me here.
• I have deliberately avoided detailed explanations. Docker, Home Assistant, Mosquitto and Frigate are too complicated to explain here. This should only serve as a quick start, further explanations can be found in the documentation of the respective projects.

The instructions below assume, that you are in an empty directory with read and write permission.

mkdir -p frigate-data/config
mkdir -p mosquitto-data/config

touch mosquitto-data/config/mosquitto.passwd
sudo chown root:root mosquitto-data/config/mosquitto.passwd 
sudo chmod 0700 mosquitto-data/config/mosquitto.passwd

Now, create the files docker-compose.yml and mosquitto-data/config/mosquitto.conf and paste the contents below. The docker-compose.yml creates a docker network and adds all containers to it. This way the containers can communicate with each other. Instead of an IP address you can use the container_name of each container.

e.g. nano docker-compose.yml

services:
  frigate:
    container_name: frigate
    restart: unless-stopped
    image: frigate-stable:latest-rk
    shm_size: "400mb"
    security_opt:
      - apparmor=unconfined
      - systempaths=unconfined
    devices:
      - /dev/dri
      - /dev/dma_heap
      - /dev/rga
      - /dev/mpp_service
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./frigate-data/config:/config
      - ./frigate-data/media:/media/frigate
      - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    ports:
      - "5000:5000"
      - "8554:8554"
      - "8555:8555"
    networks:
      - mqtt

  homeassistant:
    container_name: homeassistant
    image: ghcr.io/home-assistant/home-assistant:stable
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./homeassistant-data/config:/config
    restart: unless-stopped
    ports:
      - "8123:8123"
    networks:
      - mqtt

  mosquitto:
    container_name: mosquitto
    image: eclipse-mosquitto:latest
    restart: always
    volumes:
      - ./mosquitto-data/config:/mosquitto/config
      - ./mosquitto-data/log:/mosquitto/log
      - ./mosquitto-data/data:/mosquitto/data
    ports:
      - "1883:1883"
      - "9001:9001"
    networks:
      - mqtt

networks:
  mqtt:

e.g. nano mosquitto-data/config/mosquitto.conf

persistence true
persistence_location /mosquitto/data/
log_dest file /mosquitto/log/mosquitto.log
listener 1883 0.0.0.0

allow_anonymous false
password_file /mosquitto/config/mosquitto.passwd

Now start just the mosquitto container and add two users for Frigate and Home Assistant. Remember the password that you choose. Afterwards, create the frigate-data/config/config.yml. I highlighted some lines that you might need to change.

docker compose up -d mosquitto
docker exec -it mosquitto mosquitto_passwd /mosquitto/config/mosquitto.passwd frigate
docker exec -it mosquitto mosquitto_passwd /mosquitto/config/mosquitto.passwd homeassistant
docker compose down

e.g. nano frigate-data/config/config.yml

detectors:
  rknn:
    type: rknn
    num_cores: 3

model:
  input_pixel_format: bgr

mqtt:
  enabled: true
  host: mosquitto
  user: frigate
  password: pass #change

record:
  enabled: true
  events:
    retain:
      mode: active_objects

ffmpeg:
  hwaccel_args: preset-rk-h264 # or preset-rk-h265, depends on your cam

cameras:
  my_cam:
    ffmpeg:
      inputs:
        - path: rtsp:https://user:[email protected]:554/cam #change
          roles:
            - detect
            - record

Now start Home Assistant and install HACS, afterwards start all containers:

docker compose up -d homeassistant
docker exec homeassistant bash -c "wget -O - https://get.hacs.xyz | bash -"
docker compose down
docker compose up -d

Finally, you can open Home Assistant in your browser using http:https://IP-of-your-server:8123. Go to Settings --> Integrations --> Add integration. There you can setup HACS, afterwards MQTT, afterwards Frigate (download it from HACS first). After installing HACS you should restart Home Assistant using docker compose restart in your terminal. During the setup of MQTT you are asked to enter the broker address mosquitto as well as the username homeassistant and password that you chose earlier. During the Frigate setup you need to enter the address http:https://frigate:5000.

You can stop your setup using docker compose down, start using docker compose up -d and restart using docker compose restart if you are in the directory where you set everything up. Now, you should study the docs of each project and adopt everything to your needs.

Known issues

Streams crash or recordings are missing

This is due to a hardware limitation that prevents the video processing hardware to access memory outside the first 4GB. This is potentially fixable, see rockchip-linux/mpp#560.

[wxuan1990]

I am currently using a N4100 soft router to run Frigate. When there is no one around, the page shows an inference speed of 20-30 milliseconds, but when there are people, it becomes 70-80 milliseconds. The actual feeling is that it takes two to three seconds for Home Assistant to get the information, which is too slow.

I plan to purchase an Orange Pi 5, so I am particularly interested in the following two points:

1.Is an Orange Pi 5 with 4GB of memory enough to run Frigate?
2.Can your Frigate running on Orange Pi 5 trigger related monitoring events within one second? Or is the effect similar to my current configuration?"

[MarcA711]

Hi,

1. I have 3 4K-Cams with 2 streams each (one full res for recording, one low res for detection), run Frigate + Home Assistant on my board and it uses around 4GB memory. docker stats --no-stream reports that Frigate uses around 2GB of memory. Home Assistant uses around 300MB. I guess 4GB could work, but it is close to the limit. You could add some swap, it shouldn't slow down your system too much if you use a M.2 SSD. But keep in mind that you can't upgrade the memory, if you want to use your board for other stuff later that requires more. For an Orange Pi 5 Plus the surcharge for additional memory is rather low 4GB: $108, 8GB: $129 and 16GB: $151... So I would definitely buy 8GB or 16GB.
2. I would say that Frigate detects a person within 1 second after being almost fully visible in the camera stream. However, in my case the camera stream is some seconds delayed compared to the real world. I am not sure if this delay comes from my camera or if it comes somewhere from the software. Also, I don't know how long it takes until Home Assistant gets the event. But I would say that it is not possible to reliably trigger events within one second after they happen in the real world. The camera buffers the video, streams it to frigate, frigate (or rather ffmpeg) buffers the video, then there are additional processing and detection steps, some additional delay and buffering... I don' know how much one can optimize everything to have as low delay as possible. Maybe @NickM-27 has some info here.

[wxuan1990]

Thank you for your reply. I have tested before, triggering the frigate/<camera_name>/motion event is quite fast, and Home Assistant (HA) will receive the information very quickly. I haven’t done precise testing, but I feel it’s within a second. However, compared to the motion event, I feel that the object event will be much slower. So I previously thought that as long as the object detection is faster, the total time might still be within a second, then it can barely cooperate with other smart devices. But now that I think about it, as you said, there are other factors that affect it, making it difficult to respond reliably within a second.

[blakeblackshear]

Are these models really 320x320? I thought the YOLO NAS models were 640x640.

[MarcA711]

Hey @blakeblackshear,
do you think it is worth supporting Frigate+ for the rknn detector?

One feature of the Rockchip platform is "Model Encryption" that could help to prevent license violation. However, I haven't used it so far and don't know how it works. But I thought it might be interesting.

[blakeblackshear]

Since the model is trained on 640x640, what does changing the input_size to 320x320 actually do?

I'm considering adding YOLO-NAS as an available architecture for Frigate+ models. The models would then be compiled to ONNX format and that would enable broad support. Since they would be trained from scratch on a custom dataset, I can use them commercially.

[blakeblackshear]

Also, that export script in the README is a little different than the one on this page of the docs: https://docs.deci.ai/super-gradients/latest/documentation/source/models_export.html

[MarcA711]

I am not quite sure how pytorch adjusts the model to different input sizes. Here is the visualization of the first few layers of yolo-nas s 640x640 (left) and 320x320 (right):

I didn't notice any difference comparing both models (however, I just tested detection on a few images).

Also, that export script in the README is a little different than the one on this page of the docs: https://docs.deci.ai/super-gradients/latest/documentation/source/models_export.html

I wasn't aware of this function. I will have a look how it works and if the exported models are better.

[blakeblackshear]

I'm not sure either. I would guess that specifying a size other than the size from training would just add a step that upscales the data to the original input, but I don't really know.

[jtut1731]

Frigate version: 0.14.0-b97e274
NPU (hardware object detection) working: yes
VPU (hardware video processing) working: yes
Kernel info: Linux frigate 5.10.160-rockchip #31 SMP Mon Feb 12 15:49:56 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
RKNPU driver: RKNPU driver: v0.9.2
Distro: (not sure exactly which release...installed some time back, https://github.com/Joshua-Riek/ubuntu-rockchip)
device tree: friendlyelec,nanopc-t6rockchip,rk3588
SoC: rk3588
SBC: FriendlyElec NanoPC-T6 8GB

[jtut1731]

Frigate version: 0.14.0-b97e274
RKNPU driver: RKNPU driver: v0.9.2
SoC: rk3588
SBC: FriendlyElec NanoPC-T6 8GB
num_cores: 3
deci-fp16-yolonas_s: 17-26 ms
deci-fp16-yolonas_m: 25-36 ms
deci-fp16-yolonas_l: 31-34 ms

[bugui2007]

Frigate version: frigate:dev-d646338-rk
NPU (hardware object detection) working: yes
VPU (hardware video processing) working: yes
Linux isis 6.1.0-1014-rockchip #14-Ubuntu SMP Mon May 27 18:03:24 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
Kernel info:Linux isis 6.1.0-1014-rockchip #14-Ubuntu SMP Mon May 27 18:03:24 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
RKNPU driver: RKNPU driver: v0.9.6
Distro:Joshua-Riek/ubuntu-rockchip
device tree: rockchip,rk3588-orangepi-5-plusrockchip,rk3588
SoC: rk3588
SBC:Orange Pi 5 Plus 32GB
deci-fp16-yolonas_m: 36ms

[jtut1731]

I have tried all deci-fp16-yolonas_* models. Trying to move from coral-based system. All of these models seem to weigh heavily on "cat". Same config (except for model and detectors). In 0.13.2 coral system, dogs were dogs, and people were people. Now dogs are always cats and people are sometimes cats. Rarely are there any actual cats. Are there any other models to try with the rockchip decoder? Should I just remove cat from object detection and see if that improves classification?

[jtut1731]

Thanks! Adding a labelmap fixed the issue with the cat/dog. I've only seen the cat/people issue once or twice. Appreciate the help!

[MarcA711]

Hey @NickM-27,
I have a question in order to fix the issue mentioned above:
Is it safe to change config.model.labelmap_path from within the rknn.pys init function? Similar to how I set config.model.model_type here:

frigate/frigate/detectors/plugins/rknn.py

Line 42 in cf4517c

config.model.model_type = model_props["model_type"]

Or will this not work (or even result in a race condition, where the labelmap is sometimes loaded before and sometimes after the change)? Thank you!

[NickM-27]

See

frigate/frigate/detectors/detector_config.py

Line 80 in cf4517c

def check_and_load_plus_model(

I think it should be okay

[blakeblackshear]

Im not sure I would do that programmatically. Probably best to put it in the docs and have users reference a different labelmap path in their config. I am looking at YOLONAS for Frigate+, so I wouldn't recommend baking an opinionated approach to the labelmap in the code.

[MarcA711]

I would only set this, if the user uses the rknn model that comes with frigate. If the user provides his own model, the default value labelmap_path: /labelmap.txt will be still set and the user is able to override it. Also, I will just override the labelmap_path, if the default value /labelmap.txt is set. So, if the user provides his own labelmap_path: /config/my_labelmap.txt I won't override it, even if the user uses e.g. the yolonas.rknn model that comes with frigate.

[jtut1731]

With the known issue of 8GB and 16GB rockchip boards having crashes, isn't this really a non-starter for frigate until this is fixed?

[MarcA711]

Well, my 16gb system runs multiple days without this issue. In my setup, it occurred just a few times. But my setup uses around 4GB and rarely more. I think if you run more stuff that requires more memory, you will likely run into this issue more often.

Also, in order to reduce memory usage, you can set scale_rkrga=async_depth=0 as preset (see here. But this won't fix the problem.

[jtut1731]

At first, I figured this was the causing my issues with recordings, but then I found "Problem in 0.14.0 beta1... frigate stop recording every hour....needds to restart.. #11642". However, even if you aren't using more than 4GB, I would suspect that there is nothing telling the kernel to only use the first 4GB. It would seem that until they can fix the issue, it would be good if there is some kernel setting that would force it to only recognize the first 4GB. Do you know if this exists, or is my thinking flawed?

[MarcA711]

AFAIK, mpp requests an address within the first 4gb. The crash only happens, when there is no more space left.

[maxscheel]

With the beta v3 image, when restreaming 1024x768 mjpg stream from an esp32 camera module to m264 I'm getting about I'm 20% cpu usage on my rk3588. Feels like go2rtc ffmpeg is not using hardware acceleration yet.
Is restream still the recommended way of interfacing mjpg streams?

[MarcA711]

Yes, go2rtc doesn't support hardware acceleration for the FFmpeg fork used in frigate. See here for more information AlexxIT/go2rtc#979

However, you can use go2rtc's exec command like this:

go2rtc:
  rtsp:
  streams:
      MyCam: exec:ffmpeg -hide_banner -allowed_media_types video+audio -fflags nobuffer -flags low_delay -timeout 5000000 -rtsp_flags prefer_tcp -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -i rtsp:https://user:[email protected]:554/MainStream -c:v h264_rkmpp -g 50 -bf 0 -an -rtsp_transport tcp -f rtsp {{output}}

This transcodes h265 to h264 using hw accel and streams it to go2rtc.

[maxscheel]

Thanks for that! That should work with mjpeg as input stream as soon as their ffmpeg fork supports mjpeg decoding via mpp. Looks like that's on their to-do list.

[jemershaw]

I'm not sure if anyone knows but should the rknn detector just keep incrementally increase memory usage? It seems to go from 1% to about 8% usage after a day and then will flush back to 4% usage. Is this set somewhere in frigate? If I use a coral usb it stays around 2% usage so I wasn't sure if there was a reason it spikes so much higher vs the coral usb.

[MarcA711]

Maybe there is a memory leak in the rknpu library. I haven't noticed this before but I will check if I can reproduce the issue.

[johnp789]

Frigate version: 0.13.2-69d9a261
NPU (hardware object detection) working: yes
VPU (hardware video processing) working: no
Kernel info: Linux bda143f4e4b3 6.1.0-1021-rockchip #⁠21-Ubuntu SMP Mon Jul 29 03:52:32 UTC 2024 aarch64 GNU/Linux
RKNPU driver: v0.9.7
Distro: Ubuntu 24.04 server by Joshua Riek, https://github.com/Joshua-Riek/ubuntu-rockchip
device tree: rockchip,rk3566-orangepi-3brockchip,rk3566
SoC: rk3566
SBC: Orange Pi 3B 8GB V1.1.1

[MarcA711]

Hey, thank you for posting your specs!

VPU (hardware video processing) working: no

May I help you to get HW acceleration running? I guess it will work in the new frigate v14.

[johnp789]

Sure, I would like to try to get it working. With ghcr.io/blakeblackshear/frigate:0.14.0-rc2-rk I get

frigate  | 2024-08-05 11:30:56.935396179  [2024-08-05 11:30:56] detector.rknn                  INFO    : Starting detection process: 403
frigate  | 2024-08-05 11:30:56.936454339  Process detector:rknn:
frigate  | 2024-08-05 11:30:56.945069244  Traceback (most recent call last):
frigate  | 2024-08-05 11:30:56.945455991    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate  | 2024-08-05 11:30:56.945468824      self.run()
frigate  | 2024-08-05 11:30:56.945608824    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
frigate  | 2024-08-05 11:30:56.945616990      self._target(*self._args, **self._kwargs)
frigate  | 2024-08-05 11:30:56.945720240    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
frigate  | 2024-08-05 11:30:56.945726948      object_detector = LocalObjectDetector(detector_config=detector_config)
frigate  | 2024-08-05 11:30:56.945817947    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
frigate  | 2024-08-05 11:30:56.945822906      self.detect_api = create_detector(detector_config)
frigate  | 2024-08-05 11:30:56.945907780    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
frigate  | 2024-08-05 11:30:56.945912155      return api(detector_config)
frigate  | 2024-08-05 11:30:56.945995571    File "/opt/frigate/frigate/detectors/plugins/rknn.py", line 39, in __init__
frigate  | 2024-08-05 11:30:56.946001404      model_props = self.parse_model_input(config.model.path, soc)
frigate  | 2024-08-05 11:30:56.946116320    File "/opt/frigate/frigate/detectors/plugins/rknn.py", line 83, in parse_model_input
frigate  | 2024-08-05 11:30:56.946120404      if "/" in model_path:
frigate  | 2024-08-05 11:30:56.946634901  TypeError: argument of type 'NoneType' is not iterable
frigate  | 2024-08-05 11:30:56.947398771  Exception ignored in: <function Rknn.__del__ at 0xffff89179d30>
frigate  | 2024-08-05 11:30:56.947546353  Traceback (most recent call last):
frigate  | 2024-08-05 11:30:56.947659519    File "/opt/frigate/frigate/detectors/plugins/rknn.py", line 62, in __del__
frigate  | 2024-08-05 11:30:56.949162760      self.rknn.release()
frigate  | 2024-08-05 11:30:56.949484174  AttributeError: 'Rknn' object has no attribute 'rknn'

[MarcA711]

Could you post your config.yml?

Do you have something like this in it:

model:
  path:

Delete the empty path line or specify a correct model like: path: deci-fp16-yolonas_m

[johnp789]

Here is my config after adding a path element. I didn't have it at all before.

detectors:
  rknn:
    type: rknn
model:
  input_pixel_format: bgr
  path: deci-fp16-yolonas_m
objects:
  track:
    - person
    - car
    - dog
    - cat
    - bicyle
    - motorcycle
    - airplane
    - bus
mqtt:
  enabled: false
record:
  enabled: true
  events:
    retain:
      mode: active_objects
ffmpeg:
  hwaccel_args: preset-rk-h264 # or preset-rk-h265, depends on your cam
cameras:
  porch:
    ffmpeg:
      inputs:
        - path: rtsp:https://user:[email protected]/some-cam
          roles:
            - detect
            - record
version: 0.14

Frigate runs now, and pgrep -al ffmpeg shows that ffmpeg is running with -hwaccel rkmpp, -hwaccel_output_format drm_prime, and -vf scale_rkrga. In top the CPU is between 81% and 85% idle.

$ sudo cat /sys/kernel/debug/rkrga/load
num of scheduler = 1
================= load ==================
scheduler[0]: rga2
	 load = 7%
-----------------------------------
	 process 20: pid = 249918, name: ffmpeg [many parameters...]

[MarcA711]

Yes you are right there is an issue. Normally Frigate should default to deci-fp16-yolonas_n if there is no path specifies, but it doesn't. I will fix this. Thank you for reporting.

[johnp789]

Frigate version: 0.13.2-69d9a261
RKNPU driver: v0.9.7
SoC: rk3566
SBC: Orange Pi 3B 8GB V1.1.1
num_cores: 1
(not sure, model included with marca711/frigate:latest-rk): 62-64 ms

[johnp789]

Frigate version: 0.14.0-da913d8
RKNPU driver: v0.9.7
SoC: rk3566
SBC: Orange Pi 3B 8GB V1.1.1
num_cores: (N/A for rk3566)
deci-fp16-yolonas_m: 190 ms
deci-fp16-yolonas_s: 96 ms

[MarcA711]

It seems that there is an incompatibility between the yolov8 models from Frigate v13 and newer kernels. This results in this error:

2024-08-06 17:31:22.543948520  E RKNN: [17:31:22.543] failed to submit!, op id: 7, op name: Add:/model.2/m.0/Add, flags: 0x5, task start: 1390, task number: 6, run task counter: 0, int status: 0, please try updating to the latest version of the toolkit2 and runtime from: https://console.zbox.filez.com/l/I00fc3 (PWD: rknn) 

I recommend upgrading to Frigate v14 since yolov8 is not supported anymore. If you encounter this problem in Frigate v14, please reply to this comment.

This issue was initially reported here: nyanmisaka/ffmpeg-rockchip#95

[abratchik]

@MarcA711 Hi Marc, thanks for escalating this issue. I tested on the v14 rc2 - same result, reported here. It doesn't seem to be the problem related to the frigate version. Most likely the problem is between the RKNN toolkit and the model as suggested here.

[MarcA711]

Most likely the problem is between the RKNN toolkit and the model as suggested rockchip-linux/rknn-toolkit2#320.

Yes, this is also what I wrote above: It seems that there is an incompatibility between the yolov8 models from Frigate v13 and newer kernels.

However, upgrading frigate will also upgrade to newer models and eventually solve the problem. In your case, you have an old kernel with old rknn driver (v0.9.3). You should definitely upgrade to a newer version. I recommend this os: https://github.com/Joshua-Riek/ubuntu-rockchip

[abratchik]

Hi @MarcA711 , @ALL, FYI: I tried the 14.0 release on the 6.1 image from here.

Linux ubuntu 6.1.0-1021-rockchip #21-Ubuntu SMP Mon Jul 29 03:52:32 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Good news - inference is working on one camera with good speed. I did not try with more cameras yet, this is planned.

Frigate version: 0.14.0-da913d8
RKNPU driver: v0.9.7
SoC: rk3588
SBC: Orange Pi 5 Plus 16GB
num_cores: 8
deci-fp16-yolonas_m: 57-60 ms
deci-fp16-yolonas_m: 40-45 ms

Still I was expecting better detector performance. My other setup is running on the pretty old Intel Core i7-6700K and inference there on ov detector is 10-20 ms. Given that the Rk3588 has the hardware acceelerator with 3 NPU cores and also hardware decoding of h.265 stream, I was expecting it to be at least on par with the old Core i7.

Many thanks for your help and advice!