feat(WORK IN PROGRESS): Add ArmNN and rknn2 detectors. Add docs about orange pi 5 #5733
Conversation
✅ Deploy Preview for frigate-docs canceled.
|
|
Thanks. We will need to integrate the dependencies in the dockerfile. This won't be in the 0.12 release. We will target 0.13 for this. |
|
@blakeblackshear I'll check docker file. |
|
That will end up being a substantial amount of work as other dependencies may not be compatible with more recent versions. |
|
@blakeblackshear I was able to add dedicated apt repos in order to install dependencies within Docker container of frigate, so maybe we could add anyway Do we have any dev docs about docker configuration for a frigate? |
|
There aren't an developer docs specific to docker. You should be able to see how everything works by looking at the Makefile and Dockerfile in the repo. |
|
|
||
| ```yaml | ||
| ffmpeg: | ||
| hwaccel_args: -hwaccel drm -hwaccel_device /dev/dri/renderD128 -c:v h264_rkmpp |
There was a problem hiding this comment.
Can we make this an ffmpeg preset?
There was a problem hiding this comment.
Not yet, because we need to get specifically compiled ffmpeg for this board
There was a problem hiding this comment.
How big is the ffmpeg binary? We may be able to include both in the image.
But if this pattern starts to be a thing, we need to reconsider. One approach is to download the ffmpeg binary automatically to /data/ on container startup, or skip it if the SHA validation of the existing binary passes.
There was a problem hiding this comment.
That would be difficult for docker users since /data/ isn't mapped and we don't want to be pulling on each startup. I'm wondering how including both would work, how would frigate / go2rtc know which to use?
There was a problem hiding this comment.
On container startup we can relocate the binaries according to the Frigate config, for example.
There was a problem hiding this comment.
About /data, you are right. Maybe after #5219 the binary could be downloaded to /config.
There was a problem hiding this comment.
But wait a sec, even /data would be fine, I think. When containers are restarted in docker, they are not recreated, so the data is kept (different than add-ons where restart=recreate, but in that case /data is kept anyway).
There was a problem hiding this comment.
I see what you mean, that could definitely make sense. Perhaps ffmpeg acceleration support for these devices would make sense as a separate PR and leave this PR just for the detector addition
There was a problem hiding this comment.
But wait a sec, even /data would be fine, I think. When containers are restarted in docker, they are not recreated, so the data is kept (different than add-ons where restart=recreate, but in that case /data is kept anyway).
It is bad practice to write to storage which is not mapped to user storage and only contained in the container, so that is not an option
| @@ -15,6 +15,78 @@ ffmpeg: | |||
| hwaccel_args: preset-rpi-64-h264 | |||
| ``` | |||
|
|
|||
| ### Orange Pi 5 ( ArmNN ) | |||
|
|
|||
| Ensure you have installed | |||
There was a problem hiding this comment.
Can we make this an install script similar to the way TensorRT detector has?
There was a problem hiding this comment.
I'm afraid we need to stop overwhelming Dockerfile with if/if/else spaghetti and have Dockerfile / board / arch
There was a problem hiding this comment.
We just moved away from an approach like that to support multiarch builds with multiple detectors. I also don't see what that script would have to do with the Dockerfile
There was a problem hiding this comment.
There is no sense to have armnn related detector for x86_64 board. No need to install packages from rkmpp scope for raspberry PI board, no need to have rknpu libraries for raspberry pi
We should stop using raspberry pi PPAs for non raspberry PI boards
There was a problem hiding this comment.
As an architect I see us start decoupling frigate from Docker
frigate should become an app, which is used by specific Docker as part of the whole system, depending from the hardware available
There was a problem hiding this comment.
There is no sense to have armnn related detector for x86_64 board. No need to install packages from rkmpp scope for raspberry PI board, no need to have rknpu libraries for raspberry pi We should stop using raspberry pi PPAs for non raspberry PI boards
of course, frigate already has examples of this case like how there is only a tensorrt image for amd64. However, you may very well want to run the ArmNN detector alongside OpenVINO via NCS2, Coral, etc.
There was a problem hiding this comment.
Got it, makes sense @NickM-27
I'll check if I can add proper PPAs and script for installing proper ffmpeg for hardware acceleration on this board
|
|
||
| Also, for the CPU and GPU accelleration you should use `armnn` detector on this board [see](detectors.md) | ||
|
|
||
| Install packages [see tutorials](https://github.com/ARM-software/armnn/blob/branches/armnn_23_02/InstallationViaAptRepository.md) |
There was a problem hiding this comment.
Same as above, can this be an install script?
|
TODO - add https://github.com/rockchip-linux/rknpu2 support |
|
I was able to use rknpu2 using model I converted from tflite. Can check here https://github.com/gilankpam/frigate/tree/rk3588_ubuntu I had to modify the base image to use ubuntu one, when I install ffmpeg debs for debian, it didnt work. And for the detector code, mostly I just copied and pasted from yolov5 repo, still not sure what it does but it works lol. frigate logs |
Sounds awesome |
|
I see @gilankpam . You removed some dependencies and added your own plugin alongside what I created diff gilankpam@d39dc47#diff-791d51fadcffb7c2419b907f8d7155e0e708cd6b513a742e01fc94087a020402 Now we need to ensure this could be added to main repo for become part of the officially released frigate I guess we need your help @blakeblackshear @NickM-27 I'm working on getting ffmpeg compiled with mpp and other arm64 hardware accelerated libraries to make it universal for different boards, but with Docker we'd need your suggestions probably. |
|
@podarok this is the screenshot I also noticed that the score output by the rknn model is lower compared with |
|
@gilankpam that could be because of yolov5 model is bigger and could load CPU more. |
|
You may use armnn, rknn, and rknn2 detectors on your board What is the difference between rknn and rknn2? @gilankpam |
podarok
changed the title
|
@podarok its the same just testing if having 2 detector would increase the performance. Still not sure lol Anyway I will try to convert CPU tflite model to rknn and test. Thanks for the idea |
rknn2 NPU has 3 cores. |
|
Trying to get caught up, at this point what hardware would this PR enable? Is it only an entire board like the orangepi or would it support any type of USB accelerator? |
at this point we have not yet finished with PR, it's work in progress Plugins(detectors) seems like ready to be used if you have proper hardware and software installed @NickM-27 |
I'm asking this question to be able to give suggestions as the answer depends on how this new detector can be utilized. So, what all hardware variations are known that can take advantage of the new detector type(s) in this PR? |
|
@NickM-27 ALso all ARM64 + Mali GPUs will be supported by armnn detection plugin, created by me Which outperform CPU detection by a lot! |
|
Thanks, that's helpful. I'd say in that case the rknpu2 work will likely match how tensorrt is done in the dockerfile meaning thst a separate image will be built for that on top of the base frigate image The work for arm64 + mail GPU would (I think) fit into the base image just having the install_deps script installing the necessary deps for detection. Ffmpeg custom build will probably be a different discussion depending on what that looks like. |
|
Can provide more details this weekend if there are more questions. @felipecrs and @NateMeyer are also knowledgeable with this docker file work and may be able to provide suggestions |
|
I have a Rock 5B to run this on. I'll take a look through it and let you know what questions I have in the next few days! |
feat: Add mpp and ffmpeg compilation script for arm64
|
Just pushed mpp and ffmpeg compilation piece which helps to get rkmpp hardware acceleration for ffmpeg |
|
@4ib3r is there something I'm missing trying to get gstreamer work in @simon511000 Docker image #5733 (comment)? My OrangePi is barely keeping up with decoding full-res from cameras on CPU — it sits at 80C and I'm desperately looking for ways to shift decoding to GPU 😰 Happy to try any suggestions either for 
|
|
Ok, what I missed was that I needed combination of all these docker run --privileged --rm -it --entrypoint /bin/bash --group-add=110 --device /dev/dri/renderD128:/dev/dri/renderD128 --device /dev/rga:/dev/rgaWhere group-add is groupId of Now I see mpp plugin being initialized correctly! gst-inspect-1.0 | grep -i "video.*decoder"
mpp[5843]: mpp_info: mpp version: 0af9b5be author: xueman.ruan 2023-08-11 fix[h264d]: fix TSVC decode assert error.
mpp[5843]: mpp_info: mpp version: 0af9b5be author: xueman.ruan 2023-08-11 fix[h264d]: fix TSVC decode assert error.
mpp[5843]: mpp_info: mpp version: 0af9b5be author: xueman.ruan 2023-08-11 fix[h264d]: fix TSVC decode assert error.
mpp[5843]: mpp_info: mpp version: 0af9b5be author: xueman.ruan 2023-08-11 fix[h264d]: fix TSVC decode assert error.
dv: dvdec: DV video decoder
rockchipmpp: mppvideodec: Rockchip's MPP video decoder
flxdec: flxdec: FLX video decoder
vmnc: vmncdec: VMnc video decoder
mpeg2dec: mpeg2dec: mpeg1 and mpeg2 video decoder
theora: theoradec: Theora video decoderTrying to get the pipeline to work now |
|
I've been spending way too much time on this, but have had some success. I've just forked the latest dev and pushed build changes (thanks to everyone in this thread) which accomplishes the following:
I just got the custom ffmpeg for hardware decoding setup this evening and it's made a noticeable performance improvement. With 3-1080p cameras and a 2k camera, my Orange Pi 5 Plus was constantly at 85 degrees, but now it sits between 75 and 80. I've removed OpenVino support from the build, as I don't need it and it takes way too long to build on the Orange Pi. If you want to give this a try, this should get you up and running. Everything should be done on your Orange Pi 5 device and you need to be running Ubuntu 22.04:
I hope this works for others, but I can't say that I've run it on a clean machine, so I may have done something else to prep that I'm unaware of. Over the next couple of weeks, I plan on cleaning this up a bit, compatibility with other Linux builds, and trying to get the rknn detector working. But for now, I hope this gets some of you up and running. I'll try to get a docker image up sometime soon, too. |
|
Hey @kcembrey, thanks for posting your work! I also finally got However, recording stopped working, 10 second slices are malformed: frigate.record WARNING : Failed to probe corrupt segment /tmp/cache/cam5-20230831224139.mp4: 1 - b'[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaaf8e70bd0] moov atom not found\n/tmp/cache/cam5-20230831224139.mp4: Invalid data found when processing input\n'
frigate.record WARNING : Discarding a corrupt recording segment: /tmp/cache/cam5-20230831224139.mp4I've tried adding @kcembrey, do recordings work on your fork? |
|
As you're looking at cleaning this up, would it make sense to split this into multiple PRs? One for ArmNN that would apply to any Arm64 image, and one for the rk3588-related features? |
|
@artem-zinnatullin Yes, recording and detection are working great in my setup, although I have fewer cameras. I have seen that "moov atom not found" error on startup, but haven't looked into it yet. Doesn't seem to affect my recording. I haven't tried the jjm2473 repo for librga, as that's coming from this PPA, which I believe, uses the old official packages. I'll try some tweaks this weekend to make it friendly to more distributions. Agreed, @NateMeyer. I started setting this up for myself, but I would like to be able to contribute if I can. I'll probably split this into two branches with separate focuses. This is my first Docker project, so I'm finally getting comfortable with it. I think I have the skills to be useful for packaging the ArmNN install stuff, but I don't know if I'll ever get the rknn detector working without others' help. |
Hi @kcembrey, I tried building your image but I ended up with an error for box_priors.txt just want to confirm if this is what is needed? https://github.com/nnstreamer/nnstreamer-example/blob/main/Tizen.native/ObjectDetection/res/box_priors.txt |
|
Make sure to read https://deploy-preview-6262--frigate-docs.netlify.app/development/contributing-boards if working on this type of contribution |
|
@sylv3rblade Sorry about that. I thought I had removed all references to that, but my build was cached when I tested it. box_priors.txt is not currently required for this build. It was for a version of the rknn detector but not currently needed. I just updated the repo to remove that requirement and tested this time on my machine with the --no-cache flag and it was successful. |
|
@kcembrey |
|
I was able to build the image from your branch. Could share the config.yml? I have use the example files from your repo but the status is as follows: I see no hw acceleration. |
|
@rnoxy here's my docker-compose that's working, as long as you've compiled the image yourself using the code in my branch: @a2nt That error can be ignored, but I do think I've found a way to address it. I've been putting some work into cleaning this up and trying to get the GPU acceleration working. It's just nothing ready for use. There's a "Requested unknown backend GpuAcc" error on startup which tells me it's not using the full capability of the GPU. Maybe it's just a device that needs to be passed in Docker, but my understanding is that "Privileged" mode gives the container full access to all of the hardware, so access shouldn't be an issue. I'll try and get myself learned up on how the detector code works and get that GPU acceleration working this weekend, then post an update by Sunday evening. I'm a developer, but Docker and Python are brand new to me as of a couple of months ago, so I've got a steep learning curve. We'll get this going, though. |
I ran into that issue too, as far as I remember adjusting LD_PRELOAD and LD_LIBRARY_PATH environmental variables solved the issue. I added these to my docker-compose file: I remember I installed malirun which maybe installs some dependencies which are needed: |
|
I can confirm that on the rock5b with rk3588 armnn detector works using the cpu model. To summarize:
Using @kcembrey OrangePi5 branch with a little bit of modifications (thank you) and @hbiyik and ffmpeg-rk branch maintainer. CPU usage is around 5.8 - 6 for 4 x 4k streams and dmesg produces these logs Also, dmesg spits out so many buckets of the same log messages that it's hard to go through them. This makes me think that the VPU decoder is not utilized regardless of using hwaccel drm and hwaccel hevc or h264 encoding. |
|
I managed to build kc's image after a few tries , H264 & hevc decoding seems to be working but i was expecting less CPU usage , perhaps its due to the librga used in the build process? .. anyways.. im getting this warning when using RKNN detector W RKNN: [16:01:32.494] Output(Identity): size_with_stride larger than model origin size, if need run OutputOperator in NPU, please call rknn_create_memory using size_with_stride. |
|
Found repositories with pre-built binaries for ubuntu. Hopefully it will be helpful to create docker image |
Hi, fwiw the GpuAcc backend dynamically loads one of these libOpenCL.so libGLES_mali.so libmali.so, otherwise it will fail to register in ArmNN. You may need a symlink with one of those names linking to the actual libmali.so depending on its name on your device. The symlink should be in your LD_LIBRARY_PATH also. Francis. |
|
yes i forgot to mention i had to manually add the openCL dependencies.. you can find them in these repos: https://github.com/JeffyCN/mirrors/tree/libmali/lib/aarch64-linux-gnu |
|
|
I've merged all of the recent changes into my branch locally, and hope to push that soon, but I'm also trying to add features mentioned in the previous comments to get everything more optimized. I'll reply in here when I've pushed the new commits to the branch. |
|
Marc has pushed a commit that enables hevc and h264 encoders/decoders which are based on latest ffmpeg builds with rockchip support. His pull request has been merged to frigate main branch (dev). I'll be testing shortly. |
|
So with the closing of this issue, does that mean the dev branch (and I assume the next main release) will have full RK3588 support for video enc/dec and NN acceleration? I ask because the latter seems like the point of this issue and not covered by the PR that was mentioned, and the comments here thus far suggest its a bit of a "you can manually fudge things to make it work". |
|
@drunlade At least I work on it as much as I can. I hope to get everything working until the next big release. But I can't promise, since I don't have much time left. Yes, I plan to add full support for rknn NPU detector and FFmpeg hardware video de- and encode acceleration. However, I currently don't work on the ArmNN detector that was also implemented here. I used the ArmNN implementation for some weeks and it worked great. I think we should merge the ArmNN detector as well, since almost any SBC without an NPU could use this, including the popular Raspberry Pi 4 and the new Raspberry Pi 5. If no one implements it, will try to do it, after I finished all my work on the rockchip image (probably not before next year). |
|
By the way, basic NN detection support using rockchip NPU was already implemented in another PR before and is already merged in the dev branch. I currently work on adding some features to it. But the current implementation in dev is usable for detection and works great on my Orange Pi 5 Plus. Just clone dev branch and use "sudo make local-rk" to build the image. |
|
I closed this because the majority of work has been covered, there's been no code activity in many months, and in any case it would need to be refactored to fit in as a community supported build |
|
@MarcA711 Thank you so much for your work. The detector is working amazing for me on my Orange Pi 5+. @drunlade There's also a prebuilt container that works by setting the image property in your docker-compose file to I kept intending on working on this some more, but it was way over my head and work has been keeping me busy. I may revisit the armnn detector in the future (in a separate thread), but it's still not all that efficient, so not much of an improvement over CPU. The rk detector is way more efficient for rockchip devices, so I think the armnn detector would only benefit those without a separate NPU on their chip. Thanks so much for the work of everyone on this! |
@MarcA711 wonderfull job I had run "make local-rk" and build docker image successfully, But got errrors when run the docker image, can you share your docker image and docker-compose yaml file? |
|
Hey @zjd1988, you can use this official beta image: The beta docs mention all steps necessary to setup rknpu detector and hardware video acceleration. You should specifically read the rockchip specific sections: Rknpu detector: If there are still problems please open a a new issue and make sure to include the following information: |
|
@MarcA711 THX |
TODO: - add https://github.com/rockchip-linux/rknpu2 support
There is yolov5 support already in examples
I was able to add
armnndetectorSee docs https://github.com/ARM-software/armnn#arm-nn
Arm NN is the most performant machine learning (ML) inference engine for Android and Linux, accelerating ML
on Arm Cortex-A CPUs and Arm Mali GPUs.
Some proofs
During detection
Events screen
Frigate log
clinfo output