Adding VIM3 (Verisilicon TPU) to Frigate #5892

RichardPar · 2023-04-01T19:00:53Z

Verisolicon TPU on the Amlogic A311D SBC (Khadas VIM3)

config.yml

  vim3:
    type: vim3

Bootup Log

2023-04-01 18:51:57.023060299  [2023-04-01 18:51:57] frigate.app                    INFO    : Capture process started for test: 371
2023-04-01 18:51:57.047566671  [2023-04-01 18:51:56] detector.vim3                  INFO    : Starting detection process: 365
2023-04-01 18:51:57.047576963  [2023-04-01 18:51:57] frigate.detectors.plugins.vim3 INFO    : Attempting to register VIM3 TPU
2023-04-01 18:51:57.050100088  Vx delegate: allowed_cache_mode set to 0.
2023-04-01 18:51:57.050634838  Vx delegate: device num set to 0.
2023-04-01 18:51:57.050644004  Vx delegate: allowed_builtin_code set to 0.
2023-04-01 18:51:57.050646088  Vx delegate: error_during_init set to 0.
2023-04-01 18:51:57.050648129  Vx delegate: error_during_prepare set to 0.
2023-04-01 18:51:57.050649921  Vx delegate: error_during_invoke set to 0.
2023-04-01 18:51:57.055184045  WARNING: Fallback unsupported op 32 to TfLite
2023-04-01 18:51:57.059300878  [2023-04-01 18:51:57] frigate.detectors.plugins.vim3 INFO    : TPU found

{"before": {"id": "1680375244.063041-ia7vkl", "camera": "test", "frame_time": 1680375250.884574, "snapshot_time": 1680375246.463054, "label": "person", "sub_label": null, "top_score": 0.83984375, "false_positive": false, "start_time": 1680375244.063041, "end_time": null, "score": 0.70703125, "box": [756, 882, 898, 1078], "area": 27832, "ratio": 0.7244897959183674, "region": [530, 500, 1110, 1080], "stationary": false, "motionless_count": 2, "position_changes": 1, "current_zones": [], "entered_zones": [], "has_clip": false, "has_snapshot": false}, "after": {"id": "1680375244.063041-ia7vkl", "camera": "test", "frame_time": 1680375251.251274, "snapshot_time": 1680375246.463054, "label": "person", "sub_label": null, "top_score": 0.83984375, "false_positive": false, "start_time": 1680375244.063041, "end_time": 1680375256.474941, "score": 0.5859375, "box": [619, 961, 969, 1079], "area": 41300, "ratio": 2.9661016949152543, "region": [557, 576, 1061, 1080], "stationary": false, "motionless_count": 0, "position_changes": 1, "current_zones": [], "entered_zones": [], "has_clip": false, "has_snapshot": false}, "type": "end"}

…IB only when needed on the platform.

netlify · 2023-04-01T19:01:03Z

✅ Deploy Preview for frigate-docs canceled.

Name	Link
🔨 Latest commit	`9cff2b2`
🔍 Latest deploy log	https://app.netlify.com/sites/frigate-docs/deploys/642c163a4ef3ef00081b3831

NickM-27

Thanks for the PR, few comments left. Also, you will need to update the documentation in this PR to reflect the detector, what hardware it runs on, and example config to get it up and running.

NickM-27 · 2023-04-01T21:20:36Z

docker/rootfs/lib/vim3/libArchModelSw.so

Is there any way that these can be downloaded in the docker file, as opposed to included in the repo?

That was an open question! .. is there a method to detect hardware being run on?

Hardware : Khadas VIM3 khadas@Khadas:~/git/frigate$

The output can determine if the files are needed or not

Additionally... where to host?

the VIM3_Frigate_modules.tar.gz are hosted on Gdrive
https://drive.google.com/file/d/1um67CKxJOybbzz6xE75RlyyDR-m2YsA3/view?usp=sharing

need to be unpacked in to /lib/vim3

NOTE : I did not want to touch the Dockerfile as my version has got the OV components commented out

Hmm, google drive is interesting. Curious what @blakeblackshear thinks

I dont like it! .. but looking for solutions :D

Where does that Google drive link originate?

It would be best to build them in a docker layer. If that takes way too long, precompiled binaries can be downloaded from a separate github repository, but it would to need have a transparent build process and versioned releases.

It would be best to build them in a docker layer. If that takes way too long, precompiled binaries can be downloaded from a separate github repository, but it would to need have a transparent build process and versioned releases.

Ideally.. yes - but I would also like an end to world hunger and peace on earth :D The Verisilicon code only compiles on AMD64 with a cross compiler; which is weird; but its not mine so I have no place to critique. The build takes about 20 minutes to complete if your tongue is at the right angle. (and that is on an i7

I will put the blobs in a seperate respitory - that leaves open the question, how can I detect if its a VIM3 board. The host has /proc/cpuinfo which shows VIM3 SBC in the Hardware field.. how can this be queried in the Docker container? (is the hosts /proc/ inherited within the Docker create process?

Richard

The image is built on a GitHub server (amd64) not on the end users device. This means there are a few options:

vim3 would either always be included in the default docker builds 2. Similar to tensorrt there would be a specific image for vim3 and that would be its own docker build process on top of the default image

There will be a separate dockerfile for this board altogether.

I believe there may be some guidance around 3 as the preferred way to go, seems we are going to be getting more and more community submissions for different detector / board types. Hopefully there will be more clarification here after 0.12 releases.

The image is built on a GitHub server (amd64) not on the end users device. This means there are a few options:

1. vim3 would either always be included in the default docker builds 2. Similar to tensorrt there would be a specific image for vim3 and that would be its own docker build process on top of the default image 2. There will be a separate dockerfile for this board altogether.

I believe there may be some guidance around 3 as the preferred way to go, seems we are going to be getting more and more community submissions for different detector / board types. Hopefully there will be more clarification here after 0.12 releases.

I am totally lost on using Git for Building stuff and all the shenanigans around that! Never done that before. I dont mind including the VIM3 with the default docker, but it wont scale when more boards are added - perhaps have a script to download new libraries/objects to a docker volume for persistence.

maybe a script can be executed at the start of Frigate (run /bin/check_platform_dependencies.sh) to run only once which checks the platform and which detectors are configured - and then downloads/installs them.

Done...
made a new repo to download binaries..
added script to install files
Added documentation

frigate/detectors/plugins/vim3.py

Merge branch 'dev' of github.com:RichardPar/frigate into dev

to the ldconfig path

NickM-27 · 2023-04-04T12:58:47Z

docker/install_deps.sh

@@ -75,6 +75,18 @@ if [[ "${TARGETARCH}" == "arm64" ]]; then
 libva-drm2 mesa-va-drivers
 fi

+# Check if the platform is a VIM3 and download TF delegate drivers for NPU


This would only work if the build was actually done on a VIM3 device, it would not include VIM3 support in the official builds.

I would suggest for now leaving this as is, the 0.12 build is being finished up and then new PRs can be merged so at that point we will have a better idea for the strategy of supporting boards like this.

Thanks...

I can change the line to
if [[ "${TARGETARCH}" == "arm64" ]]

instead?

RichardPar · 2023-04-28T07:34:40Z

There is no love for the platform.. it has a working NPU and there is no desire to get the hardware video decoder working

-abandoned-

NickM-27 · 2023-04-28T11:51:46Z

There's no reason to close this. As has been said on other issues, we're not looking to personally support all SBC detectors so working on an idea for community supported detectors but that will take some time.

This was of course pinned and meant to stay open.

RichardPar · 2023-04-29T06:19:58Z

With all due respect,

The board has a working NPU and there was no engagement to get FFMPEG to work with hardware video decode. I closed it as I didnt see the point of adding a SBC that was crap and not up to the job.
I have since got the ARM-NN working on the Khadas VIM4 board. (Video decoding is working awesomely on it)

blakeblackshear · 2023-04-29T11:14:25Z

I think we were assuming you meant there was a lack of engagement from us. It's clear now that's not what you meant.

RichardPar · 2023-04-29T17:25:35Z

I think we were assuming you meant there was a lack of engagement from us. It's clear now that's not what you meant.

:)

I am preparing a new pull request for the Khadas VIM4 board running the ARMNN in the GPU. So far its running nicely... but again, its the packages specific for the SBC that slow things down. (It may work with all MALI/OpenCL based boards though!)

Richard

savikko · 2023-08-09T09:39:25Z

Khadas has announced a while ago (https://www.khadas.com/post/june-2023-update) that they are releasing new 5.15 kernel which would include support for h264 hw decode.

Just as an extra info here.

savikko · 2023-11-14T13:57:50Z

And those images are now available for testing: https://forum.khadas.com/t/vim3-3l-ubuntu-22-04-linux-5-15-testing-images-are-available-now/20600

RichardPar added 2 commits April 1, 2023 19:45

Added VIM3 components - There has to be a smarter way to select the L…

2f810ed

…IB only when needed on the platform.

Merge branch 'blakeblackshear:dev' into dev

172cc4a

NickM-27 requested changes Apr 1, 2023

View reviewed changes

RichardPar added 5 commits April 2, 2023 07:25

Add documention on how to use

ed7532d

Sync

0db2089

Merge branch 'dev' of github.com:RichardPar/frigate into dev

Removed Binaries from Rootfs and will be moved to download repository

1ec3126

Removed file as its created in the install dependencies section

5a0a19d

Added download of VIM3 files with a platform check - Adds the lib

9cff2b2

to the ldconfig path

RichardPar requested a review from NickM-27 April 4, 2023 12:48

NickM-27 reviewed Apr 4, 2023

View reviewed changes

NickM-27 added the pinned label Apr 4, 2023

NickM-27 linked an issue Apr 5, 2023 that may be closed by this pull request

[Support]: Addition of Verisilicon TPU to Frigate - working PoC ; how to incorporate - [Diff included] #5890

Closed

RichardPar closed this Apr 28, 2023

Awei2020 mentioned this pull request Oct 28, 2023

[Snyk] Security upgrade axios from 1.3.5 to 1.6.0 Awei2020/frigate#7

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding VIM3 (Verisilicon TPU) to Frigate #5892

Adding VIM3 (Verisilicon TPU) to Frigate #5892

RichardPar commented Apr 1, 2023

netlify bot commented Apr 1, 2023 •

edited

Loading

NickM-27 left a comment

NickM-27 Apr 1, 2023

RichardPar Apr 2, 2023 •

edited

Loading

NickM-27 Apr 2, 2023

RichardPar Apr 2, 2023

blakeblackshear Apr 2, 2023

blakeblackshear Apr 3, 2023

RichardPar Apr 3, 2023

NickM-27 Apr 3, 2023

RichardPar Apr 3, 2023

RichardPar Apr 4, 2023

NickM-27 Apr 4, 2023

RichardPar Apr 4, 2023

RichardPar commented Apr 28, 2023

NickM-27 commented Apr 28, 2023

RichardPar commented Apr 29, 2023

blakeblackshear commented Apr 29, 2023

RichardPar commented Apr 29, 2023 •

edited

Loading

savikko commented Aug 9, 2023

savikko commented Nov 14, 2023

Adding VIM3 (Verisilicon TPU) to Frigate #5892

Adding VIM3 (Verisilicon TPU) to Frigate #5892

Conversation

RichardPar commented Apr 1, 2023

netlify bot commented Apr 1, 2023 • edited Loading

✅ Deploy Preview for frigate-docs canceled.

NickM-27 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RichardPar Apr 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RichardPar commented Apr 28, 2023

NickM-27 commented Apr 28, 2023

RichardPar commented Apr 29, 2023

blakeblackshear commented Apr 29, 2023

RichardPar commented Apr 29, 2023 • edited Loading

savikko commented Aug 9, 2023

savikko commented Nov 14, 2023

netlify bot commented Apr 1, 2023 •

edited

Loading

RichardPar Apr 2, 2023 •

edited

Loading

RichardPar commented Apr 29, 2023 •

edited

Loading