Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jetson Nano Unknown decoder 'h264_nvmpi' #1175

Closed
KillahB33 opened this issue Jun 4, 2021 · 66 comments
Closed

Jetson Nano Unknown decoder 'h264_nvmpi' #1175

KillahB33 opened this issue Jun 4, 2021 · 66 comments
Labels

Comments

@KillahB33
Copy link

Describe the bug
Basically I am running this on a jetson nano, I have compiled ffmpeg with the jocover/jetson-ffmpeg package.
I have also confirmed that it is properly showing by running ffmpeg -encoders |grep 264 and I see it there.
Frigate say that it is an unknown decoder.

Version of frigate
0.8.4-5043040

Config file
Include your full config file wrapped in triple back ticks.

detectors:
  coral:
    type: edgetpu
    device: pci
ffmpeg:
    global_args: -hide_banner -loglevel warning
    hwaccel_args: []
    input_args: -c:v h264_nvmpi -vsync drop
    output_args:
        detect: -f rawvideo -pix_fmt rgb24
mqtt:
  host: 192.168.0.000
  user: **********
  password: *********
objects:
  # Optional: list of objects to track from labelmap.txt (default: shown below)
  track:
    - person
    - car
    - truck
    - dog
    - horse
cameras:
  front_left:
    ffmpeg:
      inputs:
        - path: rtsp:https://admin:[email protected]/Streaming/Channels/101
          roles:
            - detect
            - rtmp
            - clips
    width: 2688
    height: 1520
    fps: 1
    clips:
      enabled: true
    motion:
      mask:
        - 0,0,690,0,499,35,0,479
        - 2688,0,2688,277,2320,142,1623,0
        #Removing timestamp
        - 114,94,104,153,857,177,888,97
  front_right:
    ffmpeg:
      inputs:
        - path: rtsp:https://admin:[email protected]/Streaming/Channels/301
          roles:
            - detect
            - rtmp
            - clips
    width: 2688
    height: 1520
    fps: 1
    clips:
      enabled: true
    motion:
      mask:
        - 2688,0,2688,548,2504,316,2372,284,1810,156,1502,132,1179,118,482,135,146,180,0,243,0,0
        #Removing timestamp
        - 114,94,104,153,857,177,888,97
  back_left:
    ffmpeg:
      inputs:
        - path: rtsp:https://admin:[email protected]/Streaming/Channels/201
          roles:
            - detect
            - rtmp
    width: 2688
    height: 1520
    fps: 1
    motion:
      mask:
        - 191,496,83,55,836,69,1044,205,538,1304
  backyard:
    ffmpeg:
      inputs:
        - path: rtsp:https://192.168.0.000:554/live0
          roles:
            - detect
            - rtmp
    width: 1920
    height: 1080
    fps: 1
  garage:
    ffmpeg:
      inputs:
        - path: rtsp:https://192.168.0.000:554/live0
          roles:
            - detect
            - rtmp
    width: 1920
    height: 1080
    fps: 1

Frigate container logs

frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg sent a broken frame. memoryview assignment: lvalue and rvalue have different structures
frigate.video                  INFO    : backyard: ffmpeg process is not running. exiting capture thread...
ffmpeg.garage.detect           ERROR   : Unknown decoder 'h264_nvmpi'

Frigate stats

{
  "back_left": {
    "camera_fps": 0.0, 
    "capture_pid": 63, 
    "detection_fps": 0.0, 
    "pid": 44, 
    "process_fps": 0.0, 
    "skipped_fps": 0.0
  }, 
  "backyard": {
    "camera_fps": 0.0, 
    "capture_pid": 66, 
    "detection_fps": 0.0, 
    "pid": 55, 
    "process_fps": 0.0, 
    "skipped_fps": 0.0
  }, 
  "detection_fps": 0.0, 
  "detectors": {
    "coral": {
      "detection_start": 0.0, 
      "inference_speed": 10.0, 
      "pid": 36
    }
  }, 
  "front_left": {
    "camera_fps": 0.0, 
    "capture_pid": 57, 
    "detection_fps": 0.0, 
    "pid": 39, 
    "process_fps": 0.0, 
    "skipped_fps": 0.0
  }, 
  "front_right": {
    "camera_fps": 0.0, 
    "capture_pid": 58, 
    "detection_fps": 0.0, 
    "pid": 41, 
    "process_fps": 0.0, 
    "skipped_fps": 0.0
  }, 
  "garage": {
    "camera_fps": 0.0, 
    "capture_pid": 70, 
    "detection_fps": 0.0, 
    "pid": 56, 
    "process_fps": 0.0, 
    "skipped_fps": 0.0
  }, 
  "service": {
    "storage": {
      "/dev/shm": {
        "free": 1047.9, 
        "mount_type": "tmpfs", 
        "total": 1048.6, 
        "used": 0.7
      }, 
      "/media/frigate/clips": {
        "free": 35464.0, 
        "mount_type": "ext4", 
        "total": 62716.1, 
        "used": 24542.0
      }, 
      "/media/frigate/recordings": {
        "free": 35464.0, 
        "mount_type": "ext4", 
        "total": 62716.1, 
        "used": 24542.0
      }, 
      "/tmp/cache": {
        "free": 35464.0, 
        "mount_type": "ext4", 
        "total": 62716.1, 
        "used": 24542.0
      }
    }, 
    "uptime": 572, 
    "version": "0.8.4-5043040"
  }
}

FFprobe from your camera

Run the following command and paste output below

I don't think this is useful as my streams show fine without hwaccell

Screenshots
Video is all green with this option on

Computer Hardware

  • OS: JetPack Ubuntu
  • Install method: Docker Compose
  • Virtualization: N/A
  • Coral Version: m2 ekey
  • Network Setup: Wired

Camera Info:
Not relevant

@KillahB33
Copy link
Author

@htilly I see that you have this running so any insights on the device to pass on docker-compose or anything else would be greatly appreciated.

@blakeblackshear
Copy link
Owner

This will likely require adding an option to frigate's ffmpeg build.

@KillahB33
Copy link
Author

Ah ok so the package I mentioned would need to be used.
Would this be as part of the aarch64 package, or would we add a new build for the Jetson Nano?
If it's the latter I can contribute.

@blakeblackshear
Copy link
Owner

@KillahB33
Copy link
Author

And before I start work, master should be my base?

@blakeblackshear
Copy link
Owner

No. release-0.9.0

@KillahB33
Copy link
Author

KillahB33 commented Jun 4, 2021

Ok I'll rebase, and just to confirm for testing. I build Dockerfile.ffmpeg.aarch64, then update base with the ARCH ARG and my image, then build that, then build Dockerfile.aarch64.
The other images used in base I can use untouched.

@blakeblackshear
Copy link
Owner

Mostly. Take a look at the makefile. It may clarify some things.

@KillahB33
Copy link
Author

Oh cool, I haven't used that before.

@KillahB33
Copy link
Author

KillahB33 commented Jun 5, 2021

Hey @blakeblackshear, I am running into issues as it is looking for the following.
I am sure there is other stuff that I may have missed, should I continue with building it into aarch64 and use the jetson jetpack as a source image and pull this stuff in or should I look at making a separate image with jetpack as my base instead of vanilla ubuntu? My preference is for the latter but it's your call

set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,-rpath-link=/usr/lib/aarch64-linux-gnu/tegra")

set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,-rpath-link=/usr/local/cuda/lib64")
find_library(LIB_NVBUF nvbuf_utils PATHS /usr/lib/aarch64-linux-gnu/tegra)
find_library(LIB_V4L2 nvv4l2 PATHS /usr/lib/aarch64-linux-gnu/tegra)

I opened an incident on the other project to see if there is another way this can work and I'll let you know.

@blakeblackshear
Copy link
Owner

If it requires a separate based image, I would rather have a separate image like we do for amd64nvidia.

@KillahB33
Copy link
Author

KillahB33 commented Jun 7, 2021

Is there any libraries I can remove? Just trying to see if I can clean up the image a little but I am not sure if I will break anything.
For some context, the following is all that's required for it to work on jetson but I am not sure if the rest of the libraries are used for something else in frigate.
./configure --enable-nvmpi

I assume anything listed as a video encoder here I can remove?
http:https://underpop.online.fr/f/ffmpeg/ffmpeg-all.html.gz

@blakeblackshear
Copy link
Owner

I wouldn't remove anything.

@e1z0
Copy link

e1z0 commented Jun 12, 2021

https://github.com/e1z0/jetson-frigate i’ve do some research on it also.

@KillahB33
Copy link
Author

Thanks so much, gonna see if I can role this into what I am building here.

@blakeblackshear blakeblackshear added this to the 0.9.0 milestone Jun 17, 2021
@Esiravegna
Copy link

https://github.com/e1z0/jetson-frigate i’ve do some research on it also.

I'm afraid that, as is, the repo does build the ffmpeg binary, but doesn't work on the jetson nano as per this issue.

@KillahB33
Copy link
Author

KillahB33 commented Jun 28, 2021

So finally got back on this, hoping I'll be able to close this out thanks to e1z0's help.
Basically I built his image, the base image he is using is pretty huge in comparison so trying get the best of both and get this working.

@blakeblackshear blakeblackshear removed this from the 0.9.0 milestone Jul 2, 2021
@KillahB33
Copy link
Author

@blakeblackshear the patch file that's needed, where should it live in this repo? Should it just be in the docker folder or is there somewhere else more appropriate.
Also just realized the base image that e1z0 was using is actually fine so just gonna be a bit of rework to fit in this repo then should be finished.

@blakeblackshear
Copy link
Owner

I would put it in the docker folder.

@KillahB33
Copy link
Author

Came to update.
e1z0 was very kind to deploy an image which I got running but even with hardware acceleration on, my jetson was just getting hotter and not offloading as much as I assumed it would to the GPU.
I have since abandoned this approach and have migrated to proxmox to run it on my main server.
Sorry for the disappointing news, e1z0 has cloned the repo with his changes but as you can see in the issues the rest of us were struggling to rebuild the same image.

@0mdb
Copy link

0mdb commented Sep 27, 2021

@e1z0 @KillahB33 Is the built image hosted somewhere? I'm struggling to build jetson-frigate as well.

@blakeblackshear Do you think an official frigate aarch64nvidia for the Jetson Nano is in the cards? I have a m.2 Coral TPU on the way but if the NVEC, NVENC and potentially even the GPU can take care of decoding, the Nano + Coral could be a powerful hardware option.

@blakeblackshear
Copy link
Owner

Do you think an official frigate aarch64nvidia for the Jetson Nano is in the cards?

It's difficult for me to work through it without the hardware on hand. These ffmpeg builds are difficult to get right. I'm not sure it's a popular enough device for me to invest the time, but definitely open to PRs.

@ozett
Copy link

ozett commented Sep 28, 2021

the small jetson nano device is comparable cheap and AI powerfull on its own.
Its desinged for that. Additonalle Edge TPU would enhance this.
But it surely needs time and effort to make the frigate details stable and efficient on that device.
even in a docker-ish way. maybe i jump into this game, but i wished others had already spent this time.. 😄

@janbjorn
Copy link

janbjorn commented Sep 28, 2021

I've been looking for this as for some time since I'm running a couple jetson nanos. Is it possible to take inspiration from a similar project? https://github.com/roflcoopter/viseron/tree/master/docker/jetson-nano
I haven't had time to look into it but from a quick glance it seems they have ffmpeg compiled with gpu support for Jetson nano. I'm quite short on time right now so I'm not able to commit the develoment time for a PR, but I'd be happy to assist in testing...

@shbatm
Copy link

shbatm commented Sep 28, 2021

I'm interested in supporting any testing as needed; I have the same hardware setup as @KillahB33 (Jetson Nano + m2 Coral), and have messed around a little trying to get the hwaccel options going with @e1z0's repo but haven't made it very far yet.

@ozett
Copy link

ozett commented Sep 29, 2021

others seem to use the nano also for AI with docker. some details for fiddling:
snowzach/doods#25 (comment)

@KillahB33
Copy link
Author

Apologies y'all I would love to help but I can't seem to locate the image that I was using from e1 or find the comment where he mentions it. If you have more than 4 1080p cameras I would also suggest you look for something else as my nano couldn't handle it even after the hwaccell was setup.

@ozett
Copy link

ozett commented Nov 30, 2021

my jetson: (next is exploring the ffmpeg issue... docker is running. but frigate stops with ffmpeg-errors)
image

@ozett
Copy link

ozett commented Nov 30, 2021

image

@ozett
Copy link

ozett commented Nov 30, 2021

id did this, looks good on the host-os.
edit:
image

who can help?
can i do this inside the docker-container?
go into it with bash, and install this ffmpeg from that repo?
docker exec -it frigate /bin/bash'

image

@ozett
Copy link

ozett commented Nov 30, 2021

no hw-accel on rtsp decoding, and inference on jetson-nano CPU. its running...
not bad ... i guess...

image

@ozett
Copy link

ozett commented Nov 30, 2021

too bad, error ath the end, after inside the container install of ffmpeg from deb sources..
think this must be done at docker-build level?

edit: here goes the patch to ffmpeg...
but seems that this must be build (on docker-level?) for inside the container
https://github.com/roflcoopter/viseron/blob/master/docker/jetson-nano/Dockerfile.ffmpeg#L570

if anybody sets something up, i will test it ..

image

@ozett
Copy link

ozett commented Nov 30, 2021

i wanted to test performance/hw-accel on the host-os with ffmpeg from the repo.
but doesnt look good. as far as i see there is no hw-support... but maybe the cmd-line is totally wrong..

no hwaccell to see in jtop or tegrastats ...
what now to test?

3 tastk running as test on host-os (beside the old container):

 ffmpeg -vsync drop -c:v h264_nvv4l2dec -rtsp_transport tcp -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -stimeout 5000000 -i rtsp:https://admin:[email protected]/Streaming/channels/103 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an  -r 5 -s 1280x720 -f null -`

image

@ozett
Copy link

ozett commented Nov 30, 2021

GSTREAMER on jetson !
can frigate be modified to use gestreamer for RTSP-Decoding?
https://forums.developer.nvidia.com/t/jetson-nano-h264-decoder-performance/189704/4

== translation possible from ffmpeg rtsp -> gstreamer ?

try ffmpeg teststreaming

ffmpeg -vsync drop -c:v h264_nvv4l2dec -rtsp_transport tcp -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -stimeout 5000000 -i rtsp:https://admin:[email protected]/Streaming/channels/103 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an  -r 5 -s 1280x720 -f null -

get cam per ssh on jetson ddisplay

DISPLAY=:1 gst-launch-1.0 rtspsrc location=rtsp:https://admin:[email protected]:554/Streaming/channels/103 latency=200 drop-on-latency=true ! queue ! rtph264depay ! h264parse ! omxh264dec ! nvvidconv ! xvimagesink -e

@ozett
Copy link

ozett commented Dec 1, 2021

seems painfull to compile ffmpeg for the nano
roflcoopter/viseron#289 (comment)

@ozett
Copy link

ozett commented Dec 1, 2021

there is hope....
installed shinobi on the jetson.
seems that they managed to get ffmpeg running with hw-accell..

now somebody has to look up their source to take over the trick..
https://gitlab.com/Shinobi-Systems/Shinobi/-/blob/dev/INSTALL/ffmpeg.sh
????

image

edit: big mistake....
while testing.. i had gstreamer running, that caused the nvdec
i killed the gestreamer task and back to normal shinobi ->
shinobi does not use nvdec... seems also no hw-accell... what a pity...
image

edit2: Installing shinobis tensor-flow plugin freezes the jetson completly... will revert back to fresh image..
TensorFlow Plugin

@ozett
Copy link

ozett commented Dec 1, 2021

thought of walking away from the nano, but can this be true without hw-accel? ->
image

@ozett
Copy link

ozett commented Dec 1, 2021

dont let the jetson nano down, dive into deepstream,
a low-code workbench, and they already have a tracker in deepstream...

https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_NvMultiObjectTracker_parameter_tuning_guide.html

image

@KillahB33
Copy link
Author

thought of walking away from the nano, but can this be true without hw-accel? -> image

Yes it can be because he's not doing detection, as soon as you turn detection on that cpu is going to choke.

@ozett
Copy link

ozett commented Dec 1, 2021

but jetson nano is capable of h264/265 hardware detection in a significant amount of streams.
but mostly with jetson (hw-accel-)tools like gstreamer and doing detection with tensorRT/deepstream
seem impressive and worth exploring for 99,- - but a world beside this world of frigate.pym tensorflow-lite and corals

@ozett
Copy link

ozett commented Dec 1, 2021

really cool, (diving into deepstream)
with gstreamer instad of ffmpeg one could de-fish/de-warp...

image

@ozett
Copy link

ozett commented Dec 4, 2021

some help with deepstream, tensorRT and yolo on the nano ->
https://github.com/marcoslucianops/DeepStream-Yolo

@LordNex
Copy link

LordNex commented Dec 8, 2021

Hello, new to this feed but I have a few questions I'm hoping someone can answer without me having to try 1000 things.

I have an i5 laptop that I'm running Home Assistant on. I have the 4 gig Jetson Nano installed with the base image and docker installed. The only docker container that's running is DeepStack for detections.

Is there an approved/easy way to install Frigate in a separate container to use as an NVR or do I have to go through a bunch of hoops? I don't really need Frigate to do any serious detections. It would be nice if it could detect a person first and if that's true I can trigger DeepStack for facial recognition. But I didn't know if that's a way it can be installed.

@KillahB33
Copy link
Author

KillahB33 commented Dec 8, 2021

There are a few others doing that on here. You have home assistant so should be easy to do it with some node red setups.
If you search issues for deepstack something should come up. The container you need for the setup is the one being discussed though so you will need to use that one.

@yury-sannikov
Copy link
Contributor

hi there, what do you think about this approach?
#2440

@ozett
Copy link

ozett commented Dec 13, 2021

hi there, what do you think about this approach? #2440

GREAT!

@SheaSmith
Copy link

I managed to get FFMPEG working for me. Basically I used https://github.com/Metric-Void/jetson-ffmpeg-docker for FFMPEG, but I had to modify it at bit to get the entrypoints and install locations correct. I also had to modify Frigate a bit so that it was able to work with Python 3.6 (as the Nvidia docker container is still based on Ubuntu 18.04, which doesn't really support Python 3.8). I'm just in the process of cleaning up some of the code, but I should have some stuff pushed up soon. But I doubt I'll be able to incorporate it as a pull request, as it modifies too many core files. Hopefully once Nvidia releases a Jetpack image based on 20.04 (hopefully next year), it should be a lot simpler.

I'm getting pretty consistent use of NVDEC, with sporadic NVENC usage according to jtop:
image

@ozett
Copy link

ozett commented Dec 14, 2021

NVDEC usage from ffmpeg looks FANTASTIC...

did you stressed the rtmp decoding with more than 10 cams?
what does that looks like?

(in a 2nd pipeline maybe gstreamer could also made working on the nano for frigate)

@SheaSmith
Copy link

NVDEC usage from ffmpeg looks FANTASTIC...

did you stressed the rtmp decoding with more than 10 cams? what does that looks like?

(in a 2nd pipeline maybe gstreamer could also made working on the nano for frigate)

I haven't really stressed it too much. I've just got 6 cameras, and don't use RTMP (although it was enabled). All seemed to be running fine, with no artifacts or anything.

@ozett
Copy link

ozett commented Dec 25, 2021

8 cams seems possible with good performance..
wb666greene/AI-Person-Detector#11 (comment)

@LordNex
Copy link

LordNex commented Dec 25, 2021 via email

@eldadh
Copy link

eldadh commented Feb 22, 2022

Hey Guys,
Any Progress with having a Frigate Version for Jetson Nano?

@NickM-27
Copy link
Sponsor Collaborator

@eldadh There is an open PR #2548 where you can follow the progress, download and try to run yourself if you'd like

@NickM-27
Copy link
Sponsor Collaborator

This has been added as a community supported build for 0.13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests