[Detector Support]: Creation of Openvino models with integrated intel GPU fails on latest Linux kernel 6.8.2 #10785

marco-calautti · 2024-04-02T07:12:42Z

marco-calautti
Apr 2, 2024

Describe the problem you are having

After updating my system (Arch), the kernel got updated to 6.8.2, and frigate is not able anymore to compile the openvino detector model for my integrated intel gpu, failing with the message logs shown below.
After reverting the kernel to 6.7.8, everything works normally again. Note that ffmpeg hardware acceleration is not affected, but only the openvino detectors are.

My specs: CPU intel i3 6100T with integrated GPU HD Graphics 530.

This issue in intel compute-runtime might be related: intel/compute-runtime#710

Version

0.13.2-6476f8a

Frigate config file

mqtt:
  enabled: True
  host: mosquitto

detectors:
  ov:
    type: openvino
    device: GPU
    model:
      path: /openvino-model/ssdlite_mobilenet_v2.xml

model:
  width: 300
  height: 300
  input_tensor: nhwc
  input_pixel_format: bgr
  labelmap_path: /openvino-model/coco_91cl_bkgr.txt
  
go2rtc:
  streams:
    cam_salotto: # this needs to be named exactly as the camera, if you want this stream to be the live stream in frigate integration
      - rtsp:https://frigate:[email protected]:554/stream1
      - "ffmpeg:cam_salotto#audio=aac"
    cam_salotto_sub: #used only to detect movement and audio
      - rtsp:https://frigate:[email protected]:554/stream2
      - "ffmpeg:cam_salotto_sub#audio=aac"

    cam_terrazzo: # this needs to be named exactly as the camera, if you want this stream to be the live stream in frigate integration
      - rtsp:https://frigate:[email protected]:554/stream1
      - "ffmpeg:cam_terrazzo#audio=aac"
    cam_terrazzo_sub: #used only to detect movement and audio
      - rtsp:https://frigate:[email protected]:554/stream2
      - "ffmpeg:cam_terrazzo_sub#audio=aac"

ffmpeg:
  hwaccel_args: preset-vaapi

record:
  enabled: True
  retain:
    days: 0
    mode: all
  events:
    retain:
      default: 30
      mode: motion

# Include all cameras by default in Birdseye view
birdseye:
  enabled: True
  mode: continuous

cameras:
  cam_salotto:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp:https://127.0.0.1:8554/cam_salotto_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
        - path: rtsp:https://127.0.0.1:8554/cam_salotto
          input_args: preset-rtsp-restream
          roles:
            - record
    onvif: # this enables camera features control like object movement tracking
      host: 192.168.1.22
      port: 2020
      user: frigate
      password: xxx

    audio:
      enabled: True
      min_volume: 5
    record:
      enabled: True
    detect:
      enabled: True
    objects:
      track:
        - person
      filters:
        person:
          threshold: 0.75    
    live:
        stream_name: cam_salotto
  
  cam_terrazzo:
    ffmpeg:
      output_args:
        record: preset-record-generic-audio-copy
      inputs:
        - path: rtsp:https://127.0.0.1:8554/cam_terrazzo_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
            - audio
        - path: rtsp:https://127.0.0.1:8554/cam_terrazzo
          input_args: preset-rtsp-restream
          roles:
            - record
    onvif: # this enables camera features control like object movement tracking
      host: 192.168.1.25
      port: 2020
      user: frigate
      password: xxx

    audio:
      enabled: True
      min_volume: 5
    record:
      enabled: True
    detect:
      enabled: True
    objects:
      track:
        - person
      filters:
        person:
          threshold: 0.75
          mask:
            - 0,69,348,0,0,0
            - 597,0,588,29,532,0
    live:
        stream_name: cam_terrazzo

docker-compose file or Docker CLI command

frigate:
    container_name: frigate
    cap_add:
      - PERFMON # to see gpu stats
    restart: unless-stopped
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: ${FRIGATE_SHM_SIZE} # update for your cameras based on calculation above
    devices:
      - /dev/bus/usb:/dev/bus/usb
      - ${FRIGATE_DRI_CARD} # this is /dev/dri/card0
      - ${FRIGATE_DRI_RENDER} # this is /dev/dri/renderD128
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ${BASE_STORAGE_DIR}/${HASS_DATA_DIR}/${FRIGATE_DATA_DIR}/${FRIGATE_CONFIG_DIR}:/config
      - ${FRIGATE_STORAGE_DEVICE}:/media/frigate
      - type: tmpfs # Optional: reduces SSD/SD Card wear
        target: /tmp/cache
        tmpfs:
          size: ${FRIGATE_TMPFS_SIZE}
    ports:
      - "5000:5000"
      - "8554:8554" # RTSP feeds
      - "8555:8555/tcp" # WebRTC over tcp
      - "8555:8555/udp" # WebRTC over udp
    networks:
      - homeassistant-net

Relevant log output

frigate  | 2024-04-02 08:57:44.141725869  Process detector:ov:
frigate  | 2024-04-02 08:57:44.143750259  Traceback (most recent call last):
frigate  | 2024-04-02 08:57:44.143842791    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate  | 2024-04-02 08:57:44.143846325      self.run()
frigate  | 2024-04-02 08:57:44.143848743    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
frigate  | 2024-04-02 08:57:44.143850429      self._target(*self._args, **self._kwargs)
frigate  | 2024-04-02 08:57:44.143856204    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
frigate  | 2024-04-02 08:57:44.143886819      object_detector = LocalObjectDetector(detector_config=detector_config)
frigate  | 2024-04-02 08:57:44.143889641    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
frigate  | 2024-04-02 08:57:44.143908891      self.detect_api = create_detector(detector_config)
frigate  | 2024-04-02 08:57:44.143912045    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
frigate  | 2024-04-02 08:57:44.143914599      return api(detector_config)
frigate  | 2024-04-02 08:57:44.143968297    File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in __init__
frigate  | 2024-04-02 08:57:44.143971237      self.interpreter = self.ov_core.compile_model(
frigate  | 2024-04-02 08:57:44.143974192    File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 399, in compile_model
frigate  | 2024-04-02 08:57:44.143976497      super().compile_model(model, device_name, {} if config is None else config),
frigate  | 2024-04-02 08:57:44.144153790  RuntimeError: Failed to create plugin /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_gpu_plugin.so for device GPU
frigate  | 2024-04-02 08:57:44.144156352  Please, check your environment
frigate  | 2024-04-02 08:57:44.144158442  Check 'error_code == 0' failed at src/plugins/intel_gpu/src/runtime/ocl/ocl_device_detector.cpp:194:
frigate  | 2024-04-02 08:57:44.144160354  [GPU] No supported OCL devices found or unexpected error happened during devices query.
frigate  | 2024-04-02 08:57:44.144162338  [GPU] Please check OpenVINO documentation for GPU drivers setup guide.
frigate  | 2024-04-02 08:57:44.144163985  [GPU] clGetPlatformIDs error code: -1001

Operating system

Other Linux

Install method

Docker Compose

Coral version

CPU (no coral)

Any other information that may be helpful

No response

Answered by alexeynl

Apr 29, 2024

I use image Docker images from documentation ghcr.io/blakeblackshear/frigate:stable. I have migrated Docker host to Ubuntu 24.04 with 6.8 kernel and faced with the same issue.
To get rid from this error I have installed latest compute-runtime deb packages inside Docker container according installation steps from https://github.com/intel/compute-runtime/releases.

Download packages to temp directory of Docker host:

mkdir neo
cd neo
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.16510.2/intel-igc-core_1.0.16510.2_amd64.deb
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.16510.2/intel-igc-opencl_1.0.16510.2_amd64.deb
wget http…

View full answer

marco-calautti · 2024-04-07T09:19:13Z

marco-calautti
Apr 7, 2024
Author

Any reason why this has been ignored? Especially considering that kernel 6.7 has been declared EOL, I think it is important for frigate to work as expected on 6.8 and greater.

3 replies

NickM-27 Apr 7, 2024
Collaborator Sponsor

None of the maintainers use openvino and building your own model has even more variables to it. If the built in model doesn't work then that's very concerning, building your own model is a lot more specific and any number of things could be the reason that it doesn't work

You're asking a very specific and specialized question that few people can give an informed answer to

Edit: looks like the title wasn't correct and it is the default model being used.

NickM-27 Apr 7, 2024
Collaborator Sponsor

It looks like this issue is outside of Frigate intel/compute-runtime#710 (comment)

esand Apr 29, 2024

@NickM-27 As mentioned below by @alexeynl, it seems that updating Intel compute package might be required as well. It seems Jellyfin also updated (jellyfin/jellyfin-packaging#15) to address the issue. Not saying a kernel fix isn't also necessary, but if Intel compute is part of Frigate, I'd just like to +1 requesting it get updated.

alexeynl · 2024-04-29T07:02:10Z

alexeynl
Apr 29, 2024

I use image Docker images from documentation ghcr.io/blakeblackshear/frigate:stable. I have migrated Docker host to Ubuntu 24.04 with 6.8 kernel and faced with the same issue.
To get rid from this error I have installed latest compute-runtime deb packages inside Docker container according installation steps from https://github.com/intel/compute-runtime/releases.

Download packages to temp directory of Docker host:

mkdir neo
cd neo
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.16510.2/intel-igc-core_1.0.16510.2_amd64.deb
wget https://github.com/intel/intel-graphics-compiler/releases/download/igc-1.0.16510.2/intel-igc-opencl_1.0.16510.2_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/intel-level-zero-gpu-dbgsym_1.3.29138.7_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/intel-level-zero-gpu_1.3.29138.7_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/intel-opencl-icd-dbgsym_24.13.29138.7_amd64.ddeb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/intel-opencl-icd_24.13.29138.7_amd64.deb
wget https://github.com/intel/compute-runtime/releases/download/24.13.29138.7/libigdgmm12_22.3.18_amd64.deb

Next you need repack downloaded libigdgmm12_22.3.18_amd64.deb package as states here otherwise you will get the error during installation inside docker container:

dpkg-deb: error: archive 'libigdgmm12_22.3.18_amd64.deb' uses unknown compression for member 'control.tar.zst', giving up

sudo apt install binutils
mkdir deb_repack
cp ./libigdgmm12_22.3.18_amd64.deb ./deb_repack/libigdgmm12_22.3.18_amd64.deb
cd deb_repack
ar x libigdgmm12_22.3.18_amd64.deb
zstd -d < control.tar.zst | xz > control.tar.xz
zstd -d < data.tar.zst | xz > data.tar.xz
ar -m -c -a sdsd libigdgmm12_22.3.18_amd64_repacked.deb debian-binary control.tar.xz data.tar.xz
cp libigdgmm12_22.3.18_amd64_repacked.deb ../libigdgmm12_22.3.18_amd64.deb
cd ..
rm -rf ./deb_repack/

Run Frigate docker container as following to prevent from constant restart due the error and crash:

sudo docker run -d ghcr.io/blakeblackshear/frigate:stable tail -f /dev/null

Copy deb files to Frigate docker container:

cd ..
sudo docker cp neo frigate:/

Install deb packages inside Frigate docker container:

sudo docker exec -it frigate sh -c 'dpkg -i /neo/*.deb'

Restart the container and check its status:

sudo docker restart frigate
sudo docker ps | grep frigate

This is just workaround. For permanent fix Docker image mainteiner must build and push image with latest intel compute-runtime.

6 replies

mavacpjm Aug 7, 2024

getting error during deb install any help much appreciated:

dpkg: error processing archive /neo/intel-level-zero-gpu_1.3.29138.7_amd64.deb (--install):
trying to overwrite '/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1', which is also in package intel-opencl-icd 22.43.24595.41-1

Here is the full verbose: when i run the deb install on container frigate.

Selecting previously unselected package intel-igc-core.
(Reading database ... 13120 files and directories currently installed.)
Preparing to unpack .../intel-igc-core_1.0.16510.2_amd64.deb ...
Unpacking intel-igc-core (1.0.16510.2) ...
Selecting previously unselected package intel-igc-opencl.
Preparing to unpack .../intel-igc-opencl_1.0.16510.2_amd64.deb ...
Unpacking intel-igc-opencl (1.0.16510.2) ...
Selecting previously unselected package intel-level-zero-gpu.
Preparing to unpack .../intel-level-zero-gpu_1.3.29138.7_amd64.deb ...
Unpacking intel-level-zero-gpu (1.3.29138.7) ...
dpkg: error processing archive /neo/intel-level-zero-gpu_1.3.29138.7_amd64.deb (--install):
trying to overwrite '/usr/lib/x86_64-linux-gnu/libze_intel_gpu.so.1', which is also in package intel-opencl-icd 22.43.24595.41-1
Preparing to unpack .../intel-opencl-icd_24.13.29138.7_amd64.deb ...
Unpacking intel-opencl-icd (24.13.29138.7) over (22.43.24595.41-1) ...
Preparing to unpack .../libigdgmm12_22.3.18_amd64.deb ...
Unpacking libigdgmm12:amd64 (22.3.18) over (22.3.3+ds1-1) ...
Setting up intel-igc-core (1.0.16510.2) ...
Setting up intel-igc-opencl (1.0.16510.2) ...
Setting up libigdgmm12:amd64 (22.3.18) ...
Setting up intel-opencl-icd (24.13.29138.7) ...
Processing triggers for libc-bin (2.36-9+deb12u7) ...
Errors were encountered while processing:
/neo/intel-level-zero-gpu_1.3.29138.7_amd64.deb

mavacpjm Aug 7, 2024

Ran the script a second time like esand stated above and it installed without issue, still havent check if its resolved yet. But wanted to update that the script completed for me after a second run.

willgryan Aug 7, 2024

Ran the script a second time like esand stated above and it installed without issue, still havent check if its resolved yet. But wanted to update that the script completed for me after a second run.

Yes, I ran the same lines again after experiencing the error the first time, and it now works.

mavacpjm Aug 7, 2024

Restarted Docker container, still openvino is running off cpu, so this did not resolve for me on ubuntu 22.04 LTS .

CPU pegged over 72 % when a detection occurs. Forced GPU as well to see if AUTO was the issue, still defaults back to CPU.

mavacpjm Aug 7, 2024

It worked!!!! I gave it a few minutes after restarting frigate and things normalized - Inference went from 70ms to sub 29ms and now CPU is sitting between 8.0 and 9.0 on detector. Thanks esand for this !!! Awesome!!!

The Frigate container will need updating to resolve this long term, this only works for the interim as esand stated!

marco-calautti · 2024-05-01T10:32:18Z

marco-calautti
May 1, 2024
Author

A simpler, temporary fix, as suggested in the intel-compute-runtime repo is to set the following env variables (in the case of frigate, I guess this should be enough in the docker configuration):
NEOReadDebugKeys=1
OverrideGpuAddressSpace=48

Anyway, since the frigate image contains the intel-compute-runtime, I also believe it is frigate's responsibility to update it, to properly fix the bug.

11 replies

blakeblackshear Jun 14, 2024
Maintainer

suggested in the intel-compute-runtime repo

Can you provide a link to where this was suggested?

marco-calautti Jun 14, 2024
Author

intel/compute-runtime#710

jameslivulpi Jun 14, 2024

also for ref in changelog here: https://github.com/intel/compute-runtime/releases/tag/24.09.28717.12 under "Known issues"

bobloadmire Jun 15, 2024

posting here incase anyone has any advice, I added these to my Proxmox Frigate lxc.conf:
lxc.environment: NEOReadDebugKeys=1
lxc.environment: OverrideGpuAddressSpace=48

Didn't seem to have any effect. Any proxmox users, or am I using the correct values for my variables? Running Intel 8505 with igpu passthrough.

willgryan Jul 8, 2024

I am also experiencing this issue.

Intel N100 in privileged LXC container on Proxmox 8.2.2 / Kernel 6.8.4.3-pve using tteck install script.

ffmpeg HW acceleration works fine. OpenVINO detector set to AUTO falls back to CPU (high CPU load seen). Changing config manually to GPU causes Frigate to hang and freeze.

Update: I have fixed this by manually updating to the latest intel-compute driver in the LXC instance tteck/Proxmox#2816

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Detector Support]: Creation of Openvino models with integrated intel GPU fails on latest Linux kernel 6.8.2 #10785

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 20 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[Detector Support]: Creation of Openvino models with integrated intel GPU fails on latest Linux kernel 6.8.2 #10785

Describe the problem you are having

Version

Frigate config file

docker-compose file or Docker CLI command

Relevant log output

Operating system

Install method

Coral version

Any other information that may be helpful

Replies: 3 comments · 20 replies

marco-calautti Apr 7, 2024 Author

NickM-27 Apr 7, 2024 Collaborator Sponsor

NickM-27 Apr 7, 2024 Collaborator Sponsor

marco-calautti May 1, 2024 Author

blakeblackshear Jun 14, 2024 Maintainer

marco-calautti Jun 14, 2024 Author

Replies: 3 comments 20 replies

marco-calautti
Apr 7, 2024
Author

NickM-27 Apr 7, 2024
Collaborator Sponsor

NickM-27 Apr 7, 2024
Collaborator Sponsor

marco-calautti
May 1, 2024
Author

blakeblackshear Jun 14, 2024
Maintainer

marco-calautti Jun 14, 2024
Author