[HW Accel Support]: ffmpeg high cpu caused by "-c:a aac" in record stream #11498
-
Describe the problem you are havingI am running frigate on an odroid M1S (rockchip 3566 chipset), and huge thanks to @MarcA711 for his efforts on hardware acceleration, it works fantastically well... however I was still battling high CPU on ffmpeg - but have now figured out the cause and mitigation. The recording stream is using -c:a:aac, which is apparently triggering a surprising amount of CPU to re-encode the (already AAC) audio stream. Changing this to -c:a:copy has a dramatic effect - Bringing cpu usage down from ~85% to ~18% (on a 1920x1080 20fps stream) I was surprised that ffmpeg apparently reencodes the audio stream here rather than transparently copying it from the incoming source, which matches the format. See the log output section for details. I'm happy to help troubleshoot and can reproduce easily if there is anything else I can test to make things more optimized, in the meantime I'm overriding the 'record' output args with my fix. Thank you for all your hard work, this project is great! Version0.13.2-69D9A261 (@MarcA711 built this, uses new rockchip hwaccel params and ffmpeg) Frigate config file# yaml-language-server: $schema=http:https://127.0.0.1:5000/api/config/schema.json
birdseye:
enabled: true
mode: motion
#restream: true
mqtt:
enabled: true
host: 192.168.11.5
cameras:
driveway:
enabled: true
ffmpeg:
inputs:
- path: rtsps:https://192.168.11.182:7441/sJKCVajaBKvzO1RS # Med Res
roles:
- detect
#- record
- path: rtsps:https://192.168.11.182:7441/LvsMuZNLIj1hAeXE # High Res
roles:
- record
#- detect
motion:
mask:
- 0,0,0,408,138,391,209,257,206,193,233,131,255,75,360,90,349,58,280,41,274,0
- 1280,0,1280,26,1088,21,1002,32,960,0
zones:
near_house:
coordinates: 0,720,1280,720,1280,461,1040,283,0,387
# Masks for high res detection
# motion:
# mask:
# - 751,0,676,192,846,208,803,314,558,323,491,832,0,866,0,0
# - 2688,0,2688,283,2313,280,2037,181,2039,0
# objects:
# filters:
# person:
# mask: 717,411,757,416,760,384,712,380
detect:
fps: 5
width: 1280
height: 720
wyzecam_test:
enabled: True
ffmpeg:
inputs:
- path: rtsp:https://192.168.11.211:8554/1080p?mp4 # High Res
roles:
- record
- detect
#- path: rtsp:https://192.168.11.211:8554/360p?mp4 # Low Res
#roles:
# - detect
motion:
mask:
- 1378,1080,1920,1080,1920,1007,1378,1002 # datetime stamp on wyze cams
detect:
fps: 5
width: 1920
height: 1080
objects:
filters:
bird:
min_score: 0.4
threshold: 0.5
wyzecam_indoor:
enabled: True
ffmpeg:
inputs:
- path: rtsp:https://192.168.11.168:8554/1080p?mp4 # High Res
roles:
- record
- detect
#- path: rtsp:https://192.168.11.168:8554/360p?mp4 # Low Res
# roles:
# - detect
motion:
mask:
- 1378,1080,1920,1080,1920,1007,1378,1002 # datetime stamp on wyze cams
detect:
fps: 5
width: 1920
height: 1080
detectors:
coral1:
type: edgetpu
device: pci:0
ffmpeg:
#global_args: "-v info -loglevel info -threads 2"
hwaccel_args: preset-rk-h264
output_args:
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a copy
objects:
track:
- person
- dog
- car
- bicycle
- cat
- motorcycle
- bird
- bear
- sheep
- truck
snapshots:
enabled: True
record:
enabled: True
events:
retain:
default: 30
mode: motion docker-compose file or Docker CLI commandfrigate:
container_name: frigate
restart: unless-stopped
image: docker.io/marca711/frigate:latest-rk
shm_size: "256mb"
environment:
- "PLUS_API_KEY=REDACTED"
volumes:
- ./frigate/config:/config
- /sdcard/frigate:/media/frigate
- type: tmpfs
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "5000:5000"
- "8554:8554" # rtsp
network_mode: host
privileged: true
# TODO: I don't think this is necessary when running privileged...
devices:
- /dev/apex_0:/dev/apex_0
- /dev/dri:/dev/dri
- /dev/dma_heap:/dev/dma_heap
- /dev/mali0:/dev/mali0
- /dev/rga:/dev/rga
- /dev/mpp_service:/dev/mpp_service
- /dev/iep:/dev/iep
- /dev/mpp-service:/dev/mpp-service
- /dev/vpu_service:/dev/vpu_service
- /dev/vpu-service:/dev/vpu-service
- /dev/hevc_service:/dev/hevc_service
- /dev/hevc-service:/dev/hevc-service
- /dev/rkvdec:/dev/rkvdec
- /dev/rkvenc:/dev/rkvenc
- /dev/vepu:/dev/vepu
- /dev/h265e:/dev/h265e Relevant log outputTesting within the docker container using 'docker exec -ti frigate /bin/bash:
# This runs at ~85% CPU
ffmpeg -loglevel trace -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent "FFmpeg Frigate/0.13.2-69d9a261" -avoid_negative_ts make_zero -fflags +discardcorrupt+genpts -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp:https://192.168.11.211:8554/1080p?mp4 -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac -f mp4 -f null -
Last message repeated 1 times
[h264 @ 0x5580a366c0] decode_slice_header error
[h264 @ 0x5580a366c0] no frame!
Input #0, rtsp, from 'rtsp:https://192.168.11.211:8554/1080p?mp4':
Metadata:
title : go2rtc/1.8.5
Duration: N/A, start: 1716491616.152750, bitrate: N/A
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, 20 fps, 40 tbr, 90k tbn
Stream #0:1: Audio: aac (HE-AAC), 16000 Hz, stereo, fltp
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
Metadata:
title : go2rtc/1.8.5
encoder : Lavf60.16.100
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, q=2-31, 20 fps, 40 tbr, 90k tbn
Stream #0:1: Audio: aac (LC), 16000 Hz, stereo, fltp, 128 kb/s
Metadata:
encoder : Lavc60.31.102 aac
[aac @ 0x5580a7f050] Queue input is backward in time /s speed=N/A
[null @ 0x5580a54b20] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 62008 >= 60987
[aac @ 0x5580a7f050] Queue input is backward in time
[null @ 0x5580a54b20] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 89407 >= 88384
..... Those errors continue periodically .....
###############################################################################
# This runs at ~18% CPU
ffmpeg -loglevel info -threads 2 -hwaccel rkmpp -hwaccel_output_format drm_prime -user_agent "FFmpeg Frigate/0.13.2-69d9a261" -avoid_negative_ts make_zero -fflags +discardcorrupt+genpts -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp:https://192.168.11.211:8554/1080p?mp4 -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a copy -f mp4 -f null -
ffmpeg version 6b13265d70-20240215 Copyright (c) 2000-2023 the FFmpeg developers
built with gcc 13.2.0 (crosstool-NG 1.25.0.232_c175b21)
configuration: --prefix=/ffbuild/prefix --pkg-config-flags=--static --pkg-config=pkg-config --cross-prefix=aarch64-ffbuild-linux-gnu- --arch=aarch64 --target-os=linux --enable-nonfree --enable-gpl --enable-version3 --disable-debug --enable-iconv --enable-libxml2 --enable-zlib --enable-libfreetype --enable-libfribidi --enable-gmp --enable-openssl --enable-lzma --enable-fontconfig --enable-libharfbuzz --enable-libvorbis --enable-opencl --enable-libpulse --enable-libvmaf --enable-libxcb --enable-xlib --enable-amf --enable-libaom --enable-libaribb24 --enable-avisynth --enable-chromaprint --enable-libdav1d --disable-libdavs2 --enable-libfdk-aac --enable-ffnvcodec --enable-cuda-llvm --enable-frei0r --enable-libgme --enable-libkvazaar --enable-libaribcaption --enable-libass --enable-libbluray --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librist --enable-libssh --enable-libtheora --disable-libvpx --enable-libwebp --enable-lv2 --disable-libvpl --enable-openal --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopenmpt --enable-librav1e --enable-rkmpp --enable-rkrga --enable-librubberband --disable-schannel --enable-sdl2 --enable-libsoxr --enable-libsrt --enable-libsvtav1 --enable-libtwolame --enable-libuavs3d --enable-libdrm --disable-vaapi --enable-libvidstab --enable-vulkan --enable-libshaderc --enable-libplacebo --enable-libx264 --enable-libx265 --disable-libxavs2 --enable-libxvid --enable-libzimg --enable-libzvbi --extra-cflags=-DLIBTWOLAME_STATIC --extra-cxxflags= --extra-ldflags=-pthread --extra-ldexeflags=-pie --extra-libs='-ldl -lstdc++ -lgomp' --extra-version=20240215
libavutil 58. 29.100 / 58. 29.100
libavcodec 60. 31.102 / 60. 31.102
libavformat 60. 16.100 / 60. 16.100
libavdevice 60. 3.100 / 60. 3.100
libavfilter 9. 12.100 / 9. 12.100
libswscale 7. 5.100 / 7. 5.100
libswresample 4. 12.100 / 4. 12.100
libpostproc 57. 3.100 / 57. 3.100
[h264 @ 0x5583e7b6c0] non-existing PPS 0 referenced
Last message repeated 1 times
[h264 @ 0x5583e7b6c0] decode_slice_header error
[h264 @ 0x5583e7b6c0] no frame!
[h264 @ 0x5583e7b6c0] non-existing PPS 0 referenced
Last message repeated 1 times
.... these continue for ~4 seconds ....
Last message repeated 1 times
[h264 @ 0x5583e7b6c0] decode_slice_header error
[h264 @ 0x5583e7b6c0] no frame!
Input #0, rtsp, from 'rtsp:https://192.168.11.211:8554/1080p?mp4':
Metadata:
title : go2rtc/1.8.5
Duration: N/A, start: 1716491478.067500, bitrate: N/A
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, 20 fps, 40 tbr, 90k tbn
Stream #0:1: Audio: aac (HE-AAC), 16000 Hz, stereo, fltp
Output #0, null, to 'pipe:':
Metadata:
title : go2rtc/1.8.5
encoder : Lavf60.16.100
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, q=2-31, 20 fps, 40 tbr, 90k tbn
Stream #0:1: Audio: aac (HE-AAC), 16000 Hz, stereo, fltp
Stream mapping:
Stream #0:0 -> #0:0 (copy)
Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
.... No errors like the prior command .... FFprobe output from your cameraInput #0, rtsp, from 'rtsp:https://192.168.11.168:8554/1080p?mp4':
Metadata:
title : go2rtc/1.8.5
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0: Video: h264 (Main), yuv420p(tv, bt709, progressive), 1920x1080, 20 fps, 20.08 tbr, 90k tbn
Stream #0:1: Audio: aac (HE-AAC), 16000 Hz, stereo, fltp Operating systemOther Linux Install methodDocker Compose Network connectionWired Camera make and modelWyze Cam v3 (wz_mini_hacks) Any other information that may be helpfulNo response |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 4 replies
-
this is exactly what would be expected. -c:a copy means to copy, otherwise -c:a aac is explicitly telling it to re-encode. The incoming audio stream can be aac but encoded to a different sample rate for example. Also there is on reason for you to use manual ffmpeg args. Finally, you are running a different ffmpeg build to the one that is used in mainline frigate, perhaps this version specifically has some issue with audio encoding. In my testing audio encoding in the main build is negligible. |
Beta Was this translation helpful? Give feedback.
-
To summarize, aac was being re-encoded due to different profiles, the solution here is one of two things.
The camera was exporting an HE-AAC stream, which was being re-encoded by ffmpeg to an LC aac stream: |
Beta Was this translation helpful? Give feedback.
To summarize, aac was being re-encoded due to different profiles, the solution here is one of two things.
If you are able to change the source encoding (I'm using a wyze cam with wz_mini_hacks, so go2rtc encoding parameters are able to be changed), change the encoding profile to 'aac_low' and try to make sure the bitrate is 128k
This gives the most compatibility, as some clients cannot play back aac-he\
Set the frigate ffmpeg 'record' output arg to
preset-record-generic-audio-copy
profile to avoid re-encoding the audio track.This will work, but certain players (EG android chrome) cannot play the resulting stream
The camera was exporting an HE-AAC stream, which was being re-encoded …