FFmpeg NVENC: Add CUDA support for zero-copy 4:4:4 and RGB

[Xaymar]

Description

The NVENC encoders in FFmpeg support CUDA inputs, which also support 4:4:4 and RGBX formats. These two formats are not supported by the DirectX backend, or the OpenGL backend, so using CUDA benefits us here. We've already linked to CUDA through the use of the NVIDIA Face Tracking filter, and just have to extend the API a little bit.

Checklist

• I am willing to pay someone to implement this feature.
• This feature absolutely can't be done with already existing features.

[Xaymar]

Through further investigation, it has become clear that only two formats can be supported with zero-copy encoding: NV12, and RGBA - the other OBS video formats use additional textures which libOBS does not provide. Therefore in order to support this we would have to take in an RGBA texture and apply our own conversion to it, either as a CUDA kernel or as a Compute shader. The following conversions would need to be done:

• RGBA -> NV12 (OpenGL)
• RGBA -> YUV420 (both)
• RGBA -> YUV444 (both)
• RGBA -> XRGB (both)

At the current time, this is not yet planned. If OBS eventually supports P010 however it should be trivial to add support for encoding that using zero-copy methods.

[Xaymar]

The P010 format is available since OBS Studio 28.0, and allows for HDR capture. This is still a 4:2:0 format, so it will not allow for the unsupported formats to be used for encoding.

[Xaymar]

Rejected: Impossible

This may come as a shock to the ones following this issue, but it was inevitable once I started really investigating the issue. After a few minutes of delving today, I now know that zero-copy GPU encoding for anything but NV12 is impossible, as libOBS does not include the code path required for it. Unfortunately that means that this feature is impossible until libOBS itself supports it.

Therefore, closing as impossible since upstream does not support it.

[Xaymar]

An incomplete Pull Request that addresses this shortcoming is available, however it appears to not have seen much activity.