How do you use encoders with StreamFX (like ProRes)? Is it possible to use any encoder in OBS with StreamFX that FFmpeg supports, like x265?

[YousufSSyed]

I spent a good amount of time trying to figure out how encoders work (or what they even do) with StreamFX. As far as I can tell, StreamFX lets you record with encoders it supports, correct? If so, I want to try using ProRes, I looked at two wiki pages, but they don't say what to do and are vague.

While my end goal would be to record using x265 (H265 software encoding) on OBS, I think maybe figuring ProRes could help me do that, or at least see for myself how StreamFX encoders work. I also want to see if I could use any encoders that FFmpeg supports (whatever encoders are in said FFmpeg binary if there even is any, and if there is, does it support x265? If not, could I replace the FFmpeg binary).

[Xaymar]

The encoders StreamFX provides are all present in the encoder selection drop-down shown in Advanced Output Mode. The wiki pages explain the feature and their configuration - it is expected that the end-user already understands what OBS Studio does and how it works. See the endless amounts of tutorials available on YouTube if you do not already understand OBS Studio.

[Xaymar]

That would be out-of-scope of this discussion forum. You may have better luck in the OBS Studio forums.

[YousufSSyed]

How about with StreamFX?

[Xaymar]

As explained above, you would need to either provide a handler, or write an integration for x265 directly.

[YousufSSyed]

OK, do u have documentation on how to write a handler and how they work? Or do u have any resources to point me to?

[Xaymar]

Writing a handler is as simple as implementing the interface. Take a look at the other existing handlers to see how you can write your own - most of it is just interfacing with FFmpeg and libOBS.

[YousufSSyed]

@Xaymar How do you uninstall StreamFX? because maybe I did and then reinstalled it, the encoder options under Custom Output (FFMpeg were the same). Also, do you know of a way to forcibly refresh the UI after uninstalling StreamFX?

Also, how does OBS use FFmpeg? Does it ship a binary with the app, if so where is it? If I were to write an x265 handler, would it just work? Because I imagine it wouldn't if the FFmpeg that OBS uses doesn't have x265 in it (for obvious reasons: patents and licensing).

[Xaymar]

As for the other question(s), the FFmpeg binaries used by OBS Studio must expose the encoder you wish to use. Simple having a handler for an encoder alone does nothing, since there is no actual encoder to talk to.

Note that StreamFX does not provide Custom Output (FFmpeg), and obs-ffmpeg-encoder also did not provide that tab. The encoders are simply listed in the normal encoder drop-down.

[YousufSSyed]

Where are the OBS FFmpeg binaries on Mac? I just looked it up but couldn't find anything. Not even the Guide OBS Studio FFmpeg Upgrade page says what to do on Mac.

[Xaymar]

As OBS Studio related things are out of scope for this forum, you may want to ask this question in either the off-topic section on our Discord, or on the actual OBS Studio discussion forums.

[YousufSSyed]

But I'll say, an x265 handler wouldn't work without changing the FFmpeg binary.

[Xaymar]

That is unfortunate, but not something StreamFX cares about. The guide you linked also was just a stop-gap to get Windows's FFmpeg up to speed with all other platforms (which were shipping 4.4 or even 5.0 already).