Add monoscopic 360° video support

[defagos]

Description

This 🧑‍🎄 PR adds support for monoscopic 360° video content, based on the past work done in SRG Media Player.

360.mov

Unlike SRG Media Player no support for stereoscopic 360° video content is provided. This could be added with little effort on the video view side, but the problem with stereoscopic support is that it usually requires much more than the ability to display content for each eye. You not only need an additional VR viewing device (e.g. a cardboard headset into which you slip your phone, or crossing eyes 😵‍💫) but usually also a proper onboarding experience. Controls are also a challenge since you don't have foveal tracking with such devices, and cardboard devices lack buttons, inconveniently forcing users to extract their device for interactions.

Monoscopic 360° support has never been a Pillarbox requirement (and never will on other platforms) since 360° content is not a priority anymore at SRG SSR. But it was fun to add, proving that Pillarbox player is flexible enough to handle this kind of use case as well. Moreover can the Best player in the world™️ have no 360° support? 😉

During the implementation I also discovered an issue with our Picture in Picture implementation, described in #696. I intentionally wanted to fix the issue on a separate branch but proper 360° integration required this issue to be fixed as well. This made the PR bigger than initially expected, but that was the price to pay to ensure everything was working as expected. Sorry for that.

Remark: Thanks to Pillarbox support for Trick mode you can even seek smoothly in 360° streams containing I-frame playlists, as can be seen in the video above. You can even rotate during seeks which is quite neat, especially if you compare this user experience with the one of the YouTube app playing 360° content.

Monoscopic 360° video support design consideration

Monoscopic 360° support has been built right into VideoView. I initially intended to have a separate view but having everything handled by the VideoView transparently is far better from an integration point of view.

Unlike what was done in SRG Media Player there are no touch or gyro controls built directly into the VideoView. A modifier makes it possible to enable monoscopic support and control the orientation at which content is seen. The rest is the responsibility of the app, as can be seen in the demo, since different user experiences might be desired when displaying 360° content. A set of quaternion helpers is provided to make it easier to orient the camera or relate orientation to CoreMotion attitude.

Note that we cannot determine whether content is monoscopic or not based on some HLS playlist metadata or the video resolution. Monoscopic content is usually delivered in with 2:1 aspect ratio but this same format is also popular among filmmakers (e.g. the Morning Show examples we have in our demo). The choice has therefore to be made externally by the app, e.g. with the help of some metadata delivered with the content to be played.

Remark: The SystemVideoView does not support 360° playback. On tvOS displaying 360° content properly therefore requires the use of VideoView directly, but this also means that a dedicated user interface must be implemented (also to support 360° navigation).

Picture in Picture design considerations

I revisited our Picture in Picture implementation to solve #696, making the implementation a lot simpler in the process.

The main issue was that our reference counting strategy was performing cleanups even if PiP was never started. Our reference counting could namely not differentiate between actual parent player view dismissal or simple video view removal / insertion within a non-dismissed parent player view, leading to early deallocation.

To solve this issue while retaining our current formalism (at least with minimal API changes) I realized that basic and advanced PiPs are nearly identical, except that advanced PiP might outlive the player view during PiP itself. This is where associated player view state must be persisted, for which I introduced a dedicated mechanism based on a PictureInPicturePersistable protocol. This protocol not only ensures state persistence and restoration automatically, but also provides hooks into the Picture in Picture life cycle so that the state can be adjusted properly if needed.

One example where the PiP life cycle is handy is our multi-instance example, where two player views can be transitioned to PiP. With basic PiP there is nothing to be done. Since both views stays visible in the backgrounded app, the layer not playing in PiP will be paused automatically (if the correct policy has been set). When extending this example for advanced PiP the situation changes, as the view is removed from screen. In this case the layer not playing in PiP will be removed and not be paused, so there must be a way to pause it explicitly. This is where the life cycle methods in PictureInPicturePersistable come in handy.

The other advantage of this approach is that no code update to static properties is required from developers integrating PiP into an existing implementation, which stays almost the same. This has several benefits:

• Developers can use @StateObjects in their implementation, as they usually do. For proper restoration they must only check if a suitable PictureInPicturePersistable.persisted instance is available first.
• State is not persisted forever anymore. Once a state is not required anymore it will be deallocated automatically.
• Implementation in some tricky cases (e.g. the PlaylistView example) is now trivial and similar to other examples.

I could make the PictureInPicture singleton implementation much simpler as well. While this object is now a delegate of other PiP implementations to manage the common persistable state (like before the in-app PiP modifier is applied at a parent view level, without knowing whether a VideoView or a SystemVideoView will be used) I could remove the Mode which was feeling clunky.

I could also remove other infamous tricks like a dispatch on the next run loop to avoid a crash when PiP controllers were deallocated.

Remark: An existing PR #698 that attempted to fix #696 was closed. It was not satisfying enough from an implementation perspective anyway and introduced a new (delayed) PiP setup block that was calling for problems.

Finally I ensures that layers are properly managed to avoid unexpected pauses in some restoration scenarios. More information has been added to #612.

With all these fixes I could successfully implement advanced Picture in Picture in a multi-instance context, something we omitted in our initial Picture in Picture implementation:

PiP.mov

We also now support advanced restoration scenarios, e.g. playing the same content in Picture in Picture with a custom layout, then restoring into the system player, without any playback interruption:

PiP.mov

Several restoration scenarios that our API should make it possible to implement, involving custom and system views with or without PiP, have been listed as comment on #612, including videos.

Breaking API changes

• Instead of providing VideoView or SystemVideoView options to the view itself, these options have been moved to modifiers for better ergonomics.
• The View/enabledForInAppPictureInPictureWithCleanup(perform:) modifier has been replaced with SwiftUI/View/enabledForInAppPictureInPicture(persisting:). Implementation of a view model which can be persisted has been made simpler with the introduction of an empty PictureInPicturePersistable protocol which automatically provides persistence abilities to existing objects. Instead of implementing cleanup with a closure only a persistable object is now required to ensure proper state persistence during Picture in Picture.

Changes made for 360° support

• Add 360° support to our custom player. The point of view can be controlled with drag gestures or by tilting the device.
• Remove support for custom / system layouts to make the code simpler. System player integration can still be checked under the Showcase tab anyway.
• Prevent display sleep while playing 360° content (and if the PlayerConfiguration requires it) since this behavior is incorrectly not applied for playback involving SKVideoNode.
• Add 360° videos to the Examples tab (with dedicated thumbnails).
• Add 360° videos to the playlist content choice list.
• Add example with standard and 360° videos played simultaneously in the MultiView player.
• Display 360° videos on tvOS (with no special user interface or interactions).
• Separate VideoView and SystemVideoView implementations over several files for better readability.
• Fix Modal to avoid incorrect drag gesture detection outside the safe area boundaries (revealed when implementing 360° navigation gestures).

No explicit documentation was added for 360° support since most of the work has to be done client-side and depends on the desired user experience. The rest is quaternion stuff for which I added some documentation pointers.

Remark: Picture in Picture support is explicitly not possible with 360° content. It could work but having the raw video content in the PiP overlay is clunky at best.

Changes made to fix Picture in Picture

• Fix Picture in Picture is killed too early if no VideoView is initially present #696 by revisiting our implementation, as described above.
• Implement in-app PiP support detection at the PiP button level simpler.
• Improve restoration API to internally manage the minimal delay required for properly animated PiP restoration.
• Add advanced Picture in Picture support to the MultiView example. Also add a button to swap players between both views.
• Add examples under the Showcase tab to test various restoration scenarios between custom and system layouts, supporting Picture in Picture or not. Both the custom and system players can now optionally support Picture in Picture.
• Add setting to the opt-in demo making it possible to toggle Picture in Picture support on or off.
• Introduce proper view models for each player instead of sharing the same PlayerViewModel.
• Fix playlist PiP restoration.
• Move all view models to ObservableObject since we know that memory management issues existed prior to 17.2.
• Remove WillAppearView.
• Since we are not cleaning resources manually anymore (the state goes away automatically when not needed) the issue described in Sound continues for a while after closing the tvOS player #520 can also be seen on iOS when playing URNs and transitioning via PiP between different player views. The Player is correctly released but the underlying AVQueuePlayer might take a bit more time to be deallocated when URNs are involved (but ultimately it will be deallocated, there is no permanent resource leak). I therefore applied a safer fix by setting the volume to 0 when Player is deallocated (we cannot use isMuted since its observation would lead to different analytics at deallocation time, making a few UTs fail). This fix works for both iOS and tvOS and replaces the old one introduced in Sound continues for a while after closing the tvOS player #520.

Known issues

• The is currently no way to enter a URL or URN and asking for a 360° monoscopic layout. I added a suggestion to improve the URL / URN entry in Add certificate and license URL support to the demo #620.
• Playback of monoscopic 360° content is paused when the associated view is destroyed. This is a known issue I have documented for the time being. There might be a fix but I am not sure it is mandatory yet. The result is that playback might be paused in playlists or when swapping players in the MultiView example displaying 360° content.
• We could probably have better interactions with inertia and camera motions in our demo 360° integration. I will create a dedicated issue but this is can be seen as an improvement. This PR is already large enough.

Checklist

• APIs have been properly documented (if relevant).
• The documentation has been updated (if relevant).
• New unit tests have been written (if relevant).
• The demo has been updated (if relevant).
• The playground has been updated (if relevant).