Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EXT-X-GAP #2940

Closed
4 of 5 tasks
mtoczko opened this issue Jul 31, 2020 · 14 comments
Closed
4 of 5 tasks

EXT-X-GAP #2940

mtoczko opened this issue Jul 31, 2020 · 14 comments

Comments

@mtoczko
Copy link
Collaborator

mtoczko commented Jul 31, 2020

What version of Hls.js are you using?

0.14.7

What browser and OS are you using?

Chrome 84

Test stream:

gap: video segment 1.ts and 5.ts, audio segment 1.ts and 5.ts
https://mtoczko.github.io/hls-test-streams/test-gap-audio-video/playlist.m3u8

gap: video segment 1.ts and 5.ts
https://mtoczko.github.io/hls-test-streams/test-gap-video/playlist.m3u8

gap: audio segment 1.ts and 5.ts
https://mtoczko.github.io/hls-test-streams/test-gap-audio/playlist.m3u8

Checklist

  • The stream has correct Access-Control-Allow-Origin headers (CORS)
  • There are no network errors such as 404s in the browser console when trying to play the stream

Expected behavior

The client should not attempt to load segment.
Fill audio gaps with silence. (Safari)
Fill video gaps with black frame. (Safari)

@robwalch
Copy link
Collaborator

robwalch commented Aug 10, 2020

Thanks for the examples @mtoczko, and for creating this issue to point out the gap in our implementation.

Looking through the latest spec I see LL-HLS Partial Segments can also be flagged with a GAP attribute, so that's worth noting here since it looks like EXT-X-GAP was added at the same time as LL-HLS tags.

6.3.3. Playing the Media Playlist File
...
The client SHOULD NOT attempt to load Media Segments that have been marked with an EXT-X-GAP tag, or to load Partial Segments with a GAP=YES attribute.

@robwalch
Copy link
Collaborator

robwalch commented Aug 12, 2020

#2865 Parses GAP, although functionally the result is the same as before; All unknown tags starting with # in the playlist get added to the fragment tagList.

Filling with silence is doable. We don't have any code that can generate black frames. At best we can copy or extend the duration of existing frames.

@mtoczko
Copy link
Collaborator Author

mtoczko commented Aug 12, 2020

@robwalch Safari fills the gap with the last video frame. If the gap is in the first place, it takes the first frame of the next segment and extends the duration.

@robwalch
Copy link
Collaborator

Thanks for clarifying @mtoczko. That makes sense, and would be a great feature, not just for GAP segments, for for actual start gaps that result in stalls and require seek nudging.

@robwalch
Copy link
Collaborator

Not a Contribution

From the HLS spec:

6.3.3. Playing the Media Playlist File

...

The client SHOULD NOT attempt to load Media Segments that have been
marked with an EXT-X-GAP tag, or to load Partial Segments with a
GAP=YES attribute. Instead, clients are encouraged to look for
another Variant Stream of the same Rendition which does not have the
same gap, and play that instead.

So after avoiding segments or parts marked with a GAP, and before filling with silence or extending frames, we should attempt to find media in other renditions.

@robwalch robwalch added this to the 1.4.0 milestone Aug 5, 2022
@ok-ikaros
Copy link

hey @robwalch I see that this feature is in the list of top priorities for the next release - do you guys know what the status on this?

@robwalch
Copy link
Collaborator

Hi @lelandhwu,

The Milestone is set to 1.4.0 which is the next planned release: HLS.js Roadmap.

The work hasn't started yet. I need to wrap up #5191 first. The plan is to skip loading of segments with a GAP tag, and handle them the same way a FRAG_PARSING_ERROR would be handled.

@mtoczko
Copy link
Collaborator Author

mtoczko commented Mar 24, 2023

Hi @robwalch
I am still analyzing the algorithm of operation. But I have a few questions:

What is the source of the value of this constant? RENDITION_PENALTY_DURATION_MS = 300000
I have noticed that there is no switch to the backup stream for the audio track, is this intentional?
In the gap of an audio track, can we fill the stream with silence? Similar to what happens in the native Safari player?

@robwalch
Copy link
Collaborator

What is the source of the value of this constant? RENDITION_PENALTY_DURATION_MS = 300000

It's an arbitrary value. We could make it configurable in the future. It exists so that clients to flip back and forth between redundant streams (or hosts). If a set errors enough to be put in the "penalty box" it will not be switched back to for five minutes.

I have noticed that there is no switch to the backup stream for the audio track, is this intentional?

No? If an audio or subtitle track errors the whole rendition (the variants using that group) error, and the player switches to the backup (or should switch to a variant with a different group).

@robwalch robwalch removed this from the 1.4.0 milestone Mar 27, 2023
@robwalch
Copy link
Collaborator

v1.4.0 will include support added in #5257. This issue can be used to track additional enhancements and feedback.

@mtoczko
Copy link
Collaborator Author

mtoczko commented Mar 30, 2023

Hi @robwalch
I noticed one undesirable action: gap-controller.ts:264 [warn] > skipping hole, adjusting currentTime from 105.275674 to 170.05533300000002

To reproduce the issue, you need to watch the entire video and seek to 105.
"backBufferLength": 90
08b938f3.hls-js-dev.pages.dev-1680185898252.log

@robwalch
Copy link
Collaborator

Hi @robwalch I noticed one undesirable action: gap-controller.ts:264 [warn] > skipping hole, adjusting currentTime from 105.275674 to 170.05533300000002

Filed #5360

@mtoczko
Copy link
Collaborator Author

mtoczko commented Apr 1, 2023

Hi @robwalch

No? If an audio or subtitle track errors the whole rendition (the variants using that group) error, and the player switches to the backup (or should switch to a variant with a different group).

Maybe you have an idea why it plays the video normally once and switches the group, but when played again it skips the gap.

[log] > [audio-stream-controller]: Buffered audio sn: 100 of track 0 (frag:[200.000-202.005] > buffer:[50.005-142.016][154.005-156.011][168.000-186.005][188.011-202.005])
base-stream-controller.ts:1695 [log] > [audio-stream-controller]: PARSED->IDLE
base-stream-controller.ts:715 [log] > [audio-stream-controller]: Loading fragment 101 cc: 0 of [0-133] track: 0, target: 202.005
base-stream-controller.ts:1695 [log] > [audio-stream-controller]: IDLE->FRAG_LOADING
gap-controller.ts:304 [warn] > skipping hole, adjusting currentTime from 141.952299 to 154.05533300000002

80d1619f.hls-js-4zn.pages.dev-1680353100256.log

@robwalch
Copy link
Collaborator

robwalch commented Apr 6, 2023

Maybe you have an idea why it plays the video normally once and switches the group, but when played again it skips the gap.

I've just made some improvements to this in #5366 that allow for more variant and redundant rendition failover (back to renditions that already failed).

The thing is with redundant failover we don't maintain the level details for alternates (only roughly in the error handler now for this issue). The failover is not something we want to repeat when both renditions have errors, so there is some basic balancing and penalization of renditions selection based on error count where we do not switch back to alternates if they have errored more the the current selection and errored recently.

If you add PATHWAY-ID to the redundant variants then the player would use Pathway Switching rather than redundant failover to organize levels and handle the failover switching. This will change behavior such that each variant or level is a different object rather than one level with redundant urls and attributes grouped in the level object. The error penalty may be harsher (not allowing for failover back to pathways that errored recently) but because the data is there, fragment gaps could be compared across all options. In the future, redundant levels may be refactored discrete objects with assigned pathways, but for now, that implementation (level.urls and level.urlId) remains the same.

robwalch added a commit that referenced this issue Apr 6, 2023
…range are partial (#5366)

* Only allow large gaps to be skipped if start gap or all fragments in range are partial
Fixes #5360

* Fix adaptation and fragment tracking after fragment errors (gap tags)
#2940
@mtoczko mtoczko closed this as completed Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

3 participants