WO2012009246A1

WO2012009246A1 - Multi-component media content streaming

Info

Publication number: WO2012009246A1
Application number: PCT/US2011/043496
Authority: WO
Inventors: Zhenyu Wu; Michael Scott Deiss; Li Hua Zhu
Original assignee: Thomson Licensing
Priority date: 2010-07-13
Filing date: 2011-07-11
Publication date: 2012-01-19

Abstract

A method and a device, for encapsulating a media entity containing a plurality of content components into a plurality of transport streams, are described along with the corresponding method and device for reading content components from a transport stream. Extensions to MPEG-2 Transport Stream are proposed. The extensions include a media composition descriptor that conveys the relationship information among content components so that a receiver can know all the related content components even if they are transmitted in different transport streams. The present invention enables adaptive HTTP streaming of media entities.

Description

MULTI-COMPONENT MEDIA CONTENT STREAMING

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from U.S. Provisional Patent Application Serial No. 61/363,842, entitled "Multi-Component Media Content Streaming," filed on July 13, 2010. The teachings of the above-identified provisional patent application are expressly incorporated herein by reference.

TECHNICAL FIELD

The present invention relates generally to HTTP streaming. More specifically, the invention relates to encapsulating content components of a media entity, such as a scalable video coding (SVC) steam or a multi-view coding (MVC) stream, into separate transport streams for HTTP streaming purpose. BACKGROUND OF THE INVENTION

Traditionally all media components (audio / video / subtitles, etc.) are multiplexed into one MPEG-2 transport stream for delivery. As a result, end devices receive the same bundle of the media components.

Nowadays, multi-component media content, such as those encoded by MPEG-4 AVC extensions of SVC and MVC is gaining wider acceptance by the industry. There is a growing demand that the end device is allowed to choose its own service composition flexibly to meet its service capabilities and requirements, such as bandwidth, display resolutions, etc. Therefore, it is beneficial to encapsulate media components into separate transport streams for storage and transport, so that a client can choose a subset of the components' streams to be multiplexed and delivered.

With the current MPEG-2 Transport Stream standard, if the content components from a piece of multi-component media content are simply encapsulated into multiple streams, the MPEG-2 de-multiplexer is not able to successfully de-multiplex the streams. This is because the current MPEG-2 Transport Stream standard does not convey information as to which streams belong to the same media content, which streams need to be de-multiplexed and where to locate them. Without such information, it is not possible to successfully demultiplex the media with such encapsulation. It is desirable to provide such a mechanism to enable separate encapsulation of the content components from a piece of multi-component media content that are compatible with HTTP streaming applications. Prior solutions to the problem have not adequately been established in the art.

SUMMARY OF THE INVENTION

This invention directs to methods and apparatuses for encapsulating a media entity containing a plurality of content components into a plurality of transport streams and for reading content components from a transport stream.

According to an aspect of the present invention, there is provided a method and a device for encapsulating a media entity containing a plurality of content components into a plurality of transport streams. The plurality of content components is extracted from the media entity. Relationships among the plurality of content components are identified. The plurality of content components is packetized into separate transports streams and the identified relationships are associated with the plurality of transport streams.

According to another aspect of the present invention, there is provided a method and a device for reading content components from a transport stream. Content components' relationship information is first extracted from the transport stream. The content components that are related to the current content component are identified. For those related content components that are necessary for the processing of the current content component, they are obtained and processed along with the current content component.

BRIEF DESCRIPTION OF THE DRAWINGS

The above features of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

Figure 1 shows a process of encapsulating a media entity containing multiple content components into multiple transport streams.

Figure 2 shows the structure of an apparatus used to encapsulate a media entity which contains multiple components into multiple transport streams according to the present invention.

Figure 3 illustrates an example of using the invented descriptor. Figure 4 shows the process of reading content components from a transport stream.

Figure 5 shows the structure of a stream reader to read content components from a transport stream. DETAILED DESCRIPTION

In the present invention, a media entity, such as a media file or a set of media files or streaming media, is divided or encapsulated into multiple transport streams. Such encapsulation enables the HTTP streaming, whereby a receiver only needs to extract related transport streams according to its service capability and requirement, such as bandwidth, display resolution etc.

Fig.l shows a process of encapsulating a media entity containing multiple content components into multiple transport streams. At step 110, content components are extracted from the media entity. In step 130, each of the content components is packetized into a separate transport stream. Relationships between the current content component with other content components are identified in step 140. Such relationships are associated with the transport stream in step 150. These multiple transport streams are stored on the server, or when needed, a subset of the multiple transport streams are multiplexed and transmitted to a receiver. A special case of the subset contains all the transport streams. In some applications, the receiver may send requests for specific transport streams that the receiver desires to receive. In such a scenario, only the transport streams, including the associated relationship information, specified by the requests are multiplexed and sent to the receiver.

Fig. 2 illustrates an encapsulator that can be used to realize such an encapsulation process shown in Fig. 1. The encapsulator comprises a component extractor 210 which extracts the content components from the media entity. Each of the extracted content components is packetized by a packetizer 220 into transport stream packets which form a separate transport stream. As a result, a plurality of transport streams are generated, each of which corresponds to a content component. Further, a correlator 230 is employed to identify the relationships among the plurality of content components. Those relationships are fed into the packetizer 220 for embedding into the transport streams. The encapsulator also contains an optional multiplexer 240 to multiplex a subset of the plurality of transport streams for transmission, when necessary. As explained above, at a receiver's request, the multiplexer 240 may also multiplex the requested transport streams, including the associated relationship information, for transmission.

In one embodiment, the relationships among the content components are represented using media composition descriptors proposed as an extension to MPEG-2 standard. It would be appreciated that those skilled in the art will be able to devise various ways of representing and associating the relationships that, although not explicitly described or shown herein, embody the present principles and are included within its spirit and scope.

The media composition descriptor of the present invention signals the media composition information about multi-component media entity, including content component identifiers, inter-component relationships and component transport priorities. In the following, a program element corresponds to one or more content components that belong to the multi- component media entity. Thus the above media composition information defined for content components extends to program elements.

Table 1 - Media composition descriptor

Semantics:

descriptor_tag - The descriptor_tag is an 8 bit field which identifies each descriptor in MPEG-2 standard. For more details about descriptors in MPEG-2, see Section 2.6 of Information technology - Generic coding of moving pictures and associated audio information: systems, ISO/IEC 13818-1:2007.

descriptor_length - The descriptor_length is an 8 bit field specifying the number of bytes of the descriptor immediately following the descriptor_length field.

self_component_UUID - the 128-bit field assigns a universally unique identifier (UUID) to the program element carried within the packets with a PID value. As described earlier, PID is a unique integer value used to identify program elements of a program in a single or multi-program transport stream. Here, the value of the PID is specified by elementary_PID in the PMT descriptor loop as will be described later.

self_component_priority_id - the 8-bit field specifies the transport priority of the program element carried within the packets with the PID whose value is specified by elementary_PID. 0 indicates the highest priority, and a greater value indicates lower priority.

component_UUID - the 128-bit field specifies the universally unique identifier of a program element that consists of at least one content component that belongs to the same media entity as the content component in the program element specified by self_component_UUID. Furthermore, the program element referenced by this component_UUID must have the same system clock reference as the program element referenced by self_component_UUID.

The purpose of the UUID (self_component_UUID and component_UUID) is that multiple independent processes (for instance, the stream encapsulation process) can assign and use an ID (the UUID) without having to consult each other or a common database to avoid an ID assignment collision or duplicate. As a result, multiple streams can be associated with each other without the confusion of an ID conflict or creation of another entity that has to remember what has already been assigned. The process of using the UUID makes interoperability easier since one doesn't have to coordinate the action of many encoders if, for instance, the encoded data were to be placed on one hard drive or in one database at a future time. component_relationship - the 8-bit field specifies the relationship between two content components in two program elements specified by self_component_UUID and component_UUID. The following types are currently defined:

Table 2 Defined relationship types of media content components

Value Description

0 current program element and another program element are independent

[1..64] current program element is dependent on another program element

1 current program element extends the spatial resolution of another program

element

2 current program element extends the SNR scalability of another program

element

3 current program element extends the temporal scalability of another program element

4 Current program element extends the view scalability of another program

element

5 Current program element is the left view

6 Current program element is the right view

Γ7..64] undefined

[65..128] another program element is dependent on current program element

65 another program element extends the spatial resolution of current program

element

66 another program element extends the SNR scalability of current program

element

67 another program element extends the temporal scalability of current program element

68 another program element extends the view scalability of current program

element

[129..255] undefined component_priority_id - the 8-bit field specifies the transport priority of the program element carried within the packets with the PID whose value is specified by elementary_PID. 0 indicates the highest priority, and a greater value indicates lower priority

The descriptor shall be included in the Program Map Table (PMT) in the descriptor loop for the elementary stream that carries at least one content component so that the relationship among the content components are associated with the components. More details about PMT can be found at Section 2.4.4.8 of Information technology - Generic coding of moving pictures and associated audio information: systems, ISO/IEC 13818-1:2007.

Figure 3 shows an example of using the invented descriptor. An SVC bitstream is encapsulated into three different MPEG-2 transport streams. These streams contain the NAL units from QVGA, SD and HD1080p spatial layers respectively. The invented descriptor is embedded in the Program Map Table of each stream. The figure shows the linkage from HD1080p.m2ts towards the other two streams. With the descriptor, it is then possible for the target decoder at the receiving device to learn that there are two associated component streams, the relationships between the streams, and their transport priorities.

In one embodiment, the packetizer 220 comprises a descriptor builder and an inserter. The descriptor builder builds a descriptor, such as the media composition descriptor, for each of the transport streams using the identified relationship. The inserter inserts the descriptors into program map tables corresponding to the plurality of transport streams.

In formats such as MPEG-2 transport stream, the encoded content in each content component is packetized into packetized elementary stream (PES) packets, which are further packetized into Transport Stream (TS) packets to form the separate transport stream for each content component. Another embodiment to associate the relationship information, such as those contained in the media composition descriptor, is to embed the information into the PES packet header without using a descriptor. In one implementation, the packetizer 220 comprises an inserter and two packetizing units, one for packetizing each of the plurality of content components into packetized elementary stream packets and one for further packetizing the packetized elementary stream packets into transport stream packets which form the separate transport stream. In a different implementation, the two packetizing units can be combined into one. Further, the inserter of the packetizer 220 adds identified relationship to a packet header of each packetized elementary stream packet.

To obtain desired content components at the receiver side, relationship information among the content components can be utilized. Fig. 4 shows the process of reading content components from a transport stream. The process first extracts a content component's relationship information from the transport stream at step 410. Content components related to the current content component are identified in step 420. In one embodiment wherein the media composition descriptors are employed for representing and transmitting the relationships among content components, step 410 extracts a descriptor from the transport stream for the content component, e.g. by extracting PMT that contains the descriptor, and the related content components can be identified in step 420 by analyzing the extracted descriptor to find related content components.

In a different embodiment, wherein the relationships among content components are embedded into the packet header of the PES, the related content components are identified by analyzing a packet header of each packet in a PES, which contains the content component, of the transport stream in step 420.

For each of the identified related content components, its relation type with the current content component is analyzed at step 440. Based on the relation type, a decision step 450 is carried out to determine if it is necessary to obtain said related content component. For instance, if the current content component extends the spatial resolution of another content component, e.g. said another content component is a base layer component and the current content component is an enhancement layer component, said another content component must be obtained. On the other hand, if said another content component extends the spatial resolution of the current content component, said another content component will be obtained based on the application requirements. One example is that when the receiver does not support the extended spatial resolution or the bandwidth is limited to only a base layer, it is not necessary to obtain said another content component to extend the resolution. Apparently, if there is no resolution or bandwidth limitations, said another content component shall be obtained at the receiver to enhance viewers' experience. A special case is that the content component under the examination loop is the current content component itself. Clearly, the decision step 450 would determine that the extraction of the component is necessary. When it is determined at step 450 that obtaining the related content component is necessary, references to the related content component shall be extracted in step 460. Such references can be an identifier of the content component to be obtained, such as the self_component_UUID or the component_UUID in the media composition descriptor. With the identifier or reference of the content component, a receiver can extract the corresponding component from the transport stream that is received as shown in step 470. If decision step 450 shows that it is not necessary to obtain said related content component, the process moves on to the next related content component in step 430.

After every related content component is examined, all necessary content components are processed at step 480 according to the application. For example, the processing can be decoding the content components and rendering them using a display.

In a different embodiment, a receiver can send a request to the sender with the identifiers of desired related content components and extract those content components after the sender sends the transport stream containing the requested content components.

Fig. 5 shows the structure of a stream reader to read content components from a transport stream. The stream reader comprises an extractor 510, an analyzer 520 and a processor 530. The extractor 510 extracts a content component's relationship information from a transport stream. The analyzer 520 examines the extracted relationship information and identifies related content components. According to the relationship between the current content component and the identified related content component as well as application requirements, some content components are necessary for the current content component and need to be obtained, and some are not. In cases where obtaining is necessary, the analyzer 520 sends the references to said related content components to the extractor 510 for extraction. In one scenario, the analyzer may send the references to the sender through a request and the extractor extracts the requested content components from a transport stream sent from the sender in responding to the request. A reference can be an identifier of a component stream. The processor 530 processes the extracted content components to generate output for further applications.

Although preferred embodiments of the present invention have been described in detail herein, it is to be understood that this invention is not limited to these embodiments, and that other modifications and variations may be effected by one skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims

1. A method for encapsulating a media entity containing a plurality of content components into a plurality of transport streams, the method comprising:

identifying relationships among the plurality of content components; packetizing each of the plurality of content components into a separate transport stream to arrive at a plurality of transport streams, each of which corresponds to a content component; and

associating the identified relationships with the plurality of transport streams.

2. The method of claim 1, further comprising:

multiplexing a subset of said plurality of transport streams for transmitting thereof to a receiver.

3. The method of claim 1, wherein the associating step further comprises:

building a descriptor for each of the transport streams based on the identified relationship; and

inserting the descriptors into program map tables corresponding to the plurality of transport streams;

4. The method of claim 3, wherein the descriptor is a media composition descriptor and the identified relationship comprises component identifiers, inter-component relationships and component transport priorities.

5. The method of claim 1, wherein the packetizing step comprises:

packetizing each of the plurality of content components into packetized elementary stream packets; and

further packetizing the packetized elementary stream packets into transport stream packets to form said separate transport stream;

and wherein the associating step further comprises

adding identified relationship to a packet header of each packetized elementary stream packet.

6. The method of claim 1, further comprising:

receiving, from a receiver, a request for at least one of the plurality of transport streams;

multiplexing the requested transport streams; and

sending the multiplexed transport stream to the receiver.

7. The method of claim 6, wherein the request comprises identifiers of said at least one of the plurality of transport streams.

8. The method of claim 1, wherein the transport stream is an MPEG-2 transport stream.

9. A stream encapsulator comprising:

a component extractor for extracting a plurality of content components from a media entity;

a packetizer for packetizing each of the plurality of content components into a separate transport stream to arrive at a plurality of transport streams, each of which corresponds to a content component; and

a correlator for identifying relationship among the plurality of content components and inputting the identified relationship to the packetizer for embedding into the transport streams.

10. The stream encapsulator of claim 9, further comprising:

a multiplexer for multiplexing a subset of said plurality of transport streams for transmitting thereof to a receiver.

11. The stream encapsulator of claim 10, wherein the multiplexer multiplexes at least one of the plurality of transport streams indicated by a request sent by a receiver.

12. The stream encapsulator of claim 11, wherein the request comprises identifiers of said at least one of the plurality of transport streams.

13. The stream encapsulator of claim 9, wherein the packetizer further comprises: a descriptor builder for building a descriptor for each of the transport streams based on the identified relationship; and

an inserter for inserting the descriptors into program map tables corresponding to the plurality of transport streams.

14. The stream encapsulator of claim 13, wherein the descriptor comprises a media composition descriptor and the identified relationship comprises component identifiers, inter-component relationships and component transport priorities.

15. The stream encapsulator of claim 9, wherein the packetizer further comprises: a packetizing unit for packetizing each of the plurality of content components into packetized elementary stream packets, and for packetizing the packetized elementary stream packets into transport stream packets which form said separate transport stream; and

an inserter for adding identified relationship to a packet header of each packetized elementary stream packet.

16. The stream encapsulator of claim 9, wherein the transport stream is an MPEG-2 transport stream.

17. A method for reading content components from a transport stream, the method comprising:

extracting a content component's component relationships from the transport stream;

identifying related content components to the content component from the extracted component relationships;

if necessary, obtaining at least one of said related content components; and processing said at least one obtained related content component.

18. The method of claim 17, wherein the extracting step comprises extracting a descriptor from the transport stream for the content component; and the identifying step comprises analyzing the descriptor to find said related content components.

19. The method of claim 17, wherein the identifying step comprises:

analyzing a packet header of each packet in a packetized elementary stream of the transport stream to find said related content components, the packetized elementary stream containing the content component.

20. The method of claim 18 or 19, wherein the analyzing step comprises extracting identifiers of said related content components.

21. The method of claim 17, wherein the obtaining step comprises:

extracting said at least one related content component from the transport stream.

22. The method of claim 17, wherein the obtaining step comprises:

sending a request for a transport stream comprising said at least one related content component; and

extracting said at least one related content component from the requested transport stream.

23. The method of claim 22, wherein the request comprises identifiers of said at least one related content component.

24. The method of claim 17, wherein the transport stream is an MPEG-2 transport stream.

25. A stream reader, comprising:

an extractor for extracting a content component's component relationships from a transport stream and for extracting related content components of said content component;

an analyzer for examining the extracted component relationships to identify said related content components, and if necessary, for sending references to said related content components to the extractor for extraction; and

a processor for processing the extracted related content components.

26. The stream reader of claim 25, wherein the extractor extracts a descriptor from the transport stream for the content component; and the analyzer analyzes the descriptor to identify said related content components.

27. The stream reader of claim 25, wherein the analyzer analyzes a packet header of each packet in a packetized elementary stream of the transport stream to find the related content components, the packetized elementary stream containing the content component.

28. The stream reader of claim 26 or 27, wherein the analyzer further extracts identifiers of said related content components.

29. The stream reader of claim 25, wherein the analyzer sends a request for a transport stream comprising said at least one related content component; and the extractor extracts said at least one related content component from the requested transport stream.

30. The stream reader of claim 29, wherein the request comprises identifiers of said at least one related content components.