KR101019010B1 - Preprocessor method and apparatus - Google Patents

Abstract

FIELD OF THE INVENTION The present invention generally relates to multimedia data processing, and more particularly, to processing operations performed before or in conjunction with data compression processing. A method of processing multimedia data includes receiving interlaced video frames, obtaining metadata for the interlaced video frames, and converting the interlaced video frames into sequential video using at least a portion of the metadata. And providing at least a portion of the sequential video and metadata to an encoder used to encode the sequential video. The method also includes generating spatial information and bidirectional motion information for the interlaced video frame, and generating sequential video based on the interlaced video frame and using spatial and bidirectional motion information. do.

Multimedia, Data Processing, Data Compression, Metadata, Sequential Video

Description

Preprocessor method and apparatus {PREPROCESSOR METHOD AND APPARATUS}

Claims of Priority under 35 U.S.C. §119

This patent application is issued to U.S. Provisional Application No. 60 / 789,048, filed Apr. 3, 2006, and assigned U.S. Provisional Application No. 60, filed April 4, 2006, all assigned to the assignee herein and hereby expressly incorporated by reference. / 789,266 and US Provisional Application No. 60 / 789,377, filed April 4, 2006.

background

Field

The present application generally relates to multimedia data processing, and more particularly, to a processing operation performed before or in conjunction with a data compression process.

background

summary

Each of the devices and methods of the invention described herein has several aspects, and no single aspect of those aspects is solely responsible for its desirable attributes. Hereinafter, more prominent features of the present invention will be briefly described without limiting its scope. After considering the present description, and in particular after reading the section referred to as the "detailed description", those skilled in the art will understand how the features of the present invention provide improvements to the multimedia data processing apparatus and method.

In one aspect, a method of processing multimedia data includes receiving interlaced video frames, converting interlaced video frames into sequential video, generating metadata associated with sequential video, and sequential video and meta Providing at least a portion of the data to an encoder used to encode the sequential video. The method may further comprise encoding the sequential video using metadata. In some aspects, the interlaced video frame includes NTSC video. Converting the video frame may include deinterlacing the interlaced video frame.

In some aspects, the metadata may include bandwidth information, bidirectional motion information, bandwidth ratios, complexity values such as time and / or spatial complexity, luminance information, and the spatial information may include luminance and / or chrominance information. Can be. The method may also include generating spatial information and bidirectional motion information for the interlaced video frame, and generating sequential video based on the interlaced video frame and using spatial and bidirectional motion information. have. In some aspects, converting the interlaced video frame includes inverse telecine of a 3/2 pulldown video frame, and / or resizing the sequential video. The method can further include dividing the sequential video to determine image group information, and the dividing step can include shot detection of the sequential video. Also, in some aspects, the method includes filtering the sequential video with a noise canceling filter.

In another aspect, an apparatus for processing multimedia data includes a receiver configured to receive an interlaced video frame, a deinterlacer configured to convert the interlaced video frame into sequential video, and generate metadata associated with the sequential video, and It may include a divider configured to provide metadata to an encoder used to encode sequential video. In some aspects, the apparatus can further include an encoder configured to receive the sequential video from the communication module and to encode the sequential video using the provided metadata. The deinterlacer may be configured to perform space-time deinterlacing and / or reverse telecine. The divider may be configured to perform shot detection and generate compressed information based on the shot detection. In some aspects, the divider may be configured to generate bandwidth information. The apparatus may also include a resampler configured to resize the sequential frames. The metadata may include bandwidth information, bidirectional motion information, bandwidth ratios, luminance information, spatial complexity values associated with the content, and / or time complexity values associated with the content. In some aspects, the deinterlacer is based on the interlaced video frame and is configured to generate spatial information and bidirectional motion information for the interlaced video frame and sequential video using spatial and bidirectional motion information.

Another aspect includes an apparatus for processing multimedia data, the apparatus for processing multimedia data comprising means for receiving interlaced video frames, means for converting interlaced video frames into sequential video, and generating metadata associated with sequential video. Means, and means for providing at least some of the sequential video and metadata to an encoder used to encode the sequential video. In some aspects, the conversion means comprises an inverse telecine device and / or a space-time deinterlacer. In some aspects, the generating means is configured to perform shot detection and generate compressed information based on the shot detection. In some aspects, the generating means is configured to generate bandwidth information. In some aspects, the means for generating comprises means for resampling to resize a sequential frame.

Another aspect, when executed, causes a machine to receive an interlaced video frame, convert the interlaced video frame into sequential video, generate metadata associated with the sequential video, and generate sequential video and metadata And a machine readable medium comprising instructions for processing multimedia data, the at least one portion being provided to an encoder used to encode sequential video.

Another aspect is an encoder used to receive interlaced video, convert the interlaced video into sequential video, generate metadata associated with the sequential video, and encode at least a portion of the sequential video and metadata to sequential video. It includes the processor, including the configuration provided to. Transformation of the interlaced video can include performing space-time deinterlacing. In some aspects, the transformation of the interlaced video includes performing inverse telecine. In some aspects, the generation of metadata includes generating compressed information based on the detection of shot changes. In some aspects, the generation of the metadata includes determining compression information of the sequential video. In some aspects, the configuration includes a configuration to resample the video to produce a resized sequential frame. In some aspects, the metadata may include bandwidth information, bidirectional motion information, complexity information such as temporal or spatial complexity information based on content, and / or compression information.

Brief description of the drawings

1 is a block diagram of a communication system for delivering streaming multimedia data.

2 is a block diagram of a digital transmission facility that includes a preprocessor.

3A is a block diagram of an exemplary embodiment of a preprocessor.

3B is a flowchart illustrating a process of processing multimedia data.

3C is a block diagram illustrating means for processing multimedia data.

4 is a block diagram illustrating operation of an exemplary preprocessor.

5 is a diagram of phase determination in an inverse telecine process.

6 is a flow chart illustrating a process for reverse telecine of telecined video.

7 is a trellis diagram illustrating the phase transition.

8 is a guide for identifying each frame used to generate a plurality of matrices.

9 is a flow chart illustrating how the metric of FIG. 8 is generated.

10 is a flowchart illustrating processing a metric to obtain an estimated phase.

11 is a data flow diagram illustrating a system for generating decision variables.

12 is a block diagram illustrating variables used to evaluate branch information.

13A, 13B and 13C are flowcharts illustrating how the lower envelope is calculated.

14 is a flowchart illustrating the operation of the coincidence detector.

15 is a flow diagram illustrating a process of calculating an offset for a decision variable used to compensate for a mismatch in phase determination.

16 represents inverse telecine operation after a pull down phase is estimated.

17 is a block diagram of a deinterlacer device.

18 is a block diagram of another deinterlacer device.

19 is a diagram of a subsampling pattern of an interlaced image.

20 is a block diagram of a deinterlacer device for generating a deinterlaced frame using Wmed filtered motion estimation.

21 illustrates an aspect of an aperture that determines a static area of multimedia data.

FIG. 22 is a diagram illustrating an aspect of an aperture that determines a slow motion region of multimedia data. FIG.

23 is a diagram illustrating an aspect of motion estimation.

24 illustrates two motion vector maps used for determining motion compensation.

25 is a flowchart illustrating a method of deinterlacing multimedia data.

FIG. 26 is a flowchart illustrating a method of generating a deinterlaced frame using space-time information.

27 is a flowchart illustrating a method of performing motion compensation for deinterlacing.

28 is a block diagram of a preprocessor including a processor configured for shot detection and other preprocessing operations in accordance with some aspects.

29 shows the relationship between encoding complexity C and allocated bit B. FIG.

30 is a flow diagram illustrating a process that processes groups of pictures and may be used in some aspects to encode video based on shot detection in video frames.

31 is a flowchart illustrating a process for shot detection.

32 is a flowchart illustrating a process of determining different shot classifications of a video.

33 is a flowchart illustrating a process of assigning a frame compression scheme to video frames based on shot detection results.

34 is a flowchart illustrating a process of determining a sudden scene change.

35 is a flow chart illustrating a process of determining a slowly changing scene.

36 is a flowchart illustrating a process of determining a scene including a camera flashlight.

37 shows a motion compensation vector between the previous frame MV _P and the current frame and the next frame MV _N.

38 is a graph illustrating the relationship to variables used to determine a frame difference metric.

FIG. 39 is a block diagram illustrating data encoding and residual calculation. FIG.

40 is a block diagram illustrating determination of a frame difference metric.

41 is a flowchart illustrating a procedure of assigning a compression type to a frame.

42 shows an example of 1-D poly-phase resampling.

43 is a graphic showing the safe action area and the safe title area of a data frame.

44 is a graphic illustrating a safe action region of a data frame.

details

The following description provides a detailed description to provide a thorough understanding of the examples of the invention. However, one of ordinary skill in the art appreciates that although all the details of an example or aspect of a process or device may not be described or illustrated herein, examples of the invention may be practiced. For example, the electrical component may be shown in block diagrams, not all electrical connections or all electrical components of the electrical component are illustrated so as not to obscure the examples of the present invention in too much detail. In other instances, such components, other structures and techniques may be shown in detail to further illustrate examples of the present invention.

Disclosed herein are aspects and aspects of certain inventions for preprocessors and methods of operating preprocessors that improve the performance of existing preprocessing and encoding systems. Such preprocessors can process metadata and video in preparation for encoding, including performing deinterlacing, inverse telecine, filtering, identification of shot types, processing and generation of metadata, and generation of bandwidth information. References herein to “an aspect”, “an aspect”, “an aspect”, or “an aspect” refer to one or more of the particular features, structures, or features described in connection with the aspect of the preprocessor system. It can be included in at least one embodiment. The use of such phrases in various places in the specification is not necessarily all referring to the same aspect, nor does it refer to separate or other aspects other than the other aspects. In addition, various features are described that may appear in some aspects but not in other aspects. Similarly, various steps are described that may be steps for some aspects but not steps for other aspects.

As used herein, “multimedia data” or “multimedia” is a broad term that includes video data (which may include audio data), audio data, or both video data and audio data. As used herein, “video data” or “video” is a broad term and refers to one or more image series or sequences or one image, including text, images, and / or audio data, and unless otherwise specified, multimedia It may be used to refer to data, or may be used interchangeably.

1 is a block diagram of a communication system 100 for delivering streaming multimedia. Such a system is applied to transmitting digitally compressed video to multiple terminals, such as shown in FIG. The digital video source can be, for example, a digital cable or satellite feed or an analog source to be digitized. The video source is processed at the transmission facility 120 and encoded and modulated on the carrier for transmission over the network 140 to one or more terminals 160. Terminal 160 decodes the received video and typically displays at least a portion of the video. Network 140 refers to any type of communication network, wired or wireless, suitable for transmitting encoded data. For example, network 140 may be a cellular telephone network, a wired or wireless local area network (LAN) or wide area network (WAN), or the Internet. Terminal 160 includes data, including mobile phones, PDAs, home or commercial video display equipment, computers (portable, laptop, handheld, PC, and large-scale server-based computer systems), and personal entertainment devices capable of utilizing multimedia data. It may be any type of communication device capable of receiving and displaying a, but is not limited thereto.

2 and 3 show exemplary embodiments of the preprocessor 202. In FIG. 2, the preprocessor 202 is in the digital transmission facility 120. Decoder 201 decodes the encoded data from the digital video source and provides metadata 204 and video 205 to preprocessor 202. Preprocessor 202 performs some type of processing on video 205 and metadata 204, and processes the processed metadata 206 (eg, base layer reference frame, enhancement layer reference frame, bandwidth information). , Content information) and a video 207 to the encoder 203. Such preprocessing of multimedia data can improve visual intelligibility, anti-aliasing, and compression efficiency of the data. In general, preprocessor 202 receives the video sequence provided by decoder 201 and converts the video sequence into a sequential video sequence for subsequent processing (eg, encoding) by the encoder. In some aspects, the preprocessor 202 may include reverse telecine, deinterlacing, filtering (eg, artifact elimination, de-ringing, deblocking, and denoising), resizing (eg, QVGA (Quarter) from standard definition (SD)). Video resolution array), and GOP structure generation (eg, complexity map generation, scene transition detection, and calculation of fade / flash detection).

3A performs a preprocessing operation on metadata 204 and video 205 received with a module or component (collectively referred to herein as a “module”), and then for subsequent processing (eg, , Preprocessor 202, configured to provide processed metadata 206 and sequential video 207. The module may be implemented in hardware, software, firmware, or a combination thereof. Preprocessor 202 includes one or more modules shown, inverse telecine 301, deinterlacer 302, denoiser 303, alias remover 304, resampler, all of which are further described below. 305, a deblocker / deringer 306, and a GOP divider 307. In addition, preprocessor 202 may include other suitable modules that may be used to process video and metadata, including memory 308 and communication module 309. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other type of storage medium known in the art. . An exemplary storage medium is coupled to the processor to enable the processor to read information from and write information to the storage medium. In the alternative, the storage medium may be integrally formed with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

3B is a flowchart illustrating a process 300 for processing multimedia data. Process 300 begins and proceeds to block 320 to receive the interlaced video. The preprocessor 202 shown in FIGS. 2 and 3 may perform this step. In some aspects, a decoder (eg, decoder 201 of FIG. 2) may receive interlaced data and provide this interlaced data to preprocessor 202. In some aspects, the data receiving module 330 shown as part of the preprocessor 202 in FIG. 3C may perform this step. Process 300 then proceeds to block 322 to convert the interlaced video into sequential video. The preprocessor 202 shown in FIGS. 2 and 3A, and the module 332 of FIG. 3C may perform this step. If the interlaced video is telecined, the processing of block 322 may include performing reverse telecine to generate sequential video. Process 300 can then proceed to block 324 to generate metadata associated with the sequential video. GOP divider 307 of FIG. 3A and module 334 of FIG. 3C may perform such processing. Process 300 then proceeds to block 326 to provide at least a portion of the sequential video and metadata to the encoder for encoding (eg, compression). The preprocessor 202 shown in FIGS. 2 and 3A and the module 336 of FIG. 3C can perform this step. After providing the sequential video and associated metadata to other components for encoding, process 300 may end.

3C is a block diagram illustrating means for processing multimedia data. Such means are shown herein as being included in the preprocessor 202. Preprocessor 202 includes means for receiving a video, such as module 330. The preprocessor 202 also includes means for converting interlaced data, such as module 332, into sequential video. Such means may include, for example, a space-time deinterlacer and / or a reverse telecine device. In addition, preprocessor 202 includes means for generating metadata associated with sequential video, such as module 334. Such means may include a GOP divider 307 (FIG. 3A) capable of generating various types of metadata as described above. In addition, preprocessor 202 may include means for providing sequential video and metadata to the encoder for encoding as shown by module 336. In some aspects, such means can include the communication module 309 shown in FIG. 3A. As will be appreciated by those skilled in the art, such means can be implemented in a number of standard ways.

The preprocessor 202 can use the acquired metadata (eg, obtained from the decoder 201 or other source) for one or more of the preprocessing operations. The metadata may include information related to describing or classifying the content (“content information”) of the multimedia data. In particular, the metadata may include content classification. In some aspects, the metadata may include content information required for the encoding operation. In such a case, the preprocessor 202 may be configured to determine the content information, use the content information for the preprocessing operation, and / or provide the content information to another component, such as, for example, the encoder 203. In some aspects, preprocessor 202 may use such content information to influence GOP segmentation, determine appropriate types of filtering, and / or determine encoding parameters communicated to the encoder.

4 illustrates an example embodiment of a process block that may be included in a preprocessor, which illustrates the processing that may be performed by preprocessor 202. In this example, preprocessor 202 receives metadata and video 204, 205 and provides output data 206, 207 including (processed) metadata and video to encoder 203. Typically, there are three types of video received by the preprocessor. Firstly, the received video may be sequential video, and deinterlacing does not have to be performed. Secondly, the video data can be telecined video, and the interlaced video is converted from a 24 fps movie sequence. Third, the video may be interlaced video that is not telecined. Preprocessor 226 may process these types of video as described below.

At block 401, the preprocessor 202 determines whether the received video 204, 205 is sequential video. In some cases, this may be determined from the metadata if the metadata includes such information, or may be determined by the processing of the video itself. For example, the inverse telecine process described below can determine whether the received video 205 is sequential video. If the received video is sequential video, the process proceeds to block 407 to perform a filtering operation on the video to reduce noise, such as white Gaussian noise. If the video is not sequential video, at block 401, the process proceeds to the phase detector of block 404.

Phase detector 404 distinguishes between video generated in telecine and video originating in a standard broadcast format. If a determination is made that the video has been telecined (“yes” decision path exiting phase detector 404), the telecined video is returned in its original format at inverse telecine 406. Duplicate fields are identified and removed, and fields derived from the same video are recreated as complete images. Since the sequence of reconstructed film images is photographed at regular intervals of 1/24 seconds, the motion estimation process performed at the GOP splitter 412 or decoder uses an inverse telecine image rather than telecine data with non-uniform time axes. More accurate.

In one aspect, the phase detector 404 makes some decisions after the reception of the video frame. These decisions include (i) whether the current video is from a telecine output and whether the 3: 2 pull down phase is one of the five phases P ₀ , P ₁ , P ₂ , P ₃ , and P ₄ shown in FIG. 5. , And (ii) whether the video was generated as conventional NTSC. The crystal is represented as phase P ₅ . These crystals appear as the output of the phase detector 404 shown in FIG. The path from the phase detector 404 labeled "Yes" indicates that the correct pull down phase has been provided to operate the inverse telecine 406 to classify and combine the fields formed from the same photographic image. Similarly, the path from the phase detector 404 labeled “No” activates the deinterlacer 405, apparently dividing the NTSC frame into fields for optimal processing. In addition, reverse telecine is also a concurrent US patent application owned by the assignee of the present invention "INVERSE TELECINE ALGORITHM BASED ON STATE MACHINE" and incorporated herein by reference in its entirety [representative Docket No. QFDM.021A (050943).

Phase detector 404 can analyze video frames continuously as different types of video may be received at any time. As an example, video conforming to the NTSC standard may be inserted into the video as commercial. After inverse telecine, the resulting sequential video is transmitted to a denoiser (filter) 407 that can be used to reduce white Gaussian noise.

If normal NTSC video is recognized (“no” path from phase detector 401), it is sent to deinterlacer 405 for compression. After the deinterlacer 405 turns the interlaced fields into sequential video, a denoising operation may be performed on the sequential video.

After appropriate inverse telecine or deinterlacing processing, at block 408, the sequential video is processed for alias removal and resampling (eg, resizing).

In addition, after resampling, the progressive video proceeds to block 410 where deblocker and de-ringing operations are performed. Two types of artifacts, "blocking" and "ringing", usually occur in video compression applications. Because the compression algorithm splits each frame into blocks (e.g., 8x8 blocks), blocking artifacts occur. Each block is reconstructed with some small error, and the error at the edge of the block is often contrasted with the error at the edge of the neighboring block, allowing the block boundary to be seen. On the other hand, the ringing artifacts appear as distortions around the edges of the image feature. Ringing artifacts occur because the encoder discards too much information when quantizing the high frequency DCT coefficients. In some illustrative examples, both deblocking and deringing can hide these visible artifacts using lowpass finite impulse response (FIR) filters.

After deblocking and deringing, the sequential video is processed by the GOP divider 412. GOP positioning may include detection of shot transitions, generation of complexity maps (eg, temporal, spatial bandwidth maps), and adaptive GOP segmentation. Shot detection relates to determining when a frame in a group of picture (GOP) represents data indicating that a scene change has occurred. Instead of inserting I-frames at fixed intervals, scene change detection can be used by the video encoder to determine the appropriate GOP length and insert the I-frame based on the GOP length. In addition, the preprocessor 202 can be configured to generate a bandwidth map that can be used to encode the multimedia data. Instead, in some aspects, a content classification module located outside the preprocessor generates a bandwidth map. Adaptive GOP segmentation can adaptively change the composition of the group of pictures coded together. An exemplary embodiment of the operation shown in FIG. 4 is described below.

Station telecine

Inverse telecine processing is described below, and an exemplary embodiment of reverse telecine is provided with reference to FIGS. 4-16. Video compression provides the best results when the characteristics of the source are known and used to select the ideally matched processing type. For example, off-the-air video can occur in several ways. Broadcast video, which is generated by conventional methods in video cameras and broadcast studios, conforms to the US NTSC standard. According to the NTSC standard, each frame consists of two fields. One field consists of odd lines and the other field consists of even lines. This may be referred to as an "interlaced" format. While the frame is generated at about 30 frames / second, the field is a record of the television camera image 1/60 second away. On the other hand, the film is taken at 24 frames / second so that each frame consists of a complete image. This may be referred to as a "sequential" format. In the case of transmission on NTSC equipment, the "sequential" video is converted into a "interlaced" video format via a telecine process. In one aspect, as described further below, the system of the present invention has the advantage of determining when the video is telecined and performing the appropriate conversion to play the original sequential frame.

5 illustrates the effect of telecineting a sequential frame converted to interlaced video. F ₁ , F ₂ , F ₃ , and F ₄ are sequential video input to a telecine device. The numbers "1" and "2" below are indications of either odd or even fields. Note that some fields are repeated taking into account inconsistencies between frame rates. 4 also shows the pull-down phases P ₀ , P ₁ , P ₂ , P ₃ , and P ₄ . Phase P0 is indicated by the first of two NTSC compatible frames having the same first field. The next four frames correspond to phases P ₁ , P ₂ , P ₃ , and P ₄ . Note that the frames indicated by P ₂ and P ₃ have the same second field. Since film frame F ₁ is scanned three times, an NTSC compatible first field of two identical continuous outputs is formed. Since all NTSC fields derived from film frame F ₁ are selected from the same film image, they are selected simultaneously. Other NTSC frames derived from the film may have adjacent fields that are 1/24 seconds apart.

The phase detector 404 shown in FIG. 4 makes some decisions after the reception of the video frame. These determinations include (i) whether the current video is from a telecine output and the five phases P ₀ , P ₁ , P ₂ , P ₃ , and 3: 2 pull down phases shown in the definition 512 of FIG. Whether it is one of P ₄ ; And (ii) whether the video was generated as conventional NTSC, and the determination is indicated as phase P ₅ .

These crystals appear as the output of the phase detector 401 shown in FIG. The path from the phase detector 401 labeled "Yes" indicates that an accurate pull down phase is provided to operate the inverse telecine 406 to classify and combine the fields formed from the same photographic image. Similarly, the path from phase detector 401 labeled "No" activates deinterlacer block 405, apparently dividing the NTSC frame into fields for optimal processing.

6 is a flowchart illustrating a process 600 for reverse telecine of a video stream. In one aspect, process 600 is performed by inverse telecine 301 of FIG. 3. Beginning at step 651, inverse telecine 301 determines a plurality of metrics based on the received video. In this aspect, four metrics are formed that are the sum of the differences between fields derived from the same frame or adjacent frames. In addition, the four metrics are assembled into Euclidean measurements of the distance between the four metrics derived from the received data and the most likely values of these metrics for each of the six hypothesized phases. Euclidean sum is referred to as branch information, and there are six such amounts for each received frame. Each hypothesized phase has a phase that follows, which phase changes with each received frame in the case of a possible pull down phase.

Possible transition paths are shown in FIG. 7 and indicated at 767. There are six such routes. The decision process maintains four measurements that are equivalent to the sum of Euclidean distance for each path of the hypothesized phase. Each Euclidean distance in the sum decreases with age so that the procedure responds to the changed condition. It is thought that the phase track whose minimum sum of Euclidean distances is the operating phase track is operated. The current phase of this track is referred to as the "applicable phase". At this time, unless P ₅ , inverse telecine based on the selected phase may occur. If P ₅ is selected, then at block 405 (FIG. 4), the current frame is deinterlaced using a deinterlacer. In summary, the applicable phase is used as the current pull down phase, or as an indicator to command the deinterlacing of frames estimated to have a valid NTSC format.

For every frame received from the video input, a new value for each of the four metrics is calculated. These are defined as follows.

The term SAD is an abbreviation for the term "summed absolute differences". In figure 8 the fields which are differentiated to form a metric are illustrated. Subscripts refer to field numbers, and letters refer to previous (= P) or current (= C). Brackets in FIG. 8 refer to pair-wise differences of fields. SAD _FS refers to the difference between field 1 of the current frame labeled C ₁ and field 1 of the previous frame labeled P ₁ , spanned by parentheses labeled FS in the definition provided in FIG. 8, and SAD _SS Refers to the difference between field 2 of the current frame labeled C ₂ and field 2 of the previous frame labeled P ₂ , spanned by parentheses labeled SS, and SAD _CO of the current frame labeled C ₂ . Refers to the difference between field 1 of the current frame labeled Field 2 and C ₁ (covered by parentheses labeled CO), and SAD _PO refers to field 1 of the current frame, both of which are covered by PO labeled parentheses. And the difference between field 2 of the previous frame.

In the following, the calculation load for evaluating each SAD is described. In conventional NTSC, there are about 480 active horizontal lines. In order to have a 4: 3 aspect ratio and have the same resolution in the horizontal direction, there must be a vertical line or degrees of freedom equivalent to 480 × 4/3 = 640. The video format of 640 x 480 pixels is one of the formats accepted by the Advanced Television Standards Committee (ATSC). Thus, every frame duration of 1/30 seconds, 640 x 480 = 307,200 new pixels are created. New data is generated at a rate of 9.2 × 10 ⁶ pixels / second, meaning that hardware or software running such a system processes the data at a rate of about 10 MB or more. This is one of the high speed parts of the system. This may be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. The SAD calculator may be a standalone component, included as hardware, firmware, middleware in a component of another device, implemented in microcode or software running on a processor, or a combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments that perform the calculation may be stored on a machine-readable medium, such as a storage medium. A code segment may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by communicating and / or receiving information, data, arguments, parameters, or memory contents.

The flowchart 900 of FIG. 9 explicitly illustrates the relationship of FIG. 8, which is a pictorial representation of equations (1) to (4). This shows the storage locations 941, 942, 943 and 944 where the most recent values of SAD _FS , SAD _CO , SAD _SS and SAD _PO are maintained, respectively. These are each generated by four absolute difference sum calculators 940, respectively, so that the luminance value 931 of the previous first field data, the luminance value 932 of the current first field data, and the luminance value of the current second field data ( 933 and the luminance value 934 of the previous second field data. In addition, which defines a metric, the term "value (i, j)" means the luminance value at positions i, j and adds to all the active pixels, but adds to an important subset of the active pixels. It is not excluded.

Flowchart 1000 of FIG. 10 is a detailed flow diagram illustrating a process of detecting telecined video and reversing it to reconstruct the original scanned film image. In step 1030, the metric defined in FIG. 9 is evaluated. Proceeding to step 1083, find the lower envelope values of the four metrics. The lower envelope of the SAD metric is a dynamically determined quantity, the highest numerical floor, below which no SAD penetrates. Proceeding to step 1085, branch information defined in Equations (5) to (10) below can be determined to use previously determined metrics, lower envelope values, and experimentally determined constants A. Since the successive phase values may not match, the amount Δ is determined in step 1087 to reduce this apparent instability. The phase is considered to coincide when the phase determination sequence coincides with the model shown in FIG. After that step, the process proceeds to step 1089 to calculate the decision variable using the current Δ value. The decision variable calculator 1089 derives and evaluates the decision variable using all the information generated in the block of 1080. Step 1030, step 1083, step 1085, step 1087, and step 1089 are extensions of the metric determination 651 of FIG. 6. From these variables, the applicable phase is obtained by the phase selector 1090. As shown, decision step 1091 uses the applicable phase to reverse telecine or deinterlace the telecined video. This is a more explicit representation of the phase detector 404 operation of FIG. In one aspect, the processing of FIG. 10 is performed by the phase detector 404 of FIG. Beginning at step 1030, the phase detector 404 determines a plurality of metrics by the process described above with reference to FIG. 8 and proceeds through steps 1083, 1085, 1087, 1089, 1090, and 1091. do.

Flowchart 1000 shows a process for estimating the current phase. The flow chart at step 1083 describes calculating branch information using the determined metric and lower envelope values. The branch information may be recognized as the Euclidean distance described above. Exemplary equations that may be used to generate branch information are Equations (5) through (10) below. The branch information amount is calculated at block 1209 of FIG.

The processed video data is stored in a storage medium that may include, for example, a disk type storage medium (eg, magnetic or optical) or a chip configured storage medium (eg, ROM, RAM) connected to a processor. Can be. In some aspects, reverse telecine 406 and deinterlacer 405 may each comprise some or all of a storage medium. The amount of branch information is defined by the equation

Specific details of the branch calculation are shown in the branch information calculator 1209 of FIG. As shown in the calculator 1209, the branching information development includes an amount such as L _{S which} is the lower envelope value of SAD _FS and SAD _SS , L _{P which} is the lower envelope value of SAD _PO , and L _{C which} is the lower envelope value of SAD _CO . Use them. The lower envelopes are used as distance offsets in the branch information calculation, used alone, or used with a predetermined constant A to produce H _S , H _P and H _C. The values are kept up to date in the lower envelope tracker described below. The H offset is defined as follows.

The process of tracking the values of L _S , L _P and L _C is represented in FIGS. 13A, 13B and 13C. For example, consider the tracking algorithm 1300 for L _{P shown} at the top of FIG. 13A. The metric SAD _PO is compared at the comparator 1305 with the current L _P value plus the threshold T _P. If the metric SAD _PO exceeds, then the current L _P value does not change, as shown in block 1315. If the metric SAD _PO does not exceed, as shown in block 1313, the new L _P value is a linear combination of SAD _PO and L _P. In another aspect for block 1315, the new L _P value is L _P + T _P.

Similarly, the amounts L _S and L _C of FIGS. 13B and 13C are calculated. The processing blocks of Figures 13A, 13B and 13C having the same function are given the same numbers but are primed ('or ") to indicate that they act on different sets of variables, for example, linear of SAD _PO and L _C. If a combination is formed, the operation is shown in block 1313 '. As in the case of L _P , another aspect for 1315' will replace L _C with L _C + T _C.

However, in the case of L _S , the algorithm of FIG. 13B alternately processes SAD _PS and SAD _SS to label each X because this lower envelope is applied to both variables. When the current SAD _FS value in block 1308 is read into the position for X in block 1303, then the current SAD _SS value in block 1307 is read into the position for X in block 1302, the alternation of the SAD _FS and SAD _SS values occurs. Occurs. As in the case for L _P , another aspect for 1315 ″ will replace L _S with L _S + T _S. The threshold and amount A currently used in the lower envelope value test are predetermined by experiment.

FIG. 11 is a flow diagram illustrating an example process for performing step 1089 of FIG. 10. 11 generally illustrates a process for updating decision variables. Six decision variables (corresponding to six possible decisions) are updated according to the new information derived from the metric. The decision variable is obtained as follows.

The amount α is less than 1 and limits the dependence of the decision variable on its past values, the use of α being equivalent to reducing the effect of each Euclidean distance as the data grows older. In flowchart 1162, the decision variables to be updated are described on the left as available on lines 1101, 1102, 1103, 1104, 1105, and 1106. Then, multiply each decision variable on the phase transition path of one of the phase transition paths by α, a number less than 1 in one of the blocks 1100, and then attenuate the old, determinant value by the attenuated decision variable. It is added to the current value of the branch information variable indexed by the next phase on the phase transition path that was present. This occurs at block 1110. The variable D ₅ is offset in block 1 193 by the amount Δ, and Δ is calculated in block 1112. As described below, the amount is chosen to reduce the mismatch of the phase sequence determined by the present system. In block 1120 the smallest decision variable is found.

In summary, new information unique to each decision is added to the previous value of the appropriate decision variable multiplied by α, to obtain the value of the current decision variable. Since a new decision can be made when the new metric can be controlled, the present technology can make a new decision upon reception of fields 1 and 2 every frame. These determinants are the sum of the Euclidean distances described above.

The applicable phase is chosen to have the subscript of the smallest decision variable. Decisions based on decision variables are made explicitly at block 1090 of FIG. Any decision is allowed in the decision space. As described in block 1091, these decisions are that (i) the applicable phase is not P5 (inverse telecine of video), and (ii) the applicable phase is P5 (deinterlacing of video).

Since metrics are derived from video that is inherently variable, there may sometimes be errors in the coherent string of decisions. This technique detects a phase sequence that is inconsistent with FIG. The operation is shown schematically in FIG. The algorithm 1400 stores the subscript (= x) of the current phase determination at block 1405 and the subscript (= y) of the previous phase determination at block 1406. In block 1410, if x = y = 5 is tested, in block 1411 the following values, i.e.

x = 1, y = 0; or

x = 2, y = 1; or

x = 3, y = 2; or

x = 4, y = 3; or

Whether x = 0 and y = 4 is tested. If either of the two tests is positive, a decision is declared to match at block 1420. If neither is positive, the offset shown in block 1193 of FIG. 11 is calculated in FIG. 15 and added to D ₅ , the decision variable associated with P ₅ .

In addition, the modification to D ₅ in FIG. 15 appears as part of process 1500, which provides a corrective action for mismatch in phase sequence. Assume that the consistency test failed at block 1510 of the flowchart 1500. Proceeding along the “No” branch beginning at block 1510, the next test of block 1514 is whether D ₅ > D _i for all i <5, or alternatively at least one of the variables D _i (i <5). Is greater than D ₅ . If the first case is valid, then at block 1516 the parameter δ with δ ₀ as the initial value is changed to 3δ ₀ . If the second case is valid, then at block 1517 δ is changed to 4δ ₀ . In block 152B, the value of Δ is updated to Δ _B , where

Returning to block 1510 again, assume that the decision string is determined to match. At block 1515, the parameter δ is changed to δ ₊ , defined as follows.

At block 152A, the new value of δ is inserted into the updated relationship for Δ Δ _A. This is described as follows.

Then, at block 1593, the updated Δ value is added to decision variable D ₅ .

Figure 16 illustrates how the reverse telecine process proceeds once the pull down phase is determined. Using this information, the fields 1605 and 1605 'are identified as representing the same video field. The two fields are averaged together and combined with field 1606 to reconstruct frame 1620. The reconstructed frame has reference number 1620 '. A similar process will reconstruct frame 1622. Fields derived from frames 1621 and 1623 are not duplicated. These frames are reconstructed by combining the first frame and the second frame together.

In the above aspect, each time a new frame is received, four new metric values are obtained, and a six-ply hypothesis set is tested using the newly calculated decision variable. Other processing structures can be adapted to calculate decision variables. The Viterbi decoder adds the metrics of the branches that make up the path together to form the path metric. The decision variables defined here are formed by similar rules, each of which is a "leaky" sum of new information variables. (For Ricky sum, the previous value of the decision variable is multiplied by a number less than 1 before new information data is added.) The Viterbi decoder structure can be modified to support the operation of this procedure.

Although this aspect is described in terms of processing a typical video in which new frames appear every 1/30 second, this process may be applied to frames that are recorded and processed backward in time. The decision space remains the same, but there is an insignificant change that represents the time reversal of the input frame sequence. For example, a string of coherent telecine crystals from a time inverted mode (shown here),

P ₄ P ₃ P ₂ P ₁ P ₀

It will also be reversed in time.

Using this variant of the first aspect, the decision process can make two attempts (one attempt is forward in time and the other forward in time) upon successful decision making. While the two attempts are not independent, each attempt will be different to process the metric in a different order.

This idea can be applied in the context of a buffer maintained to store future video frames that may require additional. If one video segment is determined to give unacceptably inconsistent results in the forward processing, the procedure may attempt to resolve the difficult video stretch by deriving future frames from the buffer and processing the frames in the reverse direction. will be.

In addition, the processing of video described in this application can be applied to video in a PAL format.

Deinterlacer

As used herein, “deinterlacer” is a broad term and refers to a deinterlacing system, device, or process (e.g., software configured to perform a process, that processes a wholly or substantial portion of interlaced multimedia data to form sequential multimedia data. Firmware, or hardware).

Broadcast video that is typically generated by a video camera or broadcast studio is in accordance with the US NTSC standard. A common way of compressing video is to interlace the video. In interlaced data, each frame consists of one of two fields. One field consists of odd lines of the frame and the other field consists of even lines of the frame. While the frame is generated at about 30 frames / second, the field is a record of the image of the television camera 1/60 second away. Each frame of the interlaced video signal represents every other horizontal line of the image. When a frame is projected onto the screen, the video signal alternates between even and odd lines. When this is done fast enough, for example at about 60 frames per second, the video image looks smooth to the human eye.

Interlacing has been used for decades in analog television broadcasts based on NTSC (US) and PAL (Europe) formats. Since only half of the image needs to be transmitted in each frame, the interlaced video uses approximately half the bandwidth of the entire image. The display format of the video generated as a result inside terminal 160 is not necessarily NTSC compatible and cannot easily display interlaced data. Instead, modern pixel-based displays (e.g., LCD, DLP, LCOS, Plasma, etc.) are sequentially scanned to display sequentially scanned video sources (however, many older video devices are spherical interlaced scans). Technology). Examples of some commonly used deinterlacing algorithms are "Scan rate up-conversion using adaptive weighted median filtering," P. Haavisto, J. Juhola, and Y. Neuvo, Signal Processing of HDTV II , pp. 703-710, 1990, and "Deinterlacing of HDTV Images for Multimedia Applications," R. Simonetti, S. Carrato, G. Ramponi, and A. Polo Filisan, in Signal Processing of HDTV IV , pp. 765-772, 1993.

An example of a deinterlacing aspect for a system and method that can be used alone or in combination to improve the performance of deinterlacing, and that can be used in the deinterlacer 405 (FIG. 4) is described. Such an aspect may include determining a first temporary deinterlaced frame by deinterlacing the selected frame using space-time filtering, determining a second temporary deinterlaced frame from the selected frame using bidirectional motion estimation and motion compensation, and also Combining the first and second temporary frames to form a final sequential frame. Space-time filtering may use a Wmed (weighted median) filter that may include a horizontal edge detector that prevents blurring of horizontal or near horizontal edges. Spatial-temporal filtering of previous and subsequent fields adjacent to the "current" field produces an intensity motion-level map that classifies a portion of the selected frame into different motion levels such as, for example, static, slow-motion, and fast motion. .

In some aspects, the intensity map is generated by Wmed filtering using filtering apertures including pixels from five adjacent fields (two previous fields, current field, and two next fields). Wmed filtering can determine forward, reverse, and bidirectional static region detection to effectively handle scene transitions and haunting objects. In various aspects, the Wmed filter can be used across one or more fields having the same parity in inter-field filtering mode, and intra-field filtering mode by tweaking the threshold criteria. Can be switched to. In some aspects, motion estimation and compensation utilizes luma (pixel intensity or brightness) and chroma data (pixel color information) to improve the deinterlacing area of the selected frame, where the brightness levels are nearly uniform but the colors are different. Denoising filters can be used to increase the accuracy of motion estimation. Denoising filters can be applied to the Wmed deinterlaced temporary frames to remove alias artifacts generated by Wmed filtering. The deinterlacing methods and systems described below produce good deinterlacing results and have relatively low computational complexity, allowing for deinterlacing implementations that run at high speeds, thereby allowing such implementations to be used in mobile phones, computers, and other types of electronic or communications using displays. It is suitable for a variety of deinterlacing applications, including the system used to provide data to the device.

Aspects of the deinterlacer and deinterlacing method are described herein with reference to various components, modules, and / or steps used to deinterlace multimedia data.

FIG. 17 is a block diagram illustrating an aspect of a deinterlacer 1700 that can be used as the deinterlacer 405 of FIG. 4. The deinterlacer 1700 includes a spatial filter 1730 that filters at least a portion of the interlaced data space and time (“space-time”) and generates space-time information. For example, Wmed can be used in spatial filter 1730. Further, in some aspects, the deinterlacer 1700 includes a denoising filter (not shown), such as, for example, a Weiner filter or a wavelet reduction filter. The deinterlacer 1700 also includes a motion estimator 1732 that provides motion estimation and compensation of selected frames of interlaced data and generates motion information. The combiner 1734 receives and combines the space-time information and the motion information to form a sequential frame.

18 is another block diagram of deinterlacer 1700. The processor 1836 in the deinterlacer 1700 includes a spatial filter module 1838, a motion estimation module 1840, and a combiner module 1842. Interlaced multimedia data from an external source 1848 may be provided to the communication module 1844 in the deinterlacer 1700. The deinterlacer, and its components or steps, may be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. For example, the deinterlacer may be a standalone component, included as hardware, firmware, middleware in a component of another device, implemented in microcode or software running on a processor, or a combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform deinterlacer operations may be stored on a machine-readable medium, such as a storage medium. A code segment may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by communicating and / or receiving information, data, arguments, parameters, or memory content.

The received interlaced data may include, for example, a chip configured storage medium (eg, ROM, RAM) or disk type storage medium (eg, magnetic or optical) coupled to the processor 1836. And may be stored in a deinterlacer 1700 in a storage medium 1846. In some aspects, the processor 1836 may include some or all of the storage medium. The processor 1836 is configured to process the interlaced multimedia data to form a sequential frame, which is provided to another device or process.

Conventional analog video devices, such as televisions, render video in an interlaced manner, that is, such devices transmit scan lines with even numbers (even fields) and scan lines with odd numbers (odd fields). In terms of signal sampling, this is equivalent to space-time subsampling in the pattern described as follows.

Here, Θ denotes an original frame image, F denotes an interlaced field, and ( x, y, n ) denotes horizontal, vertical, and temporal positions of pixels, respectively.

Without losing generality, since n = 0 is an even field throughout this disclosure, equation (23) can be simplified as follows.

Since the decimation is performed in the horizontal direction, the subsampling pattern can be displayed at the next n to y coordinates. In Fig. 19, both the circle shape and the star shape indicate a position where the original full-frame image has sample pixels. The interlacing process decimates star pixels without affecting the circular pixels. Note that since the vertical position is indexed from zero, the even field is the top field and the odd field is the bottom field.

The purpose of the deinterlacer is to convert the interlaced video (field sequence) into an uninterlaced sequential frame (frame sequence). In other words, the full-frame image is "restored" or generated by interpolating even and odd fields. This can be expressed by equation (25).

Here, F _i represents the deinterlacing result for the lost pixel.

20 is a block diagram illustrating some aspects of an aspect of a deinterlacer that generates sequential frames from interlaced multimedia data using Wmed filtering and motion estimation. The upper portion of FIG. 20 can be generated using information from the current field, two previous fields (PP field and P field), and two subsequent fields (Next Field and Next Next Field). A motion intensity map 2052 is shown. As described in more detail below, motion intensity map 2052 may classify or divide the current frame into two or more different motion levels, and may be generated by space-time filtering. As described below with reference to equations (4)-(8), in some aspects, a motion intensity map 2052 is generated to identify static regions, slow-motion regions, and fast-motion regions. For example, a space-time filter, such as Wmed filter 2054, filters the interlaced multimedia data using criteria based on the motion intensity map, and generates a space-time temporary deinterlaced frame. In some aspects, the Wmed filtering process involves a horizontal neighborhood of [-1, 1], a vertical neighborhood of [-3, 3], and a time neighborhood of five adjacent fields, which is illustrated by the five fields shown in FIG. PP field, P field, current field, next field, next next field), Z- ¹ represents a delay of one field. For the current field, the next field and the P field are non-parity fields, the PP field and the next next field are parity fields. “Nearby” used for space-time filtering may refer to the spatial and temporal location of fields and pixels actually used during the filtering operation, and may be represented as, for example, “aperture” as shown in FIGS. 21 and 22. have.

The deinterlacer may also include a denoiser (noise cancellation filter) 2056. Denoiser 2056 is configured to filter the space-time temporary deinterlaced frame generated by Wmed filter 2056. Noise canceling the space-time temporary deinterlaced frame makes the subsequent motion search process more accurate, especially when the source interlaced multimedia data sequence is contaminated by white noise. In addition, an alias between even and odd rows in the Wmed image may be at least partially removed. Denoiser 2056 may also be implemented as various filters, including the wavelet reduction and wavelet winner filters described further below.

The lower part of FIG. 20 illustrates an aspect of determining motion information (eg, motion vector candidate, motion estimation, motion compensation) of interlaced multimedia data. In particular, FIG. 20 shows the "final" sequential frame (shown as deinterlaced current frame 2064), which is then used to generate a motion compensated temporary sequential frame of the selected frame and then combined with the Wmed temporary frame. A motion estimation and motion compensation scheme to form is shown. In some aspects, a motion vector (MV) candidate (or estimate) of interlaced multimedia data is provided to a deinterlacer from an external motion estimator and used to provide a starting point for a bidirectional motion estimator and compensator 2068 (ME / MC). do. In some aspects, the MV candidate selector 2072 is previously determined for the neighboring block for the MV candidate of the processing block, such as, for example, the MV of a previously processed block, such as the block in the deinterlaced previous frame 2070. Use MV. Motion compensation can be done in both directions based on the previous deinterlaced frame 2070 and the next (eg, future) Wmed frame 2058. The current Wmed frame 2060 and the motion compensated (MC) current frame 2066 are merged or combined by combiner 2062. The resulting deinterlaced current frame 2064, i.e., sequential frame, is provided back to the ME / MC 2068 to be used as the deinterlaced previous frame 2070 and also to, for example, a compression and display terminal. It is communicated outside the deinterlacer for subsequent processing such as transmission. Various aspects shown in FIG. 20 are described in more detail below.

25 illustrates a process 2500 for processing multimedia data to generate a sequence of sequential frames from a sequence of interlaced frames. In one aspect, the sequential frame is generated by the deinterlacer 405 shown in FIG. At block 2502, process 2500 (process A) generates space-time information for the selected frame. The spatio-temporal information may include information used to classify the motion level of the multimedia data to generate a motion intensity map, and may include the Wmed temporary deinterlaced frame and the information used to generate the frame (e.g., equation (26). ) To information used in formula (33). This process may be performed by the Wmed filter 2054, as shown at the top of FIG. 20, the associated process described in more detail below. In process A shown in FIG. 26, as further described below, in block 2602, regions are classified into fields of different motion levels.

Next, at block 2504 (process B), process 2500 generates motion compensation information for the selected frame. In one aspect, the bidirectional motion estimator / motion compensator 2068 shown at the bottom of FIG. 20 may perform this process. Process 2500 then proceeds to block 2506 to form a sequential frame associated with the selected frame by deinterlacing the fields of the selected frame based on the space-time information and the motion compensation information. This can be done by the coupler 2062 shown at the bottom of FIG. 20.

Motion intensity map

For each frame, the motion intensity map 2052 can be determined by processing the pixel in the current field to determine regions of different “motion”. Exemplary aspects for determining the three categories of motion intensity maps are described below with reference to FIGS. 21-24. The motion intensity map designates regions of each frame as static regions, slow-motion regions, and fast motion regions based on comparing pixels in the same parity field and different parity fields.

Static area

Determining the static region of the motion map may include processing a pixel in the vicinity of the adjacent field to determine whether the luminance difference of the given pixel (s) meets a certain criterion. In some aspects, determining the static area of the motion map includes determining of five adjacent fields (current field (C), two fields in time before the current field, and two fields in time after the current field). Processing nearby pixels to determine whether the luminance difference of the given pixel (s) satisfies a certain threshold. These five fields are shown in FIG. 20, where Z- ¹ represents the delay of one field. That is, typically five adjacent fields will be indicated in such a sequence with a time delay of Z ⁻¹ .

21 illustrates an aperture identifying a constant pixel of each of the five fields that may be used for space-time filtering in accordance with some aspects. The aperture contains a 3x3 pixel group of previous previous field (PP), previous field (P), current field (C), next field (N), and next next field (NN), from left to right. . In some aspects, the area of the current field is considered static in the motion map if it satisfies the pixel position shown in FIG. 21 and its corresponding field, the criteria described in equations (26) to (28). .

And

or

Where T ₁ is the threshold,

L _P is the luminance of pixel P located in the P field,

L _N is the luminance of pixel N located in the N field,

L _B is the luminance of pixel B located in the current field,

L _E is the luminance of pixel E located in the current field,

L _BPP is the luminance of pixel B _PP located in the PP field,

L _EPP is the luminance of pixel E _PP located in the PP field,

L _BNN is the luminance of pixel B _NN located in the NN field,

L _ENN is the luminance of pixel E _NN located in the NN field.

The threshold T ₁ may be predetermined and set to a specific value, determined and provided by a process other than deinterlacing (eg, as metadata for the deinterlaced video), or may be determined dynamically during deinterlacing.

The static region criteria described above in Equations (26), (27), and (28) use more fields than conventional deinterlacing techniques for at least two reasons. First, comparisons between the same parity fields have lower aliases and phase-inconsistencies than comparisons between different parity fields. However, the minimum time difference (and thus the correlation) between the field being processed and its vicinity near the same parity field is two fields, which is greater than the minimum time difference from near its different parity field. The combination of more reliable different parity fields and lower aliased parity fields can improve the accuracy of static region detection.

In addition, as shown in FIG. 21, five fields may be distributed symmetrically in the past and the present with respect to the pixel X in the current frame C. FIG. The static region can be subdivided into three categories: forward static (static for previous frame), reverse static (static for next frame), or bidirectional static (if both forward and reverse criteria are met). This segmentation of static regions can improve performance, especially during scene transitions and target appearances.

Slow-motion area

When the luminance value of a certain pixel does not satisfy the criteria designated as the static region but satisfies the criteria designated as the slow-motion region, it may be regarded as a slow-motion region in the motion-map. Equation (29) below defines the criteria that can be used to determine the slow-motion region. Referring to Fig. 22, the locations of pixels Ia, Ic, Ja, Jc, Ka, Kc, La, Lc, P, and N identified in equation (29) are shown in the aperture centered on pixel X. The aperture includes a neighborhood of 3x7 pixels of the current field C and a neighborhood of 3x5 of the next field N and the previous field P. FIG. The pixel X is regarded as part of the slow-motion region when the pixel in the aperture does not satisfy the criteria for the static region described above and satisfies the criteria shown in the following equation (29).

Where T ₂ is the threshold,

은 각각 화소 Ia, Ic, Ja, Jc, Ka, Kc, La, Lc, P 및 N 에 대한 휘도 값이다.

Are the luminance values for pixels Ia, Ic, Ja, Jc, Ka, Kc, La, Lc, P and N, respectively.

The threshold T ₂ may also be predetermined and set to a specific value, provided by a process other than deinterlacing (eg, as metadata for the deinterlaced video), or determined dynamically during deinterlacing.

It should be noted that because of the angle of its edge detection capability, one filter can blur edges that are horizontal (eg, greater than 45 ° from vertically aligned). For example, the edge detection capability of the aperture (filter) shown in FIG. 22 is influenced by the angle formed by the pixels A and F or C and D. Any edge that is horizontal than such an angle that will not be optimally interpolated, and therefore stepped artifacts, may appear at that edge. In some aspects, the slow-motion category can be divided into two subcategories. "Horizontal edge" and "other" describe this edge detection effect. The slow-motion pixel may be classified as a horizontal edge if it satisfies the criterion of equation (30) shown below, and may be classified as a so-called "other" category if it does not satisfy the criterion of equation (30).

Here, T ₃ is a threshold value, and LA, LB, LC, LD, LE, and LF are luminance values of pixels A, B, C, D, E, and F.

Different interpolation methods may be used for each of the horizontal edges and other categories.

Fast-motion zone

If the criteria for the static region and the criteria for the slow-motion region are not satisfied, the pixel may be considered to be in the fast-motion region.

After classifying the pixels in the selected frame, process A (FIG. 26) proceeds to block 2604 to generate a temporary deinterlaced frame based on the motion intensity map. In this aspect, the Wmed filter 2054 (FIG. 20) filters the selected field and the necessary adjacent field (s) to provide a candidate full-frame image F ₀ that can be defined as follows.

_{Where, α i (i = 0,} 1, 2, 3) is a constant weight is calculated as follows:

As shown at the bottom of FIG. 20, the Wmed filtered temporary deinterlaced frame is provided for subsequent processing in connection with the motion estimation and motion compensation process.

As shown in equation (31) and described above, static interpolation includes interfield interpolation, and slow-motion and fast-motion interpolation include intrafield interpolation. In certain aspects in which time (eg, inter-field) interpolation of the same parity field is undesirable, time interpolation sets the threshold T ₁ (Equations (4) to (6)) to 0 ( T ₁ = 0). Can be "disabled" by doing so. By processing the current field with temporal interpolation disabled, no area of the motion-level map is classified as static, and the Wmed filter 2054 (FIG. 20) is applied to the current field and two adjacent non-parity fields. Three fields shown in the aperture of FIG. 22 are used.

Denoising

In some aspects, the denoiser may be used to remove noise from the candidate Wmed frame before further processing using the motion compensation information. The denoiser can remove the noise present in the Wmed filter and maintain the existing signal regardless of the frequency content of the signal. Various types of denoising filters can be used, including wavelet filters. Wavelets are function classes used to localize a given signal in both spatial and scaling domains. The idea underlying a wavelet is to analyze the signal at different scales or resolutions such that a small change in the wavelet representation thus produces a small change in the original signal.

In some embodiments, the denoising filter is based on one aspect of a (4, 2) bi-orthogonal tertiary B-spline wavelet filter. One such filter can be defined by the following forward and inverse transforms.

And

로 표시된다. 이는 최고 주파수 부대역 계수를 0.6745 로 나눈 값의 평균 절대 편차 (median absolute deviation) 로서 추정될 수 있다. 그러한 필터의 구현은 "Ideal spatial adaptation by wavelet shrinkage," D.L. Donoho and I.M. Johnstone, Biometrika, vol. 8, pp. 425-455, 1994 에 더 설명되어 있으며, 이는 본원에서 그 전체를 참조로서 병합하고 있다.By applying a denoising filter, the accuracy of motion compensation in a noisy environment can be increased. The noise in the video sequence is assumed to be an additive white gauss. The variance of the estimated noise is

Is displayed. This can be estimated as the median absolute deviation of the highest frequency subband coefficient divided by 0.6745. Implementations of such filters are described in “Ideal spatial adaptation by wavelet shrinkage,” DL Donoho and IM Johnstone, Biometrika , vol. 8, pp. 425-455, 1994, which is hereby incorporated by reference in its entirety.

In addition, wavelet reduction or wavelet winner filters may be applied as the denoiser. Wavelet reduction denoising may involve reduction in the wavelet transform region, and typically includes three steps: linear forward wavelet transform, nonlinear reduced denoising, and linear reverse wavelet transform. The Wiener filter is an MSE-optimal linear filter that can be used to improve images degraded by additive noise and blurring. Such filters are well known in the art and described, for example, "Ideal spatial adaptation by wavelet shrinkage," and "Improvement Wavelet denoising via empirical Wiener filtering," Proceedings of SPIE , by SPGhael, AMSayeed, and RGBaraniuk. , vol 3169, pp. 389-399, San Diego, July 1997.

Motion compensation

Referring to FIG. 27, at block 2702, process B performs bidirectional motion estimation, and then at block 104 performs motion compensation using the motion estimate, which is further illustrated in FIG. 20 and illustrated below. It is described in the aspect. There is a field of "lag" between the Wmed filter and the motion compensation based deinterlacer. As shown in Fig. 23, the motion compensation information for the "lost" data (non-original rows of pixel data) of the current field "C" includes both the previous frame "P" and the next frame "N". Is predicted from the information. In the current field (FIG. 23), the solid line represents the row where the original pixel data exits, and the dotted line represents the row where the Wmed-interpolated pixel data exits. In some aspects, motion compensation is performed near four rows by eight columns of pixels. However, such a pixel neighborhood is an example for explanation, and those skilled in the art may perform motion compensation based on the pixel neighborhood including a different number of rows and a different number of columns in another aspect, and the selection is for example. It can be appreciated that a number of factors can be based on calculation speed, available processing power, or features of the deinterlaced multimedia data. Since the current field has only half of that row, the four rows that actually match correspond to an 8 pixel by 8 pixel area.

Referring to FIG. 20, the bidirectional ME / MC 2068 predicts for the Wmed current frame 2060 for the deinterlaced current frame 2070 and the Wmed next frame 2058 using sum of squared error (SSE). The similarity between the block and the predicted block can be measured. The generation of the motion compensated current frame 2066 then fills in the missing data between the original pixel lines using the pixel information from the most similar matching block. In some aspects, the bidirectional ME / MC 2068 weights or biases pixel information from the deinterlaced previous frame 2070 information, because the frame is generated by motion compensation information and Wmed information. This is because the Wmed next frame 2058 is only deinterlaced by space-time filtering.

In some aspects, one or more luma groups of pixels (eg, such as one 4 row by 8 column luma block) to improve matching performance in areas of a field having similar luma areas but with different chroma areas and A metric can be used that includes the contribution of pixel values of one or more chroma groups of pixels (such as, for example, two two row by four column chroma blocks U and V). Such an approach effectively reduces mismatch in color sensitive areas.

The motion vector MV has a granularity of 1/2 pixel in the vertical dimension and a particle size of 1/2 or 1/4 pixel in the horizontal direction. To obtain fractional pixel samples, interpolation filters can be used. For example, some filters that can be used to obtain half pixel samples are interlinear filters (1, -5, 20, 20), which are recommended by H.263 / AVC: bilinear filters (1, 1). , -5, 1), and a six-tap Hamming window sinc function filter (3, -21, 147, 147, -21, 3). The quarter-pixel samples can be generated from full and half pixel samples by applying a dual linear filter.

In some aspects, motion compensation uses various types of retrieval processes to provide data (eg, at some location in the current frame) with respect to corresponding data at different locations within another frame (eg, next frame or previous frame). , Data representing the object), and the positional difference in each frame represents the motion of the object. For example, the search process may use a full motion search that may cover a larger search area, a fast motion search that may use fewer pixels, and / or a selected pixel used in a search pattern, for example. It may have a specific shape, such as a diamond shape. In the case of a fast motion search, the search region can be centered around a motion estimate, or motion candidate, which can be used as a starting point for searching for adjacent frames. In some aspects, MV candidates may be generated from an external motion estimator and provided to a deinterlacer. In addition, the motion vector of the macroblock from the corresponding neighborhood in the previous motion compensated adjacent frame may be used from the motion estimate. In some aspects, the MV candidate can be generated by searching for the vicinity of the macroblocks (eg, 3 macroblocks x 3 macroblocks) of the corresponding previous and next frames.

FIG. 24 shows an example of two MV maps, MV _P and MV _N , which are generated during motion estimation / compensation by searching for the vicinity of the previous frame and the next frame as shown in FIG. Can be. For both MV _P and MV _N , the block to be processed to determine the motion information is the center block indicated by "X". For both MV _P and MV _N , there are nine MV candidates that can be used during motion estimation of the current block X being processed. In this example, four of the MV candidates are in the same field from the previously performed motion search and are represented by brighter colored blocks in MV _P and MV _N (FIG. 24). Five other MV candidates, represented by darker colored blocks, were duplicated from the motion information (or map) of the previously processed frame.

After motion estimation / compensation is completed, two interpolation results may arise from the missing row (indicated by the dashed line in FIG. 23): one interpolation result generated by the Wmed filter (Wmed current frame 2060 FIG. 20), and motion One interpolation result generated by the motion estimation process of the compensator (MC current frame 2066). Typically, the combiner 2062 merges the Wmed current frame 2060 and the MC current frame 2066 by using at least a portion of the Wmed current frame 2060 and the MC current frame 2066, so that the currently deinterlaced frame 2064 Create However, under certain conditions, the combiner 2062 may generate the current deinterlaced frame using only either the current frame 2060 or the MC current frame 2066. In one example, combiner 2062 merges Wmed current frame 2060 and MC current frame 2066 to produce a deinterlaced output signal as shown in equation (36).

은 ^t 가 전치 행렬인 위치

에 있는 필드

에서의 휘도 값을 위해 이용된다. 다음과 같이 정의된 클립 함수를 이용하면,here,

Is a position where ^t is a transpose matrix

Field in

Is used for the luminance value at. Using the clip function defined as

k ₁ Can be calculated as follows.

Where C ₁ is a robustness parameter and Diff is a luma between the available pixels in the predicted frame (selected from existing fields) and the predicting frame pixels. By appropriately selecting C ₁ , it is possible to adjust the relative importance of the mean squared error. k ₂ can be calculated as shown in equation (39).

는 모션 벡터이고, δ 는 0 으로 나누는 것을 방지하기 위한 작은 상수이다. 또한, 필터링을 위한 클립핑 함수를 이용하여 디인터레이싱하는 것은 "De-interlacing of video data," G.D.Haan and E.B.Bellers, IEEE Transactions on Consumer Electronics, Vol.43, NO.3, pp.819- 825, 1997 에 설명되어 있으며, 그 전체를 본원에서 참조로서 병합하고 있다.here,

Is a motion vector and δ is a small constant to prevent division by zero. Deinterlacing using a clipping function for filtering is also described in "De-interlacing of video data," GDHaan and EBBellers, IEEE Transactions on Consumer Electronics, Vol. 43, NO.3, pp.819-825, 1997. The entirety of which is incorporated herein by reference.

In some aspects, coupler 2062 can be configured to try and maintain the following equation to achieve high PSNR and robust results.

It is possible to separate the de-interlacing prediction scheme, including inter-field interpolation, from the inter-field interpolation using the Wmed + MC deinterlacing scheme. That is, space-time Wmed filtering can be used primarily for intrafield interpolation, while intrafield interpolation can be performed during motion compensation. This reduces the peak signal-to-noise ratio of the Wmed result, but makes the visual quality after motion compensation more satisfactory, because bad pixels will be removed from the Wmed filtering process as a result of inaccurate interfield prediction mode determination.

Chroma processing may be compatible with associated luma processing. In terms of motion map generation, the motion level of the chroma pixel is obtained by observing the motion level of the four associated luma pixels. The operation can be based on voting (the chroma motion level borrows the dominant luma motion level). However, we suggest using the following conventional approach. If any one of the four luma pixels has a fast motion level, the chroma motion level will be fast motion; On the other hand, if any one of the four luma pixels has a slow motion level, the chroma motion level will be slow motion; In other cases, the chroma motion level is static. The conventional approach may not achieve the highest PSNR, but avoids the risk of using INTER prediction whenever there is ambiguity in the chroma motion level.

The multimedia data sequences were deinterlaced using only the Wmed algorithm described herein, or using the Wmed algorithm and a motion compensated algorithm together. In addition, the same multimedia data sequence was deinterlaced using a pixel blending (or averaging) algorithm and a "no-deinterlacing" case, where the fields were simply combined without any interpolation or blending. The resulting frame was analyzed to determine the PSNR, which is shown in the following table.

PSNR
(dB)
sequence De-Interlacing
none Blending Wmed Wmed +
MC soccer 8.955194 11.38215 19.26221 19.50528 city 11.64183 12.93981 15.03303 15.09859 crew 13.32435 15.66387 22.36501 22.58777

Although there is only marginal improvement by deinterlacing with MC in addition to Wmed, the visual quality of the deinterlaced image produced by combining Wmed and MC interpolation results is visually more satisfactory, because, as described above, Similarly, combining Wmed and MC results suppresses aliasing and noise between even and odd fields.

In some resampling aspects, a poly-phase resampler is implemented for image size resizing. In one example of downsampling, the ratio between the original image and the resized image may be p / q , where p and q are integers that are prime to each other. The total number of phases is p . In some embodiments, the cutoff frequency of the polyphase filter is 0.6 for a resizing coefficient of about 0.5. To increase the high frequency response of the resized sequence, the cutoff frequency does not exactly match the resizing rate. This inevitably allows some aliasing. However, it is well known that humans prefer a clear but slightly aliased image over a blurred, aliased image.

42 shows an example of polyphase resampling showing the phase when the resizing ratio is 3/4. In addition, the cutoff frequency shown in FIG. 42 is 3/4. The original pixel is shown in the vertical axis in FIG. 42 above. In addition, the filter waveform is represented by showing the sinc function about its vertical axis. Since the cutoff frequency is chosen to be exactly the same as the resampling ratio, zero of the sinc function overlaps the pixel position after resizing, which is shown by the intersection in FIG. To obtain the pixel value after resizing, the contributions from the original pixel can be summed as shown in the following equation.

Where f _c is the cutoff frequency. The 1-D polyphase filter can be applied to both horizontal and vertical dimensions.

Another aspect of resampling (resizing) describes overscan. In an NTSC television signal, one image may have 486 scan lines, and in digital video, each pixel may have 720 pixels. However, because of the discrepancy between size and screen format, not all of the entire image can be seen on a television. The part of the image that is not visible is referred to as overscan.

To help broadcasters provide useful information in as many television-viewable areas as possible, SMPTE (Society of Motion Picture & Television Engineers) is referred to as the safe action area and the safe title area. The specific size of the action frame to be defined. See SMPTE RP (recommended practice) 27.3-1989 for Specifications for Safe Action and Safe Title Areas Test Pattern for Television Systems. The SMPTE defines the safe action area as the "where all important actions must take place". The safe title area is defined as an area where "all useful information can be defined to ensure visibility for multiple home television receivers." For example, as shown in FIG. 43, the safe action area 4310 occupies 90% of the screen center, giving a 5% boundary to all surroundings. The safe title area 4305 occupies 80% of the center of the screen, giving a 10% border.

Referring to FIG. 44, some broadcasts will include text in the safe action area because the safe title area is too small to add more content to the image, which is inside the white rectangular window 4415. Normally, black borders are visible in overscan. For example, in FIG. 44, black borders appear at the top 4420 and bottom 4425 of the image. These black borders can be removed from overscan because H.264 video uses boundary expansion in motion estimation. The expanded black border can increase the residual. Normally, the boundary can be cut by 2% and then resized. Accordingly, a filter for resizing may be generated. Truncation is performed to eliminate overscan before polyphase downsampling.

Deblocking / Deringing

In one example of the deblocking process, a deblocking filter may be applied to all 4 × 4 block edges of a frame except for the edge at the border of the frame and any edge at which the deblocking filter process is disabled. This filtering process will be performed based on the macroblock after completion of the frame construction process in which all macroblocks in the frame are processed in order of increasing macroblock addresses. For each macroblock, the vertical edges are filtered first, left to right, then horizontal edges from top to bottom. As shown in FIG. 39, for the horizontal and vertical directions, the luma deblocking filter process is performed on four 16-sample edges, and the deblocking filter process for each chroma component is two 8-sample edges. Is performed for. Sample values at the top and left of the current macroblock that may have already been modified by the deblocking process operation for the previous macroblock will be used as input to the deblocking filter process for the current macroblock, and also It may be modified during filtering. The sample values modified during the filtering of the vertical edges can be used as input to the filtering of the horizontal edges for the same macroblock. The deblocking process can be called separately for the luma and chroma components.

In one example of the de-ringing process, the 2-D filter may be adaptively applied to smooth out the area near the edge. Edge pixels are rarely filtered or filtered at all to avoid blurring.

GOP Splitter

Hereinafter, exemplary embodiments of processing, including adaptive GOP segmentation, shot detection, and bandwidth map generation, which may be included in a GOP divider, are described.

Generate Bandwidth Map

Human visual quality V may be a function of both encoding complexity C and allocated bit B (also referred to as bandwidth). 29 is a graph showing this relationship. It should be noted that the encoding complexity metric C takes into account spatial and temporal frequencies from the human visual point of view. In the case of distortion which is more sensitive to the human eye, the complexity value is correspondingly higher. Usually, it can be assumed that V is monotonically decreasing at C and monotonically increasing at B.

In order to achieve a constant visual quality, a bandwidth Bi is assigned to the i th object (frame or MB) and encoded to satisfy the criterion represented in the two equations immediately below.

In the two equations directly above, Ci is the encoding complexity of the i th object, B is the total available bandwidth, and V is the visual quality achieved for one object.

은 (C, V) 쌍의 근방 내에서 변하지 않는 것으로서 취급될 수 있다. 대역폭 비율 βi 는 아래에 도시된 식에서 정의된다:It is difficult to formulate human visual quality as a predetermined equation. Thus, the equation set is not precisely defined. However, assuming the 3-D model is continuous for all variables, the bandwidth ratio

May be treated as unchanged in the vicinity of the (C, V) pair. The bandwidth ratio βi is defined in the equation shown below:

Then, the bit allocation can be defined as expressed in the following equation.

Here, δ represents "near".

Encoding complexity is influenced by human visual sensitivity in both space and time. Gird's human visual model is an example of a model that can be used to define spatial complexity. These models take into account local spatial frequencies and ambient lighting. The resulting metric is referred to as D _csat . At the preprocessing point in the process, it is not known whether the image will be intra coded or inter coded, and bandwidth ratios for both are generated. Bits are allocated according to the ratio between β _INTRAs of different video objects. In the case of an intra coded image, the bandwidth ratio is represented by the following equation.

는

을 보장하기 위한 정규화 계수이다. 예를 들어, α _INTRA = 4 인 값은 양호한 시각 품질을 달성한다. 콘텐츠 정보 (예를 들어, 콘텐츠 분류) 를 이용하여, 특정 비디오 콘텐츠에 대해 원하는 양호한 시각 품질 레벨에 대응하는 값으로 α _INTRA 를 설정할 수 있다. 일 예에서, 비디오 콘텐츠가 "talking head" 뉴스 브로드캐스트를 포함하면, 비디오의 정보 이미지 또는 디스플레이 가능 부분이 오디오 부분보다 낮은 중요도를 갖는 것으로 간주될 수도 있으므로, 시각 품질 레벨을 낮게 설정할 수도 있고, 그 데이터를 인코딩하는데 더 적은 비트를 할당할 수 있다. 다른 예에 있어서, 비디오 콘텐츠가 스포츠 이벤트를 포함하면, 표시된 이미지가 시 청자에게 더 중요할 수도 있으므로, 콘텐츠 정보를 이용하여, 보다 높은 시각 품질 레벨에 대응하는 값으로 α _INTRA 를 설정할 수도 있고, 이에 따라, 그 데이터를 인코딩하는데 더 많은 비트를 할당할 수 있다.In the above formula, Y is the average luminance component of the macroblock, α _INTRA is the weighting coefficient for the luminance squared and the following d _csat term,

Is

Normalization factor to ensure For example, a value with α _INTRA = 4 achieves good visual quality. Using content information (eg, content classification), α _INTRA can be set to a value corresponding to a desired good visual quality level for specific video content. In one example, if the video content includes a "talking head" news broadcast, the visual quality level may be set lower because the information image or displayable portion of the video may be considered to have lower importance than the audio portion, and Less bits can be allocated for encoding the data. In another example, if the video content includes a sporting event, since the displayed image may be more important to the viewer, the content information may be used to set α _INTRA to a value corresponding to a higher visual quality level. Thus, more bits can be allocated to encode the data.

To understand this relationship, it should be noted that bandwidth is allocated algebraically according to encoding complexity. The term Y ^{2, which} is the luminance squared, indicates that coefficients with larger magnitudes use more bits for encoding. To prevent the log from becoming negative, 1 is added to the terms in parentheses. It is also possible to use logs with different bases.

The time complexity is determined by measuring the frame difference metric, which measures the difference between two consecutive frames, taking into account the amount of motion (for example, the motion vector) together with a frame difference metric such as sum of the absolute difference (SAD). do.

Bit allocation for the inter coded image may take into account spatial complexity as well as time complexity. This is expressed as

의 역할 을 이해하기 위해서는, 인간 시각 시스템의 다음 특징에 주목하자: 평활하며 예측 가능한 모션 (작은

) 을 경험한 영역이 주의를 끌고, 인간 눈에 의해 추적될 수 있으며, 통상 정지 영역보다 임의의 많은 왜곡을 허용할 수는 없다. 그러나, 고속이거나 예측할 수 없는 모션 (큰

) 을 경험한 영역은 추적될 수 없고, 상당한 양자화를 허용할 수 있다. 실험예는 α _INTRA = 1,

은 양호한 시각 품질을 달성한다는 것을 나타낸다.In the above formula, MV _P and MV _N are the forward and reverse motion vectors for the current MB. It can be seen that Y ² in the intra coded bandwidth formula is replaced by the sum of squared difference (SSD). In the above formula

To understand the role of, note the following features of the human visual system: smooth and predictable motion (small

The area experiencing the) can be attracted, tracked by the human eye, and typically cannot allow any more distortion than the stationary area. However, high speed or unpredictable motion (big

) May not be tracked and may allow significant quantization. Experimental example is α _INTRA = 1,

Indicates that good visual quality is achieved.

Shot detection

Hereinafter, exemplary embodiments of shot detection will be described. Such components and processes may be included in the GOP divider 412 (FIG. 4).

Motion compensator 23 can be configured to determine bidirectional motion information about a frame within the video. In addition, motion compensator 23 determines one or more difference metrics, such as, for example, sum of absolute difference (SAD) or sum of squared difference (SSD), and calculates luminance information (eg, macros) for one or more frames. Block (MB) luminance average or difference), luminance histogram difference, and frame difference metric, and other information, examples of which are described with reference to equations (1) through (3). The shot classifier may be configured to classify the frames in the video into two or more "shot" categories using information determined by the motion compensator. The encoder is configured to adaptively encode a plurality of frames based on shot classification. Hereinafter, a motion compensator, a shot classifier, and an encoder will be described with reference to equations (1) to (10).

28 is a block diagram of a preprocessor 202 that includes a processor 2831 configured for shot detection and other preprocessing operations in accordance with some aspects. The digital video source may be provided by a source external to the preprocessor 202 as shown in FIG. 4 and communicated to the communication module 2836 within the preprocessor 202. The preprocessor 202 includes a storage medium 2835 in communication with the processor 2831, and both the storage medium 2835 and the processor 2831 are in communication with the communication module 2836. Processor 2831 includes motion compensator 2832, shot classifier 2833, and other preprocessing module 2034 to generate motion information, classify shots within frames of video data, and as described herein. Operate to perform other preprocessing tests. Motion compensators, shot classifiers, and other modules may include processes similar to the corresponding modules of FIG. 4 and may process video to determine the information described below. In particular, processor 2831 obtains a metric representing a difference between adjacent frames of the plurality of video frames, the metric comprising bidirectional motion information and luminance information, and performing shot transitions of the plurality of video frames based on the metric. Determine, and adaptively encode a plurality of frames based on shot transitions. In some aspects, the metric can be calculated by a device or process that is external to the processor 2831 (which can also be external to the preprocessor 202) and directly or indirectly through another device or memory. (2831). The metric can also be calculated by a processor 2831, such as, for example, a motion compensator 2832.

Preprocessor 202 provides video and metadata, for example, to terminal 160 (FIG. 1) for further processing, encoding, and transmission. In some aspects, the encoded data can be scalable multi-layer encoded video that can include a base layer and an enhancement layer. The scalable layer encoding is owned by the assignee of the present application and is hereby incorporated by reference in its entirety, and the concurrent US patent number entitled "SCALABLE VIDEO CODING WITH TWO LAYER ENCODING AND SINGLE LAYER DECODING". Attorney docket no. 050078, which is further described.

In some aspects, various exemplary logic blocks, components, modules, and circuits described in connection with FIG. 28, and other examples and figures disclosed herein, include general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs). May be implemented or performed in a field programmable gate array (FPGA) or other programmable logic device, separate gate or transistor logic, separate hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor such as that shown in FIG. 28 may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

Typically, video encoding works for structured groups of pictures (GOP). GOP usually starts with an intra coded frame (I-frame) and proceeds to a series of P (prediction) or B (bidirectional) frames. Typically, an I-frame can store all the data for representing a frame, and a B-frame can contain data in the previous and next frames (eg, only contain data that has changed from the previous frame or differ from the data in the next frame). ), The P-frame contains the changed data from the previous frame.

In general use, I-frames are interspersed between P- and B-frames in encoded video. In terms of size (eg, the number of bits used to encode the frame), I-frames are typically much larger than P-frames, and P-frames are also larger than B-frames. For efficient encoding, transmission and decoding processing, the GOP length should be long enough to reduce the effective loss from large I-frames and short enough to resolve the discrepancy or channel damage between the encoder and decoder. Also, for the same reason, macroblocks (MBs) in P frames can be intracoded.

Scene transition detection may be used in the video encoder to determine the appropriate GOP length and insert the I-frame based on the GOP length, instead of inserting the I-frames at regular intervals. In a real streaming video system, communication channels are usually corrupted by bit errors or packet loss. Depending on where I frame or I macroblock is placed, the decoded video quality and viewing experience may have a significant impact. One encoding scheme is to use intra coded frames for an image or image portion that has a significant change from the associated previous image or image portion. Usually, these areas cannot be predicted effectively and efficiently using motion estimation, and such areas are more efficiently encoded if excluded from interframe coding techniques (e.g., encoding using B-frames and P-frames). I can do it. In the context of channel damage, those areas are likely to receive error propagation, which can be reduced or eliminated (or nearly reduced or eliminated) by intra-frame encoding.

Some of the GOP video may be classified into two or more categories, each region having different in-frame encoding criteria that may vary depending on the particular implementation. As one example, video can be classified into three categories: abrupt scene transitions, cross-fading and other slow scene transitions, and camera flashlights. Sudden scene transitions usually involve a significantly different frame from the previous frame caused by camera manipulation. Because the contents of these frames are different from the contents of the previous frame, the sudden scene transition should be encoded as an I frame. Cross-fading and other slow scene transitions usually involve slow scene transitions caused by computer processing of camera shots. Gradual blending of two different scenes may seem more satisfactory to the human eye, but is difficult to video code. Motion compensation cannot effectively reduce the bitrate of those frames, and more intra MBs can be updated for these frames.

Camera flashlight, or camera flash event, occurs when the content of a frame includes a camera flash. Such a flash has a relatively short duration (e.g., one frame), and is so bright that the pixels in the frame representing the flash exhibit very high luminance compared to the corresponding area on adjacent frames. Camera flashlights change the brightness of an image suddenly and quickly. Typically, the duration of the camera flashlight is shorter than the time masking duration of the human vision system (HVS), which is typically defined to be 44 ms. Since the human eye is not sensitive to the quality of these short brightness bursts, they may be roughly encoded. Because motion compensation cannot be used to efficiently process flashlight frames and is a poor prediction candidate for future frames, the rough encoding of these frames does not reduce the encoding efficiency of future frames. Scenes classified as flashlights should not be used to predict other frames due to "artificial" high brightness, and for the same reason other frames cannot be effectively used to predict these frames. Once identified, these frames can be eliminated because they may require a relatively large amount of processing. One option is to remove the camera flashlight frame and encode the DC coefficients instead; Such a solution is simple, fast to calculate and saves a lot of bits.

If any of the frames are detected, a shot event is declared. Shot detection is useful not only for improving encoding quality but also for identifying video content search and indexing. Hereinafter, one aspect of the shot detection process will be described.

FIG. 30 illustrates a working process 3000 for a GOP that may be used to encode video based on shot detection within a video frame in some aspects, wherein portions (or subprocesses) of the process 3000 are shown in FIG. It is described and illustrated with reference to 30-40. Processor 2831 can be configured to include a process 3000. After the process 3000 begins, flow proceeds to block 3042 to obtain a metric (information) for the video frame, which includes information indicative of the difference between adjacent frames. The metric then includes luminance based information and bidirectional motion information to determine a transition that has occurred between adjacent frames that can be used for shot classification. Such a metric can be obtained from another device or process, or calculated by, for example, the processor 2831. An embodiment of exemplary metric generation is described below with reference to process A in FIG. 31.

Process 3000 then proceeds to block 3044 to determine shot transitions of the video based on the metric. Video frames can be classified into two or more categories of which types of shots (eg, sudden scene transitions, slow scene transitions, or scenes containing high brightness values (camera flashes)) are included in the frame. The encoding of some implementations may require other categories. An illustrative example of shot classification is described with reference to process B of FIG. 32, respectively, and in more detail with reference to processes D, E, and F of FIGS. 34-36.

Once the frames have been classified, process 3000 proceeds to block 3046 to encode or specify the frames for encoding using the shot classification result. Such a result may affect whether to encode a frame into an intra coded frame or to a predictive frame (eg, a P-frame or a B-frame). Process C of FIG. 33 shows an example of an encoding scheme using a shot result.

31 shows an example of a process of obtaining a metric of a video. FIG. 31 illustrates certain steps occurring at block 3042 of FIG. 30. With continued reference to FIG. 31, at block 3152, process A obtains or determines bidirectional motion estimation and compensation information of the video. The motion compensator 2832 of FIG. 28 may be configured to perform bidirectional motion estimation for the frame and determine motion compensation information that may be used for the next shot classification. Process A then proceeds to block 3154 to generate luminance information including a luminance difference histogram for the current or selected frame and one or more adjacent frames. Then, finally, process A proceeds to block 3156 to calculate a metric representing a shot included in the frame. One such metric is the frame difference metric shown in two examples of equations (4) and (10). An example of determining motion information, luminance information, and frame difference metrics is described below.

Motion compensation

To perform bidirectional motion estimation / compensation, the video sequence matches two blocks of the frames, that is, blocks in the nearest neighboring frame (one past frame and one future frame) with all 8x8 blocks of the current frame. It can be preprocessed with a bidirectional motion compensator. The motion compensator generates motion vectors and difference metrics for every block. FIG. 37 illustrates this concept, illustrating an example of matching a pixel of a current frame C to a previous frame P and a future frame (or next) frame N, wherein the motion vector (previous motion) for the matched pixel is shown. Vector MV _P and future motion vector MV _N ). The following briefly describes exemplary aspects of bidirectional motion vector generation and associated encoding.

40 shows an example of a motion vector determination process and predictive frame encoding, for example in MPEG-4. The process shown in FIG. 40 illustrates in more detail an example process that can occur at block 3152 of FIG. 31. In FIG. 40, the current image 4034 consists of 5 x 5 macroblocks, and the number of macroblocks in this example is arbitrary. The macroblock consists of 16 x 16 pixels. The pixel may be defined by an 8 bit luminance value Y and two 8 bit color difference values Cr and Cb.

In MPEG, the Y, Cr and Cb components can be stored in 4: 2: 0 format, and the Cr and Cb components are downsampled by two in the X and Y directions. Thus, each macroblock will consist of 256 Y components, 64 Cr components and 64 Cb components. The macroblock 4036 of the current image 4034 is predicted from the reference image 4032 at a different time point than the current image 4034. To find the closest best matching macroblock 4038 in terms of the Y, Cr and Cb values for the current macroblock 4036 being encoded, search the reference image 4032. The position of the best matching macroblock 4038 within the reference image 4032 is encoded in the motion vector 4040. The reference picture 4032 may be an I-frame or a P-frame that the decoder will reconstruct before the configuration of the current picture 4034. Subtracting the optimal matching macroblock 4038 from the current macroblock 4036 (calculating the differences for each of the Y, Cr, and Cb components), a residual error 4042 is obtained. The residual error 4042 is encoded with a 2D Discrete Cosine Transform (DCT), followed by quantization 4046. Quantization 4046 may be performed to provide spatial compression, for example, by assigning more bits to the low frequency coefficients while allocating fewer bits to the high frequency coefficients. The quantized coefficients of the residual error 4042 together with the reference image 4034 and the motion vector 4040 identifying the information are encoded information representing the current macroblock 4036. The encoded information may be stored in memory for later use, processed for example for error correction or image enhancement, or transmitted over network 140.

Using the encoded quantized coefficients of the residual error 4042 together with the encoded motion vector 4040, reconstruct the current macroblock 4036 at the encoder used as part of the reference frame for subsequent motion estimation and compensation. can do. The encoder can emulate the decoder's procedure for this P-frame recovery. Due to the emulation of the decoder, both the encoder and the decoder work with the same reference picture. At this point, a restoration process is provided regardless of whether it is done at the encoder or at the decoder for later inter-coding. Reconstruction of a P-frame may begin after the reference frame (or part of the image or frame being referenced) has been reconstructed. The encoded quantized coefficients are dequantized 4050 and then subjected to 2D inverse DCT, or IDCT 4042, to obtain decoded or reconstructed residual error 4054. Decode the encoded motion vector 4040 and use it to find the best matching macroblock 4056 that has already been reconstructed from the already reconstructed reference picture 4032. Then, the reconstructed residual error 4054 is added to the reconstructed optimal matching macroblock 4056 to form a reconstructed macroblock 4058. The reconstructed macroblock 4058 may be stored in memory, displayed as one image or separately with other reconstructed macroblocks, or further processed for image enhancement.

Encoding using B-frames (or any section coded with bi-prediction) may utilize time redundancy between an area in the current picture and an optimal matching prediction area in the previous picture and an optimal matching prediction area in the subsequent picture. The subsequent best matching prediction region and the previous best matching prediction region are combined to form a combined bidirectional prediction region. The difference between the current video region and the best match combined bidirectional prediction region is the residual error (or prediction error). The position of the optimal matching prediction region in the subsequent reference image and the optimal matching prediction region in the previous reference image may be encoded into two motion vectors.

Luminance Histogram Difference

The motion compensator may generate difference metrics for all blocks. The difference metric can be a sum of square difference (SSD) or a sum of absolute difference (SAD). Here, SAD is used as an example without losing generality.

For all frames, the SAD ratio is calculated as follows.

을 생성한다.Where SAD _P and SAD _N are the sum of absolute differences of the forward and reverse difference metrics, respectively. Note that the denominator contains a small amount ε to prevent a "divide by zero" error. The molecule also contains ε to balance the effect of 1 in the denominator. For example, if the previous frame, the current frame, and the next frame are the same, the motion search will generate SAD _P = SAD _N = 0. In this case, the calculation is instead of zero or infinity

.

The luminance histogram can be calculated for every frame. Typically, multimedia images have an 8-bit luminance depth (eg, a "bin" number). The histogram can be obtained by setting the luminance depth used to calculate the luminance histogram according to some embodiments to 16. In other aspects, the brightness depth may be set to an appropriate number that may vary depending on the type of data being processed, available computational power, or other predetermined criteria. In some aspects, the luminance depth can be set dynamically based on calculated or received metrics, such as data content.

Equation (49) shows an example of calculating the luminance histogram difference (lambda).

Where N _Pi is the number of blocks in the i-th bin for the previous frame, N _Ci is the number of blocks in the i-th bin for the current frame, and N is the total number of blocks in one frame. If the luminance histogram difference between the previous frame and the current frame is completely different (disjoint), then λ = 2.

The frame difference metric D described with reference to block 56 of FIG. 5 may be calculated as shown in equation (50).

, 및

이다.Where A is a constant selected by the application,

, And

to be.

32 shows an example of process B for determining three categories of shot (or scene) transitions using metrics obtained or determined for video. 32 illustrates certain steps occurring in an aspect of block 3044 of FIG. 30. Referring back to FIG. 32, at block 3262, process B first determines whether the frame satisfies the criteria for being designated as a sudden scene transition. Process D of FIG. 34 shows an example of such a determination. Process B then proceeds to block 3264 to determine whether the frame is part of a slow transition scene. Process C of FIG. 35 shows an example of determining a scene to switch slowly. Finally, at block 3366, process B determines whether the frame includes a camera flash, that is, includes a large luminance value that is different from the previous frame. Process F of FIG. 36 shows an example of determining a frame including a camera flash. Illustrative examples of these processes are described below.

Sudden Cutaway

34 is a flowchart illustrating a process of determining a sudden scene change. FIG. 34 describes in more detail certain steps that may occur in some aspects of block 3262 of FIG. 32. In block 3482, check whether the frame difference metric D satisfies the criteria shown in equation (51).

Where A is a constant selected by the application and T ₁ is a threshold. If the criterion is satisfied, at block 3484, process D designates the frame as a sudden scene change, which in this example does not require additional shot classification.

는 커야 하고,

는 작아야 한다. 비율

를

단독 대신에 이용하여, 그 메트릭을 콘텍스트의 액티비티 레벨로 정규화할 수 있다.In one example, the simulation shows that setting A = 1, and T ₁ = 5 achieves good detection performance. If the current frame is a sudden transition frame,

Should be large,

Should be small. ratio

To

Instead of being used alone, the metric can be normalized to the activity level of the context.

It should be noted that the criterion uses the luminance histogram difference lambda λ in a nonlinear manner. 39 shows that λ * (2λ + 1) is a convex function. If λ is small (eg, close to zero), there is little preemphasis. The larger lambda is, the more emulation is performed by that function. Using this pre-emphasis for any λ greater than 1.4, if the threshold T ₁ is set to 5, then a sudden scene change is detected.

Cross fading and slow cutaway

35 also shows further details of some aspects that may occur at block 3264 of FIG. 32. Referring to FIG. 35, at block 3592, process E determines whether the frame is part of a series of frames indicating a slow scene transition. As shown in equation (52), when the frame difference metric D is less than the first threshold T _{1 and} greater than or equal to the second threshold T ₂ , process E determines that the current frame is cross fading or slow scene transition. .

For a certain number of consecutive frames,

Where T ₁ is the same threshold value used above and T ₂ is another threshold value. Usually, since differences in embodiments are possible, the exact values of T ₁ and T ₂ are determined by regular experiments. If the criterion is satisfied, at block 3594, process E classifies the frame as part of the slowly changing scene shot classification for the end of the selected frame.

Camera flashlight event

Process F shown in FIG. 36 is an example of a process that can determine whether the current frame includes a camera flashlight. In the camera of this exemplary aspect, the luminance histogram statistics are used to determine whether the current frame includes a camera flashlight. Process F determines the camera flash event in the selected frame by first determining whether the brightness of the current frame is greater than the brightness of the previous and next frames, as shown in block 3602. If the brightness of the current frame is greater than the brightness of the previous and next frames, the frame is not a camera flash event, but if the brightness of the current frame is large, the frame may be a camera flash event. In block 3604, process F determines whether the reverse difference metric is greater than threshold T ₃ , determines whether the forward difference metric is greater than threshold T ₄ , and if both conditions are met, in block 3606, process F Classifies the current frame as having a camera flashlight. In one example, at block 3602, as shown in equations (53) and (54), process F determines whether the average brightness of the current frame minus the average brightness of the previous frame is greater than or equal to the threshold T _3. The process F determines whether or not the average luminance of the current frame minus the average luminance of the next frame is greater than or equal to the threshold T ₃ .

If the criteria are not met, the current frame is not classified as containing a camera flashlight, and process F returns. If the criterion is satisfied, process F proceeds to block 3604 to determine if the reverse difference metric SAD _P and the forward difference metric SAD _N are greater than a certain threshold T ₄ , as shown in equations (55) and (56) below. Determine whether or not.

는 현재 프레임의 평균 휘도이고,

는 이전 프레임의 평균 휘도이고,

은 다음 프레임의 평균 휘도이며, SAD _P 및 SAD _N 은 현재 프레임과 연관된 순방향 및 역방향 차이 메트릭이다. 기준이 만족되지 않으면, 프로세스 F 는 리턴한다.here,

Is the average luminance of the current frame,

Is the average luminance of the previous frame,

Is the average luminance of the next frame, and SAD _P and SAD _N are the forward and reverse difference metrics associated with the current frame. If the criteria are not met, process F returns.

Typically, when the implementation of the described process results in differences in operating parameters including thresholds, the value of T ₃ is determined by regular experiments. The SAD value is included in the determination because the camera flash usually occupies only one frame, and due to the luminance difference, such frame cannot be predicted well using motion compensation from both forward and reverse directions.

In some aspects, one or more of the thresholds T ₁ , T ₂ , T _3, and T ₄ are predetermined, and such values are integrated into the shot classifier in the encoding device. Typically, these thresholds are selected through testing of a particular implementation of shot detection. In some embodiments, one or more of the thresholds T ₁ , T ₂ , T _3, and T ₄ are based on information supplied to the shot classifier (eg, metadata) or based on information calculated by the shot classifier itself. It can be set during processing (eg, dynamically).

Reference is now made to FIG. 33, which shows a process C for encoding a video or determining an encoding parameter for a video based on a shot classification of a selected frame. In block 3370, process C determines whether the selected frame has been classified as a sudden scene change. If the selected frame is classified as a sudden scene transition, then at block 3371, the current frame is classified as a sudden scene transition, can encode that frame as an I-frame, and determine a GOP boundary. If the current frame is not classified as a sudden scene transition, process C proceeds to block 3372 and, if the current frame is classified as part of a slowly changing scene, then at block 3373, the current frame, and other frames of the slowly changing scene. Can be encoded as a predictive frame (eg, P-frame or B-frame). Process C then proceeds to block 3374 to check whether the current frame has been classified as a flashlight scene that includes a camera flashlight. If the current frame is classified as a flashlight scene that includes a camera flashlight, then at block 3375, the frame is identified for special processing, e.g., removing, duplicating, or encoding a specific coefficient for that frame. can do. If the current frame is not classified as a flashlight scene that contains camera flashlights, then no classification is done for the current frame, and the selected frame can be encoded according to different criteria, which can be encoded or dropped as an I-frame. Can be. Process C may be implemented in an encoder.

In the above aspect, the difference amount between the frame to be compressed and two adjacent frames thereof is indicated by the frame difference metric D. If a significant amount of one-way brightness change is detected, it indicates a cross-fade effect within that frame. As cross-fade becomes more pronounced, more gain may be achieved using B frames. In some aspects, the modified frame difference metric is used as shown in equation (57) below.

및

은 각각 현재 프레임과 이전 프레임 간의 루마 차이이고, 현재 프레임과 다음 프레임 간의 루마 차이이고, Δ 는 정 규 실험에서 결정될 수 있는 상수 (구현예에 따라 다를 수 있음) 를 나타내고, α 는 0 과 1 사이의 값을 갖는 가중 변수이다.here,

And

Are respectively the luma difference between the current frame and the previous frame, the luma difference between the current frame and the next frame, respectively, Δ represents a constant (depending on the implementation) that can be determined in the regular experiment, and α is between 0 and 1 Weighted variable with the value of.

If a consistent trend of luma change is observed and the change intensity is large enough, the modified frame difference metric D ₁ is only different from the original frame difference metric D ₁ . D ₁ is D or less. If the change in luma is constant ( d _P = d _N ), the modified frame difference metric D ₁ is lower than the original frame difference metric D with the lowest ratio of (1-α).

Table 1 below shows the performance improvement by adding a sudden scene change detection. In both the non-scene-change (NSC) and scene-change (SC) cases, the total number of I-frames is approximately equal. In the NSC case, the I-frames are distributed evenly among the entire sequence, but in the SC case, the I-frames are only assigned to sudden scene transition frames.

It can be seen that an improvement of 0.2 to 0.3 dB can usually be achieved in PSNR units. Simulation results indicate that the shot detector is very accurate in determining the shot event described above. Simulation of five clips with normal cross-fade effect shows that at Δ = 5.5 and α = 0.4, a PSNR gain of 0.226031 dB is achieved with the same bitrate.

Sequence / metric Bitrate (kbps) Average QP PSNR (dB) Anime NSC 226.2403 31.1696 35.6426 Animation SC 232.8023 29.8171 36.4513 Music nsc 246.6394 32.8524 35.9337 Music sc 250.0994 32.3209 36.1202 Headline NSC 216.9493 29.8304 38.9804 Headline News SC 220.2512 28.9011 39.3151 Basketball NSC 256.8726 33.1429 33.5262 Basketball sc 254.9242 32.4341 33.8635

Table 1: Simulation Results of Sudden Cutaway Detection

Adaptive GOP Structure

An exemplary embodiment of the adaptive GOP structure operation is described below. Such an operation may be included in the GOP divider 412 of FIG. 4. The older video compression standard MPEG2 does not require a GOP to have a constant structure (but it can be imposed that a GOP will have a constant structure). The MPEG2 sequence always starts with an I frame encoded without reference to the previous picture. Normally, the MPEG2 GOP format is prearranged in the encoder by fixing the interval in P following the I frame or in the GOP of the predictive picture. P frame refers to an image partially predicted from a previous I or P image. The frame between the starting I frame and the subsequent P frame is encoded as a B frame. A "B" frame (B stands for bidirectional) can use the previous and next I or P pictures individually or simultaneously as a reference. On average, the number of bits used to encode an I-frame exceeds the number of bits used to encode a P-frame, and likewise, on average, the number of bits used to encode a P-frame is used to encode B-frames. Exceeds the number of bits. The skipped frame, even if used, may not use any bits for its representation.

One advantage of using P-frames and B-frames in more recent compression algorithms is that video transmission size can be reduced by skipping frames. If the time redundancy is high, for example, if there is little change between images, the video stream is efficiently represented using P, B, or skipped images, since the first decoded I or P image is later referred to. This is because another P or B image is decoded by using.

A group of picture (GOP) divider adaptively encodes frames to minimize time redundancy. The difference between the frames is quantified and a decision is made to represent the image by I, P, B or skipped frames after an appropriate test is performed on the quantified differences. Processing at the GOP divider is aided by other operations of the preprocessor 202 to provide filtering for noise removal.

The adaptive encoding process has the advantage that it is not available in the "fixed" encoding process. The fixed process ignores the possibility of small changes in the content, but the adaptation procedure allows for far more B frames to be inserted between each I frame and P frame, or between two P frames, thereby providing sufficient frame sequence. This reduces the number of bits used to represent. On the other hand, for example, in a fixed encoding process, if the change in the video content is significant, the efficiency of the P frame is greatly reduced because the difference between the predicted frame and the reference frame is too large. Under these conditions, the matching object may be out of the motion search area, and the similarity between the matching objects is reduced due to the distortion due to the change in the camera angle. The adaptive encoding process has the advantage that it may be used to optimally determine when a P frame should be encoded.

In the system disclosed herein, the type of condition described above is automatically detected. The adaptive encoding process described herein is flexible and is made to adapt to changes in these contents. The adaptive encoding process evaluates a frame difference metric, which can be considered as a measure of the distance between frames with the same additional distance characteristic. Conceptually, if frames F ₁ , F ₂ and F ₃ have interframe distances d ₁₂ and d ₂₃ , the distance between F ₁ and F ₃ is considered to be at least d ₁₂ + d ₂₃ . Frame allocation is made based on distance-like metrics and other measurements.

GOP divider 412 operates by assigning an image type to a frame upon receipt. The picture type represents a prediction method that may be used to code each block.

I-pictures are coded without reference to other pictures. Since I-pictures are standalone, they provide access points within the data stream where decoding can begin. The I encoding type is assigned to a frame if the "distance" to the previous frame exceeds the scene transition threshold.

The P-picture may use the previous I or P picture for motion compensated prediction. The P-picture uses a block in the previous field or frame that may be moved from the predicted block as a reference for encoding. After subtracting the reference block from the block under consideration, typically, the residual block is encoded using a discrete cosine transform to remove spatial redundancy. If the "distance" between the last frame and one frame assigned to be a P frame exceeds a second threshold (the second threshold is typically less than the first threshold), then the P encoding type is assigned to that frame. .

As described above, the B-frame picture may use the previous and next P-picture or I-picture for motion compensation. Blocks in a B picture may be forward, backward or bi-predicted, or may be intra-coded without reference to other frames. For H.264, the reference block can be a linear combination of 32 blocks out of multiple frames. If the frame cannot be assigned to be of type I or P, it is allocated to be of type B if the "distance" to the immediately preceding frame is greater than the third threshold (usually less than the second threshold). If the frame cannot be allocated to be an encoded B type, it is allocated to a "skip frame" state. These frames can be skipped because they are virtual copies of the previous frame.

Evaluating a metric that quantifies the difference between adjacent frames in display order is the first part of this process that occurs in GOP divider 412. This metric is the distance referenced above, and using that metric, every frame is evaluated to find its proper type. As such, the spacing between an I frame and an adjacent P frame, or two consecutive P frames, may vary. The calculation of the metric begins by processing video frames using block-based motion compensators, one block being the basic unit of video compression, usually consisting of 16 × 16 pixels, but for example 8 × 8, 4 × 4 and 8 Other block sizes, such as × 16, are possible. In the case of a frame consisting of two interlaced fields present in the output, motion compensation is done in units of fields, and the search for the reference block occurs in the field rather than in the frame. In the case of a block in the first field of the current frame, the forward reference block is found in the field of the next frame, and also the backward reference block is found in the field of the frame immediately before the current field. The current block is collected in the compensated field. The process continues in the second field of the frame. Two compensated fields are combined to form a forward and reverse compensated frame.

In the case of the frame generated in the inverse telecine 406, since only the reconstructed film frame is generated, the search for the reference block may be made frame by frame. In addition, two differences (forward and backward) from two existing blocks are obtained to generate forward and backward compensated frames. In sum, the motion compensator generates motion vectors and difference metrics for every block. Depending on whether the forward or backward difference is evaluated, the difference in the metric between the block in the field or frame that is currently under consideration and the block that best matches it, in the previous field or frame, or the field or frame immediately following it. Note that is evaluated. Only luminance values are included in this calculation.

As such, the motion compensation step produces two sets of differences. They are present in time between the blocks of current luminance values and the luminance values in the reference block selected from the frames immediately preceding the current frame and the frames in the immediately preceding frame. The absolute value of each forward and each reverse difference is determined for each pixel in the block, each of which is summed separately over the entire frame. When a deinterlaced NTSC field containing a frame is processed, both fields are included in two additions. In this way, SAD _P and SAD _N , i.e., the sum of the absolute values of the forward and reverse differences are obtained.

For all frames, the SAD ratio is calculated using the following relationship.

의 감도를 줄인다.Where SAD _P and SAD _N are the summed absolute values of the forward and reverse differences, respectively. A small amount ε is added to the molecule to prevent a "divide by zero" error. If a similar term ε is added to the denominator and SAD _P or SAD _N approaches zero,

Decrease the sensitivity of

In another aspect, the difference may be a sum of squared difference (SSD), sum of absolute difference (SAD), or SATD, and the block of pixel values is transformed by applying a two-dimensional discrete cosine transform before taking the difference in the block elements. do. The sums are evaluated over an area of active video, but in other aspects a smaller area may be used.

In addition, luminance histograms of all frames (not motion compensated) as received are calculated. The luminance histogram acts on the DC coefficients, i. Correspondingly, an average value of 256 luminance values in a 16x16 block may be used in the luminance histogram. For an image with 8 bits of luminance depth, the number of bins is set to 16. The following metric calculates the histogram difference.

는 i 번째 빈에서 이전 프레임으로부터의 블록 수이고,

는 i 번째 빈에 속하는 현재 프레임으로부터의 블록 수이며, N 은 프레임 내의 전체 블록 수이다.As above,

Is the number of blocks from the previous frame in the i th bin,

Is the number of blocks from the current frame belonging to the i th bin, and N is the total number of blocks in the frame.

These intermediate results are collected to form the current frame difference as follows.

는 현재 프레임에 기초한 SAD 비율이고,

는 이전 프레임에 기초한 SAD 비율이다. 장면이 평활한 모션을 가지며 그 루마 히스토그램이 거의 변하지 않으면,

이다. 현재 프레임이 갑작스런 장면 전환을 표시하면,

는 커질 것이고,

는 작아야 한다.

단독 대신에, 비율

를 이용하므로, 문맥의 액티비티 레벨로 메트릭이 정규화된다.here,

Is the SAD rate based on the current frame,

Is the SAD ratio based on the previous frame. If the scene has smooth motion and the luma histogram hardly changes,

to be. If the current frame shows a sudden transition,

Will grow,

Should be small.

Instead of alone, the rate

Because we use, the metric is normalized to the activity level of the context.

The dataflow 4100 of FIG. 40 illustrates certain components that may be used to calculate the frame difference metric. Preprocessor 4125 delivers the interlaced fields in the case of video with an NTSC source, and delivers frames of the film image when the video source is the result of inverse telecine to bidirectional motion compensator 4133. Bi-directional motion compensator 4133 divides into blocks of 16 by 16 pixels and operates on the field (or frame in the case of a video source of a movie) by comparing each block with all 16 by 16 blocks within a given area of the previous frame field. do. The block providing the best match is selected and removed from the current block. The absolute value of the difference is taken and the result is added over 256 pixels containing the current block. If this procedure is performed for all current blocks of the field and then for both fields, then both SAD _N , ie, reverse difference metrics, are calculated by the reverse difference module 4137. Similar procedures may be performed by the forward difference metric 4136. The forward difference module 4136 uses the frame immediately preceding the current frame in time as the reference block source to develop SAD _P , i.e., the forward difference metric. The same estimation process occurs when an input frame is formed in inverse telecine, even if performed using the reconstructed film image. A histogram that may be used to complete the calculation of the frame difference metric may be formed in the histogram difference module 4141. Each 16x16 block is assigned to a bin based on the average of its luminance values. This information is formed by adding all 256 pixel luminance values together in one block, normalizing to 256 as needed, and incrementing the count of bins where the average value is placed. The calculation is done once for each premotion compensated frame, and the histogram for the current frame becomes the histogram for the previous frame when a new current frame arrives. In histogram difference module 4141, the two histograms are differentially and normalized by the number of blocks to form [lambda] defined in equation (59). These results are combined in a frame difference combiner 4143, which is obtained in equation (60) using intermediate results obtained from histogram difference module 4139, forward and reverse difference modules 4136 and 4136. Evaluate the defined current frame difference.

The system of the flowchart 4100 and its components or steps may be implemented by hardware, software, firmware, middleware, microcode, or any combination thereof. Each function of the flowchart 4100, including the preprocessor 4125, the bidirectional motion compensator 4133, the forward and reverse difference metric modules 4136 and 4137, the histogram difference module 4141, and the frame difference metric combiner 4143. A component may be implemented as a standalone component, integrated as hardware, firmware, middleware in a component of another device, implemented in microcode or software running on a processor, or a combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform a desired task may be stored on a machine-readable medium, such as a storage medium. A code segment may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by communicating and / or receiving information, data, arguments, parameters or memory contents.

The received and processed data may include, for example, a chip-configured storage medium (eg, ROM, RAM) or a disk type storage medium (eg, magnetic or optical) connected to the processor. In some aspects, combiner 4143 can be stored in a storage medium that can include some or all of the storage medium. Flowchart 4200 of FIG. 41 shows a process for assigning a compression type to a frame. In one aspect, the current frame difference defined in equation (3) is the basis for all decisions made with respect to frame allocation. As indicated by decision block 4253, if the frame under consideration is the first frame in the sequence, the decision path marked “Yes” proceeds to block 4255, declaring that the frame is an I frame. The accumulated frame difference at block 4257 is set to zero, and the process returns to starting block 4253 (at block 4258). If the frame under consideration is not the first frame in the sequence, the path marked “No” proceeds from block 4253 where the determination is made, testing the current frame difference against the scene transition threshold in test block 4259. If the current frame difference is greater than its threshold, the decision path marked "Yes" proceeds to block 4255, again allocating an I-frame. If the current frame difference is less than the scene transition threshold, then the "no" path proceeds to block 4261, adding the current frame difference to the accumulated frame difference.

Continuing through the flowchart, at decision block 4263, the accumulated frame difference is compared with a threshold t, which is typically less than the scene transition threshold. If the accumulated frame difference is greater than t, control moves to block 4265 where the frame is allocated to be a P frame, and then the accumulated frame difference is reset in step 4267. If the accumulated frame difference is less than t, control moves from block 4263 to block 4269. In block 4269 the current frame difference is compared to τ, which is less than t. If the current frame difference is less than τ, the frame is skipped at block 4273, and if the current frame difference is greater than τ, the frame is allocated to be β.

는 다음과 같이 정의된다.In another aspect, another frame encoding complexity indicator

Is defined as

는 도 41 의 흐름도 (4200) 에서 현재 프레임 차이 대신에 이용될 것이다. 알 수 있는 바와 같이, 순방향 모션 보상이 낮은 레벨의 움직임을 나타내는 경우에만,

은 M 과 상이하다. 이 경우에, M 은 M 보다 작다.Here, α is a scalar, and, SAD _P is the SAD with forward motion information, MV _P is the sum of the measured lengths in the pixels of the motion vector from the forward motion compensation, s and m is lower than the SAD _P s or MV _P Is two threshold numbers that make the frame encoding complexity indicator zero when m is less than m .

May be used instead of the current frame difference in the flowchart 4200 of FIG. As can be seen, only if forward motion compensation exhibits low levels of motion,

Is different from M. In this case, M is less than M.

The shot detection and encoding aspects described herein may be described as a process depicted as a flowchart, flow chart, structure diagram, or block diagram. Although the flowchart depicted in the figures may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of operation may be rearranged. Normally, one process ends when the operation is completed. One process may correspond to a method, function, procedure, subroutine, subprogram, or the like. If the process corresponds to a function, the termination corresponds to what the function returns to the calling or main function.

In addition, one of ordinary skill in the art appreciates that one or more elements of a device disclosed herein may be rearranged without affecting the operation of the device. Similarly, one or more elements of a device disclosed herein may be combined without affecting the operation of the device. Those skilled in the art will appreciate that any of a variety of different technologies and techniques may be used to represent information and multimedia data. In addition, those skilled in the art will recognize that the various illustrative logical blocks, modules, and algorithm steps described in connection with the examples disclosed herein may be implemented as electronic hardware, firmware, computer software, middleware, microcode, or combinations thereof. have. To clearly illustrate the compatibility of hardware with software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed methods.

For example, the steps of a method or algorithm described in connection with the shot detection and encoding examples and figures disclosed herein may be implemented directly in hardware, in a software module executed by a processor, or in a combination of the two. The methods and algorithms are particularly applicable to communication technologies, including wireless transmission of video to cellular telephones, computers, laptop computers, PDAs, and all types of personal and business communications devices. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other type of storage medium known in the art. An exemplary storage medium is coupled to the processor to enable the processor to read information from or write information to the storage medium. In the alternative, the storage medium may be integrally formed with the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuit (ASIC). The ASIC may reside in a wireless modem. In the alternative, the processor and the storage medium may reside as discrete components in a wireless modem.

In addition, the various illustrative logic blocks, components, modules, and circuits described in connection with the examples disclosed herein may be used in general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or the like. It may be implemented or performed in other programmable logic devices, separate gate or transistor logic, separate hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in other ways, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

Those skilled in the art, with reference to the above description of the disclosed examples, may practice or use the disclosed methods and apparatus. Those skilled in the art will readily appreciate various modifications to these examples, and the principles defined herein may be applied to other examples without departing from the spirit or scope of the disclosed methods and apparatus, and additional elements may be added. The description of the aspects of the invention is illustrative, and is not intended to limit the scope of the claims.

Claims (50)

As a method of processing multimedia data, Receiving interlaced video frames; Converting the interlaced video frames into sequential video; Generating metadata associated with the sequential video; And Providing at least a portion of the sequential video and the metadata to an encoder used to encode the sequential video, Generating metadata associated with the sequential video includes performing shot detection and generating compressed information based on the shot detection. The method of claim 1, And encoding said sequential video using said metadata. The method of claim 1, And converting the interlaced video frames comprises deinterlacing the interlaced video frames. The method of claim 1, And the metadata includes bandwidth information. The method of claim 1, And the metadata includes bidirectional motion information. The method of claim 3, wherein The deinterlacing step, Generating spatial information and bidirectional motion information for the interlaced video frames; And Generating the sequential video based on the interlaced video frames and using the spatial information and the bidirectional motion information. The method of claim 4, wherein The bandwidth information includes luminance information. The method of claim 1, Wherein the metadata comprises a spatial complexity value. The method of claim 1, Wherein the metadata comprises a time complexity value. The method of claim 1, Converting the interlaced video frames comprises inverse telecine of 3/2 pull-down video frames. The method of claim 10, Wherein said metadata includes bandwidth ratio information. The method of claim 1, Resizing the sequential video. 13. The method of claim 12, And segmenting the sequential video to determine group of picture information. delete The method of claim 1, Filtering the sequential video with a denoising filter. The method of claim 1, And the metadata includes luminance and chroma information. An apparatus for processing multimedia data, A receiver configured to receive interlaced video frames; A deinterlacer configured to convert the interlaced video frames into sequential video; And A divider configured to generate metadata associated with the sequential video and to provide the sequential video and the metadata to an encoder used to encode the sequential video, And the divider is configured to perform shot detection and generate compressed information based on the shot detection. The method of claim 17, And an encoder configured to receive the sequential video from a communication module and to encode the sequential video using the provided metadata. The method of claim 17, And the deinterlacer is configured to perform space-time deinterlacing. The method of claim 17, And a noise canceling filter for noise canceling the sequential video. The method of claim 17, And said deinterlacer comprises an inverse teleciner. delete The method of claim 17, The metadata includes image group information. The method of claim 17, And a resampler configured to resize the sequential frames. The method of claim 17, The metadata includes bandwidth information. The method of claim 17, And the metadata includes bidirectional motion information. The method of claim 17, The deinterlacer, Generate spatial information and bidirectional motion information for the interlaced video frames, And generate a sequential video based on the interlaced video frames and using the spatial information and the bidirectional motion information. The method of claim 23, And said metadata comprises a bandwidth ratio. The method of claim 23, And the metadata includes luminance information. The method of claim 17, And the metadata includes a spatial complexity value. The method of claim 17, And the metadata includes a time complexity value. The method of claim 17, And the metadata includes luminance and chroma information. An apparatus for processing multimedia data, Means for receiving interlaced video; Means for converting the interlaced video into sequential video; Means for generating metadata associated with the sequential video; And Means for providing the at least a portion of the sequential video and the metadata to an encoder used to encode the sequential video, The generating means is configured to perform shot detection and generate compressed information based on the shot detection. The method of claim 33, wherein And said converting means comprises an inverse telecine device. The method of claim 33, wherein And said converting means comprises a space-time deinterlacer. delete The method of claim 33, wherein And said generating means is configured to generate bandwidth information. The method of claim 33, wherein And means for resampling to resize a sequential frame. The method of claim 33, wherein And means for encoding the sequential video using the provided metadata. The method of claim 33, wherein And means for noise canceling the sequential video. The method of claim 33, wherein The metadata includes image group information. The method of claim 33, wherein And the metadata includes bidirectional motion information. The method of claim 33, wherein The conversion means, Generate spatial information and bidirectional motion information for the interlaced video frames, And generate a sequential video based on the interlaced video frames and using the spatial information and the bidirectional motion information. The method of claim 33, wherein And said metadata comprises a bandwidth ratio. The method of claim 33, wherein And the metadata includes luminance information. The method of claim 33, wherein And the metadata includes a spatial complexity value. The method of claim 33, wherein And the metadata includes a time complexity value. The method of claim 33, wherein And the metadata includes luminance and chroma information. A machine readable medium containing instructions for processing multimedia data, the machine comprising: Instructions for processing the multimedia data, when executed, cause the machine to: Receive interlaced video frames, Convert the interlaced video frames into sequential video, Generate metadata associated with the sequential video, Provide at least a portion of the sequential video and the metadata to an encoder used to encode the sequential video, The generation of metadata associated with the sequential video includes performing shot detection and generating compressed information based on the shot detection. Receive interlaced video, Convert the interlaced video into sequential video, Generate metadata associated with the sequential video, Providing at least a portion of the sequential video and the metadata to an encoder used to encode the sequential video, The generation of metadata associated with the sequential video includes performing shot detection and generating compressed information based on the shot detection.

Images