RU2800009C2 - Set of history-based non-adjacent mvp for video encoding wave-front processing - Google Patents

Abstract

FIELD: video data encoding.

SUBSTANCE: motion information for the first row of coding tree units (CTU) of the picture is stored in the first history motion vector predictor (MVP) memory buffer. Flush the second history MVP memory buffer. After flushing the second history MVP buffer, the motion information for the second line of the picture CTU is stored in the second history MVP buffer, the second line of the CTU being different from the first line of the CTU.

EFFECT: increased efficiency of video data encoding.

58 cl, 21 dwg

Description

[0001] This application claims priority of U.S. Provisional Application No. 62/696,281, filed July 10, 2018, U.S. Provisional Application No. 62/713,944, filed August 2, 2018, and U.S. Application No. 16/506,720, filed July 9, 2019, the entire contents of each of which are incorporated herein by reference.

FIELD OF TECHNOLOGY

[0002] The present disclosure relates to video coding, including video coding and video decoding.

Prior Art

[0003] Digital video functionality can be incorporated into a wide range of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-readers, digital cameras, digital recorders, digital media players, video game devices, video game consoles, cellular or satellite radio telephones, so-called “smartphones”, video teleconferencing devices, streaming devices transmission (streaming) of video and the like. Digital video devices implement video coding techniques such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4 Part 10, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC) standard, ITU-T H.265/High Efficiency Video Coding (HEVC), and extensions to such standards. Video devices can transmit, receive, encode, decode and/or store digital video information more efficiently by implementing such video coding techniques.

[0004] Video coding techniques perform spatial (intra-picture, intra-picture) prediction and/or temporal (inter-picture, inter-picture) prediction to reduce or remove redundancy inherent in video sequences. For block-based video coding, a video slice (i.e., a video image or a portion of a video image) may be divided into video blocks, which may also be referred to as coding tree units (CTUs), coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture, or temporal prediction with respect to reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames.

Brief description of the essence of the invention

[0005] In general, the present disclosure describes technologies for encoding motion information of video data blocks. These technologies can be used during parallel wavefront processing. The motion information may include motion vectors that are predicted from history-based motion vector predictors (HMVPs). A potentially suitable (candidate) HMVP may correspond to motion information of a previously encoded block. A video encoding device (encoder or decoder) may maintain a table with multiple HMVP candidates during the encoding (encoding and decoding) process. The video encoder may empty the table when a new cut appears. When there is an inter-coded block, the video encoding device may append motion information associated with the inter-coded block to a table.

[0006] In one example, a method for encoding (encoding or decoding) video data includes storing motion information for a first row of coding tree units (CTUs) of an image in a first history-based motion vector predictor (history MVP) buffer in memory; flushing the second MVP history buffer in memory; and, after flushing the second history MVP buffer, storing motion information for the second image CTU row in the second history MVP buffer, the second CTU row being different from the first CTU row. In some examples, a first sub-process (thread) of the video encoding process may encode the first CTU line, and a second sub-process of the video encoding process, different from the first sub-process, may encode the second CTU line.

[0007] In another example, the device for encoding video data includes a memory configured to store video data; and one or more processing units implemented in the circuitry and configured to: store motion information for the first row of coding tree units (CTUs) of the picture in a first history-based motion vector predictor (history MVP) buffer in memory; flush the second history MVP memory buffer; and, after flushing the second history MVP buffer, store motion information for the second image CTU line in the second history MVP buffer, the second CTU line being different from the first CTU line.

[0008] In another example, a computer-readable storage medium has instructions stored therein that, when executed, cause the processor to: store motion information for the first row of coding tree units (CTUs) of an image in a first history-based motion vector predictor (history MVP) buffer in memory; flush the second history MVP buffer in memory; and, after flushing the second history MVP buffer, store motion information for the second image CTU line in the second history MVP buffer, the second CTU line being different from the first CTU line.

[0009] In another example, the device for encoding video data includes a memory configured to store video data; means for storing motion information for a first row of coding tree units (CTUs) of an image in a first history-based motion vector predictor (history MVP) buffer in a memory; means for flushing a second history MVP buffer in memory; and means for storing motion information for the second row CTU of the image in the second history MVP buffer after flushing the second history MVP buffer, wherein the second CTU row is different from the first CTU row.

[0010] Details of one or more examples are shown in the accompanying drawings and in the description below. Other features, objectives and advantages will be apparent from the description and drawings, as well as from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 is a block diagram illustrating an exemplary video encoding and decoding system that can perform the techniques of the present disclosure.

[0012] FIG. 2A and 2B are conceptual diagrams illustrating an exemplary quadtree-binary tree (QTBT) structure and corresponding coding tree unit (CTU).

[0013] FIG. 3 is a flowchart illustrating an exemplary process for encoding motion information using history-based motion vector predictors (HMVPs).

[0014] FIG. 4 is a conceptual diagram illustrating an example of updating an HMVP table.

[0015] FIG. 5 is a conceptual diagram illustrating an exemplary selection of non-contiguous blocks for encoding motion information.

[0016] FIG. 6 is a conceptual diagram illustrating exemplary selection of non-contiguous blocks based on a parent (parent) block.

[0017] FIG. 7 is a conceptual diagram illustrating an example of desired coding tree unit (CTU) wavefront processing.

[0018] FIG. 8 is a conceptual diagram illustrating an example of motion information used for HMVP.

[0019] FIG. 9 is a conceptual diagram illustrating an example of an image divided into multiple lines of coding tree units (CTUs).

[0020] FIG. 10A and 10B are block diagrams illustrating exemplary spatial neighbor motion vectors that are candidates for merging and advanced motion vector prediction (AMVP) modes.

[0021] FIG. 11A and 11B are conceptual diagrams for illustrating temporal motion vector prediction (TMPV) candidates.

[0022] FIG. 12 is a block diagram illustrating an example of a coding tree unit (CTU) and neighboring blocks.

[0023] FIG. 13 is a block diagram illustrating the current CU in the current CTU.

[0024] FIG. 14 is a block diagram illustrating an exemplary video encoder that may perform the techniques of the present disclosure.

[0025] FIG. 15 is a block diagram illustrating an exemplary video decoder that may perform the techniques of the present disclosure.

[0026] FIG. 16 is a flowchart illustrating an exemplary method for encoding a current block of video data in accordance with the techniques of the present disclosure.

[0027] FIG. 17 is a flowchart illustrating an exemplary method for decoding a current block of video data in accordance with the techniques of the present disclosure.

[0028] FIG. 18 is a flowchart illustrating an exemplary method for encoding (encoding or decoding) video data in accordance with the techniques of this disclosure.

DETAILED DESCRIPTION

[0029] FIG. 1 is a block diagram illustrating an exemplary video encoding and decoding system 100 that can perform techniques according to the present disclosure. The techniques of the present disclosure are generally directed to encoding (encoding and/or decoding) video data. In principle, video data includes any video processing data. Thus, video data may include raw, unencoded video, encoded video, decoded (eg, reconstructed) video, and video metadata such as signaling data.

[0030] As shown in FIG. 1, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by destination device 116 (destination), in this example. In particular, source device 102 provides video data to destination device 116 via computer-readable media 110. Source device 102 and destination device 116 may comprise any of a wide variety of devices, including desktop computers, laptops, tablet computers, set-top boxes, telephones such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, and the like. In some cases, source device 102 and destination device 116 may be equipped for wireless communication and thus may be referred to as wireless communication devices.

[0031] In the example of FIG. 1, the source device 102 includes a video source 104, a memory 106, a video encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a video decoder 300, a memory 120, and a display device 118. In accordance with the present disclosure, the video encoder 200 of the source device 102 and the video decoder 300 of the destination device 116 may be configured to apply techniques for encoding motion information. Thus, the source device 102 is an example of a video encoding device, while the destination device 116 is an example of a video decoding device. In other examples, the source device and the destination device may include other components or devices. For example, source device 102 may receive video data from an external video source such as an external camera. Likewise, destination device 116 may communicate with an external display device instead of including an embedded display device.

[0032] System 100, as shown in FIG. 1 is just one example. In principle, any digital video encoding and/or decoding device can perform techniques for encoding motion information. Source device 102 and destination device 116 are just examples of such encoders in which source device 102 generates encoded video data for transmission to destination device 116 . The present disclosure refers to an “encoder” as a device that performs encoding (encoding and/or decoding) of data. Thus, video encoder 200 and video decoder 300 are examples of encoders, specifically a video encoder and a video decoder, respectively. In some examples, source device 102 and destination device 116 may operate substantially symmetrically such that source device 102 and destination device 116 each include video encoding and decoding components. Therefore, system 100 can support unidirectional or bidirectional video transmission between source device 102 and destination device 116, such as for video streaming, video playback, video broadcasting, or video telephony.

[0033] In general, video source 104 represents a source of video data (i.e., raw, unencoded video data) and provides a sequence of pictures (also referred to as “frames”) of video data to video encoder 200, which encodes the data for the pictures. Video source 104 of source device 102 may include a video capture device such as a video camera, a video archive containing previously captured raw video, and/or a video feed interface for receiving video from a video content provider. As a further alternative, video source 104 may generate computer graphics based data as source video, or a combination of live video, archived video, and computer generated video. In each case, video encoder 200 encodes captured, pre-captured, or computer-generated video data. The video encoder 200 may reorder the pictures from the received order (sometimes referred to as "display order") into the encoding order for encoding. Video encoder 200 may generate a bitstream including encoded video data. Source device 102 may then output encoded video data via output interface 108 to computer-readable media 110 for receiving and/or retrieving data, such as input interface 122 of destination device 116.

[0034] The memory 106 of the source device 102 and the memory 120 of the destination device 116 are general purpose memory. In some examples, memories 106 and 120 may store raw video data, such as raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store program instructions executable by, for example, video encoder 200 and video decoder 300, respectively. Although shown separately from video encoder 200 and video decoder 300, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. In addition, the memories 106 and 120 may store encoded video data, such as output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 may be allocated as one or more video buffers, such as for storing raw, decoded, and/or encoded video data.

[0035] Computer-readable media 110 can be any type of media or device capable of transporting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium that allows source device 102 to transmit encoded video data directly to destination device 116 in real time, such as via an RF network or computer network. The output interface 108 may modulate a transmission signal including encoded video data, and the input interface 122 may demodulate the received transmission signal in accordance with a communication standard such as a wireless communication protocol. The communication medium may comprise any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet network such as a local area network, a wide area network, or a wide area network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communications from source device 102 to destination device 116.

[0036] In some examples, source device 102 may output encoded data from output interface 108 to storage device 112. Likewise, destination device 116 may access encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally available storage media such as hard disk drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0037] In some examples, source device 102 may output encoded video data to a file server 114 or other intermediate storage device that may store encoded video generated by source device 102. The destination device 116 can access the stored video data from the file server 114 via streaming or download. File server 114 may be any type of server device capable of storing encoded video data and transmitting this encoded video data to destination device 116 . The file server 114 may be a web server (eg, for a website), a file transfer protocol (FTP) server, a network content delivery device, or a network attached storage (NAS) device. The destination device 116 can access the encoded video data from the file server 114 via any standard data connection, including an Internet connection. This may include a wireless link (e.g., a Wi-Fi connection), a wired connection (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both that is suitable for accessing encoded video data stored on file server 114. File server 114 and input interface 122 may be configured to operate in accordance with a streaming protocol, a download transfer protocol, or a combination of both.

[0038] Output interface 108 and input interface 122 may be wireless transmitters/receivers, modems, wired network components (e.g., Ethernet cards), wireless communications components operating in accordance with any of a variety of IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and the input interface 122 contain wireless components, the output interface 108 and the input interface 122 may be configured to transmit data, such as encoded video data, in accordance with a cellular standard such as 4G, 4G-LTE (Long Term Evolution), LTE-Advanced (Extended LTE), 5G, or the like. In some examples, if the output interface 108 includes a wireless transmitter, the output interface 108 and the input interface 122 may be configured to transmit data, such as encoded video data, in accordance with other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee™), the Bluetooth™ standard, or the like. In some examples, source device 102 and/or destination device 116 may include respective single-chip (SoC) devices. For example, source device 102 may include an SoC device for performing functionality related to video encoder 200 and/or output interface 108, and destination device 116 may include an SoC device for performing functionality related to video decoder 300 and/or input interface 122.

[0039] The techniques of the present disclosure may be applied to video encoding in support of any of a variety of multimedia applications such as terrestrial television transmissions, cable television transmissions, satellite television transmissions, Internet video streaming such as dynamic adaptive streaming over HTTP (DASH), digital video that is encoded on a storage medium, decoding of digital video stored on a storage medium, or other applications.

[0040] The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (eg, storage device 112, file server 114, or the like). The encoded video bitstream may include signaling information defined by video encoder 200 that is also used by video decoder 300, such as syntax elements having values that describe characteristics and/or processing of video blocks or other encoded units (e.g., slices, pictures, groups of pictures, sequences, or the like). The display device 118 displays decoded images of the decoded video data to the user. Display device 118 may be any of a variety of display devices such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light emitting diode (OLED) display, or other type of display device.

[0041] Although not shown in FIG. 1, in some examples, video encoder 200 and video decoder 300 may be integrated with an audio encoder and/or audio decoder and may include appropriate multiplexing-demuxing (MUX-DEMUX) modules or other hardware and/or software for processing multiplexed streams, including both audio and video, in a common data stream. If applicable, MUX-DEMUX modules may conform to the ITU H.223 multiplexer protocol or other protocols such as User Datagram Protocol (UDP).

[0042] Both video encoder 200 and video decoder 300 may be implemented as any of a variety of suitable encoder and/or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the methods are partially implemented in software, the device may store the instructions for the software on a suitable, non-transitory, computer-readable medium and execute the instructions in hardware using one or more processors to execute the methods of the present disclosure. Each of the video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder/decoder (CODEC) in a respective device. The device including the video encoder 200 and/or video decoder 300 may include an integrated circuit, a microprocessor, and/or a wireless communication device such as a cellular phone.

[0043] Video encoder 200 and video decoder 300 may operate in accordance with a video coding standard such as ITU-T H. 265, also referred to as High Efficiency Video Coding (HEVC), or extensions thereof, such as multiview and/or scalable video coding extensions. Alternatively, video encoder 200 and video decoder 300 may operate in accordance with other proprietary or industry standards such as the Joint Exploration Test Model (JEM) or ITU-T H.266, also referred to as Versatile Video Coding (VVC). A recent draft VVC standard is described in Bross, et al. “Versatile Video Coding (Draft 5)”, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, ^14th Meeting: Geneva, CH, 19-27 March 2019, JVET-N1001-v3 (hereinafter “VVC Draft 5”). However, the techniques of the present disclosure are not limited to any particular coding standard.

[0044] In general, video encoder 200 and video decoder 300 may perform block coding of pictures. The term “block” generally refers to a structure containing data to be processed (eg, encoded, decoded, or otherwise used in an encoding and/or decoding process). For example, the block may include a two-dimensional matrix of luma and/or chrominance data samples. In general, video encoder 200 and video decoder 300 may encode video data represented in YUV format (eg, Y, Cb, Cr). That is, instead of encoding the red, green, and blue (RGB) data for the image samples, video encoder 200 and video decoder 300 may encode luminance and chrominance components into itself, where the chrominance components may include both red tint and blue tint chrominance components. In some examples, video encoder 200 converts the received RGB data to a YUV representation prior to encoding, and video decoder 300 converts the YUV representation to RGB. Alternatively, these transformations may be performed by pre- and post-processing modules (not shown).

[0045] The present disclosure may generally refer to encoding (eg, encoding and decoding) of pictures, including the process of encoding or decoding image data. Likewise, the present disclosure may refer to picture block coding, including the process of encoding or decoding data for blocks, such as prediction and/or residual coding. The encoded video bitstream typically includes a sequence of values for syntax elements representing coding decisions (eg, coding modes) and picture blocking. Thus, references to encoding an image or block should generally be understood as encoding values for the syntax elements that make up the image or block.

[0046] HEVC defines various blocks, including coding units (CU), prediction units (PU), and transform units (TU). According to HEVC, a video encoder (such as video encoder 200) partitions a coding tree unit (CTU) into CUs according to a quadtree structure. That is, the video encoder splits the CTU and CU into four equal, non-overlapping squares, and each quadtree node has either zero or four child nodes. Nodes without child nodes may be referred to as "leaf nodes" and the CUs of such leaf nodes may include one or more PUs and/or one or more TUs. The video encoder may further split the PU and TU. For example, in HEVC, the residual quadtree (RQT) is a partition of the TU. In HEVC, PUs represent inter-prediction data while TUs represent residual data. CUs that are intra predicted include intra prediction information such as an intra mode indication.

[0047] As another example, video encoder 200 and video decoder 300 may be configured to operate in accordance with JEM or VVC. According to JEM or VVC, a video encoder (such as video encoder 200) splits an image into multiple coding tree units (CTUs). The video encoder 200 may partition the CTU according to a tree structure, such as a quadtree-binary tree (QTBT) structure or a multiple type tree (MTT) structure. The QTBT structure eliminates the concept of multiple types of partitioning, such as partitioning between CUs, PUs, and TUs in HEVC. The QTBT structure may include two levels: a first level divided according to a quadtree split and a second level divided according to a binary tree split. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of binary trees correspond to coding units (CUs).

[0048] In an MTT split structure, blocks may be split using a quadtree split (QT), a binary tree split (BT), and one or more types of ternary tree splits (TT). A ternary tree split is a split when a block is split into three subblocks. In some examples, splitting a ternary tree divides a block into three subblocks without dividing the original block through the center. Partition types in MTT (for example, QT, BT, and TT) can be symmetric or asymmetric.

[0049] In some examples, video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent both luminance and chrominance components, while in other examples, video encoder 200 and video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luma component and another QTBT or MTT structure for both chrominance components (or two Q structures). TBT or MTT for the respective chrominance components).

[0050] Video encoder 200 and video decoder 300 may be configured to use HEVC quadtree splitting, JEM QTBT splitting, or other splitting structures. For purposes of explanation, a description of the methods of the present disclosure is provided in relation to splitting a QTBT. However, it should be understood that the techniques of the present disclosure may also be applied to video encoders configured to use quadtree partitioning or other types of partitioning.

[0051] The present disclosure may use “N×N” and “N by N” interchangeably to refer to sample sizes of a block (such as a CU or other video block), in terms of vertical and horizontal dimensions, such as 16×16 samples or 16 by 16 samples. In general, a 16×16 CU will have 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Similarly, an N×N CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. Samples in a CU can be arranged in rows and columns. In addition, CUs do not need to have the same number of samples in the horizontal direction and in the vertical direction. For example, a CU may contain N×M samples, where M is not necessarily equal to N.

[0052] The video encoder 200 encodes video data for CUs representing prediction information and/or residual information and other information. The prediction information indicates how the CU should be predicted in order to generate a prediction block for the CU. The residual information is typically the sample-to-sample differences between pre-coding CU samples and the prediction block.

[0053] For CU prediction, video encoder 200 may typically generate a prediction block for the CU via inter-prediction or intra-prediction. Inter-prediction generally refers to CU prediction from previously encoded image data, while intra-prediction generally refers to CU prediction from previously encoded same image data. To perform inter prediction, video encoder 200 may generate a prediction block using one or more motion vectors. The video encoder 200 can typically perform a motion search to identify a reference block that closely matches the CU, eg, in terms of differences between the CU and the reference block. Video encoder 200 may compute a difference metric using sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), root mean square differences (MSD), or other similar difference calculations to determine if a reference block is closely matched to the current CU. In some examples, video encoder 200 may predict the current CU using unidirectional prediction or bidirectional prediction.

[0054] JEM also provides an affine motion compensation mode, which can be thought of as an inter-prediction mode. In the affine motion compensation mode, video encoder 200 may determine two or more motion vectors that represent non-translational motion such as zoom in or out, rotation, perspective movement, or other types of irregular motion.

[0055] To perform intra prediction, video encoder 200 may select an intra prediction mode to generate a prediction block. JEM provides sixty-seven intra-prediction modes, including various directional modes as well as planar and DC modes. In general, video encoder 200 selects an intra-prediction mode that describes adjacent samples for the current block (eg, CU) from which the current block samples should be predicted. Such samples may typically be above, above and to the left or left of the current block in the same picture as the current block, assuming that video encoder 200 encodes CTUs and CUs in raster scan order (left to right, top to bottom).

[0056] The video encoder 200 encodes data representing the prediction mode for the current block. For example, for inter prediction modes, video encoder 200 may encode data representing which of the various available inter prediction modes is in use, as well as motion information for the corresponding mode. For example, for unidirectional or bidirectional inter-prediction, video encoder 200 may encode motion vectors using advanced motion vector prediction (AMVP) or merge mode. Video encoder 200 may use similar modes to encode motion vectors for the affine motion compensation mode.

[0057] After prediction, such as intra-prediction or inter-prediction of a block, video encoder 200 may calculate residual data for the block. Residual data, such as a residual block, represents the sample-to-sample differences between a given block and a prediction block for a given block generated using the corresponding prediction mode. Video encoder 200 may apply one or more transforms to the residual block to obtain transformed data in the transform region instead of the sample region. For example, video encoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the video residual data. In addition, the video encoder 200 may apply a secondary transform after the first transform, such as mode-dependent indivisible secondary transform (MDNSST), signal-dependent transform, Karhunen-Loeve transform (KLT), and the like. Video encoder 200 generates transform coefficients after applying one or more transforms.

[0058] As noted above, after any transformations to obtain transform coefficients, video encoder 200 may perform quantization of the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to reduce the amount of data used to represent the coefficients as much as possible, allowing for further compression. By performing a quantization process, video encoder 200 may reduce the bit depth associated with some or all of the coefficients. For example, video encoder 200 may round an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, video encoder 200 may perform a bitwise right shift of the value to be quantized.

[0059] After quantization, video encoder 200 may scan the transform coefficients, forming a one-dimensional vector from a two-dimensional matrix including the quantized transform coefficients. The scan can be designed to place the higher energy (and therefore lower frequency) transform coefficients at the beginning of the vector, and to place the lower energy (and hence higher frequency) transform coefficients at the end of the vector. In some examples, video encoder 200 may use a predefined scan order to scan the quantized transform coefficients to obtain a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other examples, video encoder 200 may perform adaptive scanning. After scanning the quantized transform coefficients to generate a one-dimensional vector, video encoder 200 may entropy encode the one-dimensional vector, such as in accordance with context adaptive binary arithmetic coding (CABAC). Video encoder 200 may also entropy encode values for syntax elements describing metadata associated with encoded video data for use by video decoder 300 when decoding video data.

[0060] To perform CABAC, video encoder 200 may assign a context within a context model to a character to be transmitted. The context may relate, for example, to whether adjacent symbol values are null or not. The probability determination may be based on the context assigned to the symbol.

[0061] The video encoder 200 may further generate syntax data such as block-based syntax data, image-based syntax data, and sequence-based syntax data for the video decoder 300, such as in a picture header, block header, slice header, or other syntax data such as a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS). The video decoder 300 may also decode such syntax data to determine how to decode the corresponding video data.

[0062] Thus, video encoder 200 can generate a bitstream including encoded video data, such as syntax elements describing the partitioning of an image into blocks (eg, CUs), and prediction information and/or residual information for the blocks. Ultimately, the video decoder 300 may receive the bitstream and decode the encoded video data.

[0063] In principle, the video decoder 300 performs a reverse process to the process performed by the video encoder 200 to decode encoded video bitstream data. For example, video decoder 300 may decode values for bitstream syntax elements using CABAC in a substantially similar, albeit inverse manner to the CABAC coding process of video encoder 200. The syntax elements may define partitioning information of an image into CTUs and partitioning of each CTU according to an appropriate partitioning structure, such as a QTBT structure, to define a CU in a CTU. The syntax elements may further define prediction and residual information for blocks (eg, CUs) of video data.

[0064] The residual information may be represented, for example, by quantized transform coefficients. Video decoder 300 may inversely quantize and inversely transform the quantized block transform coefficients to recover a residual block for that block. Video decoder 300 uses the signaled prediction mode (intra or inter prediction) and associated prediction information (eg, motion information for inter prediction) to generate a prediction block for that block. Video decoder 300 may then combine the prediction block and the residual block (sample-to-sample basis) to reconstruct the original block. Video decoder 300 may perform additional processing, such as performing a deblocking process to reduce visual artifacts along block boundaries.

[0065] The present disclosure may generally refer to "signaling" certain information, such as syntax elements. The term “signaling” may generally refer to reporting values for syntax elements and/or other data used to decode encoded video data. That is, video encoder 200 may signal values for syntax elements in the bitstream. In general, signaling refers to the generation of a value in a bitstream. As noted above, source device 102 may transport the bitstream to destination device 116 in substantially real time or non-real time, as may occur when syntax elements are stored in storage device 112 for later retrieval by destination device 116.

[0066] In accordance with the techniques of the present disclosure, video encoder 200 and video decoder 300 may be configured to perform wavefront processing in parallel when encoding an image of video data. In general, parallel wavefront processing may involve encoding separate strings of coding tree units (CTUs) using separate processing sub-processes. For example, the first sub-process executed by video encoder 200 or video decoder 300 may process the first CTU row, the second sub-process may process the second CTU row, and so on. CTU encoding includes, among other things, encoding motion information for motion prediction coding units (CUs) in a CTU that may refer to motion information within the same CTU or a previously coded CTU (eg, left and/or upper neighbor CTU). Such motion information may be stored in a motion vector predictor (MVP) buffer. In accordance with the techniques of the present disclosure, video encoder 200 and video decoder 300 may be configured to flush the MVP buffer for the current CTU line before encoding the video data of the current CTU line. The MVP buffer may be a separate MVP buffer for the current CTU row, or a common MVP buffer may be used for multiple CTU rows.

[0067] In some examples, when storing motion information in the MVP buffer, video encoder 200 and video decoder 300 may only store unique motion information in the MVP buffer. For example, video encoder 200 and video decoder 300 may encode the current CU using the current motion vector, determine whether the motion vector is currently stored in the MVP buffer for the current CU, and if so, disable storing the motion vector in the MVP buffer, and if not, store the motion vector in the MVP buffer.

[0068] In some examples, video encoder 200 and video decoder 300 may use a first-come-first-served (FIFO) rule to remove motion vectors from the MVP buffer when the MVP buffer becomes full. That is, in order to add a new motion vector to the MVP buffer, video encoder 200 and video decoder 300 may remove the earliest input motion vector from the MVP buffer and insert a new motion vector into the MVP buffer. Thus, the MVP buffer can implement a queue-like behavior.

[0069] In some examples, video encoder 200 and video decoder 300 may maintain separate MVP buffers for each of a variety of different motion model types. For example, video encoder 200 and video decoder 300 may support an affine MVP buffer for an affine motion model, an intra-block copy MVP buffer for intra-block copy mode motion information, a lighting compensation MVP buffer for local lighting compensation motion information, a sub-block MVP buffer for sub-block MVP, and/or a temporal MVP buffer for temporal motion prediction.

[0070] In some examples, video encoder 200 and video decoder 300 may generate a synthetic MVP from two or more MVPs in one or more MVP buffers and input the synthetic MVP into one of the MVP buffers. Two or more MVPs may correspond to the same or different traffic models (that is, have different types of traffic information).

[0071] FIG. 2A and 2B are a conceptual diagram illustrating an exemplary quadtree-binary tree (QTBT) structure 130 and corresponding coding tree unit (CTU) 132. Solid lines represent quadtree partitioning, and dotted lines indicate binary tree partitioning. At each split (ie, non-leaf) node of the binary tree, one flag is signaled indicating which type of split (ie, horizontal or vertical) is used, where 0 indicates horizontal split and 1 indicates vertical split in this example. To split a quadtree, it is not necessary to specify a split type, since the quadtree nodes divide the block horizontally and vertically into 4 equally sized subblocks. Accordingly, video encoder 200 may encode and video decoder 300 may decode syntax elements (such as split information) for the area tree layer of the QTBT structure 130 (i.e., solid lines) and syntax elements (such as split information) for the prediction tree layer of the QTBT structure 130 (i.e., dotted lines). Video encoder 200 may encode and video decoder 300 may decode video data, such as prediction and transform data, for the CUs represented by leaf leaf nodes of the QTBT structure 130.

[0072] In general, CTU 132 in FIG. 2B may be associated with parameters defining block sizes corresponding to the nodes of the QTBT structure 130 at the first and second levels (eg, region tree level and prediction tree level). These parameters may include the CTU size (representing the size of the CTU 132 in samples), the minimum quadtree size (MinQTSize, representing the minimum allowable quadtree leaf node size), the maximum binary tree size (MaxBTSize, representing the maximum allowable binary tree root node size), the maximum binary tree depth (MaxBTDepth, representing the maximum allowable binary tree depth), and the minimum binary tree size (MinBTSize, representing the minimum allowable th size of the leaf node of the binary tree).

[0073] The root node of the QTBT structure corresponding to the CTU may have four child nodes at the first level of the QTBT structure, each of which may be partitioned according to a quadtree split. That is, the first-level nodes are either leaf nodes (having no children) or have four children. An example structure of QTBT 130 represents nodes such as including a parent node and child nodes having solid lines for branches. If the nodes of the first level are not larger than the maximum allowable size of the root node of the binary tree (MaxBTSize), then the nodes can be further split into the corresponding binary trees. Splitting a binary tree of one node may be repeated until the nodes resulting from the split reach the minimum allowable binary tree leaf node size (MinBTSize) or the maximum allowable binary tree depth (MaxBTDepth). An example structure of QTBT 130 represents such nodes as having dashed lines for branches. A leaf node of a binary tree is referred to as a coding unit (CU) which is used for prediction (eg intra-image or inter-image prediction) and transformation without any further partitioning. As discussed above, CUs may also be referred to as “video blocks” or “blocks”.

[0074] In one example of a QTBT split structure, the CTU size is set to 128x128 (luminance samples and two corresponding 64x64 chrominance samples), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. Quadtree splitting is applied to the CTU first to form a leaf new nodes of the quadtree. The leaf nodes of the quadtree can be from 16x16 (ie MinQTSize) to 128x128 (ie CTU size). If a quadtree leaf node is 128x128, it will not be further split into a binary tree because the size is larger than MaxBTSize (ie 64x64, in this example). Otherwise, the leaf node of the quadtree will be further split into a binary tree. Thus, the leaf node of the quadtree is also the root node for the binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (4, in this example), no further splitting is allowed. When a binary tree node has a width equal to MinBTSize (4, in this example), this means that no further vertical splitting is allowed. Similarly, a binary tree node having a height equal to MinBTSize implies that no further horizontal splitting is allowed for that binary tree node. As noted above, the leaf nodes of the binary tree are referred to as CUs and are further processed according to prediction and transformation without further partitioning.

[0075] FIG. 3 is a flowchart illustrating an exemplary process for encoding motion information using history motion vector predictors (HMVPs). Initially, a video encoder, such as video encoder 200 or video decoder 300, downloads the HMVP candidate table (140). The video encoder then encodes the video data block using HMVP candidates (142). The video encoder then updates the table with motion information of the encoded block (144).

[0076] FIG. 4 is a conceptual diagram illustrating an example of updating an HMVP table. In JVET-K0104, the table size is set to 16 and the first-in-first-out (FIFO) rule is applied. Fig. 4 depicts an example in which a FIFO rule is applied to remove an HMVP candidate and add a new one to the table used in the example methods of the present disclosure.

[0077] A video encoding device such as video encoder 200 or video decoder 300 may insert HMVP candidates from the last entry into the first entry in the table after a temporal motion vector prediction (TMVP) candidate in the candidate list. The video encoder may apply clipping (elimination) to HMVP candidates. The video encoding device may terminate the trimming process when the total number of available merge candidates reaches the signaled maximum allowable number of merge candidates.

[0078] In the example of FIG. 4, the pre-update table contains history MVP ₀ (HMVP ₀ ) to history MVP _L-1 (HMVP _L-1 ), where subscripts 0 to L-1 represent the order in which history MVPs are added. C _L-1 represents the new story MVP to be added to the table. Thus, according to the FIFO rule, HMVP ₀ is removed from the table before C _L-1 is added, in this example.

[0079] FIG. 5 is a conceptual diagram illustrating an exemplary selection of non-contiguous blocks for encoding motion information. In the example in FIG. 5, the current block, denoted by “Curr”, represents the current coding unit (CU) for which motion information can be encoded using adjacent and/or non-adjacent adjacent blocks, denoted A _i , B _j , and NA _k . Non-adjacent motion vector prediction is described, for example, in US Application No. 16/003,269, filed June 8, 2018. A video encoder may apply a FIFO rule and a maximum size of a motion candidate buffer for non-adjacent blocks.

[0080] FIG. 6 is a conceptual diagram illustrating exemplary selection of non-contiguous boxes based on a parent box. That is, the parent block is a block divided into subblocks, including the current block. For example, the parent unit may be a CTU or a subunit into which the CTU has been partitioned. Like FIG. 5 in FIG. 6, the current CU is denoted as “Curr”, and non-contiguous blocks from which motion information can be extracted and used to predict motion information of the current CU are denoted as “NA _i,j ”.

[0081] The motion vectors of adjacent spatial blocks of the aligned block can be used as motion vector prediction (MVP) candidates for the merge mode, in addition to the motion vectors H and C (i.e., the motion vectors at the center and bottom-right of the aligned block).

[0082] The techniques of the present disclosure can be used to improve motion vector prediction, for example, by adding candidates used for AMVP and/or merge coding modes, where the added candidates can be taken from non-contiguous blocks. For example, the added candidates may correspond to any of NA _1.1 to NA _1.9 in FIG. 6.

[0083] FIG. 7 is a conceptual diagram illustrating an example of desired coding tree unit (CTU) wavefront processing. As shown in FIG. 7, different sub-processes can be assigned to handle different CTU strings. That is, a video encoder, such as video encoder 200 or video decoder 300, may execute many different sub-processes, such as wavefront parallel processing (WPP) when encoding different CTU lines. In some examples, certain probabilities used for context encoding (e.g., CABAC encoding) of motion information of inter-predicted blocks could not be determined if the probabilities were to be determined from the last block of the previous CTU row, for example, assuming that the last block had not yet been encoded. Thus, in accordance with the techniques of the present disclosure, video encoder 200 and video decoder 300 may flush the CTU buffer for the CTU line before encoding the CTU line to ensure that the CTU lines can be processed correctly.

[0084] FIG. 8 is a conceptual diagram illustrating an example of motion information used for HMVP. Fig. 8 illustrates how the use of FIFO can remove the motion vectors of blocks closer to the current block from the list of candidates, while the motion vectors of subsequent blocks can be taken into account. In particular, in FIG. 8, X represents motion information encoded at the current time, and MVs of shaded blocks are in the history buffer. The present disclosure recognizes that conventional HMVP techniques do not fully utilize motion vectors of non-contiguous blocks, at least in part due to the use of the FIFO rule as shown in FIG. 8.

[0085] In particular, when encoding block X, motion information of non-adjacent blocks (TL0, T0, T1, TR0, TR1, TR2, TR3) of the top-left CTU, top CTU, and top-right CTU was removed from the history buffer. Therefore, motion information of these blocks is not considered to be added to the candidate list even if non-adjacent blocks are closer to X than, for example, CTU LL0, CTU LL1 and CTU F0-F3 whose motion vectors are in the history buffer.

[0086] The present disclosure also recognizes that a single buffer for HVMP is not applicable for parallel wavefront processing. If only one buffer is used, the buffer size must be very large to accommodate potential spatial candidates for blocks processed in each subprocess (eg, CTU rows). For example, if four sub-processes are configured to execute in parallel, the buffer size can be up to 64. As a result, more bits will be required to signal the MVP index to the video decoder 300. Similarly, redundant entries may occur. That is, an entry in the history buffer may be potentially useful for a block in that line, but may not be useful for blocks in other lines (eg, X and F in FIG. 8). Therefore, it may be difficult to find the optimal candidate for a block.

[0087] FIG. 9 is a conceptual diagram illustrating an example of an image broken into multiple coding tree unit (CTU) lines. In particular, in the example of FIG. 9, picture 150 includes CTU lines 152A-152E (CTU lines 152). Each of the CTU 152 strings includes a corresponding set of CTUs: CTU string 152A includes CTUs 154A-154J, CTU string 152B includes CTUs 156A-156J, CTU string 152C includes CTUs 158A-158J, CTU string 152D includes CTUs 160A-160J, and CTU string 152E includes CTU 162A-162J.

[0088] Video encoder 200 and video decoder 300 may be configured in accordance with the techniques of the present disclosure to use multiple buffers for history-based MVP. In some examples, video encoder 200 and video decoder 300 may maintain separate history MVP buffers for each of the CTU 152 rows (each of which may be processed by a separate respective processing sub-process), or there may be a single buffer that is flushed at the beginning of each CTU row when parallel wavefront processing is applied.

[0089] In one example, CTU 158C may be the current CTU. CTU motion information 154F-154F, 156A-156D, 158A, and 158B (shown using gray shading in FIG. 9) may be available in one or more respective history MVP buffers for use in encoding CTU 158C motion information.

[0090] Additionally or alternatively, video encoder 200 and video decoder 300 may perform history MVP buffer initialization using any or all of the following methods, alone or in combination. Video encoder 200 and video decoder 300 may flush the history MVP buffer of each CTU row for cleanup. Video encoder 200 and video decoder 300 may pre-populate the history MVP buffer of each CTU row with null motion vectors with different reference frame index and/or inter-prediction directions or other predefined or derived motion information. Video encoder 200 and video decoder 300 may prefill the history buffer MVP of each CTU row with motion information from encoded frames (pictures) in the same time layer or lower time layers (reference pictures available for the current frame/picture).

[0091] The video encoder 200 and the video decoder 300 may scale the motion information, eg, based on temporal distance, or process/modify the motion information, eg, combine this motion information with another MV. In general, video encoder 200 and video decoder 300 may combine motion information with motion information from the previous history MVP buffer in coded frames/pictures, or motion information from a co-located area (may be a CTU or larger than a specific block size, e.g., a 4×4 block) in coded frames/pictures. Video encoder 200 and video decoder 300 may prefill the CTU top line history MVP buffer when the top-right CTU from the current CTU is encoded. Video encoder 200 and video decoder 300 may use null motion vectors with different reference frame indices and/or inter-prediction directions, or other predefined or derived motion information.

[0092] Additionally or alternatively, video encoder 200 and video decoder 300 may, when some CTU of the CTU rows is encoded (encoded or decoded), use an associated history buffer MVP to initialize or modify CTU row history buffers below the current CTU row.

[0093] Additionally or alternatively, video encoder 200 and video decoder 300 may apply a FIFO rule to remove an entry from an associated history MVP buffer.

[0094] Additionally or alternatively, video encoder 200 and video decoder 300 may clear the history MVP buffer when the associated CTU string is fully encoded/decoded.

[0095] Video encoder 200 and video decoder 300 may support an MVP buffer size that is larger than AMVP/merge candidate lists or other modes. Any MV or multiple MVs from the buffer may be selected as MV candidate(s) for a candidate list used in a particular mode, such as AMPV, merge mode, affine, or any other inter-mode. The rule may specify how to select MVs from the buffer, for example, take the last N MVs added to the buffer, or take a few from the start of the buffer and/or a few from the middle of the buffer and/or a few from the end of the buffer. Alternatively, signaling may be used to indicate which MVs are selected (eg, video encoder 200 may encode the signaled data and video decoder 300 may decode the signaled data). The MVP buffer size may be signaled in any parameter set (eg, video parameter set, sequence parameter set, image parameter set, etc.), slice header, or elsewhere. An MVP buffer may be associated with a clip, image, and/or video sequence.

[0096] When the video encoder 200 and the video decoder 300 process an inter-coded block, the MVs used in the block may be added to the MVP buffer, and only unique MVs may be stored in the buffer. When the buffer is full, older MVs can be removed from the buffer when new MVs are added. There may be a rule that MVs can be added to the buffer, for example, only signaled MVs can be added, as in AMVP mode, and if the block is encoded in merge mode, then the MV of the block is not added to the buffer.

[0097] Video encoder 200 and video decoder 300 may add MVs to an already existing MV or multiple MVs in the buffer. For example, if already existing MVs in the buffer are unidirectional, then when a new MV is added, these existing MVs can be modified to be bidirectional by adding the new MV.

[0098] Some MV processing may be applied when a new MV is added. For example, if the new MV is close to existing MVs in the buffer, then those close MVs may be removed. Close can mean closeness when comparing the values of the MV components (eg x- and y-components). In some examples, only MVs that differ from already existing MVs in the buffer by a threshold value may be added to the buffer. The same threshold can be configured for different buffers.

[0099] The motion vectors in the buffer can be unidirectional (L0 or L1), bidirectional, or represent the MV of any other motion model.

[0100] The mode information may be associated with the MV in the buffer, and if the index of the MV in the buffer is signaled in the block or another rule is applied to obtain the MV from the buffer, then the mode information may be obtained from the data associated with this MV information. For example, if the information is merge mode, then the block is encoded in merge mode with the specified MV.

[0101] The present disclosure further recognizes that traditional, history-based MVPs contain only regular motion predictors and are only used for regular motion prediction without modifying motion information. In accordance with the techniques of the present disclosure, video encoder 200 and video decoder 300 may use at least one history MVP buffer that contains not only encoded motion information, but also other types of motion predictors, such as for affine motion model, intra-block copy mode motion information, local illumination compensation motion information, sub-block MVP, or temporal motion predictor.

[0102] Additionally or alternatively, video encoder 200 and video decoder 300 may use a plurality of history MVP buffers for different motion models, such as an affine motion model, intra-block copy mode motion information, local lighting compensation motion information, sub-block MVP, or temporal motion predictor.

[0103] Additionally or alternatively, a synthetic motion vector based on the current MVP and another motion predictor such as another spatial MVP or temporal MVP may also be added to the list of candidates.

[0104] Additionally or alternatively, video encoder 200 and video decoder 300 may generate a synthetic MVP from two or more MVPs in the story MVP buffer, or one or more MVPs in the story MVP buffer with other types of MVPs, such as spatial or temporal MVPs.

[0105] Video encoder 200 and video decoder 300 may implement a block partitioning scheme. In HEVC, pictures are divided into a sequence of coding tree units (CTUs). For an image that has three sample arrays, the CTU includes an N×N luminance sample block along with two corresponding chrominance sample blocks. The CTU is divided into coding units (CU) using a tree structure. Each leaf-CU may be further partitioned into one, two, or four prediction units (PU) according to the partition type of the PU. After obtaining the residual block, by applying a prediction process based on the division type of the PU, the leaf-CU can be divided into transformation units (TUs).

[0106] In VVC, the multiple type nested tree quadtree segmentation structure using binary and ternary partitioning replaces the concepts of multiple partition unit types, i.e. multiple type nested tree splitting removes the concepts of CU, PU, and TU splitting, except for those needed for CUs that are too large for the maximum transform length and maintain more flexibility for CU splitting forms. In the coding tree structure, the CU may be square or rectangular.

[0107] Video encoder 200 and video decoder 300 may use the motion information to predict a video data block. For each block, a set of motion information may be available. The motion information set contains motion information for forward and backward prediction directions. Here, forward and backward prediction directions are two prediction directions corresponding to reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1) of the current picture or slice. The terms "forward" and "inverse" do not necessarily have a geometric meaning. Instead, they are used to distinguish which reference picture the motion vector is based on. Forward prediction means a prediction generated from reference list 0, while backward prediction means a prediction generated from reference list 1. When both reference list 0 and reference list 1 are used to generate a prediction for a given block, this is called bidirectional prediction.

[0108] For a given image or slice, if only one reference picture list is used, each block within the picture or slice is forward predicted. If both reference picture lists are used for a given picture or slice, a block within the picture or slice may be forward predictable or backward predictable or bidirectionally predictable.

[0109] For each prediction direction, the motion information contains a reference index and a motion vector. The reference index is used to identify the reference picture in the corresponding reference picture list (eg, RefPicList0 or RefPicList1). The motion vector has both a horizontal and a vertical component, each indicating an offset value along the horizontal and vertical directions, respectively. In some descriptions, for simplicity, the word "motion vector" may be used interchangeably with motion information to indicate both the motion vector and its associated reference index.

[0110] The Picture Order Count (POC) is widely used in video coding standards to identify the display order of an image. While there are cases where two pictures in the same video coded sequence may have the same POC value, this usually does not happen within a video coded sequence. When multiple encoded video sequences are present in a bitstream, pictures with the same POC value may be closer to each other in terms of decoding order.

[0111] In HEVC, there are two inter-prediction modes called merge mode (with skipping being considered a special case of merge mode) and advanced motion vector prediction (AMVP) mode, respectively for the PU.

[0112] In AMVP or merge mode, a motion vector (MV) candidate list is maintained for a plurality of motion vector predictors. The motion vector(s), as well as the reference indexes in merge mode, of the current PU are generated by taking one candidate from the MV candidate list.

[0113] The MV candidate list contains up to 5 candidates for the merge mode and only two candidates for the AMVP mode. The merge candidate may contain a set of motion information, such as motion vectors, corresponding to both reference picture lists (list 0 and list 1) and reference indices. If the merge candidate is identified by the merge index, then the reference pictures are used to predict the current blocks and the associated motion vectors are also determined. However, in the AMVP mode, for each potential prediction direction from list 0 or list 1, the reference index is explicitly signaled, along with the MVP index, to the MV candidate list, since the AMVP candidate contains only a motion vector. In the AMVP mode, the predicted motion vectors can be further refined.

[0114] As can be seen from the above, the merge candidate corresponds to a complete set of motion information, while the AMVP candidate contains only one motion vector for a particular prediction direction and reference index. Candidates for both modes can be similarly derived from the same spatial and temporal neighboring blocks.

[0115] FIG. 10A and 10B are block diagrams illustrating exemplary spatial neighbor motion vector candidates for merging and advanced motion vector prediction (AMVP) modes. Fig. 10A shows an example of spatial neighbor MV candidates for the merge mode, and FIG. 10B shows an example of spatial neighbor MV candidates of the AMVP mode. Spatial MV candidates are derived from adjacent blocks as shown in FIG. 10A and 10V. For a particular PU (PU0), the methods for generating block candidates differ between merge and AMVP modes.

[0116] In the merge mode, up to four MV spatial candidates can be output in the order shown in FIG. 10A. Specifically, the order is left (0), top (1), top right (2), bottom left (3), and top left (4), as shown in FIG. 10A.

[0117] In AVMP mode, neighboring blocks are divided into two groups. The first group is the left group which includes blocks 0 and 1. The second group is the top group which includes blocks 2, 3 and 4 as shown in FIG. 10V. For each group, a candidate candidate in an adjacent block referring to the same reference picture as indicated by the signaled reference index has the highest priority to be selected to form the final group candidate. It is possible that all neighboring blocks do not contain a motion vector pointing to the same reference picture. Therefore, if no such candidate can be found, the first available candidate will be scaled to form the final candidate, so differences in time distance can be compensated for.

[0118] FIG. 11A and 11B are conceptual diagrams illustrating temporal motion vector prediction (TMPV) candidates. Fig. 11A shows an example of a TMPV candidate. The TMVP candidate, if enabled and available, is added to the list of MV candidates after the space motion vector candidates. The motion vector derivation process for a TMPV candidate is the same for merge and AMPV modes, however, the target reference index for a TMPV candidate in merge mode is always set to 0.

[0119] The primary block location for deriving a TMVP candidate is the lower right block outside the collocated PU, as shown in FIG. 11A as a "T" block to compensate for the offset to the top and left blocks used to generate spatial neighbor candidates. However, if this block is located outside the current CTB row or motion information is not available, the block is replaced by the central PU.

[0120] The motion vector for the TMVP candidate is derived from the merged PU of the merged image specified at the slice level. The motion vector for a shared PU is called a shared MV.

[0121] FIG. 11B shows an example of MV scaling. To derive a TMVP candidate motion vector, the co-located MV may require scaling to compensate for time distance differences, as shown in FIG. 11V.

[0122] Some other aspects of the merge modes and AMVP deserve mention as follows. For example, video encoder 200 and video decoder 300 may perform motion vector scaling. It is assumed that the value of the motion vectors is proportional to the distance between the images during the presentation. The motion vector associates two images, a reference image and an image containing the motion vector (namely, an enclosing image). When a motion vector is used to predict another motion vector, the distance between the containing picture and the reference picture is calculated based on Picture Order Count (POC) values.

[0123] For motion vector prediction, its associated containing picture and reference picture may be different. Therefore, a new distance is calculated (based on POC). And the motion vector is scaled based on these two POC distances. For a spatial neighbor candidate, the embedding pictures for the two motion vectors are the same, while the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporal neighbor candidates.

[0124] As another example, video encoder 200 and video decoder 300 may generate artificial motion vector candidates. If the list of motion vector candidates is not complete, artificial motion vector candidates are generated and inserted at the end of the list until it contains all candidates.

[0125] In the merge mode, there are two types of artificial MV candidates: a combined candidate derived only for B-slices, and null candidates used only for AMVP if the first type does not provide enough artificial candidates. For each pair of candidates that are already in the candidate list and have the required motion information, bidirectional combined motion vector candidates are derived by combining the motion vector of the first candidate referring to the image in list 0 and the motion vector of the second candidate referring to the image in list 1.

[0126] As another example, video encoder 200 and video decoder 300 may perform a trimming process to insert a candidate. Candidates from different blocks may end up being the same, which reduces the effectiveness of the merger/AMVP candidate list. To solve this problem, the cropping process is applied. It compares one candidate with others in the current list of candidates to avoid inserting an identical candidate to a certain extent. To reduce complexity, only a limited number of pruning processes are applied instead of comparing each potential candidate with all other existing candidates. If applicable, only the following comparisons apply: the top merge candidate is compared to the left merge candidate, the top right merge candidate is compared to the top merge candidate, the bottom left merge candidate is compared to the left merge candidate, the top left merge candidate is compared to the left merge candidate and the top merge candidate.

[0127] Video encoder 200 and video decoder 300 may also use other motion prediction techniques. In the development of multipurpose video coding (VVC), a history-based motion vector prediction (HMVP) method has been proposed in L. Zhang etc., “CE4-related: History-based Motion Vector Prediction”, Joint Video Experts Team Document: JVET-K0104 (hereinafter “K0104”). The HMVP method allows each block to find its MV predictor from a list of MVs decoded from the previous one, in addition to directly adjacent causal adjacent motion fields. A table with multiple HMVP candidates is maintained during the encoding/decoding process. The table is emptied when a new cut is found. Whenever there is an inter-coded block, the associated motion information is inserted into the table in a first-in-first-out (FIFO) manner as a new HMVP candidate. A limited FIFO rule can then be applied. When inserting an HMVP into a table, a redundancy check may first be applied to find if there is an identical HMVP in the table. If it is found, then that particular HMVP can be removed from the table and all HMVP candidates are then moved.

[0128] The HMVP candidates may be used in the process of building a merge candidate list. For example, all HMVP candidates from the last entry to the first entry in the table may be inserted after the TMVP candidate. Pruning can be applied to HMVP candidates. Once the total number of available merge candidates reaches the signaled maximum allowable number of merge candidates, the merge candidate list building process ends.

[0129] Similarly, HMVP candidates can also be used in the process of building a list of AMVP candidates. The motion vectors of the last K HMVP candidates in the table may be inserted after the TMVP candidate. In some examples, only HMVP candidates with the same reference picture as the target AMVP reference picture are used to build the list of AMVP candidates. Pruning can be applied to HMVP candidates.

[0130] In HEVC, the encoding of the current CTU may only depend on the CTUs left, top-left, top, and top-right. Thus, parallel wavefront processing (WPP) can be supported in HEVC. However, the HMVP method in K0104 may cause a dependency between the current block and all previously encoded CTUs in the slice. Thus, WPP cannot be applied if the HMVP method is used. This disclosure describes methods for using HMVP with CTU initialization, in which the dependencies remain the same as in HEVC. This disclosure also describes HMVP methods with CTU string initialization (reset).

[0131] According to the techniques of the present disclosure, video encoder 200 and video decoder 300 may perform HMVP with CTU initialization. The HMVP table is initialized at the beginning of each CTU. The initialization may add MVs from the current CTU's immediately adjacent coded units to the HMVP table. An immediately adjacent coded block may be to the left, top, top-left, or top-right of the current CTU, as in HEVC. If temporal motion vector prediction is enabled, then the immediately adjacent encoded block may also be a co-located block in the reference picture.

[0132] In FIG. 12 is a block diagram illustrating an example coding tree unit (CTU) and neighboring blocks. In one example, only the spatial and temporal merge candidates for the current CTU are used to initialize the HMVP table. An example of using spatial and temporal HEVC merge candidates is shown in FIG. 12. The insertion order is as follows: left (0), top (1), top-right (2), and top-left (4). The location of the temporary merger candidate is indicated as "T". Note that the lower right temporary merge candidate and the lower left (3) candidate are not available because their locations are below the current CTU line.

[0133] In another example, the merge candidate derivation process for the current CTU is used to initialize the HMVP table. In addition to the spatial and temporal merge candidates, other merge candidates (eg, artificial motion vector candidates) may also be used for initialization.

[0134] FIG. 13 is a block diagram illustrating the current CU in the current CTU. In some examples, the HMVP table is initialized to be empty at the start of encoding the current CTU. However, after the first CU is encoded, the spatial and temporal merge candidates of the first CU, as shown in FIG. 13 are added to the HMVP table. And then, the MV of the first CU is also added if the CU is encoded by the inter-prediction method. It should be noted that if the first CU is not equal to the current CTU, then two temporary merge candidates “T0” and “T1” may be added in order. Fig. 13 shows an example of first CU merge candidates in CTU.

[0135] In another example, the HMVP table is initialized to be empty at the start of encoding the current CTU. However, after encoding the first CU, all merge candidates of the first CU are added to the HMVP table. And then the MV of the first CU is also added if it is encoded by the inter-prediction method.

[0136] Video encoder 200 and video decoder 300 can also perform HMVP with CTU string initialization. In another example, the HMVP CTU initialization described above only applies to the first CTU in the CTU line. Similar to HMVP in K0104, a pruning process can be applied to an initialized table to remove some or all of the duplicates. The pruning process may also not be applied to the initialized table to reduce complexity.

[0137] FIG. 14 is a block diagram illustrating an exemplary video encoder 200 that may perform the techniques of the present disclosure. Fig. 14 is provided for purposes of explanation and should not be construed as limiting the methods broadly illustrated and described in this disclosure. For purposes of explanation, the present disclosure describes the video encoder 200 in the context of video coding standards such as the HEVC video coding standard and the emerging H.266 video coding standard. However, the techniques of the present disclosure are not limited to these video coding standards and apply in general to video coding and decoding.

[0138] In the example of FIG. 14, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy encoding unit 220. Any or all of the video data memory 230, mode selection module 202, residual generation module 204, transform processing module 206, quantization module 208, inverse quantization module 210, inverse transform processing module 212, reconstruction module 214, filter module 216, DPB 218, and entropy encoding module 220 may be implemented in one or more processors or processing circuits. In addition, video encoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

[0139] The video data memory 230 may store video data to be encoded by the components of the video encoder 200. Video encoder 200 may receive video data stored in video data memory 230 from, for example, video source 104 (FIG. 1). DPB 218 may act as a reference picture memory that stores reference video data for use in prediction of subsequent video data by video encoder 200. Video data memory 230 and DPB 218 may be formed by any of a variety of memory devices such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 230 and DPB 218 may be provided by the same memory device or by separate memory devices. In various examples, video data memory 230 may be on-chip (single-chip) with other components of video encoder 200, as illustrated, or off-chip with respect to these components.

[0140] In this disclosure, reference to video data memory 230 is not to be interpreted as being limited to memory internal to video encoder 200, unless specifically described as such, or memory external to video encoder 200, unless specifically described as such. Rather, reference to video data memory 230 is to be understood as a reference memory that stores video data that video encoder 200 receives for encoding (eg, video data for the current block to be encoded). Memory 106 in FIG. 1 may also provide temporary storage of output data from various modules of the video encoder 200.

[0141] The various modules in FIG. 14 are illustrated to aid in understanding the operations performed by the video encoder 200. The modules may be implemented as fixed function circuits, programmable circuits, or a combination of both. Fixed function charts refer to circuits that provide certain functionality and are preconfigured for operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, the programmable circuits may execute software or firmware that causes the programmable circuits to operate in a manner determined by the instructions of the software or firmware. Fixed function circuits can execute program instructions (for example, accept parameters or output parameters), but the types of operations that fixed function circuits perform are usually immutable. In some examples, one or more modules may be separate circuit blocks (fixed functional blocks or programmable blocks), and in some examples, one or more modules may be integrated circuits.

[0142] The video encoder 200 may include arithmetic logic units (ALUs), elementary functional units (EFUs), digital circuits, analog circuits, and/or programmable cores formed by programmable circuits. In examples where operations of video encoder 200 are performed using software executable by programmable circuitry, memory 106 (FIG. 1) may store software object code that video encoder 200 receives and executes, or other memory in video encoder 200 (not shown) may store such instructions.

[0143] The video data memory 230 is configured to store received video data. The video encoder 200 may extract an image of the video data from the video data memory 230 and transmit the video data to the residual generation unit 204 and the mode selection unit 202 . The video data in the video data memory 230 may be raw video data to be encoded.

[0144] The mode selection module 202 includes a motion estimation module 222, a motion compensation module 224, and an intra-prediction module 226. The mode selector 202 may include additional functional modules for performing video prediction in accordance with other prediction modes. By way of example, mode select module 202 may include a palette module, an intrablock copy module (which may be part of motion estimation module 222 and/or motion compensation module 224), an affine module, a linear model (LM) module, or the like.

[0145] Mode selection module 202 typically coordinates multiple encoding passes to test combinations of encoding parameters and the resulting rate-distortion values for such combinations. The coding parameters may include splitting the CTU into CUs, prediction modes for the CUs, transform types for the CU residual data, quantization parameters for the CU residual data, and so on. The mode selector 202 may ultimately select a coding parameter combination having rate-distortion values that are better than the other tested combinations.

[0146] Video encoder 200 may partition an image retrieved from video data memory 230 into a sequence of CTUs and encapsulate one or more CTUs within a slice. The mode selector 202 may partition the CTUs of the image according to a tree structure such as the QTBT structure or the HEVC quad tree structure described above. As described above, video encoder 200 may generate one or more CUs from a split of CTUs according to a tree structure. Such a CU may also be referred to generically as a “video block” or “block”.

[0147] In general, mode selection module 202 also controls its components (e.g., motion estimation module 222, motion compensation module 224, and intra-prediction module 226) to generate a prediction block for the current block (e.g., the current CU or, in HEVC, the overlapping portion of the PU and TU). For inter-prediction of the current block, motion estimation module 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (eg, one or more of the previously encoded pictures stored in DPB 218). In particular, the motion estimation module 222 may calculate a value representing how similar the potential reference block is to the current block, for example, according to the sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), root mean square differences (MSD), or the like. Motion estimation module 222 can typically perform these calculations using the sample-to-sample differences between the current block and the reference block under consideration. Motion estimation module 222 may identify the reference block having the smallest value obtained from these calculations, indicating the reference block that most closely matches the current block.

[0148] The motion estimation module 222 may generate one or more motion vectors (MVs) that define the positions of the reference blocks in the reference pictures relative to the position of the current block in the current picture. Motion estimation module 222 may then provide motion vectors to motion compensation module 224 . For example, for unidirectional inter prediction, motion estimation module 222 may provide one motion vector, while for bidirectional inter prediction, motion estimation module 222 may provide two motion vectors. Motion compensation module 224 may then generate a prediction block using the motion vectors. For example, motion compensation module 224 may extract reference block data using a motion vector. As another example, if the motion vector has fractional sampling precision, motion compensation module 224 may interpolate values for the prediction block in accordance with one or more interpolation filters. In addition, for bidirectional inter-prediction, motion compensation module 224 may extract data for two reference blocks identified by respective motion vectors and combine the extracted data, for example, by sample-to-sample averaging or weighted averaging.

[0149] In accordance with the techniques of the present disclosure, decoded picture buffer 218 may include one or more history MVP buffers for CTU lines. That is, each CTU row can be allocated its own MVP buffer, or one MVP buffer can be used for multiple CTU rows. In any case, the video encoder 200 may flush the MVP buffer for the CTU line at the start of decoding the CTU line video data. Motion compensation module 224 or another module of video encoder 200 may be configured to store only unique motion vectors in the MVP buffer. As discussed above, motion compensation module 224 or another module of video encoder 200 may be configured to use a FIFO rule to control motion information stored in the MVP buffer such that when a motion vector is added to the MVP buffer, if the MVP buffer is full, motion compensation module 224 may remove the earliest added motion vector from the MVP buffer. In some examples, video encoder 200 may maintain different corresponding MVP buffers for each of a plurality of motion models, such as, for example, affine motion model, intra-block copy mode motion information, local illumination compensation motion information, sub-block MVP, and temporal motion prediction.

[0150] As another example, for intra-prediction or intra-prediction coding, intra-prediction module 226 may generate a prediction block from samples adjacent to the current block. For example, for directional modes, intra-prediction module 226 can typically mathematically combine adjacent sample values and populate the current block with these computed values in a particular direction to form a prediction block. As another example, for the DC mode, intra-prediction module 226 may compute an average of adjacent samples for the current block and generate a prediction block to include that average for each prediction block sample.

[0151] The mode selection module 202 provides a prediction block to the residual generation module 204 . Residual generation module 204 receives a raw, uncoded version of the current block from video data memory 230 and a prediction block from mode selection module 202 . The residual generation module 204 calculates a sample-to-sample difference between the current block and the prediction block. The resulting sample-to-sample differences determine the residual block for the current block. In some examples, residual generation module 204 may also determine differences between sample values in the residual block to generate a residual block using residual differential pulse code modulation (RDPCM). In some examples, the remainder generator 204 may be formed using one or more subtraction circuits that perform binary subtraction.

[0152] In examples where mode selector 202 partitions a CU into PUs, each PU may be associated with a luminance prediction unit and corresponding chrominance prediction units. Video encoder 200 and video decoder 300 may support PUs having different sizes. As mentioned above, the CU size may refer to the size of the luminance coding unit of the CU, and the size of the PU may refer to the size of the luminance prediction unit of the PU. Assuming a particular CU size is 2Nx2N, video encoder 200 may support 2Nx2N or NxN PU sizes for intra-prediction and symmetrical PU sizes of 2Nx2N, 2NxN, Nx2N, NxN or similar for inter-prediction. Video encoder 200 and video decoder 300 may also support asymmetric partitioning for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-prediction.

[0153] In examples where the mode selector does not further partition the CU into PUs, each CU may be associated with a luminance coding block and corresponding chrominance coding blocks. As stated above, the size of the CU may refer to the size of the luminance coding unit of the CU. Video encoder 200 and video decoder 120 may support CU sizes of 2N×2N, 2N×N, or N×2N.

[0154] For other video coding methods, such as intra-block copy mode coding, affine mode coding, and linear model (LM) mode coding, as a few examples, the mode selection module 202, through the respective modules associated with the coding methods, generates a prediction block for the current block being coded. In some examples, such as palette mode coding, mode selector 202 may not generate a prediction block, but instead generate syntax elements indicating how the block is reconstructed based on the selected palette. In such modes, mode selector 202 may provide these syntax elements to entropy encoding module 220 for encoding.

[0155] As described above, the residual generation unit 204 receives video data for the current block and the corresponding prediction block. Then, the residual generation module 204 generates a residual block for the current block. To generate a residual block, residual generation module 204 calculates the sample-to-sample differences between the prediction block and the current block.

[0156] The transform processing module 206 applies one or more transforms to the residual block to generate a transform coefficient block (referred to herein as a “transform coefficient block”). Transform processing module 206 may apply various transforms to the residual block to generate a block of transform coefficients. For example, transform processing module 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve (KLT) transform, or a conceptually similar transform to the residual block. In some examples, the transform processing module 206 may perform multiple transforms to the residual block, such as a primary transform and a secondary transform such as a rotation transform. In some examples, transform processing module 206 does not apply transforms to the residual block.

[0157] Quantization unit 208 may quantize the transform coefficients in the transform coefficient block to generate a block of quantized transform coefficients. Quantization module 208 may quantize the transform coefficients of a block of transform coefficients in accordance with a quantization parameter (QP) value associated with the current block. Video encoder 200 (eg, via mode selector 202) may adjust the amount of quantization applied to transform coefficient blocks associated with the current block by adjusting the QP value associated with the CU. The quantization may introduce loss of information, and thus the quantized transform coefficients may have lower accuracy than the original transform coefficients generated by the transform processing unit 206 .

[0158] The inverse quantization unit 210 and the inverse transform processing unit 212 may apply inverse quantization and inverse transforms to the block of quantized transform coefficients, respectively, to recover the residual block from the block of transform coefficients. Restoration module 214 may generate a reconstructed block corresponding to the current block (although possibly with some degree of distortion) based on the reconstructed residual block and the prediction block generated by mode selection module 202 . For example, reconstruction module 214 may add samples of the reconstructed residual block to corresponding samples from the prediction block generated by mode selection module 202 to form a reconstructed block.

[0159] The filtering module 216 may perform one or more filtering operations on the recovered blocks. For example, filter module 216 may perform deblocking operations to reduce deblocking artifacts along the edges of the CU. The operations of filter module 216 may be omitted in some examples.

[0160] The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in cases where the operations of the filter module 216 are not required, the reconstruction module 214 may store the reconstructed blocks in the DPB 218. In examples where the operations of the filter module 216 are required, the filter module 216 may store the filtered reconstructed blocks in the DPB 218. The motion estimation module 222 and the motion compensation module 224 may extract the reference picture from DPB 218 formed from the reconstructed (and potentially filtered) blocks to inter-predict blocks of subsequent coded pictures. In addition, the intra-prediction module 226 may use the reconstructed blocks in the DPB 218 of the current picture to intra-predict other blocks in the current picture.

[0161] In general, entropy encoding module 220 may entropy encode syntax elements received from other functional components of video encoder 200. For example, entropy encoding module 220 may entropy encode blocks of quantized transform coefficients from quantization module 208 . As another example, entropy encoding unit 220 may entropy encode prediction syntax elements (eg, motion information for inter prediction or intra mode information for intra prediction) from mode selection unit 202 . Entropy encoding module 220 may perform one or more entropy encoding operations on syntax elements, which are another example of video data, to generate entropy encoded data. For example, entropy encoding module 220 may perform a context adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable to variable length (B2B) coding operation, a syntax-based context adaptive binary arithmetic coding (SBAC) operation, a probability interval partitioning (PIPE) entropy coding operation, an exponential Golomb coding operation, or another type of data entropy encoding operation. In some examples, entropy encoding module 220 may operate in bypass mode where the syntax elements are not entropy encoded.

[0162] The video encoder 200 may output a bitstream that includes the entropy encoded syntax elements needed to reconstruct slice or picture blocks. In particular, entropy encoding unit 220 may output a bit stream.

[0163] The operations described above are described with respect to a block. Such description is to be understood as operations for the luminance coding block and/or chrominance coding blocks. As described above, in some examples, the luminance and chrominance coding blocks are the luminance and chrominance components of the CU. In some examples, the luma coding block and the chrominance coding blocks are the luma and chrominance components of the PU.

[0164] In some examples, operations performed on a luminance coding block do not need to be repeated for chrominance coding blocks. As one example, the operations for identifying a motion vector (MV) and a reference picture for a luminance coding block need not be repeated for identifying an MV and a reference picture for chrominance blocks. Rather, the MV for the luminance coding block may be scaled to determine the MV for the chrominance blocks, and the reference picture may be the same. As another example, the intra-prediction process may be the same for the luminance coding block and the chrominance coding blocks.

[0165] The video encoder 200 is an example of an apparatus configured to encode video data, including a memory configured to store video data and one or more processing units implemented in circuits and configured to store motion information for a first row of coding tree units (CTUs) in a first history motion vector predictor (MVP) memory buffer and store motion information for a second CTU row in a second history MVP memory buffer, wherein the second CTU row is different from the first CTU row.

[0166] The video encoder 200 also represents an example of an apparatus configured to encode video data, including a memory configured to store video data, and one or more processing units implemented in circuits and configured to store encoded motion information in a history motion vector predictor (MVP) buffer, store another type of motion information in the history MVP buffer, and encode motion information of a video data block using the motion information of the history MVP buffer.

[0167] The video encoder 200 also represents an example of an apparatus configured to encode video data, including a memory configured to store video data and one or more processing modules implemented in circuits and configured to store a plurality of different types of motion information in respective different history motion vector predictor (MVP) buffers.

[0168] FIG. 15 is a block diagram illustrating an exemplary video decoder 300 that may perform the techniques of the present disclosure. Fig. 15 is provided for purposes of explanation and does not limit the methods broadly illustrated and described in this disclosure. For purposes of explanation, the present disclosure describes a video decoder 300 in accordance with JEM and HEVC techniques. However, the methods of the present disclosure may be performed by video coding devices that are configured in accordance with other video coding standards.

[0169] In the example of FIG. 15, the video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filtering unit 312, and a decoded picture buffer (DPB) 314. Any or all of the CPB memory 320, entropy decoding module 302, prediction processing module 304, inverse quantization module 306, inverse transform processing module 308, reconstruction module 310, filtering module 312, and DPB 314 may be implemented in one or more processors or processing circuits. In addition, video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

[0170] The prediction processing module 304 includes a motion compensation module 316 and an intra prediction module 318 . Prediction processing unit 304 may include adders for performing prediction in accordance with other prediction modes. As examples, prediction processing module 304 may include a palette module, an intra-block copy module (which may form part of motion compensation module 316), an affine module, a linear model (LM) module, or the like. In other examples, video decoder 300 may include more, less, or other functional components.

[0171] The CPB memory 320 may store video data, such as an encoded video bitstream, for decoding by components of the video decoder 300. The video data stored in the CPB memory 320 may be obtained, for example, from a computer-readable medium 110 (FIG. 1). CPB memory 320 may include a CPB that stores encoded video data (eg, syntax elements) from an encoded video bitstream. In addition, the CPB memory 320 may store video data other than the syntax elements of the encoded picture, such as temporary data representing outputs from various modules of the video decoder 300. DPB 314 typically stores decoded pictures that video decoder 300 can output and/or use as video reference data when decoding subsequent data or pictures of the encoded video bitstream. CPB memory 320 and DPB 314 may be formed by any of a variety of memory devices such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The CPB memory 320 and DPB 314 may be provided by the same memory device or by separate memory devices. In various examples, CPB memory 320 may be on-chip with other components of video decoder 300, or off-chip with respect to these components.

[0172] Additionally or alternatively, in some examples, video decoder 300 may retrieve encoded video data from memory 120 (FIG. 1). That is, memory 120 may store data as discussed above for CPB memory 320. Similarly, memory 120 may store instructions to be executed by video decoder 300 when some or all of the functionality of video decoder 300 is implemented in software to be executed by video decoder 300 processing circuitry.

[0173] The various modules shown in FIG. 15 are illustrated to aid in understanding the operations performed by the video decoder 300. The modules may be implemented as fixed function circuits, programmable circuits, or combinations of both. Like FIG. 14, fixed functional circuits refer to circuits that provide certain functionality and are pre-configured for operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality in the operations that can be performed. For example, the programmable circuits may execute software or firmware that causes the programmable circuits to operate in a manner determined by the instructions of the software or firmware. Fixed function circuits can execute program instructions (for example, accept parameters or output parameters), but the types of operations that fixed function circuits perform are usually immutable. In some examples, one or more modules may be separate circuit blocks (fixed functional blocks or programmable blocks), and in some examples, one or more modules may be integrated circuits.

[0174] Video decoder 300 may include ALUs, EFUs, digital circuits, analog circuits, and/or programmable cores formed by programmable circuits. In instances where the operations of the video decoder 300 are performed by software running on programmable circuits, the on-chip (on-chip) or off-chip memory may store instructions (eg, object code) of the software that the video decoder 300 receives and executes.

[0175] The entropy decoding module 302 may receive the encoded video data from the CPB and entropy decode the video data to reproduce the syntax elements. Prediction processing module 304, inverse quantization module 306, inverse transform processing module 308, reconstruction module 310, and filter module 312 may generate decoded video data based on syntax elements extracted from the bitstream.

[0176] In general, video decoder 300 reconstructs an image on a block-by-block basis. The video decoder 300 may perform the recovery operation on each block individually (where the block that is being recovered, ie, decoded, at the current time may be referred to as the “current block”).

[0177] Entropy decoding module 302 can entropy decode syntax elements defining quantized transform coefficients of a block of quantized transform coefficients, as well as transform information such as quantization parameter (QP) indication(s) and/or transform mode. Inverse quantization module 306 may use the QP associated with the block of quantized transform coefficients to determine the degree of quantization and, similarly, the degree of inverse quantization to be applied by inverse quantization module 306 . Inverse quantizer 306 may, for example, perform a bitwise left shift operation to inverse quantize the quantized transform coefficients. Here, inverse quantizer 306 may generate a transform coefficient block including transform coefficients.

[0178] After inverse quantizer 306 has generated a transform coefficient block, inverse transform processing module 308 may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, inverse transform processing unit 308 may apply an inverse DCT, inverse integer transform, inverse Karhunen-Loeve (KLT), inverse rotation, inverse directional transform, or other inverse transform to the coefficient block.

[0179] In addition, the prediction processing unit 304 generates a prediction block in accordance with the prediction information syntax elements that have been entropy decoded by the entropy decoding unit 302 . For example, if the prediction information syntax elements indicate that the current block is inter-predicted, motion compensation module 316 may generate a prediction block. In this case, the prediction information syntax elements may indicate a reference picture in the DPB 314 from which the reference block should be retrieved, as well as a motion vector identifying the location of the reference block in the reference picture relative to the location of the current block in the current picture. Motion compensation module 316 may, in principle, perform the inter-prediction process in a manner that is essentially the same as that described with respect to motion compensation module 224 (FIG. 14).

[0180] In accordance with the techniques of the present disclosure, decoded picture buffer 314 may include one or more history MVP buffers for CTU lines. That is, each CTU row can be allocated its own MVP buffer, or one MVP buffer can be used for multiple CTU rows. In any case, the video decoder 300 may flush the MVP buffer for the CTU line at the start of encoding the video data of the CTU line. Motion compensation module 316 or another module of video decoder 300 may be configured to store only unique motion vectors in the MVP buffer. As discussed above, the motion compensation module 316 or another module of the video decoder 300 may be configured to use a FIFO rule to control the motion information stored in the MVP buffer such that when a motion vector is added to the MVP buffer, if the MVP buffer is full, the motion compensation module 316 may remove the earliest added motion vector from the MVP buffer. In some examples, video decoder 300 may maintain different respective MVP buffers for each of a plurality of motion models, such as, for example, affine motion model, intrablock copy mode motion information, local illumination compensation motion information, subblock MVP, and temporal motion prediction.

[0181] As another example, if the prediction syntax elements indicate that the current block is intra-predicted, intra-prediction module 318 may generate a prediction block in accordance with the intra-prediction mode indicated by the prediction syntax elements. Again, intra-prediction module 318 can, in principle, perform the intra-prediction process in a manner that is substantially the same as described with respect to intra-prediction module 226 (FIG. 14). Intra-prediction module 318 may extract data from adjacent samples for the current block from DPB 314.

[0182] Recovery module 310 may recover the current block using the prediction block and the residual block. For example, reconstruction module 310 may add residual block samples to corresponding prediction block samples to restore the current block.

[0183] Filtering module 312 may perform one or more filtering operations on the recovered blocks. For example, filter module 312 may perform deblocking operations to reduce deblocking artifacts along the edges of the reconstructed blocks. The operations of filter module 312 are not necessarily performed in all examples.

[0184] The video decoder 300 may store the reconstructed blocks in the DPB 314. For example, in cases where operations of the filter module 312 are not required, the recovery module 310 may store the reconstructed blocks in the DPB 314. In cases where operations of the filter module 312 are required, the filter module 312 may store the filtered reconstructed blocks in the DPB 314. As discussed above, the DPB 314 may provide reference information such as samples the current image for intra-prediction and the previously decoded images for subsequent motion compensation, to the prediction processing module 304 . In addition, video decoder 300 may output decoded pictures from DPB 314 for subsequent presentation on a display device, such as display device 118 in FIG. 1.

[0185] The video decoder 300 is an example of an apparatus configured to decode video data, including a memory configured to store video data, and one or more processing units implemented in circuitry and configured to store motion information for a first row of coding tree units (CTU) (which may be processed by a first sub-process of the video encoding process) in a first history motion vector predictor (MVP) memory buffer, and store motion information for a second row of CTUs (which may be processed by a second video encoding process subprocess) in the second history MVP memory buffer, with the second CTU row different from the first CTU row. The second sub-process may be different from the first sub-process.

[0186] The video decoder 300 also represents an example of an apparatus configured to decode video data, including a memory configured to store video data, and one or more processing units implemented in circuits and configured to store encoded motion information in a history motion vector predictor (MVP) buffer, store another type of motion information in the history MVP buffer, and encode motion information for a block of video data using the motion information of the history MVP buffer.

[0187] The video decoder 300 also represents an example of an apparatus configured to decode video data, including a memory configured to store video data and one or more processing modules implemented in circuits and configured to store a plurality of different kinds of motion information in respective history motion vector predictor (MVP) buffers.

[0188] FIG. 16 is a flowchart illustrating an example of a method for encoding a current block in accordance with the methods of the present disclosure. The current block may contain the current CU. While the description has been made with respect to the video encoder 200 (FIGS. 1 and 14), it should be understood that other devices may be configured to perform a method similar to that shown in FIGS. 16.

[0189] In this example, the video encoder 200 initially predicts the current block using motion information (350). For example, video encoder 200 may generate a prediction block for the current block using the motion information. Video encoder 200 may then calculate a residual block for the current block (352). To calculate the residual block, video encoder 200 may calculate the difference between the original, uncoded block and the prediction block for the current block. Video encoder 200 may then transform and quantize the coefficients of the residual block (354). Video encoder 200 may then scan the quantized transform coefficients of the residual block (356). During or after scanning, video encoder 200 may entropy encode coefficients and motion information (358) using the techniques of the present disclosure. Video encoder 200 may encode the coefficients using CAVLC or CABAC.

[0190] The video encoder 200 may generate a motion information candidate list including, for example, HMVP candidates in accordance with any or all of the methods of the present disclosure, select a candidate index representing a predictor for the block motion information, and entropy encode the candidate index. In accordance with the techniques of the present disclosure, the video encoder 200 may flush the MVP buffer before using the MVP buffer to store the motion information of the corresponding CTU line. In some examples, each CTU row may have its own MVP buffer, or one MVP buffer may be used for multiple CTU rows. In addition, the video encoder 200 may store multiple types of motion information in MVP buffers, such as the same buffer or different buffers of respective motion models. The video encoder 200 may encode the motion information of the current block using motion vector predictors selected from the MVP buffer data. Video encoder 200 may then output entropy encoded block data (360), for example including data for coefficients and motion information such as a candidate index.

[0191] Thus, the method according to FIG. 16 shows an example of a method including storing motion information for a first row of coding tree units (CTU) image in a first history motion vector predictor (MVP) memory buffer; flushing the second MVP history buffer; and after flushing the second history MVP buffer, storing motion information for the second CTU row image in the second history MVP buffer, the second CTU row being different from the first CTU row.

[0192] The method of FIG. 16 also shows an example of a method including storing motion information in a history motion vector predictor (MVP) buffer; storing a different type of motion information in the history MVP buffer; and encoding the motion information of the video data block using the motion information of the history buffer MVP.

[0193] The method of FIG. 16 also shows an example of a method including storing a plurality of different types of motion information in respective different history motion vector predictor (MVP) buffers.

[0194] FIG. 17 is a flowchart illustrating an example of a method for decoding a current block of video data in accordance with the techniques of the present disclosure. The current block may contain the current CU. While the description has been made with respect to the video decoder 300 (FIGS. 1 and 15), it should be understood that other devices may be configured to perform a method similar to that shown in FIGS. 17.

[0195] The video decoder 300 may receive entropy coded data for the current block, such as entropy coded prediction information and entropy coded data for residual block coefficients corresponding to the current block (370). As discussed above, the entropy coded prediction information may include, for example, a candidate list candidate index, which may include HMVP candidates in accordance with the techniques of the present disclosure. The video decoder 300 may entropy decode the entropy encoded data to determine prediction information for the current block and recover coefficients of the residual block (372). The video decoder 300 may predict the current block (374), for example, using the inter-prediction mode as indicated by the prediction information for the current block to compute a prediction block for the current block.

[0196] Specifically, video decoder 300 may generate a candidate list including HMVP candidates as described above, and then determine a candidate from the candidate list to use as a motion vector predictor for the current block using the decoded candidate index. In accordance with the techniques of the present disclosure, the video decoder 300 may flush the MVP buffer before using the MVP buffer to store motion information of the corresponding CTU line. In some examples, each CTU row may have its own MVP buffer, or one MVP buffer may be used for multiple CTU rows. In addition, the video decoder 300 may store multiple types of motion information in MVP buffers, such as the same buffer or different buffers corresponding to motion models. Video decoder 300 may select using motion vector predictors from MVP buffer data.

[0197] The video decoder 300 may then reconstruct the motion vector for the current block using the motion vector predictor, and then predict the current block using the motion vector to generate a prediction block. Video decoder 300 may then backscan the reconstructed coefficients (376) to create a block of quantized transform coefficients. Video decoder 300 may then inversely quantize and inversely transform the coefficients to obtain a residual block (378). Video decoder 300 may eventually decode the current block by combining the prediction block and the residual block (380).

[0198] Thus, the method of FIG. 17 shows an example of a method including storing motion information for a first row of coding tree units (CTUs) of a picture in a first history motion vector predictor (MVP) memory buffer; flushing the second MVP history memory buffer; and after flushing the second history MVP buffer, storing the motion information for the second image CTU line in the second history MVP buffer, the second CTU line being different from the first CTU line.

[0199] The method of FIG. 17 also shows an example of a method including storing motion information in a history predictor motion vector (MVP) buffer; storing a different type of motion information in the history MVP buffer; and encoding the motion information of the video data block using the motion information of the history buffer MVP.

[0200] The method of FIG. 17 also shows an example of a method including storing a plurality of different types of motion information in respective history motion vector predictor (MVP) buffers.

[0201] In FIG. 18 is a flowchart illustrating an exemplary method for encoding (encoding or decoding) video data in accordance with the techniques of the present disclosure. For example, the method according to FIG. 18 may be performed during step 350 in FIG. 16 or step 374 in FIG. 17. By way of example and explanation, the method of FIG. 18 is explained in relation to video decoder 300, although video encoder 200 may also perform this or a similar method.

[0202] The video decoder 300 may encode blocks of the first row CTU of the picture (390), for example, using intra- or inter-prediction. The video decoder 300 stores the motion information of the first row of image coding tree units (CTUs) in a first buffer (392), for example, DPB 314. The video decoder 300 may use the first buffer's motion information to encode the motion information used during inter-prediction encoding. In some examples, the first sub-process of the video encoding process performed by the video decoder 300 may encode the first line of the CTU.

[0203] The video decoder 300 may also flush the second buffer (394), such as DPB 314. The second buffer may be the same as the first buffer or a different buffer. Video decoder 300 may also encode blocks of the second line CTU (396). The video decoder 300 may store the motion information of the second line of the CTU in a second buffer (398). In some examples, a second subprocess of the video encoding process performed by video decoder 300 may encode a second CTU string, with the second subprocess different from the first subprocess.

[0204] Thus, the method of FIG. 18 shows an example of a method including storing motion information for a first row of coding tree units (CTUs) of a picture in a first history motion vector predictor (MVP) memory buffer; flushing the second MVP history memory buffer; and after flushing the second history MVP buffer, storing the motion information for the second image CTU line in the second history MVP buffer, the second CTU line being different from the first CTU line.

[0205] It should be borne in mind that, depending on the example, some actions or events of any of the methods described here may be performed in a different sequence, may be added, combined, or excluded altogether (for example, not all of the described actions or events are necessary for the practical implementation of the methods). In addition, in some examples, actions or events may be performed simultaneously, such as through multi-threading, interrupt handling, or multiple processors, rather than sequentially.

[0206] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware processing unit. Computer-readable media may include computer-readable media that corresponds to a tangible medium, such as a storage medium, or a communication medium, including any medium that enables the transfer of a computer program from one place to another, for example, in accordance with a communication protocol. Thus, computer-readable media can typically correspond to (1) tangible computer-readable storage media that is non-volatile, or (2) a communication medium such as a signal or carrier wave. Storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and/or data structures for implementing the methods described in this disclosure. The computer program product may include a computer-readable medium.

[0207] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. In addition, any connection is properly defined as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of media. However, it should be understood that computer-readable storage media and storage media do not include connections, carrier waves, signals, or other temporary media, but are instead directed to non-temporal, tangible media. The magnetic disc (disk) and the optical disc (disc) as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disc, and Blu-ray disc, where magnetic discs usually reproduce data magnetically while optical discs reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0208] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Accordingly, the terms "processor" and "processing circuit" as used herein may refer to any of the above structures or any other structure suitable for implementing the methods described here. In addition, in some aspects, the functionality described herein may be provided in dedicated hardware and/or software modules configured for encoding and decoding or embedded in a combined codec. In addition, the methods may be fully implemented in one or more circuits or logic elements.

[0209] The techniques of the present disclosure may be implemented in a wide variety of devices or devices, including a wireless telephone, an integrated circuit (IC), or a stack of integrated circuits (eg, a chipset). Various components, modules, or blocks are described in this disclosure with emphasis on the functional aspects of devices configured to perform the disclosed methods, but not necessarily requiring implementation by different hardware modules. Rather, as described above, various modules may be combined into a codec hardware module or provided with a set of cooperating hardware modules, including one or more processors, as described above, in association with suitable software and/or firmware.

[0210] Various examples have been described. These and other examples are within the scope of the following claims.

Claims (143)

1. A method for encoding video data, comprising the steps of: encoding motion information for a first row of coding tree units (CTUs) of the picture, wherein the motion information for the first CTU row includes first one or more motion vectors for the first CTU row; generating first one or more prediction blocks for the first one or more coding units (CUs) of the first CTU row using first reference blocks identified by motion information for the first CTU row; encoding the first one or more residual blocks for the first one or more CUs, the first one or more residual blocks representing respective differences between the first one or more prediction blocks and the first one or more CUs; storing motion information for the first CTU row of said image in a first history-based motion vector predictor (history MVP) buffer in memory, including storing the first one or more motion vectors for the first CTU row in the first history MVP buffer for use as first motion vector predictors to predict the first one or more other motion vectors of said image; flushing the second history MVP buffer in memory; encoding motion information for a second CTU row of said picture, the motion information for the second CTU row including second one or more motion vectors for the second CTU row; generating second one or more prediction blocks for the second one or more CUs of the second CTU row using second reference blocks identified by traffic information for the second CTU row; encoding second one or more residual blocks for the second one or more CUs, wherein the second one or more residual blocks represent respective differences between the second one or more prediction blocks and the second one or more CUs; And after resetting the second history MVP buffer, motion information for the second CTU line of the said image is stored in the second history MVP buffer, the second CTU line being different from the first CTU line, including storing the second one or more motion vectors for the second CTU line for use as second motion vector predictors to predict the second one or more other motion vectors of the said image. 2. The method according to claim 1, wherein the second MVP history buffer differs from the first MVP history buffer, wherein said storing the motion information for the first CTU row comprises storing, by means of the first subprocess of the video encoding process, motion information for the first CTU row, and said storing the motion information for the second CTU row comprises storing, by means of the second subprocess of the video encoding process, motion information for the second CTU row, the first subprocess being different from the second subprocess. 3. The method of claim 1, wherein the second history MVP buffer is different from the first history MVP buffer, wherein in said flushing of the second history MVP buffer, the second history MVP buffer is flushed in response to processing the beginning of the second CTU row during parallel wavefront processing. 4. The method according to claim 1, in which in said storing the motion information for the first row, the CTUs store only the motion information unique within the first history MVP buffer into the first history MVP buffer, and in said storing the motion information for the second row, the CTUs only store the motion information unique within the second history MVP buffer into the second history MVP buffer. 5. The method of claim 1, further comprising the steps of: when the first history MVP buffer becomes full, deleting one or more of the earliest inputted motion vectors from the first history MVP buffer in accordance with a first-come-first-served (FIFO) rule; And when the second history MVP buffer becomes full, one or more of the earliest input motion vectors are removed from the second history MVP buffer in accordance with the FIFO rule. 6. The method according to claim 1, in which when said storing motion information for the first row, the CTUs store motion information for each of a plurality of different types of motion patterns in respective different history MVP buffers of the first plurality of history MVP buffers, including the first history MVP buffer; And in said storing motion information for the second row, the CTUs store motion information for each of the plurality of different types of motion patterns in respective different history MVP buffers of the second plurality of history MVP buffers including the second history MVP buffer. 7. The method of claim 6, wherein said plurality of different types of motion models comprises one or more of an affine motion model, intrablock copy mode motion information, local illumination compensation motion information, subblock motion vector prediction (MVP), and temporal motion prediction. 8. The method of claim 1, further comprising generating a synthetic MVP from two or more MVPs in one or more history MVP buffers. 9. The method of claim 8, wherein said two or more MVPs have different types of motion information comprising two or more of encoded motion information, affine motion model motion information, intra-block copy mode motion information, local lighting compensation motion information, sub - block MVP motion information, temporal motion predictor motion information, synthetic motion vector information based on the MVP motion information, spatial MVP motion information, and temporal MVP motion information. 10. The method according to claim 1, in which said storing the motion information for the first row of the CTU further comprises pre-filling the first history MVP buffer with the first predetermined motion information, the first predetermined motion information comprising a first null motion vector with a first reference frame index and a first inter-prediction direction, and said storing the motion information for the second row of the CTU further comprises pre-filling the second history MVP buffer with the second predetermined motion information, the second predetermined motion information comprising a second null motion vector with a second reference frame index and a second inter-prediction direction. 11. The method of claim 1, wherein the picture including the first CTU line and the second CTU line is the first temporal layer, wherein said storing the motion information for the first row of the CTU further comprises pre-filling the first history buffer MVP with the first motion information from the first one or more coded pictures in the first time layer or one or more time layers lower than the first time layer, and said storing the motion information for the second row of the CTU further comprises pre-filling the second history buffer MVP with the second motion information from the second one or more encoded pictures in the first time layer or one or more time layers lower than the first time layer, the method further comprising scaling at least one of the first motion information and the second motion information according to the time differences. 12. The method of claim 1, further comprising the steps of: initializing the first history buffer MVP with a first null motion vector having a first reference frame index and a first inter-prediction direction; And initializing the second history buffer MVP with a second zero motion vector having a second reference frame index and a second inter-prediction direction. 13. The method of claim 1, wherein the second CTU row is immediately below the first CTU row, the method further comprising initializing or modifying the second history MVP buffer using the first history MVP buffer. 14. The method of claim 1, further comprising the steps of: decoding the coding tree units (CUs) of the coding tree units (CTUs) of the first row of the CTUs using the first history MVP buffer; And decoding the coding tree units (CUs) of the coding tree units (CTUs) of the second row of CTUs using the second history buffer MVP. 15. The method of claim 1, further comprising the steps of: encoding coding tree units (CUs) of coding tree units (CTUs) of the first row of CTUs using the first history buffer MVP; And encoding coding tree units (CUs) of coding tree units (CTUs) of the second row of CTUs using the second history buffer MVP. 16. The method of claim 1, further comprising the steps of: encoding first motion information of coding tree units (CUs) of coding tree units (CTUs) of the first CTU row using the first history MVP buffer, the encoding of the first motion information comprising encoding the first motion information using an advanced motion vector prediction (AMVP) mode, a merge mode, or an affine mode; And encoding second motion information of coding tree units (CUs) of coding tree units (CTUs) of the second row of CTUs using a second history MVP buffer, the encoding of the second motion information comprising encoding the second motion information using an AMVP mode, a merge mode, or an affine mode. 17. The method of claim 1, further comprising the steps of: adding one or more motion vectors to the first one or more motion vectors of the first unidirectional motion information in the first history buffer MVP to convert the first unidirectional motion information to the first bidirectional motion information, and adding one or more motion vectors to the second one or more motion vectors of the second unidirectional motion information in the second history buffer MVP to convert the second unidirectional motion information to the second bidirectional motion information. 18. The method according to claim 1, in which in said storing the motion information for the first row, the CTUs store only those motion vectors that differ from other motion vectors in the first history MVP buffer by a threshold value, in the first history MVP buffer, and in said storing the motion information for the second row, the CTUs only store those motion vectors that differ from other motion vectors in the second history MVP buffer by a threshold value, in the second history MVP buffer. 19. The method according to claim 1, in which in said storing the motion information for the first row, the CTUs store data indicating whether motion vectors of the motion information for the first row of the CTU are associated with merge-coded coding units (CUs) or AMVP-coded CUs, and in said storing the motion information for the second row, the CTUs store data indicating whether the motion vectors of the motion information for the second row of the CTUs are associated with merge-coded CUs or AMVP-coded CUs. 20. The method of claim 1, wherein the first line of the CTU has a length less than the full width of the image. 21. The method of claim 1, wherein said storing motion information for the first CTU row stores motion information for a number of CUs in the first CTU row that is less than the total number of CUs in the first CTU row. 22. Device for encoding video data, containing: a memory adapted to store video data; And one or more processing modules implemented in circuits and configured to: encode motion information for a first row of coding tree units (CTUs) of the picture, wherein the motion information for the first CTU row includes first one or more motion vectors for the first CTU row; generate first one or more prediction blocks for the first one or more coding units (CUs) of the first CTU row using the first reference blocks identified by the motion information for the first CTU row; encode first one or more residual blocks for the first one or more CUs, wherein the first one or more residual blocks represent respective differences between the first one or more prediction blocks and the first one or more CUs; store motion information for the first row of CTUs of said image in a first history-based motion vector predictor (history MVP) buffer in memory, wherein the motion information for the first row of CTUs includes first one or more motion vectors for the first row of CTUs for use as first motion vector predictors to predict the first one or more other motion vectors of said image; flush the second history MVP buffer in memory; encode motion information for a second CTU row of said picture, wherein the motion information for the second CTU row includes second one or more motion vectors for the second CTU row; generate second one or more prediction blocks for the second one or more CUs of the second CTU row using second reference blocks identified by traffic information for the second CTU row; encode second one or more residual blocks for the second one or more CUs, wherein the second one or more residual blocks represent respective differences between the second one or more prediction blocks and the second one or more CUs; And after resetting the second history MVP buffer, store the motion information for the second CTU row of said image in the second history MVP buffer, wherein the second CTU row is different from the first CTU row, wherein the motion information for the second CTU row includes one or more second motion vectors for the second CTU row for use as second motion vector predictors to predict the second one or more other motion vectors of the said image. 23. The apparatus of claim 22, wherein the second history MVP buffer differs from the first history MVP buffer, wherein one or more processing modules execute the first video encoding process sub-process to store motion information for the first CTU row, and one or more processing modules execute the second video encoding process sub-process to store motion information for the second CTU row, wherein the first sub-process is different from the second sub-process. 24. The apparatus of claim 22, wherein the second history MVP buffer is different from the first history MVP buffer, wherein in order to flush the second history MVP buffer, one or more processing modules are configured to flush the second history MVP buffer in response to processing the start of the second CTU row during parallel wavefront processing. 25. The device according to claim 22, in which one or more processing modules are configured to store only motion information unique within the first history MVP buffer in the first history MVP buffer, and the one or more processing modules are configured to store only motion information unique within the second history MVP buffer in the second history MVP buffer. 26. The apparatus of claim 22, wherein the one or more processing modules are further configured to: when the first history MVP buffer becomes full, delete one or more of the earliest input motion vectors from the first history MVP buffer in accordance with a first-in-first-out (FIFO) rule; And when the second history MVP buffer becomes full, delete one or more of the earliest inputted motion vectors from the second history MVP buffer in accordance with the FIFO rule. 27. The device according to claim 22, in which one or more processing modules are configured to store motion information for each of a plurality of different types of motion patterns in respective different history MVP buffers of a first plurality of history MVP buffers including a first history MVP buffer; And the one or more processing modules are configured to store motion information for each of the plurality of different types of motion patterns in respective different history MVP buffers from a second plurality of history MVP buffers including a second history MVP buffer. 28. The apparatus of claim 27, wherein said plurality of different types of motion models comprises one or more of an affine motion model, intrablock copy mode motion information, local illumination compensation motion information, subblock motion vector prediction (MVP), and temporal motion prediction. 29. The apparatus of claim 22, wherein the one or more processing modules are further configured to generate a synthetic MVP from two or more MVPs in one or more history MVP buffers. 30. The apparatus of claim 29, wherein the two or more MVPs have different types of motion information comprising two or more of encoded motion information, affine motion model motion information, intra-block copy mode motion information, local lighting compensation motion information, sub - block MVP motion information, temporal motion predictor motion information, synthetic motion vector information based on MVP motion information, spatial MVP motion information, and temporal MVP motion information. 31. The apparatus of claim 22, further comprising a display device configured to display video data. 32. The apparatus of claim 22, further comprising a camera capable of capturing video data. 33. The apparatus of claim 22, wherein the apparatus comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box. 34. The device of claim 22, wherein the device comprises at least one of: integrated circuit; microprocessor and wireless communication devices. 35. The apparatus of claim 22, wherein the first line of the CTU has a length less than the full width of the image. 36. The apparatus of claim 22, wherein in order to store motion information for the first CTU row, one or more processing units are configured to store motion information for a number of CUs in the first CTU row that is less than the total number of CUs in the first CTU row. 37. A computer-readable storage medium in which instructions are stored that, when executed, instruct the processor: encode motion information for a first row of coding tree units (CTUs) of the picture, wherein the motion information for the first CTU row includes first one or more motion vectors for the first CTU row; generate first one or more prediction blocks for the first one or more coding units (CUs) of the first CTU row using the first reference blocks identified by the motion information for the first CTU row; encode first one or more residual blocks for the first one or more CUs, wherein the first one or more residual blocks represent respective differences between the first one or more prediction blocks and the first one or more CUs; store motion information for the first row of CTUs of said image in a first history-based motion vector predictor (history MVP) buffer in memory, wherein the motion information for the first row of CTUs includes first one or more motion vectors for the first row of CTUs for use as first motion vector predictors to predict the first one or more other motion vectors of said image; flush the second history MVP buffer in memory; encode motion information for a second CTU row of said picture, wherein the motion information for the second CTU row includes second one or more motion vectors for the second CTU row; generate second one or more prediction blocks for the second one or more CUs of the second CTU row using second reference blocks identified by traffic information for the second CTU row; encode second one or more residual blocks for the second one or more CUs, wherein the second one or more residual blocks represent respective differences between the second one or more prediction blocks and the second one or more CUs; And after flushing the second history MVP buffer, store motion information for the second CTU line of the said image in the second history MVP buffer, wherein the second CTU line differs from the first CTU line, while the motion information for the second CTU line includes one or more second motion vectors for the second CTU line for use as second motion vector predictors. 38. The computer-readable storage medium of claim 37, wherein the second history MVP buffer is different from the first history MVP buffer, wherein the instructions that cause the processor to store motion information for the first CTU row comprise instructions that cause the processor to execute a first video encoding process subprocess to store motion information for the first CTU row, and wherein the instructions that cause the processor to store motion information for the second CTU row contain instructions that cause the processor to execute a second video encoding process subprocess to store information movement for the second CTU row, with the first subprocess different from the second subprocess. 39. The computer-readable storage medium of claim 37, wherein the second history MVP buffer is different from the first history MVP buffer, wherein the instructions that cause the processor to flush the second history MVP buffer comprise instructions that cause the processor to flush the second history MVP buffer in response to processing the beginning of the second CTU row during parallel wavefront processing. 40. A computer-readable storage medium according to claim 37, wherein instructions that cause the processor to store motion information for the first row of the CTU contain instructions that cause the processor to store only motion information unique within the first history MVP buffer in the first history MVP buffer, and the instructions that cause the processor to store motion information for the second row of the CTU contain instructions that cause the processor to store only the motion information unique within the second history MVP buffer in the second history MVP buffer. 41. The computer-readable storage medium of claim 37, further comprising instructions that cause the processor to: when the first history MVP buffer becomes full, delete one or more of the earliest input motion vectors from the first history MVP buffer in accordance with a first-in-first-out (FIFO) rule; And when the second history MVP buffer becomes full, delete one or more of the earliest inputted motion vectors from the second history MVP buffer in accordance with the FIFO rule. 42. The computer-readable storage medium of claim 37, wherein the instructions that cause the processor to store motion information for the first row of the CTU comprise instructions that cause the processor to store motion information for each of a plurality of different types of motion patterns in respective different history MVP buffers of the first plurality of history MVP buffers including the first history MVP buffer; And instructions that cause the processor to store motion information for the second row of the CTU comprise instructions that cause the processor to store motion information for each of a plurality of different types of motion patterns in respective different history MVP buffers of a second plurality of history MVP buffers including a second history MVP buffer. 43. The computer-readable storage medium of claim 42, wherein said plurality of different types of motion models comprises one or more of an affine motion model, intra-block copy mode motion information, local illumination compensation motion information, sub-block motion vector prediction (MVP), and temporal motion prediction. 44. The computer-readable storage medium of claim 37, further comprising instructions that cause the processor to generate a synthetic MVP from two or more MVPs in one or more history MVP buffers. 45. The machine-readable storage medium of claim 44, wherein said two or more MVPs have different types of motion information comprising two or more of encoded motion information, affine motion model motion information, intra-block copy mode motion information, local lighting compensation motion information, sub - block MVP motion information, temporal motion predictor motion information, synthetic motion vector information based on MVP motion information, spatial MVP motion information, and temporal MVP motion information. 46. The computer-readable storage medium of claim 37, wherein the first line of the CTU has a length less than the full width of the image. 47. The computer-readable storage medium of claim 37, wherein the instructions that cause the processor to store motion information for the first CTU row comprise instructions that cause the processor to store motion information for a number of CUs in the first CTU row that is less than the total number of CUs in the first CTU row. 48. A device for encoding video data, comprising: a memory adapted to store video data; means for encoding motion information for a first row of coding tree units (CTUs) of the picture, wherein the motion information for the first CTU row includes first one or more motion vectors for the first CTU row; means for generating the first one or more prediction blocks for the first one or more coding units (CUs) of the first CTU row using the first reference blocks identified by the motion information for the first CTU row; means for encoding the first one or more residual blocks for the first one or more CUs, the first one or more residual blocks representing respective differences between the first one or more prediction blocks and the first one or more CUs; means for storing motion information for the first CTU row of said image in a first history-based motion vector predictor (history MVP) buffer in memory, including means for storing the first one or more motion vectors for the first CTU row in the first history MVP buffer for use as first motion vector predictors to predict the first one or more other motion vectors of said image; means for flushing a second history MVP buffer in memory; means for encoding motion information for a second CTU row of said picture, the motion information for the second CTU row including second one or more motion vectors for the second CTU row; means for generating a second one or more prediction blocks for the second one or more CUs of the second CTU row using second reference blocks identified by motion information for the second CTU row; means for encoding second one or more residual blocks for the second one or more CUs, wherein the second one or more residual blocks represent respective differences between the second one or more prediction blocks and the second one or more CUs; And means for storing motion information for the second CTU row of the said image in the second history MVP buffer after resetting the second history MVP buffer, wherein the second CTU row differs from the first CTU row, including means for storing the second one or more motion vectors for the second CTU row in the second history MVP buffer for use as second motion vector predictors to predict the second one or more other motion vectors of the said image. 49. The apparatus of claim 48, wherein the second history MVP buffer differs from the first history MVP buffer, wherein the means for storing motion information for the first CTU line comprises a first video encoding process sub-process, and the means for storing motion information for the second CTU line comprises a second video encoding process sub-process. 50. The apparatus of claim 48, wherein the second history MVP buffer is different from the first history MVP buffer, wherein means for flushing the second history MVP buffer comprises means for flushing the second history MVP buffer in response to processing the beginning of the second CTU row during parallel wavefront processing. 51. The device according to claim 48, in which means for storing motion information for the first row of the CTU comprises means for storing only motion information unique within the first history MVP buffer in the first history MVP buffer, and means for storing motion information for the second row of the CTU comprises means for storing only motion information unique within the second history MVP buffer in the second history MVP buffer. 52. The device according to claim 48, further comprising: means for deleting, when the first history MVP buffer becomes full, one or more of the earliest inputted motion vectors from the first history MVP buffer in accordance with a first-in-first-out (FIFO) rule; And means for deleting, when the second history MVP buffer becomes full, one or more of the earliest inputted motion vectors from the second history MVP buffer in accordance with a FIFO rule. 53. The device according to claim 48, in which means for storing motion information for the first row of the CTU comprises means for storing motion information for each of a plurality of different types of motion patterns in respective different history MVP buffers of the first plurality of history MVP buffers including the first history MVP buffer; And means for storing motion information for the second row of the CTU comprises means for storing motion information for each of the plurality of different types of motion patterns in respective different history MVP buffers of the second plurality of history MVP buffers including the second history MVP buffer. 54. The apparatus of claim 53, wherein said plurality of different types of motion models comprise one or more of an affine motion model, intrablock copy mode motion information, local illumination compensation motion information, subblock motion vector prediction (MVP), and temporal motion prediction. 55. The apparatus of claim 54, further comprising means for generating a synthetic MVP from two or more MVPs in one or more history MVP buffers. 56. The apparatus of claim 55, wherein said two or more MVPs have different types of motion information comprising two or more of encoded motion information, affine motion model motion information, intra-block copy mode motion information, local lighting compensation motion information, sub - block MVP motion information, temporal motion predictor motion information, synthetic motion vector information based on motion MVP information, spatial MVP motion information, and temporal MVP motion information. 57. The apparatus of claim 48, wherein the first line of the CTU has a length less than the full width of the image. 58. The apparatus of claim 48, wherein the means for storing motion information for the first CTU row comprises means for storing motion information for a number of CUs in the first CTU row that is less than the total number of CUs in the first CTU row.

Images