KR20240050408A - Method for correcting motion information and device therefor - Google Patents

Abstract

A video signal decoding apparatus includes a processor, wherein the processor obtains a first motion information list including one or more first motion information related to a current block, corrects the one or more first motion information, and performs one or more first corrections. Obtaining motion information, obtaining one or more first cost values for each of the one or more first corrected motion information, and rearranging the one or more first corrected motion information based on the first cost values. Obtaining a second motion information list, obtaining one or more second motion information based on the first motion information included in the second motion information list, and correcting the one or more second motion information to perform one or more second corrections Obtaining motion information, obtaining one or more second cost values for each of the second corrected motion information, and predicting the current block based on second motion information determined based on the second cost values. do.

Description

Method for correcting motion information and device therefor

The present invention relates to a method and device for processing video signals, and more particularly, to a method and device for processing video signals for encoding or decoding video signals.

Compression encoding refers to a series of signal processing technologies for transmitting digitized information through communication lines or storing it in a form suitable for storage media. Targets of compression coding include audio, video, and text. In particular, the technology for performing compression coding on video is called video image compression. Compression coding for video signals is accomplished by removing redundant information by considering spatial correlation, temporal correlation, and probabilistic correlation. However, due to recent developments in various media and data transmission media, more highly efficient video signal processing methods and devices are required.

The purpose of this specification is to increase the coding efficiency of video signals by providing a video signal processing method and apparatus for the same.

This specification provides a video signal processing method and a device therefor.

In this specification, a video signal decoding device includes a processor, wherein the processor obtains a first motion information list including one or more first motion information related to a current block, and corrects the one or more first motion information. Obtain one or more first corrected motion information, obtain one or more first cost values for each of the one or more first corrected motion information, and obtain the one or more first cost values for the one or more first corrected motion information. Obtaining a second motion information list by rearranging the values, obtaining one or more pieces of second motion information based on first motion information included in the second motion information list, and correcting the one or more pieces of second motion information. Obtain one or more pieces of second corrected motion information, obtain one or more second cost values for each of the second corrected motion information, and based on the second motion information determined based on the second cost values. It is characterized by predicting the current block.

In this specification, a video signal encoding device includes a processor, and the processor obtains a bitstream to be decoded by a decoding method, and the decoding method includes first motion information including one or more first motion information related to the current block. Obtaining a motion information list; Correcting the one or more pieces of first motion information to obtain one or more pieces of first corrected motion information; Obtaining one or more first cost values for each of the one or more pieces of first corrected motion information; Obtaining a second motion information list by rearranging the one or more pieces of first corrected motion information based on the first cost values; Obtaining one or more pieces of second motion information based on first motion information included in the second motion information list; Correcting the one or more pieces of second motion information to obtain one or more pieces of second corrected motion information; Obtaining one or more second cost values for each of the second corrected motion information; and predicting the current block based on second motion information determined based on the second cost values.

In the present specification, in a computer-readable non-transitory storage medium storing a bitstream, the bitstream is decoded by a decoding method, wherein the decoding method includes one or more first motion information related to the current block. 1 Obtaining a motion information list; Correcting the one or more pieces of first motion information to obtain one or more pieces of first corrected motion information; Obtaining one or more first cost values for each of the one or more pieces of first corrected motion information; Obtaining a second motion information list by rearranging the one or more pieces of first corrected motion information based on the first cost values; Obtaining one or more pieces of second motion information based on first motion information included in the second motion information list; Correcting the one or more pieces of second motion information to obtain one or more pieces of second corrected motion information; Obtaining one or more second cost values for each of the second corrected motion information; and predicting the current block based on second motion information determined based on the second cost values.

The one or more first cost values are values related to the similarity between the current block and the reference block corresponding to each of the one or more first corrected motion information, and the one or more second cost values are related to the similarity between the current block and the one or more first corrected motion information. It is characterized in that it is a value related to the similarity between reference blocks corresponding to each of the second corrected motion information.

The second motion information list is characterized in that cost values corresponding to each of the one or more pieces of first corrected motion information are sorted in ascending order.

The first motion information is characterized as corrected motion information corresponding to the smallest cost value among the first cost values.

The second motion information is characterized as corrected motion information corresponding to the smallest cost value among the second cost values.

The one or more pieces of first motion information are respectively included in different pictures, and the one or more pieces of second motion information are respectively included in different pictures.

The one or more first corrected motion information and the one or more second corrected motion information may include Motion Vector Difference (MVD), Template Matching (TM), Bilateral Matching (BM), Merge mode with MVD (MMVD), and MMVD. It is characterized by correction based on at least one of (Merge mode with MVD)-based TM, optical flow-based TM, and multi-pass DMVR (Multi pass Decoder-side Motion Vector Refinement).

When the encoding mode of the current block is a merge mode, the one or more first motion information and the one or more second motion information are characterized in that they are motion information derived by MMVD (Merge mode with MVD). .

Blocks corresponding to each of the one or more pieces of first motion information are blocks located in a first search area, blocks corresponding to each of the one or more pieces of second motion information are blocks located in a second search area, and the blocks corresponding to each of the one or more pieces of second motion information are blocks located in a second search area. The first search area and the second search area are characterized in that they are different from each other.

The first cost values are sequentially calculated for each of the one or more first corrected motion information, and the second cost values are sequentially calculated for each of the one or more second corrected motion information. do.

When the first cost values are calculated sequentially and a cost value smaller than the preset first value is calculated, cost values for the remaining corrected motion information are not calculated, and the second cost values are calculated sequentially. When a cost value smaller than the preset second value is calculated, cost values for the remaining corrected motion information are not calculated.

This specification provides a method for efficiently processing video signals.

The effects that can be obtained in this specification are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 is a schematic block diagram of a video signal encoding device according to an embodiment of the present invention.
Figure 2 is a schematic block diagram of a video signal decoding device according to an embodiment of the present invention.
Figure 3 shows an embodiment in which a coding tree unit is divided into coding units within a picture.
Figure 4 shows one embodiment of a method for signaling splitting of quad trees and multi-type trees.
Figures 5 and 6 show the intra prediction method according to an embodiment of the present invention in more detail.
Figure 7 is a diagram showing the positions of neighboring blocks used to construct a motion candidate list in inter prediction.
Figure 8 is a diagram showing a method for correcting motion information according to an embodiment of the present invention.
Figure 9 is a diagram illustrating a method of correcting motion information for a current block by recursively performing a motion correction method according to an embodiment of the present invention.
Figure 10 is a diagram showing the order in which the TM method is performed according to an embodiment of the present invention.
FIG. 11 is a diagram illustrating a method for setting a search area for a TM method based on initial motion information according to an embodiment of the present invention.
Figure 12 is a diagram showing the location of a motion candidate searched within a search area according to an embodiment of the present invention.
Figure 13 is a diagram showing the process of searching for the location of a motion candidate according to an embodiment of the present invention.
Figure 14 is a diagram showing a process for evaluating search candidates according to an embodiment of the present invention.
Figure 15 is a diagram showing movement characteristics of the boundary portion of an object in the current picture according to an embodiment of the present invention.
Figure 16 is a diagram showing a method of determining a reference picture using template matching according to an embodiment of the present invention.
Figures 17 and 18 are diagrams showing the process of performing template matching according to an embodiment of the present invention.
FIG. 19 is a diagram illustrating a case where initial motion information derived from neighboring blocks of the current block are attached to each other according to an embodiment of the present invention.
20 and 21 are diagrams showing a method of changing initial motion information according to an embodiment of the present invention.
Figure 22 is a diagram showing the location where a motion candidate derived through initial motion information is searched according to an embodiment of the present invention.
Figure 23 is a diagram illustrating a method of changing the search location where motion candidates are searched according to an embodiment of the present invention.
Figure 24 is a diagram showing TM based on the cost value of a motion vector according to an embodiment of the present invention.
Figures 25 and 26 are diagrams showing a method of correcting motion information using DMVR according to an embodiment of the present invention.
Figure 27 is a diagram showing a process in which multiple DMVR is performed according to an embodiment of the present invention.
FIG. 28 is a diagram illustrating a search method for obtaining a cost value related to corrected motion information of a coding block according to an embodiment of the present invention.
Figure 29 is a diagram showing a 3x3 Square search method according to an embodiment of the present invention.
Figure 30 is a diagram showing that the search area is divided into zones in the integer global search process according to an embodiment of the present invention.
Figures 31 and 32 are diagrams showing a global search process in integer units according to an embodiment of the present invention.
Figures 33 and 34 are diagrams showing a method of performing correction of motion information based on BDOF according to an embodiment of the present invention.
Figures 35 and 36 are diagrams showing a method of generating a prediction block for a current block based on BDOF according to an embodiment of the present invention.
Figure 37 is a diagram showing a subblock to which DMVR is applied according to an embodiment of the present invention.
Figure 38 is a diagram showing a method of generating a prediction block for a current block while performing multiple DMVR according to an embodiment of the present invention.

The terms used in this specification are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention of a person skilled in the art, custom, or the emergence of new technology. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in the description of the relevant invention. Therefore, we would like to clarify that the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, not just the name of the term.

In this specification, ‘A and/or B’ may be interpreted as meaning ‘including at least one of A or B.’

Some terms in this specification may be interpreted as follows. Coding can be interpreted as encoding or decoding depending on the case. In this specification, a device that performs encoding (encoding) of a video signal to generate a video signal bitstream is referred to as an encoding device or encoder, and a device that performs decoding (decoding) of a video signal bitstream to restore a video signal is referred to as a decoder. Referred to as a device or decoder. Additionally, in this specification, a video signal processing device is used as a term that includes both an encoder and a decoder. Information is a term that includes values, parameters, coefficients, elements, etc., and the meaning may be interpreted differently depending on the case, so the present invention is not limited thereto. 'Unit' is used to refer to a basic unit of image processing or a specific location of a picture, and refers to an image area containing at least one of a luminance (luma) component and a chrominance (chroma) component. Additionally, 'block' refers to an image area containing specific components among the luminance component and chrominance component (i.e., Cb and Cr). However, depending on the embodiment, terms such as 'unit', 'block', 'partition', 'signal', and 'area' may be used interchangeably. Additionally, in this specification, 'current block' refers to a block currently scheduled to be encoded, and 'reference block' refers to a block for which encoding or decoding has already been completed and is used as a reference in the current block. Additionally, in this specification, terms such as 'luma', 'luma', 'luminance', and 'Y' may be used interchangeably. In addition, in this specification, terms such as 'chroma', 'chroma', 'color difference', and 'Cb or Cr' may be used interchangeably, and since the color difference component is divided into two types, Cb and Cr, each color difference component will be used separately. You can. Additionally, in this specification, a unit may be used as a concept that includes all coding units, prediction units, and transformation units. A picture refers to a field or frame, and depending on the embodiment, the above terms may be used interchangeably. Specifically, when the captured image is an interlaced image, one frame is divided into an odd (or odd, top) field and an even (or even, bottom) field, and each field consists of one picture unit. and can be encoded or decoded. If the captured image is a progressive image, one frame can be configured as a picture and encoded or decoded. Additionally, in this specification, terms such as 'error signal', 'residual signal', 'residual signal', 'residual signal', and 'difference signal' may be used interchangeably. Additionally, in this specification, terms such as 'intra prediction mode', 'intra prediction directional mode', 'intra-screen prediction mode', and 'intra-screen prediction directional mode' may be used interchangeably. Additionally, in this specification, terms such as 'motion' and 'movement' may be used interchangeably. In addition, in this specification, 'left', 'upper left', 'upper', 'upper right', 'right', 'lower right', 'bottom', and 'lower left' mean 'left', 'upper left', ' It can be used interchangeably with 'top', 'top right', 'bottom right', 'bottom right', 'bottom', and 'bottom left'. Additionally, element and member can be used interchangeably. POC (Picture Order Count) represents temporal location information of a picture (or frame), can be the playback order displayed on the screen, and each picture can have a unique POC.

Figure 1 is a schematic block diagram of a video signal encoding device 100 according to an embodiment of the present invention. Referring to Figure 1, the encoding device 100 of the present invention includes a transform unit 110, a quantization unit 115, an inverse quantization unit 120, an inverse transform unit 125, a filtering unit 130, and a prediction unit 150. ) and an entropy coding unit 160.

The conversion unit 110 obtains a conversion coefficient value by converting the residual signal, which is the difference between the input video signal and the prediction signal generated by the prediction unit 150. For example, Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), or Wavelet Transform may be used. Discrete cosine transform and discrete sine transform perform transformation by dividing the input picture signal into blocks. In transformation, coding efficiency may vary depending on the distribution and characteristics of values within the transformation region. The transformation kernel used for transformation of the residual block may be a transformation kernel with separable characteristics of vertical transformation and horizontal transformation. In this case, transformation for the residual block can be performed separately into vertical transformation and horizontal transformation. For example, the encoder can perform vertical transformation by applying a transformation kernel in the vertical direction of the residual block. Additionally, the encoder can perform horizontal transformation by applying a transformation kernel in the horizontal direction of the residual block. In this disclosure, a transform kernel may be used as a term to refer to a set of parameters used for transforming a residual signal, such as a transform matrix, transform array, transform function, or transform. For example, the transformation kernel may be any one of a plurality of available kernels. Additionally, transformation kernels based on different transformation types may be used for each of vertical transformation and horizontal transformation.

Higher conversion coefficients are distributed toward the top left of the block, and coefficients closer to '0' are distributed toward the bottom right of the block. As the size of the current block increases, there is a possibility that there will be more coefficients of '0' in the lower right area. In order to reduce the conversion complexity of large blocks, only the upper left area can be left and the remaining areas can be reset to '0'.

Additionally, error signals may exist only in some areas of the coding block. In this case, the conversion process may be performed only for some arbitrary areas. As an example, in a block of size 2Nx2N, an error signal may exist only in the first 2NxN block, and a conversion process is performed only on the first 2NxN block, but the conversion process is not performed on the second 2NxN block and may not be encoded or decoded. Here N can be any positive integer.

The encoder may perform additional transformations before the transform coefficients are quantized. The above-described transformation method may be referred to as a primary transform, and additional transformation may be referred to as a secondary transform. Secondary transformation may be optional for each residual block. According to one embodiment, the encoder may improve coding efficiency by performing secondary transformation on a region where it is difficult to concentrate energy in the low-frequency region only through primary transformation. For example, secondary transformation may be additionally performed on a block whose residual values appear large in directions other than the horizontal or vertical direction of the residual block. Unlike primary transformation, secondary transformation may not be performed separately into vertical transformation and horizontal transformation. This secondary transform may be referred to as Low Frequency Non-Separable Transform (LFNST).

The quantization unit 115 quantizes the transform coefficient value output from the transform unit 110.

In order to increase coding efficiency, rather than coding the picture signal as is, the picture is predicted using the already coded area through the prediction unit 150, and the residual value between the original picture and the predicted picture is added to the predicted picture to create a reconstructed picture. A method of obtaining is used. To prevent mismatches between the encoder and decoder, information available in the decoder must be used when performing prediction in the encoder. For this purpose, the encoder performs a process of restoring the current encoded block. The inverse quantization unit 120 inversely quantizes the transform coefficient value, and the inverse transform unit 125 restores the residual value using the inverse quantized transform coefficient value. Meanwhile, the filtering unit 130 performs a filtering operation to improve the quality of the reconstructed picture and improve coding efficiency. For example, deblocking filters, sample adaptive offset (SAO), and adaptive loop filters may be included. The filtered picture is output or stored in a decoded picture buffer (DPB, 156) to be used as a reference picture.

A deblocking filter is a filter for removing distortion within blocks created at the boundaries between blocks in a restored picture. The encoder can determine whether to apply a deblocking filter to the edge based on the distribution of pixels included in several columns or rows based on an arbitrary edge within the block. When a deblocking filter is applied to a block, the encoder can apply a long filter, strong filter, or weak filter depending on the deblocking filtering strength. Additionally, horizontal filtering and vertical filtering can be processed in parallel. Sample adaptive offset (SAO) can be used to correct the offset from the original image on a pixel basis for a residual block to which a deblocking filter has been applied. In order to correct the offset for a specific picture, the encoder divides the pixels included in the image into a certain number of areas, determines the area to perform offset correction, and uses a method (Band Offset) to apply the offset to the area. You can. Alternatively, the encoder can use a method of applying an offset (Edge Offset) by considering the edge information of each pixel. Adaptive Loop Filter (ALF) is a method of dividing pixels included in an image into predetermined groups, then determining a filter to be applied to the group, and performing differential filtering for each group. Information related to whether to apply ALF may be signaled in units of coding units, and the shape and filter coefficients of the ALF filter to be applied may vary for each block. Additionally, an ALF filter of the same type (fixed type) may be applied regardless of the characteristics of the target block to be applied.

The prediction unit 150 includes an intra prediction unit 152 and an inter prediction unit 154. The intra prediction unit 152 performs intra prediction within the current picture, and the inter prediction unit 154 performs inter prediction using the reference picture stored in the decoded picture buffer 156. Perform. The intra prediction unit 152 performs intra prediction from the reconstructed areas in the current picture and transmits intra encoding information to the entropy coding unit 160. Intra encoding information may include at least one of an intra prediction mode, a Most Probable Mode (MPM) flag, an MPM index, and information about a reference sample. The inter prediction unit 154 may again include a motion estimation unit 154a and a motion compensation unit 154b. The motion estimation unit 154a refers to a specific region of the reconstructed reference picture, finds the part most similar to the current region, and obtains a motion vector value that is the distance between regions. Motion information (reference direction indication information (L0 prediction, L1 prediction, bi-directional prediction), reference picture index, motion vector information, etc.) about the reference area obtained from the motion estimation unit 154a is transmitted to the entropy coding unit 160. so that it can be included in the bitstream. Using the motion information transmitted from the motion estimation unit 154a, the motion compensation unit 154b performs inter-motion compensation to generate a prediction block for the current block. The inter prediction unit 154 transmits inter encoding information including motion information about the reference region to the entropy coding unit 160.

According to an additional embodiment, the prediction unit 150 may include an intra block copy (IBC) prediction unit (not shown). The IBC prediction unit performs IBC prediction from the reconstructed samples in the current picture and transmits IBC encoding information to the entropy coding unit 160. The IBC prediction unit refers to a specific region in the current picture and obtains a block vector value indicating a reference region used for prediction of the current region. The IBC prediction unit may perform IBC prediction using the obtained block vector value. The IBC prediction unit transmits IBC encoding information to the entropy coding unit 160. IBC encoding information may include at least one of reference area size information and block vector information (index information for block vector prediction of the current block within the motion candidate list, block vector difference information).

When the above picture prediction is performed, the transform unit 110 obtains a transform coefficient value by transforming the residual value between the original picture and the predicted picture. At this time, transformation can be performed on a specific block basis within the picture, and the size of a specific block can vary within a preset range. The quantization unit 115 quantizes the transform coefficient value generated by the transform unit 110 and transmits the quantized transform coefficient to the entropy coding unit 160.

The quantized transform coefficients in the form of a two-dimensional array can be rearranged into a one-dimensional array for entropy coding. The scanning method for the quantized transform coefficient may be determined depending on the size of the transform block and the intra-screen prediction mode. As an example, diagonal, vertical, and horizontal scans may be applied. This scan information can be signaled in block units and can be derived according to already established rules.

The entropy coding unit 160 generates a video signal bitstream by entropy coding information representing quantized transform coefficients, intra encoding information, and inter encoding information. The entropy coding unit 160 may use a variable length coding (VLC) method or an arithmetic coding method. The variable length coding (VLC) method converts input symbols into continuous codewords, and the length of the codewords may be variable. For example, frequently occurring symbols are expressed as short codewords, and infrequently occurring symbols are expressed as long codewords. As a variable length coding method, Context-based Adaptive Variable Length Coding (CAVLC) can be used. Arithmetic coding converts consecutive data symbols into a single decimal number using the probability distribution of each data symbol. Arithmetic coding can obtain the optimal decimal bits needed to express each symbol. As arithmetic coding, context-based adaptive binary arithmetic code (CABAC) can be used.

CABAC is a method of binary arithmetic encoding using multiple context models created based on probabilities obtained through experiments. First, if the symbols are not in binary form, the encoder binarizes each symbol using exp-Golomb, etc. Binarized 0 or 1 can be described as a bin. The CABAC initialization process is divided into context initialization and arithmetic coding initialization. Context initialization is a process of initializing the probability of occurrence of each symbol, and is determined depending on the type of symbol, quantization parameter (QP), and slice type (whether I, P, or B). The context model with this initialization information can use probability-based values obtained through experimentation. The context model provides the probability of occurrence of LPS (Least Probable Symbol) or MPS (Most Probable Symbol) for the symbol currently being coded and information (valMPS) about which empty value among 0 and 1 corresponds to the MPS. One of several context models is selected through a context index (ctxIdx), and the context index can be derived through information on the current block to be encoded or information on surrounding blocks. Initialization for binary arithmetic coding is performed based on the probability model selected from the context model. Binary arithmetic coding is divided into probability intervals using the probability of occurrence of 0 and 1, and then coding is carried out through the process where the probability interval corresponding to the bin to be processed becomes the entire probability interval for the next bin to be processed. Location information within the probability interval where the last bin was processed is output. However, since the probability interval cannot be divided indefinitely, when it is reduced to within a certain size, a renormalization process is performed, the probability interval is widened, and the corresponding location information is output. Additionally, after each bin is processed, a probability update process may be performed in which the probability of the next bin to be processed is newly set through information on the processed bin.

The generated bitstream is encapsulated in a NAL (Network Abstraction Layer) unit as a basic unit. NAL units are divided into VCL (Video Coding Layer) NAL units containing video data and non-VCL NAL units containing parameter information for decoding video data. There are various types of VCL or non-VCL NAL units. . The NAL unit consists of NAL header information and data, RBSP (Raw Byte Sequence Payload), and the NAL header information includes summary information about the RBSP. The RBSP of the VCL NAL unit includes an encoded integer number of coding tree units. In order to decode a bitstream in a video decoder, the bitstream must first be separated into NAL units, and then each separated NAL unit must be decoded. Meanwhile, the information required for decoding the video signal bitstream will be transmitted in a picture parameter set (PPS), sequence parameter set (SPS), video parameter set (VPS), etc. You can.

Meanwhile, the block diagram of FIG. 1 shows the encoding device 100 according to an embodiment of the present invention, and the separately displayed blocks show elements of the encoding device 100 logically distinguished. Accordingly, the elements of the above-described encoding device 100 may be mounted as one chip or as a plurality of chips depending on the design of the device. According to one embodiment, the operation of each element of the above-described encoding device 100 may be performed by a processor (not shown).

Figure 2 is a schematic block diagram of a video signal decoding device 200 according to an embodiment of the present invention. Referring to FIG. 2, the decoding device 200 of the present invention includes an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 225, a filtering unit 230, and a prediction unit 250.

The entropy decoding unit 210 entropy decodes the video signal bitstream and extracts transform coefficient information, intra encoding information, and inter encoding information for each region. For example, the entropy decoder 210 may obtain a binarization code for transform coefficient information of a specific area from a video signal bitstream. Additionally, the entropy decoding unit 210 inversely binarizes the binarization code to obtain a quantized transform coefficient. The inverse quantization unit 220 inversely quantizes the quantized transform coefficient, and the inverse transform unit 225 restores the residual value using the inverse quantized transform coefficient. The video signal processing device 200 restores the original pixel value by summing the residual value obtained from the inverse transform unit 225 with the predicted value obtained from the prediction unit 250.

Meanwhile, the filtering unit 230 improves image quality by performing filtering on the picture. This may include a deblocking filter to reduce block distortion and/or an adaptive loop filter to remove distortion of the entire picture. The filtered picture is output or stored in the decoded picture buffer (DPB, 256) to be used as a reference picture for the next picture.

The prediction unit 250 includes an intra prediction unit 252 and an inter prediction unit 254. The prediction unit 250 generates a prediction picture using the coding type decoded through the entropy decoding unit 210, transform coefficients for each region, intra/inter coding information, etc. To restore the current block on which decoding is performed, the current picture including the current block or the decoded area of other pictures can be used. Only the current picture is used for reconstruction, that is, a picture (or tile/slice) that performs intra prediction or intra BC prediction is used as an intra picture or I picture (or tile/slice), intra prediction, and both inter prediction and intra BC prediction are used. A picture (or tile/slice) that can be performed is called an inter picture (or tile/slice). A picture (or tile/slice) that uses at most one motion vector and a reference picture index to predict sample values of each block among inter pictures (or tiles/slices) is called a predictive picture or a P picture (or , tile/slice), and a picture (or tile/slice) using up to two motion vectors and a reference picture index is called a bi-predictive picture or B picture (or tile/slice). In other words, a P picture (or tile/slice) uses at most one set of motion information to predict each block, and a B picture (or tile/slice) uses at most two sets of motion information to predict each block. Use a set. Here, the motion information set includes one or more motion vectors and one reference picture index.

The intra prediction unit 252 generates a prediction block using intra encoding information and reconstructed samples in the current picture. As described above, intra encoding information may include at least one of an intra prediction mode, a Most Probable Mode (MPM) flag, and an MPM index. The intra prediction unit 252 predicts sample values of the current block using reconstructed samples located to the left and/or above the current block as reference samples. In this disclosure, reconstructed samples, reference samples, and samples of the current block may represent pixels. Additionally, sample values may represent pixel values.

According to one embodiment, the reference samples may be samples included in neighboring blocks of the current block. For example, the reference samples may be samples adjacent to the left border and/or samples adjacent to the upper boundary of the current block. In addition, the reference samples are samples of neighboring blocks of the current block, which are located on a line within a preset distance from the left border of the current block and/or are located on a line within a preset distance from the upper border of the current block. These may be samples that do. At this time, the surrounding blocks of the current block are the left (L) block, upper (A) block, Below Left (BL) block, Above Right (AR) block, or Above Left block adjacent to the current block. AL) may include at least one block.

The inter prediction unit 254 generates a prediction block using the reference picture and inter encoding information stored in the decoded picture buffer 256. Inter-encoding information may include a set of motion information (reference picture index, motion vector information, etc.) of the current block with respect to the reference block. Inter prediction may include L0 prediction, L1 prediction, and bi-prediction. L0 prediction refers to prediction using one reference picture included in the L0 picture list, and L1 prediction refers to prediction using one reference picture included in the L1 picture list. This may require one set of motion information (eg, motion vector and reference picture index). In the pair prediction method, a maximum of two reference regions can be used, and these two reference regions may exist in the same reference picture or in different pictures. That is, in the pair prediction method, up to two sets of motion information (e.g., a motion vector and a reference picture index) can be used, and the two motion vectors may correspond to the same reference picture index or may correspond to different reference picture indices. It may be possible to respond. At this time, the reference pictures are pictures located temporally before or after the current picture, and may be pictures that have already been reconstructed. According to one embodiment, the two reference regions used in the bi-prediction method may be regions selected from the L0 picture list and the L1 picture list, respectively.

The inter prediction unit 254 may obtain a reference block of the current block using a motion vector and a reference picture index. The reference block exists in a reference picture corresponding to a reference picture index. Additionally, the sample value of the block specified by the motion vector or its interpolated value may be used as a predictor of the current block. For motion prediction with sub-pel unit pixel accuracy, for example, an 8-tap interpolation filter can be used for the luminance signal and a 4-tap interpolation filter can be used for the chrominance signal. However, the interpolation filter for motion prediction in subpel units is not limited to this. In this way, the inter prediction unit 254 performs motion compensation to predict the texture of the current unit from the previously restored picture. At this time, the inter prediction unit can use a motion information set.

According to an additional embodiment, the prediction unit 250 may include an IBC prediction unit (not shown). The IBC prediction unit can reconstruct the current region by referring to a specific region containing reconstructed samples in the current picture. The IBC prediction unit may perform IBC prediction using the IBC encoding information obtained from the entropy decoding unit 210. IBC encoding information may include block vector information.

The predicted value output from the intra prediction unit 252 or the inter prediction unit 254 and the residual value output from the inverse transform unit 225 are added to generate a restored video picture. That is, the video signal decoding apparatus 200 restores the current block using the prediction block generated by the prediction unit 250 and the residual obtained from the inverse transform unit 225.

Meanwhile, the block diagram of FIG. 2 shows a decoding device 200 according to an embodiment of the present invention, and the separately displayed blocks show elements of the decoding device 200 logically distinguished. Accordingly, the elements of the above-described decoding device 200 may be mounted as one chip or as a plurality of chips depending on the design of the device. According to one embodiment, the operation of each element of the above-described decoding device 200 may be performed by a processor (not shown).

Meanwhile, the technology proposed in this specification is applicable to both encoder and decoder methods and devices, and parts described as signaling and parsing may be described for convenience of explanation. In general, signaling can be described as encoding each syntax from an encoder's perspective, and parsing can be described as interpreting each syntax from a decoder's perspective. That is, each syntax can be signaled by being included in the bitstream from the encoder, and the decoder can parse the syntax and use it in the restoration process. At this time, the sequence of bits for each syntax arranged according to the prescribed hierarchical structure can be referred to as a bitstream.

One picture may be divided into sub-pictures, slices, tiles, etc. and encoded. A subpicture may include one or more slices or tiles. When one picture is divided into multiple slices or tiles and encoded, it can be displayed on the screen only when all slices or tiles in the picture have been decoded. On the other hand, when one picture is encoded with several subpictures, only arbitrary subpictures can be decoded and displayed on the screen. A slice may contain multiple tiles or subpictures. Alternatively, a tile may include multiple subpictures or slices. Subpictures, slices, and tiles can be encoded or decoded independently of each other, which is effective in improving parallel processing and processing speed. However, there is a disadvantage in that the bit amount increases because encoded information of other adjacent subpictures, other slices, and other tiles cannot be used. Subpictures, slices, and tiles can be divided into multiple Coding Tree Units (CTUs) and encoded.

Figure 3 shows an embodiment in which a Coding Tree Unit (CTU) is divided into Coding Units (CUs) within a picture. In the process of coding a video signal, a picture can be divided into a sequence of coding tree units (CTUs). A coding tree unit may be composed of a luma coding tree block (CTB), two chroma coding tree blocks, and its encoded syntax information. One coding tree unit may consist of one coding unit, or one coding tree unit may be divided into multiple coding units. One coding unit may be composed of a luminance coding block (CB), two chrominance coding blocks, and its encoded syntax information. One coding block can be divided into several sub-coding blocks. One coding unit may consist of one transform unit (TU), or one coding unit may be divided into several transform units. One transformation unit may be composed of a luminance transformation block (Transform Block, TB), two chrominance transformation blocks, and its encoded syntax information. A coding tree unit may be divided into a plurality of coding units. A coding tree unit may be a leaf node without being split. In this case, the coding tree unit itself may be a coding unit.

A coding unit refers to a basic unit for processing a picture in the video signal processing process described above, that is, intra/inter prediction, transformation, quantization, and/or entropy coding. The size and shape of a coding unit within one picture may not be constant. The coding unit may have a square or rectangular shape. A rectangular coding unit (or rectangular block) includes a vertical coding unit (or vertical block) and a horizontal coding unit (or horizontal block). In this specification, a vertical block is a block whose height is greater than its width, and a horizontal block is a block whose width is greater than its height. Additionally, in this specification, a non-square block may refer to a rectangular block, but the present invention is not limited thereto.

Referring to Figure 3, the coding tree unit is first divided into a quad tree (Quad Tree, QT) structure. That is, in a quad tree structure, one node with a size of 2NX2N can be divided into four nodes with a size of NXN. In this specification, a quad tree may also be referred to as a quaternary tree. Quad-tree partitioning can be performed recursively, and not all nodes need to be partitioned to the same depth.

Meanwhile, the leaf nodes of the aforementioned quad tree can be further divided into a multi-type tree (MTT) structure. According to an embodiment of the present invention, in a multi-type tree structure, one node may be divided into a binary or ternary tree structure with horizontal or vertical division. That is, there are four division structures in the multi-type tree structure: vertical binary division, horizontal binary division, vertical ternary division, and horizontal ternary division. According to an embodiment of the present invention, the width and height of the nodes in each tree structure may both have values that are powers of 2. For example, in a Binary Tree (BT) structure, a node of size 2NX2N may be divided into two NX2N nodes by vertical binary division and into two 2NXN nodes by horizontal binary division. Additionally, in the Ternary Tree (TT) structure, a node of size 2NX2N is divided into nodes of (N/2)X2N, NX2N and (N/2)X2N by vertical ternary division, and horizontal ternary division By division, it can be divided into nodes of 2NX(N/2), 2NXN, and 2NX(N/2). This multi-type tree partitioning can be performed recursively.

Leaf nodes of a multi-type tree can be coding units. If the coding unit is not larger than the maximum transformation length, the coding unit can be used as a unit of prediction and/or transformation without further division. As an example, if the width or height of the current coding unit is greater than the maximum transform length, the current coding unit may be split into a plurality of transform units without explicit signaling regarding splitting. Meanwhile, in the above-described quad tree and multi-type tree, at least one of the following parameters may be defined in advance or transmitted through an RBSP of a higher level set such as PPS, SPS, VPS, etc. 1) CTU size: the root node size of the quad tree, 2) minimum QT size (MinQtSize): minimum allowed QT leaf node size, 3) maximum BT size (MaxBtSize): maximum allowed BT root node size, 4) Maximum TT Size (MaxTtSize): Maximum TT root node size allowed, 5) Maximum MTT Depth (MaxMttDepth): Maximum allowed depth of MTT split from leaf nodes of QT, 6) Minimum BT Size (MinBtSize): Allowed Minimum BT leaf node size, 7) Minimum TT size (MinTtSize): Minimum TT leaf node size allowed.

Figure 4 shows one embodiment of a method for signaling splitting of quad trees and multi-type trees. Preset flags can be used to signal division of the above-described quad tree and multi-type tree. Referring to Figure 4, a flag 'split_cu_flag' indicating whether to split a node, a flag 'split_qt_flag' indicating whether to split a quad tree node, a flag 'mtt_split_cu_vertical_flag' indicating the splitting direction of a multi-type tree node, or a multi-type tree node. At least one of the flags 'mtt_split_cu_binary_flag' that indicates the split shape of the type tree node can be used.

According to an embodiment of the present invention, 'split_cu_flag', a flag indicating whether to split the current node, may be signaled first. If the value of 'split_cu_flag' is 0, it indicates that the current node is not split, and the current node becomes a coding unit. If the current node is a coating tree unit, the coding tree unit includes one undivided coding unit. If the current node is a quad tree node 'QT node', the current node is a leaf node 'QT leaf node' of the quad tree and becomes a coding unit. If the current node is a multi-type tree node 'MTT node', the current node is a leaf node 'MTT leaf node' of the multi-type tree and becomes a coding unit.

If the value of 'split_cu_flag' is 1, the current node can be divided into nodes of a quad tree or multi-type tree according to the value of 'split_qt_flag'. The coding tree unit is the root node of the quad tree and can be first divided into a quad tree structure. In the quad tree structure, 'split_qt_flag' is signaled for each node 'QT node'. If the value of 'split_qt_flag' is 1, the node is split into 4 square nodes. If the value of 'split_qt_flag' is 0, the node becomes a leaf node 'QT leaf node' of the quad tree, and the node becomes a multi-square node. -Divided into type nodes. According to an embodiment of the present invention, quad tree division may be limited depending on the type of the current node. Quad tree splitting may be allowed if the current node is a coding tree unit (root node of the quot tree) or a quot tree node, and quot tree splitting may not be allowed if the current node is a multi-type tree node. Each quad tree leaf node 'QT leaf node' can be further divided into a multi-type tree structure. As described above, if 'split_qt_flag' is 0, the current node can be split into multi-type nodes. To indicate the split direction and split shape, 'mtt_split_cu_vertical_flag' and 'mtt_split_cu_binary_flag' may be signaled. If the value of 'mtt_split_cu_vertical_flag' is 1, vertical splitting of the node 'MTT node' is indicated, and if the value of 'mtt_split_cu_vertical_flag' is 0, horizontal splitting of the node 'MTT node' is indicated. Additionally, if the value of 'mtt_split_cu_binary_flag' is 1, the node 'MTT node' is divided into two rectangular nodes, and if the value of 'mtt_split_cu_binary_flag' is 0, the node 'MTT node' is divided into three rectangular nodes.

In the tree division structure, the luminance block and the chrominance block can be divided into the same form. That is, the chrominance block can be divided by referring to the division type of the luminance block. If the current chrominance block is smaller than a certain size, the chrominance block may not be divided even if the luminance block is divided.

In the tree division structure, the luminance block and the chrominance block may have different forms. At this time, division information for the luminance block and division information for the chrominance block may be signaled, respectively. Additionally, not only the division information but also the encoding information of the luminance block and the chrominance block may be different. As an example of an embodiment, at least one intra coding mode of a luminance block and a chrominance block, encoding information for motion information, etc. may be different.

A node to be divided into the smallest unit can be processed as one coding block. When the current block is a coding block, the coding block may be divided into several sub-blocks (sub-coding blocks), and the prediction information of each sub-block may be the same or different. As an example embodiment, when the coding unit is an intra mode, the intra prediction mode of each subblock may be the same or different from each other. Additionally, when the coding unit is in inter mode, the motion information of each sub-block may be the same or different. Additionally, each sub-block may be encoded or decoded independently from each other. Each sub-block can be distinguished through a sub-block index (sbIdx). Additionally, when a coding unit is divided into sub-blocks, it may be divided horizontally or vertically or diagonally. In intra mode, the mode in which the current coding unit is divided into 2 or 4 sub-blocks horizontally or vertically is called ISP (Intra Sub Partitions). In inter mode, the mode in which the current coding block is divided diagonally is called GPM (Geometric partitioning mode). In GPM mode, the position and direction of the diagonal line are derived using a predetermined angle table, and the index information of the angle table is signaled.

Picture prediction (motion compensation) for coding is performed on coding units that are no longer divided (i.e., leaf nodes of coding tree units). The basic unit that performs such prediction is hereinafter referred to as a prediction unit or prediction block.

Hereinafter, the term unit used in this specification may be used as a replacement for the prediction unit, which is a basic unit for performing prediction. However, the present invention is not limited to this, and can be understood more broadly as a concept including the coding unit.

Figures 5 and 6 show the intra prediction method according to an embodiment of the present invention in more detail. As described above, the intra prediction unit predicts sample values of the current block using reconstructed samples located to the left and/or above the current block as reference samples.

First, Figure 5 shows an example of reference samples used for prediction of the current block in intra prediction mode. According to one embodiment, the reference samples may be samples adjacent to the left boundary and/or samples adjacent to the upper boundary of the current block. As shown in Figure 5, when the size of the current block is WXH and samples of a single reference line adjacent to the current block are used for intra prediction, a maximum of 2W+2H+1 located to the left and/or above the current block Reference samples can be set using the surrounding samples.

Meanwhile, pixels of multiple reference lines may be used for intra prediction of the current block. Multiple reference lines may be composed of n lines located within a preset range from the current block. According to one embodiment, when pixels of multiple reference lines are used for intra prediction, separate index information indicating lines to be set as reference pixels may be signaled, and may be called a reference line index.

Additionally, when at least some samples to be used as reference samples have not yet been reconstructed, the intra prediction unit may obtain reference samples by performing a reference sample padding process. Additionally, the intra prediction unit may perform a reference sample filtering process to reduce the error of intra prediction. That is, filtered reference samples can be obtained by performing filtering on surrounding samples and/or reference samples obtained through a reference sample padding process. The intra prediction unit predicts samples of the current block using the reference samples obtained in this way. The intra prediction unit predicts samples of the current block using unfiltered or filtered reference samples. In this disclosure, peripheral samples may include samples on at least one reference line. For example, neighboring samples may include adjacent samples on a line adjacent to the boundary of the current block.

Next, Figure 6 shows an example of prediction modes used for intra prediction. For intra prediction, intra prediction mode information indicating the intra prediction direction may be signaled. Intra prediction mode information indicates one of a plurality of intra prediction modes constituting an intra prediction mode set. If the current block is an intra prediction block, the decoder receives intra prediction mode information of the current block from the bitstream. The intra prediction unit of the decoder performs intra prediction on the current block based on the extracted intra prediction mode information.

According to an embodiment of the present invention, the intra prediction mode set may include all intra prediction modes used for intra prediction (eg, a total of 67 intra prediction modes). More specifically, the intra prediction mode set may include a planar mode, a DC mode, and multiple (e.g., 65) angular modes (i.e., directional modes). Each intra prediction mode may be indicated through a preset index (i.e., intra prediction mode index). For example, as shown in FIG. 6, intra prediction mode index 0 indicates planar mode, and intra prediction mode index 1 indicates DC mode. Additionally, intra prediction mode indices 2 to 66 may respectively indicate different angle modes. The angle modes each indicate different angles within a preset angle range. For example, the angle mode may indicate an angle within an angle range between 45 degrees and -135 degrees clockwise (i.e., a first angle range). The angle mode can be defined based on the 12 o'clock direction. At this time, intra prediction mode index 2 indicates horizontal diagonal (HDIA) mode, intra prediction mode index 18 indicates horizontal (HORizontal, HOR) mode, and intra prediction mode index 34 indicates diagonal (DIA) mode. The mode is indicated, and intra prediction mode index 50 indicates vertical (VER) mode, and intra prediction mode index 66 indicates vertical diagonal (VDIA) mode.

Meanwhile, the preset angle range may be set differently depending on the shape of the current block. For example, if the current block is a rectangular block, a wide-angle mode indicating an angle exceeding 45 degrees or less than -135 degrees clockwise may be additionally used. If the current block is a horizontal block, the angle mode may indicate an angle within an angle range (i.e., a second angle range) between (45+offset1) degrees and (-135+offset1) degrees in a clockwise direction. At this time, angle modes 67 to 76 outside the first angle range may be additionally used. Additionally, if the current block is a vertical block, the angle mode may indicate an angle within an angle range between (45-offset2) degrees and (-135-offset2) degrees clockwise (i.e., a third angle range). . At this time, angle modes -10 to -1 outside the first angle range may be additionally used. According to an embodiment of the present invention, the values of offset1 and offset2 may be determined differently depending on the ratio between the width and height of the rectangular block. Additionally, offset1 and offset2 can be positive numbers.

According to a further embodiment of the present invention, the plurality of angle modes constituting the intra prediction mode set may include a basic angle mode and an extended angle mode. At this time, the extended angle mode may be determined based on the basic angle mode.

According to one embodiment, the basic angle mode corresponds to the angle used in intra prediction of the existing HEVC (High Efficiency Video Coding) standard, and the extended angle mode corresponds to the angle newly added in intra prediction of the next-generation video codec standard. It may be a mode that does this. More specifically, the default angle mode is the intra prediction mode {2, 4, 6,... , 66}, and the extended angle mode is the intra prediction mode {3, 5, 7,... , 65} may be an angle mode corresponding to one of the following. That is, the extended angle mode may be an angle mode between basic angle modes within the first angle range. Accordingly, the angle indicated by the extended angle mode can be determined based on the angle indicated by the basic angle mode.

According to another embodiment, the basic angle mode may be a mode corresponding to an angle within a preset first angle range, and the extended angle mode may be a wide angle mode outside the first angle range. That is, the default angle mode is the intra prediction mode {2, 3, 4, … , 66}, and the extended angle mode is the intra prediction mode {-14, -13, -12,... , -1} and {67, 68, … , 80} may be an angle mode corresponding to one of the following. The angle indicated by the extended angle mode may be determined as the angle opposite to the angle indicated by the corresponding basic angle mode. Accordingly, the angle indicated by the extended angle mode can be determined based on the angle indicated by the basic angle mode. Meanwhile, the number of expansion angle modes is not limited to this, and additional expansion angles may be defined depending on the size and/or shape of the current block. Meanwhile, the total number of intra prediction modes included in the intra prediction mode set may vary depending on the configuration of the basic angle mode and extended angle mode described above.

In the above embodiment, the spacing between extended angle modes may be set based on the spacing between corresponding basic angle modes. For example, the extended angle modes {3, 5, 7, … , 65} are the corresponding fundamental angular modes {2, 4, 6, … , 66} can be determined based on the interval between them. Additionally, the extended angle modes {-14, -13, … , -1} are the corresponding opposite fundamental angular modes {53, 53,... , 66} is determined based on the spacing between the extended angle modes {67, 68,... , 80} are the corresponding opposite fundamental angular modes {2, 3, 4, … , 15} can be determined based on the interval between them. The angular spacing between the extended angle modes may be set to be equal to the angular spacing between the corresponding basic angle modes. Additionally, the number of extended angle modes in the intra prediction mode set may be set to less than the number of basic angle modes.

According to an embodiment of the present invention, the extended angle mode may be signaled based on the basic angle mode. For example, a wide angle mode (i.e., extended angle mode) may replace at least one angle mode (i.e., basic angle mode) within the first angle range. The basic angle mode that is replaced may be an angle mode corresponding to the opposite side of the wide angle mode. That is, the basic angle mode that is replaced is an angle mode that corresponds to an angle in the opposite direction of the angle indicated by the wide-angle mode or to an angle that differs from the angle in the opposite direction by a preset offset index. According to an embodiment of the present invention, the preset offset index is 1. The intra-prediction mode index corresponding to the replaced basic angle mode may be remapped to the wide-angle mode to signal the corresponding wide-angle mode. For example, wide angle mode {-14, -13, … , -1} is the intra prediction mode index {52, 53, … , 66}, respectively, and the wide-angle mode {67, 68, … , 80} is the intra prediction mode index {2, 3, … , 15} can be signaled respectively. In this way, the intra prediction mode index for the basic angle mode signals the extended angle mode, so that even if the configurations of the angle modes used for intra prediction of each block are different, the same set of intra prediction mode indexes are used for signaling of the intra prediction mode. can be used Accordingly, signaling overhead due to changes in intra prediction mode configuration can be minimized.

Meanwhile, whether to use the extended angle mode may be determined based on at least one of the shape and size of the current block. According to one embodiment, if the size of the current block is larger than the preset size, the extended angle mode may be used for intra prediction of the current block, otherwise, only the basic angle mode may be used for intra prediction of the current block. According to another embodiment, if the current block is a non-square block, the extended angle mode may be used for intra prediction of the current block, and if the current block is a square block, only the basic angle mode may be used for intra prediction of the current block.

The intra prediction unit determines reference samples and/or interpolated reference samples to be used for intra prediction of the current block, based on intra prediction mode information of the current block. When the intra prediction mode index indicates a specific angle mode, a reference sample or an interpolated reference sample corresponding to the specific angle from the current sample of the current block is used for prediction of the current pixel. Accordingly, different sets of reference samples and/or interpolated reference samples may be used for intra prediction depending on the intra prediction mode. After intra prediction of the current block is performed using reference samples and intra prediction mode information, the decoder restores the sample values of the current block by adding the residual signal of the current block obtained from the inverse transformer to the intra prediction value of the current block. .

Movement (motion) information used for inter prediction may include reference direction indication information (inter_pred_idc), reference picture indices (ref_idx_l0, ref_idx_l1), and motion (motion) vectors (mvL0, mvL1). Reference picture list utilization information (predFlagL0, predFlagL1) may be set according to the reference direction indication information. As an example of an embodiment, in the case of unidirectional prediction using an L0 reference picture, predFlagL0=1 and predFlagL1=0 may be set. In the case of unidirectional prediction using an L1 reference picture, predFlagL0=0 and predFlagL1=1 can be set. In the case of bidirectional prediction using both L0 and L1 reference pictures, predFlagL0=1 and predFlagL1=1 can be set.

When the current block is a coding unit, the coding unit may be divided into several sub-blocks, and the prediction information of each sub-block may be the same or different. As an example embodiment, when the coding unit is an intra mode, the intra prediction mode of each subblock may be the same or different from each other. Additionally, when the coding unit is in inter mode, the motion information of each sub-block may be the same or different. Additionally, each sub-block may be encoded or decoded independently from each other. Each sub-block can be distinguished through a sub-block index (sbIdx).

The motion vector of the current block is likely to be similar to the motion vector of neighboring blocks. Therefore, the motion vector of the neighboring block can be used as a motion vector predictor (mvp), and the motion vector of the current block can be derived using the motion vector of the neighboring block. Additionally, in order to increase the accuracy of the motion vector, the motion vector difference (mvd) between the optimal motion vector of the current block found in the original image by the encoder and the motion prediction value may be signaled.

The motion vector may have various resolutions, and the resolution of the motion vector may vary on a block basis. Motion vector resolution can be expressed in integer units, half-pixel units, 1/4 pixel units, 1/16 pixel units, integer pixel units of 4, etc. Since images such as screen content are in the form of simple graphics such as text, there is no need to apply an interpolation filter, so integer units and integer pixel units of 4 can be selectively applied on a block basis. Blocks encoded in affine mode, which can express rotation and scale, have significant changes in shape, so integer units, 1/4 pixel units, and 1/16 pixel units can be selectively applied on a block basis. Information on whether to selectively apply motion vector resolution on a block basis is signaled with amvr_flag. If applied, which motion vector resolution to apply to the current block is signaled with amvr_precision_idx.

For blocks to which bidirectional prediction is applied, when applying weight average, the weights between the two prediction blocks can be applied the same or different, and information about the weights is signaled through bcw_idx.

To increase the accuracy of motion prediction values, merge or advanced motion vector prediction (AMVP) methods can be selectively used on a block basis. The Merge method is a method that configures the motion information of the current block to be the same as the motion information of neighboring blocks adjacent to the current block. The Merge method has the advantage of increasing the coding efficiency of motion information by spatially propagating motion information without change in a motion region with homogeneity. There is. On the other hand, the AMVP method is a method that predicts motion information in the L0 and L1 prediction directions respectively and signals the most optimal motion information in order to express accurate motion information. The decoder derives motion information for the current block through the AMVP or Merge method and then uses the reference block located in the motion information derived from the reference picture as a prediction block for the current block.

A method of deriving motion information in Merge or AMVP may be a method in which a motion candidate list is constructed using motion prediction values derived from neighboring blocks of the current block, and then index information for the optimal motion candidate is signaled. In the case of AMVP, since motion candidate lists are derived for each of L0 and L1, the optimal motion candidate indices (mvp_l0_flag, mvp_l1_flag) for each of L0 and L1 are signaled. In the case of Merge, since one motion candidate list is derived, one merge index (merge_idx) is signaled. The motion candidate list derived from one coding unit may vary, and a motion candidate index or merge index may be signaled for each motion candidate list. At this time, a mode in which there is no information about the remaining blocks in blocks encoded in Merge mode can be called Merge Skip mode.

SMVD (Symmetric MVD) is a method of reducing the amount of bits of transmitted motion information by ensuring that the MVD (Motion Vector Difference) values in the L0 and L1 directions are symmetrical in the case of bi-directional prediction. MVD information in the L1 direction, which is symmetrical to the L0 direction, is not transmitted, and in addition, reference picture information in the L0 and L1 directions is not transmitted and is derived during the decoding process.

OBMC (Overlapped Block Motion Compensation) generates prediction blocks for the current block using the motion information of neighboring blocks when the motion information between blocks is different, and then weight averages the prediction blocks to create the final prediction block for the current block. How to create it. This has the effect of reducing the blocking phenomenon that occurs at the block boundaries of motion compensated images.

In general, merge movement candidates have low movement accuracy. To increase the accuracy of these merge motion candidates, the MMVD (Merge mode with MVD) method can be used. The MMVD method is a method of correcting motion information using one candidate selected from among several motion difference value candidates. Information on the correction value of the motion information obtained through the MMVD method (eg, an index indicating one candidate selected from motion difference value candidates, etc.) may be included in the bitstream and transmitted to the decoder. Compared to including existing motion information differential values in the bitstream, the amount of bits can be saved by including information about the correction value of motion information in the bitstream.

The TM (Template Matching) method is a method of compensating motion information by constructing a template using surrounding pixels of the current block and finding a matching area with the highest similarity to the template. Template matching (TM) is a method of performing motion prediction in a decoder without including motion information in the bitstream in order to reduce the size of the encoded bitstream. At this time, since the decoder does not have the original image, it can roughly derive motion information about the current block using already restored neighboring blocks.

The DMVR (Decoder-side Motion Vector Refinement) method is a method of correcting motion information through the correlation of already restored reference images in order to find more accurate motion information. It uses the bidirectional motion information of the current block to compare the two reference pictures. This is a method of using the point with the best matching between reference blocks in a reference picture within a certain area as a new bidirectional movement. When such DMVR is performed, the encoder corrects the motion information by performing DMVR in one block unit, then divides the block into sub-blocks and performs DMVR in each sub-block unit to correct the motion information of the sub-block again. This can be done, and this can be called MP-DMVR (Multi-pass DMVR).

The LIC (Local Illumination Compensation) method is a method of compensating for luminance changes between blocks. It derives a linear model using surrounding pixels adjacent to the current block, and then compensates for the luminance information of the current block through the linear model.

Existing video coding methods perform motion compensation considering only horizontal, vertical, and horizontal movements, so coding efficiency deteriorates when coding videos that include movements such as enlargement, reduction, and rotation that are commonly encountered in reality. To express such movements for enlargement, reduction, and rotation, an Affine model-based motion prediction technology that uses a 4 (rotation) or 6 (enlargement, reduction, rotation) parameter model can be applied.

BDOF (Bi-Directional Optical Flow) is used to correct the prediction block by estimating the amount of pixel change based on optical-flow from the reference block of the block composed of bi-directional movement. The motion of the current block can be corrected using motion information derived from the BDOF of this VVC.

PROF (Prediction refinement with optical flow) is a technology to improve the accuracy of sub-block-level affine motion prediction to be similar to the accuracy of pixel-level motion prediction. PROF, similar to BDOF, is a technology that obtains the final prediction signal by calculating correction values in pixel units for affine motion compensated pixel values in sub-block units based on optical flow.

When generating a prediction block for the current block, the CIIP (Combined Inter-/Intra-picture Prediction) method performs a weighted average of the prediction blocks generated by the intra-picture prediction method and the prediction blocks generated by the inter-picture prediction method to create the final prediction block. This is a method to create .

The IBC (Intra Block Copy) method is a method that finds the part most similar to the current block in an already reconstructed area in the current picture and uses the corresponding reference block as a prediction block for the current block. At this time, information related to the block vector, which is the distance between the current block and the reference block, may be included in the bitstream. The decoder can calculate or set the block vector for the current block by parsing information related to the block vector contained in Beaststream.

The BCW (Bi-prediction with CU-level Weights) method does not generate a prediction block by averaging two prediction blocks that have been motion-compensated from different reference pictures, but applies weights adaptively on a block-by-block basis to compensate for motion. This is a method of performing a weighted average on two prediction blocks.

The MHP (Multi-hypothesis prediction) method is a method of performing weight prediction using various prediction signals by transmitting additional motion information to unidirectional and bidirectional motion information when predicting between screens.

CCLM (Cross-component linear model) is a method of constructing a linear model using the high correlation between a luminance signal and a chrominance signal located at the same location as the luminance signal, and then predicting the chrominance signal through the linear model. After constructing a template using restored blocks among neighboring blocks adjacent to the current block, parameters for the linear model are derived through the template. Next, depending on the video format, the restored current luminance block is selectively down-sampled to fit the size of the chrominance block. Finally, the chrominance block of the current block is predicted using the down-sampled luminance block and the corresponding linear model. At this time, the method of using two or more linear models is called MMLM (Multi-model Linear mode).

In independent scalar quantization, the restored coefficients t' _k for input coefficients t _k depend only on the associated quantization index q _k . That is, the quantization index for any restored coefficient has a different value from the quantization indexes for other restored coefficients. At this time, t' _k may be a value including the quantization error at t _k and may be different or the same depending on the quantization parameter. Here, _t'k may be named a restored transform coefficient or a dequantized transform coefficient, and the quantization index may be named a quantized transform coefficient.

In Uniform Reconstruction Quantizers (URQ), the reconstructed coefficients have the characteristic of being arranged at equal intervals. At this time, the distance between two adjacent restored values can be referred to as the quantization step size. The restored values may include 0, and the entire set of available restored values may be uniquely defined depending on the quantization step size. The quantization step size may vary depending on the quantization parameter.

In the existing method, the set of allowable restored transform coefficients decreases due to quantization, and the number of elements of this set may be finite. Because of this, there is a limit to minimizing the average error between the original image and the restored image. Vector quantization can be used as a method to minimize this average error.

A simple form of vector quantization method used in video encoding is sign data hiding. This is a method in which the encoder does not encode the sign of one non-zero coefficient, and the decoder determines the sign for the corresponding coefficient depending on whether the sum of the absolute values of all coefficients is even or odd. To this end, at least one coefficient may be increased or decreased by '1' in the encoder, and at least one coefficient is selected to be optimal in terms of cost for rate-distortion, so that the value is It can be adjusted. As an example embodiment, a coefficient having a value close to the boundary of the quantization interval may be selected.

Another vector quantization method is Trellis-Coded Quantization, and in video coding, it is used as an optimal path search technique to obtain an optimized quantization value in dependent quantization. On a block basis, quantization candidates for all coefficients within the block are placed in a trellis graph, and the optimal trellis path between optimized quantization candidates is taken into consideration the cost of rate-distortion. and explore. Specifically, dependent quantization applied to video encoding may be designed such that the set of allowable restored transform coefficients for a transform coefficient depends on the value of the transform coefficient that precedes the current transform coefficient in reconstruction order. At this time, by selectively using multiple quantizers according to the transformation coefficient, the average error between the original image and the restored image is minimized, thereby increasing coding efficiency.

Among intra prediction coding technologies, the MIP (Matrix Intra Prediction) method is a matrix-based intra prediction method. Unlike prediction methods that have directionality from pixels of neighboring blocks adjacent to the current block, pixels on the left and top of neighboring blocks are used as a predefined matrix. This is a method of obtaining a prediction signal using the and offset values.

In order to derive the intra prediction mode of the current block, based on a template, which is a random area restored while adjacent to the current block, the intra prediction mode for the template derived through the surrounding pixels of the template is used to restore the current block. It can be used for. First, the decoder can generate a prediction template for the template using surrounding pixels (references) adjacent to the template, and use the intra prediction mode, which generates a prediction template most similar to the already restored template, to restore the current block. This method can be called TIMD (Template intra mode derivation).

In general, an encoder can determine a prediction mode for generating a prediction block and generate a bitstream containing information about the determined prediction mode. The decoder can set the intra prediction mode by parsing the received bitstream. At this time, the bit amount of information about the prediction mode may be about 10% of the total bitstream size. In order to reduce the bit amount of information about the prediction mode, the encoder may not include information about the intra prediction mode in the bitstream. Accordingly, the decoder can derive (determine) an intra prediction mode for restoration of the current block using the characteristics of the surrounding blocks, and can restore the current block using the derived intra prediction mode. At this time, in order to derive the intra prediction mode, the decoder applies a Sobel filter horizontally and vertically to each neighboring pixel (pixel) adjacent to the current block to infer directionality information, and then converts the directionality information into the intra prediction mode. A mapping method can be used. The method by which the decoder derives an intra prediction mode using neighboring blocks can be described as DIMD (Decoder side intra mode derivation).

Figure 7 is a diagram showing the positions of neighboring blocks used to construct a motion candidate list in inter prediction.

Surrounding blocks may be blocks in a spatial location or blocks in a temporal location. Surrounding blocks that are spatially adjacent to the current block are Left (A1) block, Left Below (A0) block, Above (B1) block, Above Right (B0) block, or Above Left. , B2) It can be at least one of the blocks. The neighboring block temporally adjacent to the current block may be a block containing the upper left pixel position of the bottom right (BR) block of the current block in the corresponding picture (Collocated picture). If a neighboring block temporally adjacent to the current block is encoded in intra mode or a neighboring block temporally adjacent to the current block exists in an unusable position, the horizontal and vertical dimensions of the current block in the picture corresponding to the current picture (Collocated picture) A block containing the center (Ctr) pixel position of can be used as a temporal neighboring block. Motion candidate information derived from the corresponding picture may be referred to as TMVP (Temporal Motion Vector Predictor). Only one TMVP can be derived from one block, and after dividing one block into several sub-blocks, each TMVP candidate can be derived for each sub-block. The TMVP derivation method on a sub-block basis may be referred to as sbTMVP (sub-block Temporal Motion Vector Predictor).

Whether the methods described herein will be applied depends on slice type information (e.g., whether it is an I slice, a P slice, or a B slice), whether it is a tile, whether it is a subpicture, the size of the current block, the depth of the coding unit, and the current block. It may be determined based on at least one of information about whether it is a luminance block or a chrominance block, whether it is a reference frame or a non-reference frame, reference order, and temporal hierarchy according to the hierarchy. Information used to determine whether the methods described in this specification will be applied may be information previously agreed upon between the decoder and encoder. Additionally, this information may be determined according to profile and level. This information can be expressed as variable values, and the bitstream can include information about the variable values. That is, the decoder can determine whether the above-described methods are applied by parsing information about variable values included in the bitstream. For example, it may be determined whether the above-described methods will be applied based on the horizontal or vertical length of the coding unit. If the horizontal or vertical length is 32 or more (e.g., 32, 64, 128, etc.), the above-described methods can be applied. Additionally, the above-described methods can be applied when the horizontal or vertical length is less than 32 (e.g., 2, 4, 8, 16). Additionally, the above-described methods can be applied when the horizontal or vertical length is 4 or 8.

Figure 8 is a diagram showing a method for correcting motion information according to an embodiment of the present invention.

Figure 8(a) shows the process of revising motion information derived from neighboring blocks of the current block and outputting new motion information. Referring to FIG. 8(a), the decoder can obtain corrected motion information by correcting motion information about neighboring blocks of the current block using various motion correction methods. Referring to FIG. 8(b), the decoder derives a motion candidate list from neighboring blocks of the current block, then corrects one or more motion candidates in the derived motion candidate list using various motion correction methods to obtain a corrected motion candidate list. You can. The motion candidate list can be constructed using motion information derived from neighboring blocks of the current block. The decoder may obtain a corrected motion candidate list including one or more corrected motion candidates by performing a motion correction process on each or all of the one or more motion candidates in the motion candidate list. Referring to FIG. 8(c), the decoder can obtain corrected motion information by correcting the initial motion information for the current block using various motion correction methods.

Figure 9 is a diagram illustrating a method of correcting motion information for a current block by recursively performing a motion correction method according to an embodiment of the present invention.

The decoder can correct the initial motion information for the current block by recursively performing a motion correction method. The decoder may construct a motion candidate list for the current block using neighboring blocks of the current block, and may correct motion information by recursively performing one or more motion correction methods. At this time, one or more motion correction methods include MVD (Motion Vector Difference), TM (Template Matching), BM (Bilateral Matching), MMVD (Merge mode with MVD), MMVD (Merge mode with MVD)-based TM, and Optical flow-based TM. , Multi pass DMVR, etc. MVD may be a method in which the encoder generates a correction value for the motion information by including it in a bitstream, and the decoder obtains the correction value for the motion information through the bitstream to correct the motion information (MV difference value in FIG. 9 correction). TM may be a method in which a decoder constructs a template based on surrounding pixels of the current block, finds a matching area with the highest similarity to the constructed template, and corrects motion information. BM may be a method in which a decoder corrects motion information based on the similarity between a reference block in a picture included in an L0 picture list derived based on motion information of the current block and a reference block in a picture included in an L1 picture list. The MMVD method is a method of correcting motion information using one of one or more motion difference value candidates. The encoder may generate a bitstream including information about an index representing one of one or more motion difference value candidates. The decoder may parse information about the index included in the bitstream, obtain a difference value candidate indicated by the index, and correct motion information based on the obtained difference value candidate. In the MMVD-based TM method, an extended motion candidate list consisting of a motion candidate list and one or more motion difference value candidates is reordered based on the TM cost value, and motion information of the current block is obtained using the motion candidates in the reordered list. This is a way to correct. The encoder may generate a bitstream containing information about an index indicating one of the candidates in the reordered list. The decoder can parse information about the index included in the bitstream and use the motion correction candidate indicated by the index as the correction value of the motion information of the current block. The optical flow-based TM method may be a method in which the decoder configures a template of an area adjacent to the current block into an optical flow map, finds an area similar to the optical flow map of the reference picture, and corrects motion information.

One or more motion compensation methods may be applied in merge or AMVP mode. Referring to FIG. 9, the decoder can derive (construct) a motion candidate list (eg, merge candidate list) for the current block. Additionally, the decoder may correct each or all of the motion information (e.g., merge candidates) in the motion candidate list using one or more motion correction methods. The decoder may rearrange the motion candidate list based on the cost value for the corrected motion information. As described above, the decoder may perform the above-described correction method on each or all of the motion candidates in the rearranged motion candidate list and rearrange the motion candidate list. That is, the decoder can recursively perform correction for the above-described motion candidates and rearrangement of the motion candidate list. When this method is applied, the accuracy of motion information for motion candidates in the motion candidate list increases, and as a result, the residual signal can be reduced, which may have the effect of reducing the amount of bits for the residual signal. The decoder can separately receive an index indicating one of the motion candidates in the reordered motion candidate list and predict (restore) the current block based on the motion candidate indicated by the index. Alternatively, the decoder may select the motion candidate with the lowest cost value and predict (restore) the current block based on the motion candidate with the lowest cost value.

In order to increase the accuracy of the merge candidate, the encoder may generate a bitstream including information about an index indicating one of the correction values of motion information obtained using the MMVD (Merge mode with MVD) method, and the decoder may generate a bitstream The correction value of the motion information can be obtained through the index obtained by parsing the information about the index included in the bitstream, and the correction value of the motion information can be used to predict (restore) the current block. The MMVD method is a method of selecting one of a plurality of motion difference value candidates. Although it is less accurate than accurately transmitting the existing motion information difference value, it has the effect of saving a large amount of bits. In order to slightly increase accuracy, the decoder adds TM, BM, MMVD (Merge mode with MVD), and MMVD (Merge mode with MVD) to the first corrected motion information, which is corrected based on the correction value of the motion information obtained using the MMVD method. The second corrected motion information can be obtained by additionally applying at least one of the TM-based TM, Optical flow-based TM, and Multi pass DMVR methods. Alternatively, the encoder corrects the motion information by applying at least one method among TM, BM, MMVD (Merge mode with MVD), MMVD (Merge mode with MVD)-based TM, Optical flow-based TM, and Multi pass DMVR method. Afterwards, a bitstream can be generated by additionally including information on correction values for motion information obtained by applying at least one of the above methods.

For AMVP, the difference value of motion information may be included in the beast stream. The decoder can generate a prediction block for the current block using the difference value of the motion information included in the bitstream. Since the difference value of motion information is included in the bitstream, there is a problem that the bit amount increases. For this purpose, the above-described method can also be applied to the AMVP candidate list. In other words, each or all of the one or more candidates in the candidate list obtained using AMVP are TM, BM, MMVD (Merge mode with MVD), MMVD (Merge mode with MVD)-based TM, Optical flow-based TM, and Multi pass DMVR. It can be corrected based on at least one method. Since the corrected motion information is used, the bit amount for the difference value of the motion information included in the actual bit stream is reduced.

The MVD method is performed first, and then a correction method using at least one of TM, BM, MMVD (Merge mode with MVD), MMVD (Merge mode with MVD)-based TM, Optical flow-based TM, and Multi pass DMVR can be performed. there is. Alternatively, the encoder can use MVD to generate a bitstream containing correction values for the initial motion information. The decoder may perform the above-described motion correction method based on the correction value for the initial motion information included in the bitstream.

Below, MVD (Motion Vector Difference), TM (Template Matching), BM (Bilateral Matching), MMVD (Merge mode with MVD), MMVD (Merge mode with MVD)-based TM, Optical flow-based TM, Multi pass DMVR, etc. Explains how to apply it.

The motion correction method to be used may be determined based on the encoding mode (prediction mode) of the current block. For example, if the current block is encoded in GPM mode, at least one of TM, BM, and optical flow-based TM methods may be performed on the MV difference value corrected using MMVD. For example, if the current block is encoded in AMVP mode, at least one of TM, BM, and optical flow-based TM methods may be performed on the MV difference value corrected using merge mode MMVD. At this time, MVD in AMVP mode may not be applied.

For example, if the motion information of neighboring blocks adjacent to the current block is the same or similar to each other, correction for the MV difference value is not performed, and at least one of TM, BM, and optical flow-based TM methods may be performed. This is because the movement of the current block is likely to be similar to the movement of surrounding blocks. Meanwhile, if the distribution of motion information of neighboring blocks adjacent to the current block is not similar to each other, at least one of TM, BM, and optical flow-based TM methods may be performed on the MV difference value corrected using MVD or MMVD. . This is because the movement of the current block may be different from the surrounding blocks.

For example, the size of the current block, whether the current block is a luminance component block or a chrominance component block, quantization parameter information for the current block, motion resolution information of the current block, whether a differential signal exists in the current block, current block A motion compensation method (e.g., Motion Vector Difference (MVD), Template Matching (TM), Bilateral Matching (BM), Optical flow-based TM, Multi pass DMVR, etc.) can be selected. If the size of the current block is larger than a certain size or the motion resolution of the current block is 1/16 pixel unit, the TM method may not be selected. This is because the TM method has high complexity. If the current block is a chrominance component block, the TM method is not performed, and motion information corrected through the TM method can be used in the luminance component block of the current block. For example, motion information corrected through the TM method in the luminance component block of the current block may be scaled and used in the chrominance block of the current block.

For example, depending on the characteristics of the current block, a motion correction method (e.g., Motion Vector Difference (MVD), Template Matching (TM), Bilateral Matching (BM), Merge mode with MVD (MMVD), Merge mode with MVD (MMVD) TM-based TM, Optical flow-based TM, Multi pass DMVR, etc.) can be selected. This is because there is a trade-off between complexity and accuracy for each motion compensation method. For example, TM has high complexity and is not capable of parallel processing but shows the highest performance, BM has lower performance than TM but allows parallel processing, and Optical flow has low complexity and is capable of parallel processing but has poor performance. It has the disadvantage of being low. At this time, the selected motion correction method may be signaled separately. For example, the decoder can determine the motion correction method based on syntax elements included in the bitstream. At this time, the syntax element may be signaled at the SPS level, PPS level, picture level, slice level, or CU (Coding Unit) level.

Figure 10 is a diagram showing the order in which the TM method is performed according to an embodiment of the present invention.

Referring to FIG. 10, the decoder can obtain initial motion information (initial MV, reference index) derived from neighboring blocks. The decoder can set a search area within the reference image based on the initial motion information. The decoder can select several candidate locations within the search area according to a predefined search pattern. The decoder may construct a template for the current block using neighboring blocks of the current block, and may construct a reference image template of the same size as the template for the current block based on the candidate location. The decoder can obtain the cost value between the template for the current block and the reference video template. The cost value can be obtained through Sum of Absolute Differences (SAD) or Mean-Removed SAD (MRSAD). The decoder can obtain cost values for all candidate positions in the search area and use information on the motion candidate at the position corresponding to the minimum cost value as final motion information (improved motion information in FIG. 10). In this specification, the meaning of obtaining a cost value may be the same as the meaning of the decoder calculating the cost value.

Figure 11 is a diagram illustrating a method for setting a search area for a TM method based on initial motion information according to an embodiment of the present invention.

Referring to FIG. 11, in order to find a reference block corresponding to the current block in the reference picture, the decoder may set the position moved by the initial motion information (initial MV) based on the upper left position of the current block as the position of the reference block. The decoder can set a search range of an arbitrary (m x n) size based on the upper left position of the reference block. At this time, m x n may be 16 x 16. For example, the search area may range from -8 to 8 in the horizontal direction and from -8 to 8 in the vertical direction, based on the position of the initial motion information. Specifically, if the position indicated by the initial motion information is expressed in the form of horizontal and vertical coordinates, it may be (x, y). At this time, the horizontal coordinates of the search area may range from x-8 to x+8, and the vertical coordinates may range from y-8 to y+8. Afterwards, the decoder can construct a left template of the current block and an upper template of the current block using blocks adjacent to the current block. Additionally, the decoder configures the left template of the reference block and the upper template of the reference block based on the position of the reference block set in the reference picture. At this time, the size of the left template of the current block and the left template of the reference block may be the same, and the sizes of the upper template of the current block and the upper template of the reference block may be the same.

Figure 12 is a diagram showing the location of a motion candidate searched within a search area according to an embodiment of the present invention.

Referring to FIG. 12, the position of the motion candidate searched within the search area for the TM method may be set based on the position of the initial motion information (center point in FIG. 12). The location where the motion candidate is searched may vary depending on the search pattern. Search patterns may include DIAMOND patterns, CROSS patterns, etc. In FIG. 12, ◇ may indicate a position where a motion candidate is searched according to a diamond pattern, and + may indicate a position where a motion candidate is searched according to a cross pattern. The spacing of the patterns may be set to become wider or narrower as the distance from the initial motion information increases.

Figure 13 is a diagram showing the process of searching for the location of a motion candidate according to an embodiment of the present invention.

Specifically, Figure 13 is a flowchart showing the search process for the location where motion candidates for the TM method are searched. Referring to FIG. 13, the decoder can obtain (calculate) a pixel-based cost value for initial motion information. If the number of repetitions of search performance is '0', the decoder can terminate without performing the TM method. Otherwise (i.e., if the number of iterations of search performance is not '0'), the decoder may perform the TM method. At this time, the number of repetitions may be any integer value of 1 or more. Additionally, the number of repetitions, initial search pattern, and initial search interval can be set before the search performance process described with reference to FIG. 13 is performed. For example, the number of repetitions can be set to '375', the initial search pattern can be set to 'DIAMOND', and the initial search interval can be set to '6'.

Next, the search pattern and search interval can be reset. The search pattern and search interval are at least one of the following: the size of the current block, whether the current block is a luminance component block or a chrominance component block, the current motion resolution, the number of repetitions, and the distribution of cost values for the motion candidate positions calculated in the previous iteration step. It can be decided based on. Below, we will explain how to set the search pattern and search interval.

The search interval may be determined according to the motion information resolution of the current block. The motion information resolution may be in units of 1 integer pixel, 4 integer pixel, 1/2 integer pixel, 1/4 integer pixel, and 1/16 integer pixel. If the motion information resolution is 1/4 integer pixels, the initial search interval may be set to 6, and in other cases, the initial search interval may be set to 4.

The search pattern may be determined as a diamond pattern or a cross pattern. The search interval may be adjusted by decreasing or increasing an arbitrary interval from the initial search interval. For example, search patterns and search intervals may vary depending on the iteration step. The repetition step may indicate how many times the reset of the search pattern and search interval is repeated if the repetition number is not 0. In other words, the search pattern and search interval may vary depending on the number of repetition steps. For example, in the first iteration step, the search pattern and search interval can be set to a diamond search pattern and an initial search interval. In the second step, the search pattern and search interval can be set to a cross pattern and a search interval reduced by 1 from the initial search interval. In stages after the second stage, the search pattern and search interval can be set to a cross pattern and a search interval reduced by 1 compared to the previous stage.

The search pattern and search interval can be set according to the color component of the current block. The search pattern and search interval for the chrominance component may be set wider than the search pattern and search interval for the luminance component. This is because the spatial correlation between the chrominance component (signal) and the luminance component (signal) is high. Alternatively, to improve performance, the search pattern and search interval for the chrominance component may be set shorter than the search pattern and search interval for the luminance component.

The search pattern and search interval can be set based on the size of the current block. If the size of the current block is larger than a certain value, a cross pattern is used as the search pattern and the search interval can be set wider than the initial search interval. For example, the search interval may be '7'. Alternatively, to improve performance, if the size of the current block is larger than a certain number, a diamond pattern may be used as the search pattern and the search interval may be set shorter than the initial search interval. For example, the search interval may be '5'. The size of the current block can be 16x16 or 32x32, and the search pattern and search interval can be set based on the sum of the horizontal and vertical sizes of the current block.

Next, the decoder can set a correction value (offset) for the position of the searched motion candidate using the search pattern and search interval and perform evaluation on the searched motion candidates. Evaluation in this specification may mean obtaining a cost value. The location where the motion candidate is searched may vary depending on the search pattern. For example, if the search pattern is a cross pattern, the correction values are (0, 1), (1, 0), (0, -1), (-1, 0), and if the search pattern is a diamond pattern, Correction values are (0, 2), (1, 1), (2, 0), (1, -1), (0, -2), (-1, -1), (-2, 0), It is (-1, 1). At this time, the correction value (x, y) may be (horizontal, vertical), where x may be a correction value for the horizontal direction and y may be a correction value for the vertical direction.

The method described with reference to FIG. 13 can be performed recursively as many as an arbitrary number of repetitions. For example, if the number of iterations is 1, it can be performed more than once. If the decoder searches for motion candidates and evaluates all of the searched motion candidates, it can use the motion information of the motion candidate corresponding to the smallest cost value as the final motion information.

The initial motion candidate in FIG. 13 may be reset based on the smallest cost value in the previous iteration step. The initial motion candidate for the next step may be reset based on the motion candidate corresponding to the smallest cost value in the first step. For example, if the motion candidate corresponding to the smallest cost value in the first step is the motion candidate located in the upper left corner, the decoder may perform evaluation on the motion candidate adjacent to the motion candidate located in the upper left corner in the next iteration step.

The search performance process described with reference to FIG. 13 may proceed differently depending on whether the current block is a coding unit (block) or a sub-block (sub-coding block).

If the current block is a coding block and the AMVP mode is applied to the current block, an L0 motion candidate list for L0 prediction and an L1 motion candidate list for L1 prediction of the current block may be derived. A search process may be performed on some or all motion candidates of the derived candidate list to derive corrected motion information. Meanwhile, when the current block is a coding block and Merge mode is applied to the current block, one motion candidate list for L0 and L1 prediction of the current block can be derived. A search process may be performed on some or all motion candidates in one derived motion candidate list. L0 described in this specification may mean L0 prediction, and L1 may mean L1 prediction.

L0 unidirectional prediction, L1 unidirectional prediction, or bidirectional prediction may be applied to the current block. Whether L0 unidirectional prediction, L1 unidirectional prediction, or bidirectional prediction is applied to the current block may be indicated by reference direction indication information. Reference direction information may be reset based on the cost value. For example, bidirectional prediction is performed using the cost value of the prediction block generated using the initial motion information of L0, the cost value of the prediction block generated using the initial motion information of L1, and the initial motion information of L0 and L1. Cost value of the prediction block generated by weighted average of two prediction blocks, cost value of the prediction block generated using the corrected motion information of L0, cost value of the prediction block generated using the corrected motion information of L1, By performing bidirectional prediction through the corrected movements of L0 and L1, the motion information and reference direction information corresponding to the smallest cost value among the cost values of the prediction blocks generated by weighting the two prediction blocks are reset to the current block. You can.

The current coding block may be divided into several sub-blocks. The initial motion information of each sub-block may be reset to corrected motion information according to the search performance process. The template may be different for each sub-block, and a pixel of an adjacent sub-block may be used as a template. However, since the decoder can perform a search for the next sub-block only when an adjacent sub-block is restored, there is a problem in that the search process is not processed in parallel for each sub-block. To this end, the search process can be performed only for sub-blocks located at the boundary of the current block. Alternatively, for sub-blocks located on the boundary of the current block, the decoder derives corrected motion information using the TM method, and for sub-blocks not located on the boundary of the current block, the decoder uses BM, optical flow-based TM, and multi-pass DMVR methods. Corrected motion information can be derived using any one or more of these.

If the current block is processed as a coding block, the decoder can calculate the cost value for the entire current block using the initial motion information for L0 and L1 and derive the corrected motion information based on the calculated cost value. . At this time, the movement of the lower right part of the block processed as a coding block may be slightly different from the overall movement of the coding block. Depending on how the template is constructed, the search performance process may vary, and the corrected motion information may also vary. Therefore, even if the current block is processed as a coding block, the corrected motion information can be derived in sub-block units based on the cost value for the template configured based on the sub-block.

Figure 14 is a diagram showing a process for evaluating search candidates according to an embodiment of the present invention.

Specifically, FIG. 14 is a flowchart showing a process of evaluating search candidates using correction values for candidate locations selected according to the search pattern and search interval described with reference to FIG. 13. The decoder may store initial motion information as final motion information. And the decoder can perform the process described later on correction values for all search candidates.

The decoder can select one of the correction values for the candidate location to search. The correction value can be reset to suit the motion resolution to be used in the current block. The decoder may reconstruct the motion information to be evaluated by adding the reset correction value to the initial motion information. A cost value can be obtained based on the reconstructed motion information. At this time, the cost value obtained based on the motion information is calculated by adding the difference between the absolute values of the horizontal components and the absolute values of the vertical components between the initial motion information and the reconstructed motion information, and then multiplying by an arbitrary weight value. can be calculated. An arbitrary weight value may be '4'. The decoder can calculate the pixel (pixel)-based cost value for the reconstructed motion information only when the cost value obtained based on the motion information is smaller than the cost value of the initial motion information obtained based on the pixel (pixel).

After evaluating correction values for all search candidates, the decoder may set the motion information corresponding to the smallest cost value as the final motion information.

Each of the cost values obtained based on the motion information may be obtained through a difference value between the motion information obtained by adding a correction value to each of a plurality of motion candidates corresponding to a position selected according to the initial motion information. The cost value obtained based on motion information may vary depending on the size of the correction value. That is, the smaller the correction value, the smaller the cost value can be. Since the motion information corresponding to the smallest cost value is set as the final motion information, evaluation can be performed only on motion candidates surrounding the position indicated by the initial motion information with a small correction value. However, a motion candidate with a large correction value may be the optimal motion candidate. Therefore, in order to select the optimal motion candidate by evaluating various motion candidates, the cost value can be obtained using a method described later. The cost value can be obtained using the difference between the motion information values of each neighboring block, the quantization parameter, the size of the current block, etc.

The cost value can be obtained using the distribution of motion information of neighboring blocks. To evaluate various motion candidates, the decoder can obtain a cost value using the difference value between the corrected motion information and the motion information of neighboring blocks. For example, a cost value can be obtained according to the comparison result by comparing the difference value between the corrected motion information and the motion information of the surrounding block with an arbitrarily determined value. Specifically, if the difference value between the corrected motion information and the motion information of the surrounding block is greater than (or less than, or equal to) a certain value, the decoder can obtain a cost value. At this time, the neighboring block may be a neighboring block adjacent to the current block or a temporal neighboring block located at the same position as the current block in the corresponding picture (Collocated picture).

The cost value can be obtained based on the size of the current block. For example, the weight for obtaining the above-described cost value may be set according to the size of the current block. The weight can be set in inverse proportion to the size of the current block. That is, the larger the size of the current block, the lower the weight can be set. This is to evaluate a wider range of motion candidates in order to select an appropriate motion candidate. Meanwhile, the weight can be set proportional to the size of the current block. That is, the larger the size of the current block, the higher the weight can be set. This is to reduce complexity. For example, the size of the current block can be 16x16, 32x32, and can be set to the sum of the horizontal and vertical sizes of the current block. Weights can be integer values such as 1, 2, 3, 4, 5, 6, etc. Additionally, as the weight increases, the cost value increases, so if the weight is greater than a certain value, the decoder may not perform evaluation to obtain the cost value.

Figure 15 is a diagram showing movement characteristics of the boundary portion of an object in the current picture according to an embodiment of the present invention.

In general, the motion information of the current block has a high correlation with the motion information of surrounding blocks. However, if the current block is located at the border of an object, the current block may be a background area. In this case, the movement of surrounding blocks may be different from the movement of the current block. Additionally, in the case of an image captured without changing the position (viewpoint) of the camera, the background does not move, so the motion information about the background may be a zero vector (0, 0). The decoder can search for motion candidates for TM using motion information of neighboring blocks. At this time, if the current block corresponds to the background, searching for the zero vector may be more effective than searching for motion candidates using motion information of surrounding blocks. Therefore, when TM is performed, a zero vector may be included in the searched motion candidate.

In the case of an image captured while the camera moves at a constant speed, the motion information about the background may have a constant value (global motion). In this case, global motion candidates may be included in the motion candidates searched for TM.

The reference picture for the current picture may be configured differently depending on the environment in which the content is used. For example, in a real-time broadcasting environment, a reference picture such as low delay B with a low delay rate can be used. That is, only decoded pictures located in the past in the picture reproduction time order can be used as reference pictures. Additionally, since random access is important in streaming services such as Video on Demand (VoD), a reference picture configuration such as random access in which I and P pictures are encoded in GOP (Group of Picture) units can be used. That is, decoded pictures located in the past and future in the picture reproduction time order can be used as reference pictures. Since the method of configuring a reference picture for the current picture is signaled through a slice header, the decoder can already check pictures to be used as reference pictures when encoding the current picture. At this time, the reference picture used on a block basis may vary, and information on which reference picture is used may be signaled on a block basis.

Figure 16 is a diagram showing a method of determining a reference picture using template matching according to an embodiment of the present invention.

A reference picture is a picture that contains the block most similar to the current block, and the decoder can find the reference picture through template matching. Information related to the reference picture is not included in the bitstream and can be derived through template matching.

First, the decoder can select reference picture candidates to be evaluated and motion candidates to be evaluated. Next, the decoder can evaluate motion candidates for each reference picture. At this time, each motion candidate may be scaled based on the distance between the reference picture and the current picture and the distance between the current picture and the reference picture of the motion candidates (i.e., Picture Order Count (POC) difference). The decoder may perform evaluation on the scaled motion candidates. If the evaluation of the combination of all reference picture candidates and all motion candidates has been completed, the motion information of the motion candidate of the reference picture corresponding to the smallest cost value may be the final motion information. For example, if the current picture is a B picture (when two-way prediction is used), the decoder uses the motion information of the motion candidate of the reference picture corresponding to the smallest cost value and the reference picture corresponding to the second smallest cost value. Bidirectional motion information can be constructed using the motion information of the motion candidate. Alternatively, final motion information can be obtained using the motion candidate of each reference picture with the smallest cost value among the reference pictures in each reference picture list. The number of reference pictures in the reference picture list may be limited in terms of complexity. At this time, the limited number of reference pictures can be set through separate signaling.

In another method, if the evaluation of the combination of all reference picture candidates and all motion candidates has been completed, the encoder constructs a list of combination information, performs reordering in ascending order based on the cost value, and then selects the optimal combination information. A bitstream including index information can be created. The decoder can determine a reference picture or motion candidate for the current block based on index information included in the bitstream.

Figure 16(a) shows a reference picture configuration in a random access environment. Referring to FIG. 16(a), the reference block of the first reference picture (reference index 0) among the L0 reference pictures may have the smallest cost value, and the reference block of the second reference picture (reference index 1) among the L1 reference pictures may have the lowest cost value. The reference block may have the lowest cost value. The decoder uses motion information about the reference block of the first reference picture (reference index 0) among L0 reference pictures and motion information about the reference block of the second reference picture (reference index 1) among L1 reference pictures to determine the current block. Movement information can be obtained. At this time, reference picture index information for L0 and L1 may not be signaled through a bitstream but may be derived through template matching.

Figure 16(b) shows the reference picture configuration in a low-delay environment. Referring to FIG. 16(b), the L0 reference picture and the L1 reference picture may be configured identically. Among the L0 reference pictures, the reference block of the second reference picture (reference index 1) may have the smallest cost value, and among the L1 reference pictures, excluding the L0 reference block, the reference block of the third reference picture (reference index 2) This can have the smallest cost value. The decoder uses motion information about the reference block of the second reference picture (reference index 1) among L0 reference pictures and motion information about the reference block of the third reference picture (reference index 2) among L1 reference pictures to determine the current block. Movement information can be obtained. At this time, reference picture index information for L0 and L1 may not be signaled through a bitstream but may be derived through template matching.

The number of reference pictures in the reference picture list to which the above method is applied can be limited. Complexity can be reduced by applying only within a limited number of reference pictures. The limited number can be set to a predetermined value or signaled at the SPS level, PPS level, picture level, slice level, or CU (Coding Unit) level and included in the bitstream, and the decoder acquires a limited number from the bitstream and adapts. In effect, the number of reference pictures in the reference picture list can be changed.

Figures 17 and 18 are diagrams showing the process of performing template matching according to an embodiment of the present invention.

Referring to FIG. 17, motion candidates can first be derived from neighboring blocks of the current block.

The first initial movement candidate can be derived through the upper block of the current block. A position for searching for a motion candidate within a reference picture may be selected based on the first initial motion candidate. The location where the motion candidate is searched may be a location marked with ◇ (diamond pattern) or + (cross pattern). The decoder may perform evaluation in order from the motion candidate at the furthest position from the initial motion candidate to the motion candidate at the closest position. Additionally, the position of the motion candidate corrected in the previous iteration step may be set as the initial motion candidate position of the next iteration step. Among the motion candidates, the motion information of the motion candidate corresponding to the smallest cost value may become the first corrected motion candidate 1710.

Next, a second initial motion candidate can be derived through the block to the left of the current block. The decoder may use template matching to derive the second corrected motion candidate 1720 corresponding to the smallest cost value.

The corrected motion candidate corresponding to the smaller cost value between the cost value of the first corrected motion candidate 1710 and the cost value of the second corrected motion candidate 1720 becomes the final motion candidate for the current block.

The wider the range for searching motion candidates and the more positions searched, the more accurate the final motion candidate can be obtained. Even in the TM described herein, different results may be derived depending on the search range and search location. Referring to FIG. 18, when the distributions of the initial motion information (1801, 1802, 1803) derived from the neighboring blocks of the current block are distant from each other, the searched positions do not overlap with each other, so a diverse and wide range of motion candidates can be evaluated. and the most optimal movement information can be derived.

The decoder can further evaluate motion candidates at the new location. Below, the new location of the motion candidate that is additionally evaluated will be described.

A motion candidate for the center position derived using motion information derived from neighboring blocks of the current block may be additionally evaluated. That is, the decoder can additionally evaluate a motion candidate corresponding to the central position of two or more motion candidates.

The decoder can additionally evaluate motion information derived from neighboring blocks of the current block and motion candidates at different positions. The decoder may set a global search range of a larger area that includes all search ranges for each of the initial motion candidates, and then additionally evaluate motion candidates indicating empty spaces within the global search range. First, the decoder can set areas including all reference blocks indicated by motion candidates derived from neighboring blocks of the current block as the global search range. Alternatively, the decoder may set a pre-arranged range as the global search range based on the upper left pixel position of the current block and the size of the current block. The global search range is divided into four equal sizes, and if the number of motion candidates in each quadrant is less than a certain number, the center position of the quadrant corresponding to the number of motion candidates less than a certain number is used as the new motion candidate position. Once set, the decoder can perform evaluation of the motion candidate at the new motion candidate location.

If the current block is encoded in Merge mode, the decoder can perform additional evaluation on all motion candidates that can be expressed by the MMVD method. MMVD is a distance table for predetermined motion difference values and a sign for the vertical or horizontal direction. This is a method of defining a table and signaling index information about the distance and direction of the difference value. All motion candidates that can be derived through MMVD can be used in TM. At this time, the motion candidate corresponding to the smallest cost value among the motion candidates can be used for prediction of the current block without separate signaling. Alternatively, the encoder constructs a motion candidate list using all motion candidates that can be derived through MMVD, then rearranges the motion candidates in the motion candidate list in ascending order based on the cost value, and then adds them to the index for the optimal motion candidate. A bitstream containing information can be obtained. The decoder can parse information about the index from the bitstream and predict (restore) the current block through the motion candidate indicated by the index.

FIG. 19 is a diagram illustrating a case where initial motion information derived from neighboring blocks of the current block are attached to each other according to an embodiment of the present invention.

If the initial motion information 1901, 1902, and 1903 are close together, some of the search positions set by the initial motion information may overlap. However, the locations where motion candidates are searched are more detailed, so precise evaluation of motion candidates can be performed, which has the effect of making it possible to obtain precise motion information. This can be applied in environments (processes) that require more precise motion information.

The distribution of initial motion information is determined according to the motion information of blocks surrounding the current block. Therefore, when the initial motion information is close to each other or too far apart, the decoder can adjust the search range by modifying the position of the initial motion information. At this time, the distribution of motion information can be determined based on a specific value. For example, if the distance between the motion information pieces is greater than a specific value, the motion pieces of information may be determined to be separated, and if the distance between the motion pieces of information is smaller than a certain value, the motion pieces of information may be determined to be close to each other. At this time, the specific value is a value that varies depending on the motion resolution and may be an integer or decimal value.

20 and 21 are diagrams showing a method of changing initial motion information according to an embodiment of the present invention.

FIG. 20 shows an example of changing the initial motion information to make it closer when the initial motion information is distant, and FIG. 21 shows an example of changing the initial motion information to move it closer when the initial motion information is close. Referring to Figures 20 and 21, the dotted arrows represent the initial motion candidates (information) (Initial MV Candidate 1, Initial MV Candidate 2), and the solid arrows represent the modified motion candidates (information) (Modified MV Candidate 1, Modified MV Candidate 2). represents. The dotted circle represents the search area according to the initial motion candidate, and the solid circle represents the search area according to the changed motion information. By adjusting the horizontal or vertical values of the initial motion information, the motion information can be changed to become closer or farther apart.

Figure 22 is a diagram showing the location where a motion candidate derived through initial motion information is searched according to an embodiment of the present invention.

The decoder can search for motion candidates at positions separated by an arbitrary predetermined interval based on the initial motion information. At this time, any given interval can be set in various ways. Motion candidates of positions that have changed by a certain set interval in the horizontal direction can be searched, motion candidates of positions that have changed by a certain set interval in the vertical direction can be searched, and motion candidates of positions that have changed by a certain set interval in the horizontal and vertical directions can be searched. Movement candidates may be searched. For example, Figure 22(a) shows the basic position where a motion candidate derived through initial motion information is searched. Figure 22(b) shows the position of a motion candidate to be searched that has changed by a certain interval in the vertical direction. Referring to FIG. 22(b), the position where the motion candidate is searched may be reduced by 1/2 in the vertical direction. Figure 22(c) shows the position of the searched motion candidate changed by a certain interval in the horizontal direction. Referring to FIG. 22(c), the position where the motion candidate is searched may be reduced by 1/2 in the horizontal direction.

Figure 23 is a diagram illustrating a method of changing the search location where motion candidates are searched according to an embodiment of the present invention.

Referring to FIG. 23, the location where the motion candidate is searched can be adaptively set using motion information (Initial MV Candidate 1, Initial MV Candidate 2) derived from neighboring blocks of the current block. The search interval and search position for searching motion candidates can be changed to the positions of points where the derived motion information is connected to each other in a straight line. The decoder may change the position where the motion candidate is searched by reducing the first search range for the first initial motion information in at least one of the vertical and horizontal directions. Additionally, the decoder may rotate the second search range for the second motion information to change the position where the motion candidate is searched within the second search range to be located on a straight line with the changed motion candidate search position for the first initial motion information. Afterwards, the decoder may evaluate motion candidates of the changed motion candidate search positions. Likewise, the decoder may change the position where the motion candidate is searched by reducing the second search range for the second initial motion information in at least one of the vertical and horizontal directions. Additionally, the decoder may rotate the first search range for the first motion information to change the position where the motion candidate is searched within the first search range to be located on a straight line with the changed motion candidate search position for the second initial motion information. Afterwards, the decoder may evaluate motion candidates of the changed motion candidate search positions. Even if the position where the motion candidate is searched is within the search range, if it is located outside the picture, the motion candidate located outside the picture is not evaluated. If the search range changes (reduces or increases), the location where the motion candidate is searched may not match the motion resolution. In this case, the position where the motion candidate is searched may be adjusted to match the nearest motion resolution.

Since TM uses a cost value obtained using neighboring pixels (pixels) adjacent to the current block, neighboring blocks of the current block must be restored. Therefore, since the current block and surrounding blocks are dependent on each other, TM cannot be processed in parallel, which is a problem in processing speed. To solve this problem, it is necessary to remove the dependency between the current block and neighboring blocks. Specifically, in order to remove dependency, the decoder does not use the surrounding pixels of the current block to obtain the cost value, but can derive the motion information of the surrounding blocks on a pixel basis and obtain the cost value using the motion information. . In this case, the motion information of neighboring blocks can be restored in parallel as the decoder parses the bitstream, which has the effect of enabling parallel restoration between blocks. That is, the decoder can perform block restoration using TM in parallel.

Figure 24 is a diagram showing TM based on the cost value of a motion vector according to an embodiment of the present invention.

Referring to FIG. 24, a search area and location for searching a motion candidate can be set using initial motion information (Initial MV Candidate) derived from neighboring blocks of the current block. In order to obtain the cost value of the motion vector, a pixel-level optical flow MV map can be created using motion information around the reference block indicated by the initial motion information. Even for surrounding templates adjacent to the current block, a pixel-level optical flow MV template can be created through surrounding motion information. The decoder can obtain the optical flow MV reference template that is most similar to the current block MV template among the positions for motion candidate search and use it as the corrected motion information. Cost values between MV templates can be obtained in various ways. For example, the cost value between MV templates can be calculated through the absolute sum of the MV differences in the horizontal or vertical direction. In order to reduce complexity, the MV template may be derived in units of 2x2 or 4x4 rather than units of pixels.

Below, a method for correcting motion information using DMVR will be described.

Figures 25 and 26 are diagrams showing a method of correcting motion information using DMVR according to an embodiment of the present invention.

DMVR is a method of obtaining corrected motion information for the current block using the BM (Bilateral Matching) method. The BM (Bilateral Matching) method corrects the initial motion information by finding the most similar parts in the surrounding search area of the L0 reference block and the surrounding search area of the L1 reference block in blocks with bidirectional movement, and corrects the corrected movement. This is a method of using information to predict the current block. The size of the search area can be set to an arbitrary (m x n) size based on a specific point of the reference block. For example, the specific point may be the upper left location of the reference block or the center location of the reference block, and an arbitrary size may be 16 x 16. The most similar part may be the point corresponding to the smallest cost value by calculating the pixel-unit cost value between each block. The cost value can be calculated using SAD (Sum of Absolute Differences) or MRSAD (Mean-Removed SAD). The cost value may vary depending on the search area, and the corrected motion information may also vary. The decoder can divide the current block into a plurality of sub-blocks and correct motion information using DMVR for each sub-block. This is because the motion information for small-sized blocks is more accurate than for large-sized blocks. At this time, DMVR is not performed on large blocks, and DMVR can be performed only on small divided blocks (eg, sub-blocks). Referring to FIG. 26, one block can be divided into four sub-blocks. The decoder can obtain corrected motion information using DMVR for each divided sub-block. Alternatively, the decoder can use a multiple DMVR method that uses the corrected motion information found through DMVR in the large block to derive more accurate motion information through DMVR in the divided small blocks.

The multiple DMVR method is described below.

After the parsing step for the current coding block and before starting decoding for the current coding block, the decoder determines whether to perform multiple DMVR correction by judging the conditions described later in the step of deriving motion information for the current coding block. You can. The conditions for performing multiple DMVR are i) when the motion in merge mode is bidirectional, ii) when the distance between reference pictures and the current picture is the same, iii) when a motion difference value in merge mode exists, iv) weight between reference blocks. This may be a block to which prediction has not been applied. The decoder can perform multiple DMVR when any one of conditions i) to iv) is satisfied.

If the motion difference between the first merge motion candidate and other merge candidates in the merge candidate list of the current coding block is smaller than an arbitrary threshold, multiple DMVR may not be performed. Additionally, when there are a plurality of merge motion candidates different from the first merge motion candidate in the merge candidate list, if there is even one case where the motion difference is smaller than an arbitrary threshold, multiple DMVR may not be performed. At this time, among the merge candidates in the merge candidate list, only the merge candidate indicated by an index lower than the index indicating the merge motion candidate of the current coding block can be compared for motion differences. This is to reduce complexity. That is, the motion difference between each of the merge motion candidates in the merge candidate list and the merge candidate of the current coding block is calculated, and if all of the calculated motion differences are greater than a certain specified threshold, the decoder can perform multiple DMVR. Comparing the similarity between the merge motion candidate of the current coding block and the merge motion candidates in the merge candidate list is to determine whether neighboring blocks have various motions. In general, if the movements of surrounding blocks are similar, there is a high possibility that the movement of the current block will be similar to the movements of the surrounding blocks. In this case, the surrounding blocks and the current block can be inferred as blocks within the object with the same movement. Conversely, if the movements of neighboring blocks are not similar and show different movement characteristics, the neighboring blocks, including the current coding block, may not be the same object, so a process of finding more accurate motion may be necessary.

Figure 27 is a diagram showing a process in which multiple DMVR is performed according to an embodiment of the present invention.

Figure 27(a) shows the general process of performing multiple DMVR, and Figure 27(b) shows the general process of multiple DMVR in more detail.

Referring to FIG. 27(a), multiple DMVR can obtain one or more corrected motion information by performing DMVR in units of coding units (blocks) based on the initial motion information (S2710). When TM is applied to the current coding block, the decoder may perform TM using one or more corrected motion information obtained in step S2710 (S2720). If the corrected motion information of the current coding block determined by performing TM has changed from bidirectional to unidirectional, the DMVR process cannot be performed, so the steps (S2730, S2740) after S2720 are not performed, and the movement of the current block is unidirectional. is finally decided. As a result of performing TM, if the corrected motion information of the current coding block is bidirectional, the decoder can perform DMVR on a sub-block basis to obtain the corrected motion information on a sub-block basis for each sub-block (S2730). Then, the decoder can re-correct the sub-block-unit corrected motion information obtained in step S2740 based on the BDOF, and finally obtain the corrected motion information based on the BDOF.

Meanwhile, if the smallest cost value among the cost values for one or more pieces of corrected motion information obtained in step S2710 is greater than a random value, step S2730 may be performed. Meanwhile, if the smallest cost value is equal to or smaller than a random value, the decoder stores one or more corrected motion information obtained in step S2710 in sub-block units, resets the motion information of the current coding block to the initial motion information, and performs S2710. Steps through S2740 can be performed. At this time, one or more pieces of corrected motion information stored in sub-block units can be used for motion correction in step S2740. Finally, by comparing the cost value of the coding block using the initial motion information with the cost value of the coding block using motion corrected in sub-block units obtained by performing step S2740, the block unit with a small cost value and the corresponding motion information are calculated. Finally, it can be used in the current coding block. The arbitrary value may be the number of pixels in the current coding block or the size of the current coding block (either horizontal or vertical, or the sum of horizontal and vertical).

Referring to FIG. 27(b), step S2710 of FIG. 27(a) includes a step of performing DMVR in units of coding units (blocks) using integer-unit search (S2701) and coding using search in half-pixel units. It can be subdivided into a step (S2702) of performing DMVR on a unit (block) basis. At this time, the motion resolution may be set in integer units in step S2701 and in half-pixel units in step S2702. In step S2701, the decoder may obtain corrected motion information using the initial motion information and an integer correction value. In step S2702, the decoder may obtain corrected motion information using the initial motion information and the half-pixel unit correction value. The decoder can use the corrected motion information obtained in step S2701 as reference motion information for step S2702. That is, the decoder may generate a new motion candidate based on the corrected motion information obtained in step S2701 and perform evaluation on the new motion candidate. At this time, if the cost value for the new motion candidate is greater than the cost value of the corrected motion information obtained in step S2701, the corrected motion information obtained in step S2701 may be the output of the DMVR in coding unit (block) units. Conversely, if the cost value for the new motion candidate is smaller than the cost value of the corrected motion information obtained in step S2701, the corrected motion information obtained in step S2702 may be the output of the DMVR in coding unit (block) units. That is, the corrected motion information corresponding to the smaller cost value between the cost value obtained in step S2701 and the cost value obtained in step S2702 can be used for TM (S2703, S2720). At this time, when calculating cost values for all new motion candidates, there is a problem of increased complexity. To this end, the decoder calculates an MV (motion information)-based cost value, and can calculate a pixel-based cost value for a new motion candidate only if the cost value obtained based on the motion information is less than a random value. . The cost value obtained based on the motion information may be obtained by multiplying the sum of the absolute values of the differences in horizontal and vertical component values between the new motion candidate and the initial motion information by the weight. Additionally, the cost value obtained based on the motion information may be obtained using the difference between the new motion candidate and the corrected motion information obtained in step S2701.

Each of the cost values obtained based on the motion information may be obtained through a difference value between the motion information obtained by adding a correction value to each of a plurality of motion candidates corresponding to a position selected according to the initial motion information. The cost value obtained based on motion information may vary depending on the size of the correction value. That is, the smaller the correction value, the smaller the cost value can be. Since the motion information corresponding to the smallest cost value is set as the final motion information, evaluation can be performed only on motion candidates surrounding the position indicated by the initial motion information with a small correction value. However, a motion candidate with a large correction value may be the optimal motion candidate. Therefore, in order to select the optimal motion candidate by evaluating various motion candidates, the cost value can be obtained using a method described later. The cost value can be obtained using the difference between the motion information values of each neighboring block, the quantization parameter, the size of the current block, etc.

The cost value can be obtained using the distribution of motion information of neighboring blocks. To evaluate various motion candidates, the decoder can obtain a cost value using the difference value between the corrected motion information and the motion information of neighboring blocks. For example, a cost value can be obtained according to the comparison result by comparing the difference value between the corrected motion information and the motion information of the surrounding block with an arbitrarily determined value. Specifically, if the difference value between the corrected motion information and the motion information of the surrounding block is greater than (or less than, or equal to) a certain value, the decoder can obtain a cost value. At this time, the neighboring block may be a neighboring block adjacent to the current block or a temporal neighboring block located at the same position as the current block in the corresponding picture (Collocated picture).

The cost value can be obtained based on the size of the current block. For example, the weight for obtaining the above-described cost value may be set according to the size of the current block. The weight can be set in inverse proportion to the size of the current block. That is, the larger the size of the current block, the lower the weight can be set. This is to evaluate a wider range of motion candidates in order to select an appropriate motion candidate. Meanwhile, the weight can be set proportional to the size of the current block. That is, the larger the size of the current block, the higher the weight can be set. This is to reduce complexity. For example, the size of the current block can be 16x16, 32x32, and can be set to the sum of the horizontal and vertical sizes of the current block. Weights can be integer values such as 1, 2, 3, 4, 5, 6, etc. Additionally, as the weight increases, the cost value increases, so if the weight is greater than a certain value, the decoder may not perform evaluation to obtain the cost value.

Hereinafter, step S2730 of FIG. 27(a) will be described in detail.

In step S2730, the decoder divides the current coding block into several sub-blocks, and then steps S2704 and S2705 of FIG. 27(b) may be performed for each sub-block. The width of the subblock may be a smaller value compared to the width of the coding block and the preset maximum width. The height of the subblock may be compared to the height of the coding block and the preset maximum height, and the minimum value may be the height of the subblock. At this time, the subblock size can be set to a maximum of 16x16.

The decoder may set the corrected motion information obtained in step S2730 as initial motion information for steps S2704 and S2705. The decoder can perform a global search in integer units using the initial motion information and obtain optimal motion information for the current sub-block and a cost value for the optimal motion information (S2704). In order to obtain optimal motion information, a cost value for each piece of motion information for a sub-block can be obtained. If the cost value for a random subblock is smaller than a random value, motion information for the current subblock obtained through global search in integer units is stored and subsequent processes may not be performed. If the cost value for any subblock is equal to or greater than the random value, the cost value for the random subblock may be set to another preset value. The arbitrary value can be the number of pixels of the current sub-block or the size of the current sub-block. For example, the size of a subblock may be 16 x 16. Additionally, other preset values may be the maximum value within the range of numbers that the encoder and decoder can express. After step S2704, the decoder can perform a 3x3 Square search in half-pixel (1/2) units. The motion information obtained through step S2704 can be used as the reference motion information in step S2705. That is, a new motion candidate is obtained based on the information obtained through step S2704, and the decoder can evaluate the new motion candidate (S2705). The decoder may perform evaluation on a new motion candidate and store the final motion information for the current sub-block. Steps S2704 and S2705 may be performed repeatedly for all subblocks. The DMVR process on a sub-block basis has the advantage that all sub-blocks can perform DMVR in parallel because there is no dependency between sub-blocks.

FIG. 28 is a diagram illustrating a search method for obtaining a cost value related to corrected motion information of a coding block according to an embodiment of the present invention.

Referring to FIG. 28, the decoder may obtain initial motion information and set motion candidates to search based on the obtained initial motion information. And the decoder can obtain corrected motion information based on the cost value obtained by evaluating the set motion candidates. And the decoder can obtain the final corrected motion information using model-based fractional MVD optimization according to the motion information resolution of the current block. The corrected motion information obtained in steps S2701, S2702, and S2705 of FIG. 27(b) can be obtained through the method described with reference to FIGS. 28 and 29.

Figure 29 is a diagram showing a 3x3 Square search method according to an embodiment of the present invention.

Below, with reference to FIG. 29, the method of obtaining the final corrected motion information described with reference to FIG. 28 will be described in more detail.

The decoder can obtain (calculate) a cost value for the motion information corrected in the previous step (S2901). At this time, if there is a cost value for the previously obtained corrected motion information, step S2901 can be omitted. The cost value obtained in step S2901 may be obtained in different ways depending on the motion resolution. For example, if the motion resolution is an integer unit, it can be obtained as a pixel-based cost value for the corrected motion information and is used as a reference cost value. If the motion resolution is a half-pixel (1/2) unit, the decoder obtains a cost value based on the motion information obtained using the initial motion information and the corrected motion information and obtains a pixel-based cost value for the corrected motion information. . Next, if the cost value based on the motion information is greater than the cost value calculated based on the pixel (pixel), the corrected motion information is stored and the next process is not performed and is skipped. If the cost value based on motion information is equal to or smaller than the cost value calculated based on pixel (pixel), the cost value obtained by adding the cost value based on motion information to the pixel (pixel) based cost value is used as the reference cost value. The decoder may set correction values for motion candidates to be searched (S2902). Specifically, motion correction candidate values can be derived according to a determined 3x3 Square search pattern. At this time, the motion correction candidate value may be set based on the motion resolution. Depending on the motion resolution, the motion correction candidate value may be scaled to expand the range. For example, the horizontal and vertical directions of a given 3x3 Square search pattern are Mv(-1, 1), Mv(0, 1), Mv(1, 1), Mv(1, 0), Mv(1, -1) ), Mv(0, -1), Mv(-1, -1), Mv(-1, 0). If the motion resolution is half pixel (1/2), evaluation may be performed only on odd numbers in the 3x3 Square search pattern, and evaluation may not be performed on even numbers in the 3x3 Square search pattern, in order to reduce computational complexity. The decoder can set a new motion candidate using the corrected motion candidate and the motion correction candidate value. The decoder can reconstruct motion information about the reference block of the picture in the L0 picture list and motion information about the reference block of the picture in the L1 picture list (S2904). In order to evaluate L0 and L1 motions with linear characteristics, the decoder can generate a new motion candidate by adding motion correction candidate values to the corrected L0 motion candidate and subtracting the motion correction candidate values from the corrected L1 motion candidate. there is. Additionally, in order to evaluate L0 and L1 motions with non-linear characteristics, the decoder may generate a new motion candidate by adding the motion correction candidate value to the corrected L1 motion candidate without modifying the corrected L0 motion candidate. Alternatively, the decoder may generate a new motion candidate by adding the motion correction candidate value to the corrected L0 motion candidate without modifying the corrected L1 motion candidate. The decoder may calculate a cost value based on motion information for a new motion candidate (S2905). The decoder may compare the cost value based on motion information for the new motion candidate with the reference cost value (S2906). If the cost value based on motion information is greater than the reference cost value, the decoder may store a value in memory that is twice the cost value based on motion information. And the decoder can repeat the process again from step S2902 without performing the process after step S2906. If the cost value based on motion information is smaller than the reference cost value, the decoder can obtain a pixel-based cost value for a new motion candidate (S2907). Additionally, the reference cost value can be updated to a value that is the sum of the cost value based on motion information and the pixel-based cost value for the new motion candidate. If the motion resolution is a half-pixel (1/2) unit, the decoder can repeatedly perform from step S2901 to evaluate other new motion candidates. That is, if the motion resolution is in half-pixel (1/2) units, the decoder may only store the evaluation results for new motion candidates and may not use them as optimal motion candidates. The evaluation results (cost values) for the stored new motion candidates can be used in the model-based fractional MVD optimization step that can be performed later. If the motion resolution is an integer unit, the sum of the cost value based on motion information for the new motion candidate and the pixel-based cost value is smaller than the pixel-based cost value for the initially corrected motion information. , the decoder can store the new motion candidate as optimal motion information. If the motion resolution is an integer unit, the decoder can store the pixel-based cost value for the new motion candidate obtained in step S2907 as the cost value for optimal motion information. Afterwards, the decoder can check whether there are motion candidates that have not yet been explored (S2908). If there are motion candidates that have not yet been searched, the decoder can repeat step S2901 to evaluate new motion candidates. If all motion candidates have been evaluated, it corresponds to the smallest cost value among the cost values for the searched motion candidates. The motion candidate can be saved as the final corrected motion (S2909). If the motion resolution is in half-pixel (1/2) units, the decoder performs a model-based fractional MVD optimization process using the cost values for the motion candidates stored in the previous step to recalculate the optimal half-pixel-level compensation motion information. (S2910). Step S2910 may be similar to the process of deriving movement in units of fractional pixels using a pixel (pixel)-based cost value in integer units in DMVR of Versatile Video Coding (VVC). The decoder can obtain optimal motion information using cost values for surrounding (left, right, up, down) motion information adjacent to the location of optimal motion information.

The search method described with reference to FIG. 29 can be repeatedly performed any set number of times, and when repeated, the decoder can only search for locations surrounding the optimal motion information obtained in the previous search.

Figure 30 is a diagram showing the search area divided into zones in the integer global search process according to an embodiment of the present invention.

Referring to FIG. 30, global search in integer units may be a search performed by dividing the search area into several zones centered on the center position and setting different weights for each zone. The center position of the search area may be the top left pixel (pixel) position of the reference block indicated by the initial motion information. The weight for each zone may be set differently depending on the movement characteristics of the current block and neighboring blocks of the current block. For example, if the similarity between the movement of the current block and neighboring blocks of the current block is high, the weight may be set to increase from the center area to the edge area. Conversely, if the similarity between the movement of the current block and neighboring blocks of the current block is low, the weights may be set to decrease in order from the center area to the edge area. If the pixel-based cost value for the corrected motion information obtained in step S2710 of FIG. 27(a) is smaller than an arbitrary value, the global search in integer units may not be performed. Additionally, if the pixel-based cost value for a random motion compensation value within the search area is smaller than a random value, the random motion compensation value becomes the final corrected motion information, and the global search in integer units can be terminated. there is. The arbitrary value can be the number of pixels (pixels) of the current block or the size of the current block. For example, the size of the current block may be 16x16.

Figures 31 and 32 are diagrams showing a global search process in integer units according to an embodiment of the present invention.

Referring to FIG. 31, the decoder can set motion candidates to be searched based on the input initial motion information. Even so, the decoder can obtain final corrected motion information by performing evaluation on motion candidates. Global search in integer units can be performed for all locations in each zone in the order from the center zone to the edge zone, as shown in FIG. 30.

Below, global search in integer units will be described in more detail with reference to FIG. 32. Referring to FIG. 32, the decoder can reset the search position for each zone within the search area according to the size of the block (S3201). At this time, if the size of the block is smaller than a random value, the decoder evaluates only the search area reduced by a random value based on the center position of FIG. 30 to reduce computational complexity. The random value can be half the size of the current block or a positive integer, for example '8'. After calculating the pixel-based cost value using the initial motion information, set it as the minimum cost value. The decoder may set a motion correction value for the search position of the motion candidate and reset the motion correction value to match the motion resolution to be used in the current block (S3202, S3203). The decoder can set a new motion candidate using the motion correction value reset in step S3203. The decoder can reconstruct motion information about the reference block of the picture in the L0 picture list and motion information about the reference block of the picture in the L1 picture list (S3204). In order to evaluate L0 and L1 motions with linear characteristics, the decoder can generate a new motion candidate by adding motion compensation candidate values to the corrected L0 motion candidate and subtracting the motion compensation candidate value from the corrected L1 motion candidate. there is. Additionally, in order to evaluate L0 and L1 motions with non-linear characteristics, the decoder may generate a new motion candidate by adding the motion correction candidate value to the corrected L1 motion candidate without modifying the corrected L0 motion candidate. Additionally, the decoder may generate a new motion candidate by adding the motion correction candidate values to the corrected L0 motion candidate without modifying the corrected L1 motion candidate. The decoder can obtain a pixel-based cost value for a new motion candidate (S3205). In order to differentiate the cost value for each zone, the decoder can reacquire the cost value using a predefined weight value for each zone (S3206). The weight value is a value obtained through experiment and can be a positive integer value. For example, weight values can be '1', '2',... , could be '63'. If the reacquired cost value is less than the minimum cost value, the decoder may update the minimum cost value with the reacquired cost value. The decoder may evaluate whether evaluation of all motion candidates has been completed (S3207). If there are motion candidates for which evaluation has not yet been completed, the process can be performed again from step S3202 using the minimum cost value. If the evaluation of all motion candidates is completed, the decoder can set the motion candidate information corresponding to the smallest cost value among the cost values of each motion candidate as the final corrected motion information. An early termination method can be applied to global search in integer units to reduce complexity. If the reacquired cost value in step S3206 is less than the minimum cost value, and the difference value obtained by subtracting the minimum cost value from the reacquired cost value is less than an arbitrary value, step S3208 is performed without checking step S3207, and the integer unit Global search may be terminated. The arbitrary value can be the size of the current block or the number of pixels, for example 16x16.

33 and 34 are diagrams showing a method of performing correction of motion information based on BDOF according to an embodiment of the present invention.

Correction of motion information based on BDOF in FIGS. 33 and 34 specifically shows correction of motion information based on BDOF in steps S2740 and S2706 of FIG. 27.

Referring to FIG. 33, the decoder may divide the current block into sub-blocks and then calculate the correction value of the BDOF-based motion information based on the input initial motion information to obtain final corrected motion information. BDOF can be used to correct a prediction block by estimating the amount of change in a pixel from the reference block of a block composed of bidirectional motion. The motion of the current block can be corrected using motion information derived from BDOF. If the current block is encoded in at least one mode among affine, LIC, OBMC, sub-block MC, CIIP, SMVD, BCW with different weight, and MMVD, BDOF-based motion compensation may not be performed. Meanwhile, BDOF-based motion compensation can be performed when any one of the following conditions is satisfied. The conditions for performing BDOF-based motion compensation are: i) when motion in merge mode is bidirectional, ii) when the distance between reference pictures and the current picture is the same, and iii) when weight prediction between reference blocks is not applied. In the case of a block, iv) the size of the current block may be larger than a certain size. Any given size can be the width and height of the block. For example, the horizontal size of the block may be '8' and the vertical size may be '8'. Additionally, BDOF-based motion compensation can be performed on a sub-block basis, and the size of a sub-block can be up to 16x16.

Below, the process of performing BDOF-based motion compensation for each sub-block is described. Referring to FIG. 34, motion information may be the same between adjacent sub-blocks, and BDOF can be performed only once by merging sub-blocks with the same motion information (S3401, S3402). Depending on the size difference between the width and height of the current block, you can decide whether to merge subblocks horizontally or vertically. For example, if the height is greater than the width of the current block, the sub-blocks can be merged vertically, and if the width is greater than the height of the current block, the sub-blocks can be merged horizontally. The decoder may obtain corrected motion information or a prediction block by repeatedly performing steps S3403 to S3414 for each sub-block or merged sub-blocks.

The decoder can obtain BDOF parameters using the L0 reference block and L1 reference block (S3403). BDOF parameters may include the amount of change between pixel values. The decoder may divide the current block into sub-blocks of up to 8x8 in size (S3404) and then repeat steps S3405 to S3412 for each sub-block.

The decoder can obtain a pixel-based cost value between L0 and L1 reference blocks (S3405). The decoder may compare the cost value with a preset limit value (S3406). If the cost value as a result of the comparison is less than the preset limit value, the decoder can generate a prediction block for the sub-block using the L0 and L1 reference blocks by substituting (0,0) for the motion compensation value without performing motion compensation. There is (S3407, S3409). Conversely, if the cost value as a result of the comparison is equal to or greater than the preset limit value, the decoder can obtain a motion compensation value based on BDOF (S3408). The decoder can check whether the motion compensation value is (0,0) (S3410). If the motion correction value is (0,0), the decoder can generate a prediction block based on BDOF, and if the motion correction value is not (0,0), the correction value obtained in step S3408 can be stored (S3411, S3412). The corrected motion information stored in step S3412 may be the final corrected motion information of FIG. 33.

The decoder can check whether steps S3403 to S3412 have been performed for every 8x8 subblock (S3413). If there is an 8x8 subblock for which steps S3405 to S3412 have not been performed, the decoder may repeatedly perform steps S3405 to S3412. If steps S3405 to S3412 have been performed for all 8x8 sub-blocks, the decoder can check whether steps S3403 to S3412 have been performed for all 16x16 sub-blocks (S3414). If there is a 16x16 subblock for which steps S3403 to S3412 have not been performed, the decoder may repeatedly perform steps S3403 to S3412. If steps S3403 to S3412 have been performed on all 16x16 sub-blocks, motion compensation based on BDOF can be terminated. At this time, the 8x8 subblock may be a subblock constituting a 16x16 subblock.

Figures 35 and 36 are diagrams showing a method of generating a prediction block for a current block based on BDOF according to an embodiment of the present invention.

Referring to FIG. 35, the decoder can generate a prediction block for the current block using the final corrected motion information obtained by performing the process described in FIGS. 33 and 34. The decoder can divide the current block into sub-blocks and use the final corrected motion information for each sub-block to generate a prediction block for the current block based on BDOF.

Below, the specific process of generating a prediction block based on BDOF is described.

Referring to FIG. 36, the decoder can divide the current block into 8x8 subblocks (S3601). The decoder may obtain final corrected motion information by adding the corrected motion information obtained by performing steps S2730, S2704, and S2705 of FIG. 27 and the motion correction value obtained by performing steps S2740 or S2706 (S3602). At this time, the decoder can check whether the motion correction value obtained in step S2740 or S2706 of FIG. 27 is (0,0) (S3603). As a result of confirmation, if the motion compensation value is (0,0), the decoder can use the prediction block obtained in steps S3409 and S3411 of FIG. 34 as a prediction block for the luminance component (S3604). Additionally, if the motion correction value is (0,0) as a result of confirmation, the decoder can generate a prediction block for the chrominance component (S3606). Meanwhile, if the motion correction value is not (0,0) as a result of confirmation, the decoder can generate a prediction block for the luminance component and chrominance component based on BDOF (S3605). The decoder checks whether the above-described steps S3602 to S3606 have been performed for all 8x8 subblocks, and if there are subblocks that have not been performed, steps S3602 to S3606 may be repeatedly performed (S3607).

Figure 37 is a diagram showing a subblock to which DMVR is applied according to an embodiment of the present invention.

Hereinafter, various methods of performing the above-described multiple DMVR will be described with reference to FIG. 27.

Whether each step of the DMVR described with reference to FIG. 27 is performed depends on the size of the current block, the color component of the current block (whether the current block is a luminance component block or a chrominance component block), quantization parameter information for the current block, and the movement of the current block. It may be determined based on at least one of resolution information and information related to the presence or absence of a residual signal of a neighboring block of the current block. For example, if the current block is a chrominance component block or the residual signal of a neighboring block of the current block is within a certain value, some of the steps in FIG. 27 may be omitted or steps after a specific step may be omitted. This is to reduce computational complexity.

Whether or not to perform each step of DMVR described with reference to FIG. 27 may be determined depending on the size of the current block. As the size of the block increases, the number of divided sub-blocks increases, so if the size of the current block is equal to or greater than a random value, some or all of the steps of DMVR may not be performed. At this time, the arbitrary value may be the sum of the width and height sizes of the current block, and specifically may be 128.

Meanwhile, if the size of the current block (e.g., coding block) is larger than an arbitrary value, the size of the sub-block can be set to be divided proportionally large, and each step of DMVR described with reference to FIG. 27 can be performed. The arbitrary value at this time may be 16. If the current block is larger than a random value, the subblock may be divided into the expanded size by expanding the width and height of the subblock by n (e.g., a natural number) times the preset size. Alternatively, the maximum number of sub-blocks that can be applied to the current block may be set, and the width and height of the sub-block may be determined based on the maximum number.

Whether or not to perform each step of DMVR described with reference to FIG. 27 may be determined depending on the number of allowable parallel processes of the video signal processing device. This is because the number of resources (eg, processes and threads) that can be processed in parallel for each video signal processing device is limited. Accordingly, whether or not DMVR is performed step by step may vary under limited processing resources, and DMVR may be selectively applied to each block. For example, if the number of resources that can be processed is equal to or smaller than an arbitrary value, some of each step of the DMVR described with reference to FIG. 27 may be omitted or steps after a specific step may be omitted. Additionally, DMVR can be applied only to subblocks at specific locations among subblocks. The specific location may be an odd or even block among the divided sub-blocks. Alternatively, as shown in FIG. 37, the subblocks to which DMVR is applied may be subblocks located at the top left, top right, bottom left, and bottom right. Motion information for a sub-block to which DMVR is not applied can be obtained using corrected motion information of neighboring blocks of a block to which DMVR is applied.

The size of the divided subblock may be determined depending on the number of resources that can be processed in parallel. The greater the number of divided sub-blocks, the slower the processing speed of the video signal processing device may be. The number of resources that can be processed in parallel and the size of the subblock may be inversely proportional. For example, the larger the number of resources that can be processed in parallel, the smaller the size of the divided subblock can be. In other words, the greater the number of resources that can be processed in parallel, the greater the number of divided sub-blocks. Conversely, the smaller the number of resources that can be processed in parallel, the larger the size of the divided subblock can be. In addition, the number of resources that can be processed in parallel can be determined by whether it is larger or smaller than an arbitrary value, and the size of the subblock can also be determined according to the number of resources determined by an arbitrary value. In addition, the size of the subblock is not only the number of resources that can be processed in parallel, but also the quantization parameter information of the current block, the size of the current block or the color component (luminance or chrominance component) of the current block, the motion resolution information of the current block, and the quantization parameter information of the current block. It may be determined based on at least one of information related to the presence or absence of a residual signal of a neighboring block. At this time, the size of the subblock and the number of divided subblocks may be signaled separately. For example, the decoder may parse syntax elements included in the bitstream to determine the size of the subblock and the number of divided subblocks. At this time, the syntax element may be signaled at the SPS level, PPS level, picture level, slice level, or CU (Coding Unit) level.

In the multi-DMVR process, the motion compensation process may be performed only on the luminance component of the current block and not on the chrominance component. At this time, the corrected motion information for the luminance component of the current block obtained through multiple DMVR can be used for the chrominance component of the current block.

Figure 38 is a diagram showing a method of generating a prediction block for a current block while performing multiple DMVR according to an embodiment of the present invention.

The DMVR method is a method for finding more accurate motion information, and the decoder can ultimately generate one prediction block using the corrected motion information. Since DMVR is a method of correcting motion information through correlation between already restored reference images, accuracy may be lower than that of motion information found in the actual original image. In order to compensate for this low accuracy, referring to FIG. 38, the decoder can generate multiple prediction blocks by selectively using motion information at each stage obtained in the process of performing DMVR. The decoder can finally generate a prediction block for the current block through weighted average between prediction blocks. The controller can decide whether to use prediction blocks generated at each stage, and can set the weight of each prediction block at the weight average stage. Whether or not to use prediction blocks at each stage can be determined based on the pixel-based cost value already obtained when generating the prediction block. Additionally, the number of prediction blocks used may be determined according to the result of comparing the cost value with a random value. For example, if the cost value is less than a random value, X preset prediction blocks can be used, and if the cost value is greater than a random value, Y preset prediction blocks can be used. At this time, X and Y may be the same or different integer values. The controller can also decide whether to perform each step of DMVR. Whether or not each step of DMVR is performed is related to the quantization parameters of the current block, the size of the current block, the color component of the current block (whether it is a luminance component or a chrominance component), the motion of the current block, resolution information, and the presence or absence of residual signals in blocks surrounding the current block. It may be determined based on at least one of the information.

Figure 39 is a flowchart showing a method for predicting a current block according to an embodiment of the present invention.

FIG. 39 shows a method of predicting the current block based on the cost values described with reference to FIGS. 1 to 38.

Referring to FIG. 39, the decoder may obtain a first motion information list including one or more pieces of first motion information related to the current block (S3910). The decoder may obtain one or more pieces of first corrected motion information by correcting the one or more pieces of first motion information (S3920). The decoder may obtain one or more first cost values for each of the one or more first corrected motion information (S3930). The decoder may obtain a second motion information list by rearranging the one or more pieces of first corrected motion information based on the first cost values (S3940). The decoder may obtain one or more pieces of second motion information based on the first motion information included in the second motion information list (S3950). The decoder may obtain one or more second corrected motion information by correcting the one or more second motion information (S3960). The decoder may obtain one or more second cost values for each of the second corrected motion information (S3970). The decoder may predict the current block based on second motion information determined based on the second cost values (S3980).

At this time, the one or more first cost values may be values related to the similarity between the current block and the reference block corresponding to each of the one or more pieces of first corrected motion information. The one or more second cost values may be values related to the similarity between the current block and the reference block corresponding to each of the one or more pieces of second corrected motion information.

The first corrected motion information in the second motion information list may be arranged in ascending order of cost values corresponding to each of the one or more pieces of first corrected motion information.

The first motion information may be corrected motion information corresponding to the smallest cost value among the first cost values, and the second motion information may be corrected motion information corresponding to the smallest cost value among the second cost values. there is.

The one or more pieces of first motion information may each be included in different pictures, and the one or more pieces of second motion information may each be included in different pictures.

The one or more first corrected motion information and the one or more second corrected motion information may include Motion Vector Difference (MVD), Template Matching (TM), Bilateral Matching (BM), Merge mode with MVD (MMVD), and MMVD. It may be corrected based on at least one of (Merge mode with MVD)-based TM, optical flow-based TM, and multi-pass DMVR (Multi pass Decoder-side Motion Vector Refinement).

When the encoding mode of the current block is a merge mode, the one or more pieces of first motion information and the one or more pieces of second motion information may be motion information derived by Merge mode with MVD (MMVD).

Blocks corresponding to each of the one or more pieces of first motion information are blocks located in a first search area, blocks corresponding to each of the one or more pieces of second motion information are blocks located in a second search area, and the blocks corresponding to each of the one or more pieces of second motion information are blocks located in a second search area. The first search area and the second search area may be different from each other.

The first cost values may be sequentially calculated for each of the one or more pieces of first corrected motion information, and the second cost values may be sequentially calculated for each of the one or more pieces of second corrected motion information. At this time, if a cost value smaller than the preset first value is calculated while the first cost values are sequentially calculated, cost values for the remaining corrected motion information may not be calculated. If a cost value smaller than the preset second value is calculated while the second cost values are sequentially calculated, cost values for the remaining corrected motion information may not be calculated.

The methods described above in this specification may be performed through a processor of a decoder or encoder. Additionally, the encoder can generate a bitstream that is decoded by the methods described above. Additionally, the bitstream generated by the encoder may be stored in a computer-readable non-transitory storage medium (recording medium).

Although this specification is mainly described from the perspective of a decoder, it can be operated equally in an encoder. The term parsing in this specification has been described with a focus on the process of obtaining information from the bitstream, but from the encoder perspective, it can be interpreted as configuring the information in the bitstream. Therefore, the term parsing is not limited to the decoder operation, but can also be interpreted as the act of constructing a bitstream in the encoder. Additionally, this bitstream may be stored and configured in a computer-readable recording medium.

Embodiments of the present invention described above can be implemented through various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the method according to embodiments of the present invention uses one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), and Programmable Logic Devices (PLDs). , can be implemented by FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In the case of implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above. Software code can be stored in memory and run by a processor. The memory may be located inside or outside the processor, and may exchange data with the processor through various known means.

Some embodiments may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures or other data of modulated data signals such as program modules, or other transmission mechanisms, and includes any information delivery medium.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above are illustrative in all respects and should be interpreted as limited. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (20)

In the video signal decoding device,
Contains a processor,
The processor,
Obtain a first motion information list including one or more pieces of first motion information related to the current block,
Correcting the one or more first motion information to obtain one or more first corrected motion information,
Obtaining one or more first cost values for each of the one or more first corrected motion information,
Obtain a second motion information list by rearranging the one or more pieces of first corrected motion information based on the first cost values,
Obtaining one or more pieces of second motion information based on first motion information included in the second motion information list,
Correcting the one or more second motion information to obtain one or more second corrected motion information,
Obtain one or more second cost values for each of the second corrected motion information,
A video signal decoding device, characterized in that predicting the current block based on second motion information determined based on the second cost values. According to clause 1,
The one or more first cost values are values related to the similarity between the current block and the reference block corresponding to each of the one or more first corrected motion information,
The one or more second cost values are values related to the similarity between the current block and the reference block corresponding to each of the one or more second corrected motion information. According to clause 1,
The second motion information list is arranged in ascending order of cost values corresponding to each of the one or more pieces of first corrected motion information. According to clause 1,
A video signal decoding device, characterized in that the first motion information is corrected motion information corresponding to the smallest cost value among the first cost values. According to clause 1,
A video signal decoding device, characterized in that the second motion information is corrected motion information corresponding to the smallest cost value among the second cost values. According to clause 1,
The one or more pieces of first motion information are each included in different pictures,
A video signal decoding device, wherein the one or more pieces of second motion information are included in different pictures. According to clause 1,
The one or more first corrected motion information and the one or more second corrected motion information include:
MVD (Motion Vector Difference), TM (Template Matching), BM (Bilateral Matching), MMVD (Merge mode with MVD), MMVD (Merge mode with MVD)-based TM, optical flow-based TM, multi A video signal decoding device characterized in that correction is made based on at least one of multi-pass decoder-side motion vector refinement (DMVR). According to clause 1,
If the encoding mode of the current block is merge mode,
A video signal decoding device, characterized in that the one or more first motion information and the one or more second motion information are motion information derived by MMVD (Merge mode with MVD). According to clause 1,
Blocks corresponding to each of the one or more pieces of first motion information are blocks located within a first search area,
Blocks corresponding to each of the one or more pieces of second motion information are blocks located in a second search area,
A video signal decoding device, wherein the first search area and the second search area are different from each other. According to clause 1,
The first cost values are sequentially calculated for each of the one or more first corrected motion information,
The second cost values are sequentially calculated for each of the one or more pieces of second corrected motion information. According to clause 10,
When the first cost values are sequentially calculated and a cost value smaller than the preset first value is calculated, cost values for the remaining corrected motion information are not calculated,
A video signal decoding device, wherein when a cost value smaller than a preset second value is calculated while the second cost values are sequentially calculated, cost values for the remaining corrected motion information are not calculated. In the video signal encoding device,
Contains a processor,
The processor,
Obtaining a bitstream decoded by a decoding method,
The decoding method is,
Obtaining a first motion information list including one or more pieces of first motion information related to the current block;
Correcting the one or more pieces of first motion information to obtain one or more pieces of first corrected motion information;
Obtaining one or more first cost values for each of the one or more pieces of first corrected motion information;
Obtaining a second motion information list by rearranging the one or more pieces of first corrected motion information based on the first cost values;
Obtaining one or more pieces of second motion information based on first motion information included in the second motion information list;
Correcting the one or more pieces of second motion information to obtain one or more pieces of second corrected motion information;
Obtaining one or more second cost values for each of the second corrected motion information; and
A video signal encoding device comprising predicting the current block based on second motion information determined based on the second cost values. According to clause 12,
The second motion information list is arranged in ascending order of cost values corresponding to each of the one or more pieces of first corrected motion information. According to clause 12,
A video signal encoding device, wherein the first motion information is corrected motion information corresponding to the smallest cost value among the first cost values. According to clause 12,
The second motion information is corrected motion information corresponding to the smallest cost value among the second cost values. According to clause 12,
The one or more pieces of first motion information are each included in different pictures,
A video signal encoding device, wherein the one or more pieces of second motion information are each included in different pictures. According to clause 12,
If the encoding mode of the current block is merge mode,
A video signal encoding device, characterized in that the one or more first motion information and the one or more second motion information are motion information derived by MMVD (Merge mode with MVD). According to clause 12,
The first cost values are sequentially calculated for each of the one or more first corrected motion information,
The second cost values are sequentially calculated for each of the one or more pieces of second corrected motion information. According to clause 12,
When the first cost values are sequentially calculated and a cost value smaller than the preset first value is calculated, cost values for the remaining corrected motion information are not calculated,
A video signal encoding device, wherein when a cost value smaller than a preset second value is calculated while the second cost values are sequentially calculated, cost values for the remaining corrected motion information are not calculated. A computer-readable non-transitory storage medium storing a bitstream, wherein the bitstream is decoded by a decoding method,
The decoding method is,
Obtaining a first motion information list including one or more pieces of first motion information related to the current block;
Correcting the one or more pieces of first motion information to obtain one or more pieces of first corrected motion information;
Obtaining one or more first cost values for each of the one or more pieces of first corrected motion information;
Obtaining a second motion information list by rearranging the one or more pieces of first corrected motion information based on the first cost values;
Obtaining one or more pieces of second motion information based on first motion information included in the second motion information list;
Correcting the one or more pieces of second motion information to obtain one or more pieces of second corrected motion information;
Obtaining one or more second cost values for each of the second corrected motion information; and
A non-transitory storage medium comprising predicting the current block based on second motion information determined based on the second cost values.

Images