KR20230150274A - Machine learning-based flow determination for video coding - Google Patents

Abstract

Systems and techniques are described herein for processing video data. In some aspects, a method may include obtaining input video data by a machine learning system. Input video data includes one or more luminance components for the current frame. The method includes determining, by a machine learning system, motion information for the luminance component(s) of the current frame and motion information for one or more chrominance components of the current frame using the luminance component(s) for the current frame. It can be included. In some cases, the method may include determining motion information for luminance component(s) based on the luma component(s) of the current frame and at least one reconstructed luma component(s) of the previous frame. In some cases, the method may further include determining motion information for the chrominance component(s) of the current frame using the motion information determined for the luminance component(s) of the current frame.

Description

Machine learning-based flow determination for video coding

This disclosure generally relates to image and video coding, including encoding (or compression) and decoding (decompression) of images and/or video. For example, aspects of the present disclosure relate to techniques for determining flow information for luma and chroma components of one or more image frames or pictures (e.g., video frames/pictures).

Many devices and systems enable video data to be processed and output for consumption. Digital video data contains large amounts of data to meet the demands of consumers and video providers. For example, consumers of video data want high quality video with high fidelity, resolution, frame rates, etc. As a result, the large amounts of video data needed to meet these demands place a burden on communication networks and devices that process and store the video data.

Various coding techniques may be used to compress video data. The goal of video coding is to compress video data into a form that utilizes lower bit rates while avoiding or minimizing degradation to video quality. As constantly evolving video services become available, encoding techniques with superior coding efficiency are needed.

Systems and techniques are described for coding (e.g., encoding and/or decoding) image and/or video content using one or more machine learning systems. According to at least one example, a method for processing video is provided. The method includes obtaining, by a machine learning system, input video data, wherein the input video data includes at least one luminance component for the current frame; and determining, by the machine learning system, using the at least one luminance component for the current frame, motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame. do.

In another example, at least one memory (e.g., configured to store data such as virtual content data, one or more images, etc.) and one or more processors (e.g., implemented in circuitry) coupled to the at least one memory An apparatus for processing video data is provided, including: The one or more processors are configured to obtain, using the machine learning system, input video data, wherein the input video data includes at least one luminance component for a current frame; and, using the machine learning system, determine, using the at least one luminance component for the current frame, motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame. You can do something like that.

In another example, when executed by one or more processors, cause the one or more processors to: Using a machine learning system, obtain input video data, wherein the input video data includes at least one luminance component for the current frame. Ham -; Instructions for using a machine learning system to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using the at least one luminance component for the current frame. A non-transitory computer-readable medium storing data is provided.

In another example, an apparatus for processing video data is provided. The apparatus includes means for obtaining input video data, wherein the input video data includes at least one luminance component for a current frame; and means for determining, using the at least one luminance component for the current frame, motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame.

In some aspects, one or more of the methods, devices, and computer-readable media described above may be used to generate, by a machine learning system, motion information for at least one luminance component of a current frame and one or more chrominance components of the current frame. determining a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame using the motion information for the current frame; and determining one or more inter-frame predictions for the current frame using the warping parameters for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame.

In some aspects, the one or more inter-frame predictions are interpolated using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. It is determined at least in part by applying an operation.

In some aspects, the interpolation operation includes a trilinear interpolation operation.

In some aspects, the warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame include spatial-scale flow (SSF) warping parameters.

In some aspects, SSF warping parameters include learned scale-flow vectors.

In some aspects, the at least one luminance component for the current frame is used to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame, as described above. One or more of the methods, devices, and computer-readable media may include: generating motion information for at least one luminance component of a current frame based on at least one luminance component of the current frame and at least one reconstructed luma component of a previous frame; to decide; and determining motion information for one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame.

In some aspects, motion information for one or more chrominance components of the current frame is determined using a convolutional layer of a machine learning system.

In some aspects, one or more of the methods, devices, and computer-readable media described above may be used to determine motion information for one or more chrominance components of a current frame. It further includes sampling the determined motion information.

In some aspects, the current frame includes a video frame.

In some embodiments, the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component.

In some aspects, the current frame has a luminance-chrominance (YUV) format. In some cases, the YUV format is YUV 4:2:0 format.

In some aspects, the devices described herein may be a mobile device (e.g., a mobile phone or so-called “smartphone,” a tablet computer, or other type of mobile device), a wearable device, an extended reality device (e.g., includes or includes a virtual reality (VR) device, augmented reality (AR) device, or mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device in a vehicle), or another device. It may be part of it. In some aspects, a device includes at least one camera to capture one or more images or video frames. For example, a device may include a camera (e.g., an RGB camera) or multiple cameras to capture one or more videos and/or one or more images containing video frames. In some aspects, a device includes a display for displaying one or more images, video, notifications, or other displayable data. In some aspects, an apparatus includes a transmitter configured to transmit one or more video frames and/or syntax data to at least one device over a transmission medium. In some aspects, a processor includes a neural processing unit (NPU), central processing unit (CPU), graphics processing unit (GPU), or other processing device or component.

This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation in determining the scope of the claimed subject matter. The subject matter should be understood by reference to the appropriate portions of the entire specification, any or all drawings, and each claim of this patent.

The foregoing, along with other features and embodiments, will become more apparent upon reference to the following specification, claims and accompanying drawings.

Exemplary embodiments of the present application are described in detail below with reference to the following drawings.
1 shows an example implementation of a system-on-chip (SOC).
Figure 2a shows an example of a fully connected neural network.
Figure 2b shows an example of a locally connected neural network.
Figure 2C shows an example of a convolutional neural network.
Figure 2D illustrates a detailed example of a deep convolutional network (DCN) designed to recognize visual features from images.
Figure 3 is a block diagram for explaining a deep convolutional network (DCN).
4 is a diagram illustrating an example of a system including a device operable to perform image and/or video coding (encoding and decoding) using a neural network-based system, according to some examples.
5 is a diagram illustrating an example of an end-to-end neural network based image and video coding system for input with red-green-blue (RGB) format, according to some examples.
6 illustrates one or more luminance-chrominance (YUV) input formats (e.g., 4:2:0 YUV input format) that may be part of an end-to-end neural network based image and video coding system, according to some examples. Diagram illustrating an example of a space-scale flow (SSF) neural network architecture configured to process.
7A is a diagram illustrating an example of a machine-learning based flow engine operating with luma input, according to some examples.
FIG. 7B is a diagram illustrating an example of subsampling of luma motion information to obtain chroma motion information, according to some examples.
FIG. 8A is a diagram illustrating an example of a machine learning based architecture with YUV (eg, YUV 4:2:0) residuals, according to some examples.
8B is a diagram illustrating example operation of a 1x1 convolutional layer, according to some examples.
9 illustrates an example of an end-to-end neural network based image and video coding system (e.g., an end-to-end neural network based image and video coding system) that operates directly with YUV inputs (Y, U, and V), such as a YUV 4:2:0 input, according to some examples. ) Diagram illustrating an example of a machine learning-based architecture.
10 illustrates an example of an end-to-end neural network based image and video coding system (e.g., an end-to-end neural network based image and video coding system) that operates directly with YUV inputs (Y, U, and V), such as a YUV 4:2:0 input, according to some examples. ) This diagram illustrates another example of a machine learning-based architecture.
11 is a flow diagram illustrating an example of a process for processing video data, according to some examples.
12 illustrates an example computing device architecture of an example computing device that can implement various techniques described herein.

Certain aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments may be applied independently and some of them may be applied in combination as will be apparent to those skilled in the art. In the following description, for purposes of explanation, specific details are set forth to provide a thorough understanding of embodiments of the present application. However, it will be clear that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting.

The following description provides example embodiments only and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the description of the example embodiments will provide those skilled in the art with possible instructions for implementing the example embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

Digital video data can contain large amounts of data, especially as the demand for high quality video data continues to increase. For example, consumers of video data typically desire increasingly higher quality video with higher fidelity, resolution, frame rates, etc. However, the large amounts of video data required to satisfy these demands can place a significant burden on communication networks as well as devices that process and store the video data.

Various techniques can be used to code video data. Video coding may be performed according to specific video coding standards. Exemplary video coding standards include High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Expert Group (MPEG) coding, and Versatile Video Coding (VVC). Video coding often uses prediction methods such as inter-prediction or intra-prediction, which exploit redundancies present in video images or sequences. A common goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradations to video quality. As demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

Machine learning (ML) based systems can be used to perform image and/or video coding. In general, ML is a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences, without the use of explicit instructions. One example of an ML system is a neural network (also referred to as an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks are used in a variety of applications, such as image and/or video coding, image analysis and/or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, service robots, and /or can be used for devices.

Individual nodes within a neural network can emulate biological neurons by taking input data and performing simple operations on the data. The results of simple operations performed on input data are selectively transmitted to other neurons. Weight values are associated with each vector and node in the network, and these values constrain how input data is related to output data. For example, the input data of each node can be multiplied by the corresponding weight value. The sum of the products can be adjusted by an optional bias, and an activation function can be applied to the result, yielding the node's output signal or “output activation” (sometimes referred to as an activation map or feature map). Weight values may initially be determined by iterative flow of training data through the network (e.g., weight values are set during the training phase where the network learns to identify specific classes by their typical input data characteristics). ).

In particular, different types of neural networks exist, such as convolutional neural networks (CNN), recurrent neural networks (RNN), generative adversarial networks (GAN), and multilayer perceptron (MLP) neural networks. For example, a convolutional neural network (CNN) is a type of feed-forward artificial neural network. Convolutional neural networks may include sets of artificial neurons that collectively tile the input space, each having a receptive field (e.g., a spatially localized region of the input space). RNNs work on the principle of storing the output of a layer and feeding this output back to the input to help predict the outcome of the layer. GAN is a form of generative neural network that can learn patterns from input data, so the neural network model can generate new synthetic outputs that could reasonably have been obtained from the original dataset. A GAN may involve two neural networks working together, including a generative neural network that produces a synthesized output and a discriminative neural network that evaluates the output for authenticity. In MLP neural networks, data can be fed to an input layer, and one or more hidden layers provide levels of abstraction for the data. Predictions can then be made on the output layer based on the abstracted data.

In layered neural network architectures (referred to as deep neural networks when there are multiple hidden layers), the output of a first layer of artificial neurons becomes the input to a second layer of artificial neurons, and the second layer of artificial neurons The output of that layer becomes the input to a third layer of artificial neurons, and so on. CNNs can, for example, be trained to recognize hierarchies of features. Computation in CNN architectures may be distributed across a population of processing nodes, which may consist of one or more computation chains. These multi-layered architectures may be trained one layer at a time and fine-tuned using back propagation.

In many cases, deep learning based systems include an autoencoder sub-network (encoder sub-network) and a second sub-network (in some cases It is designed as a combination of (decoder sub-network) (also referred to as a hyperprior network). In some cases, other sub-networks of decoders may exist. This deep learning-based system architecture can be viewed as a combination of a transform plus quantization module (or encoder sub-network) and an entropy modeling sub-network module.

Most existing deep learning-based architectures for video compression are designed to operate on non-subsampled input formats such as RGB, YUV 4:4:4, or other non-subsampled input formats. However, video coding standards such as HEVC and VVC are designed to support YUV 4:2:0 color format in their respective main profiles. To support the 4:2:0 YUV format, deep learning-based architectures designed to operate on non-subsampled input formats must be modified.

ML-based systems (e.g., deep-learning based systems) that can use the color component of one of one or more frames (e.g., a video frame) to estimate information about the color component of the frame and other color components Described herein are systems, devices, processes (also referred to as methods), and computer-readable media (collectively referred to as “systems and techniques”) that provide. In some aspects, an ML-based system can be designed to process input data with luminance-chrominance (YUV) input formats. In these aspects, an ML-based system may be used to estimate motion information (e.g., flow information, such as optical flow information) for both a luma component and one or more chroma components (e.g., an ML-based system The luma component of both the previously-reconstructed frame (reconstructed by) and the current frame can be used. In some cases, after learning motion information for the luma component, a convolutional layer with down-sampling can be used to learn motion information (e.g., flow information) for one or more chroma components. In some cases, motion information for one or more chroma components may be obtained by directly subsampling motion information for the luma component (eg, without using a convolutional layer). This technique can be performed on all components of the frame. Using these techniques, an ML-based system can determine chroma motion information (e.g., flow information) without the need to have chroma information coded as part of the latent data or bitstream (e.g., chroma information and together reduces the need to transmit side information).

As described above, ML-based systems can be designed to process input data with YUV input format. The YUV format includes a luminance channel (Y) and a pair of chrominance channels (U and V). The U channel may be referred to as the chrominance (or chroma)-blue channel, and the U channel may be referred to as the chrominance (or chroma)-red channel. In some cases, the luminance (Y) channel or component may also be referred to as a luma channel or component. In some cases, chrominance (U and V) channels or components may also be referred to as chroma channels or components. YUV input formats may include YUV 4:2:0, YUV 4:4:4, YUV 4:2:2, among others. In some cases, the systems and techniques described herein may have a Y-chroma blue (Cb)-chroma red (Cr) (YCbCr) format, a red-green-blue (RGB) format, and/or other formats. It can be designed to handle different input formats such as data. The ML-based system described herein can encode and/or decode video data and/or stand-alone frames (also referred to as images) containing multiple frames.

Additional details and additional aspects of the disclosure will be described in conjunction with the drawings.

1 is an exemplary illustration of a system-on-chip (SOC) 100 that may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Implementation is shown. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., a neural network with weights), delays, frequency bin information, task, among other information. Information may include memory blocks associated with neural processing unit (NPU) 108, memory blocks associated with CPU 102, memory blocks associated with graphics processing unit (GPU) 104, and digital signal processor (DSP) 106. It may be stored in a memory block, memory block 118, and/or may be distributed across multiple blocks. Instructions executing on CPU 102 may be loaded from program memory associated with CPU 102 or may be loaded from memory block 118.

SOC (100) also includes GPU (104), DSP (106), 5th Generation (5G) connectivity, 4th Generation Long Term Evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, and Bluetooth connectivity. connectivity block 110, which may include, and the like, and additional processing blocks tailored for specific functions, such as multimedia processor 112, which may detect and recognize gestures, for example. In one implementation, the NPU is implemented in CPU 102, DSP 106, and/or GPU 104. SOC 100 may also include sensor processor 114, image signal processors (ISPs) 116, and/or navigation module 120, which may include a global positioning system.

SOC 100 may be based on the ARM instruction set. In one aspect of the present disclosure, instructions loaded into CPU 102 may include code for retrieving a stored multiplication result from a lookup table (LUT) corresponding to a multiplication product of an input value and a filter weight. Instructions loaded into CPU 102 may also include code to disable the multiplier during a multiply operation of a multiply product when a lookup table hit of the multiply product is detected. Additionally, instructions loaded into the CPU 102 may include code for storing the calculated multiplication product of the input value and the filter weight when a lookup table miss of the multiplication product is detected.

SOC 100 and/or its components may perform video compression and/or decompression (also referred to as video encoding and/or decoding, collectively) using machine learning techniques in accordance with aspects of the disclosure discussed herein. may be configured to perform (referred to as video coding). By using deep learning architectures to perform video compression and/or decompression, aspects of the present disclosure can increase the efficiency of video compression and/or decompression on a device. For example, a device using the video coding techniques described may compress video more efficiently using machine learning-based techniques and transmit the compressed video to another device, which may use the techniques described herein. Machine learning-based techniques can be used to decompress compressed video more efficiently.

As described above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from input nodes of an input layer, processing is performed by hidden nodes of one or more hidden layers, and output is generated through output nodes of an output layer. Deep learning networks typically include multiple hidden layers. Each layer of the neural network may include feature maps or activation maps that may contain artificial neurons (or nodes). Feature maps may include filters, kernels, etc. Nodes may contain one or more weights that are used to indicate the importance of one or more nodes in the layers. In some cases, a deep learning network may have a series of many hidden layers, with initial layers used to determine simple, low-level features of the input, and later layers building a hierarchy of more complex and abstract features.

Deep learning architectures can also learn hierarchies of features. For example, when presented with visual data, the first layer may learn to recognize relatively simple features, such as edges, in the input stream. In another example, when presented with auditory data, the first layer may learn to recognize spectral power at specific frequencies. The second layer, taking the output of the first layer as input, may learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers may learn to represent complex shapes in visual data or words in auditory data. Still higher layers may learn to recognize common visual objects or spoken phrases.

Deep learning architectures may perform particularly well when applied to problems that have a natural hierarchical structure. For example, classification of motorized vehicles may benefit from first learning to recognize wheels, windshields and other features. These features may be combined in higher layers in different ways to recognize cars, trucks, and airplanes.

Neural networks may be designed with various connectivity patterns. In feed-forward networks, information is passed from a lower layer to a higher layer, and each neuron in a given layer communicates with neurons in higher layers. A hierarchical representation may be built on successive layers of a feed-forward network, as described above. Neural networks may also have recurrent or feedback (also called top-down) connections. In a circular connection, the output from a neuron in a given layer may be communicated to other neurons in the same layer. Recursive architectures can also help recognize patterns that span more than one chunk of input data that is passed to a neural network in sequence. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. A network with many feedback connections may be helpful when recognition of high-level concepts may assist in identifying specific low-level features of the input.

Connections between layers of a neural network may be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In a fully connected neural network 202, a neuron in the first layer may communicate its output to all neurons in the second layer such that each neuron in the second layer receives input from all neurons in the first layer. will receive. Figure 2B shows an example of a locally connected neural network 204. In a locally connected neural network 204, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, a locally connected layer of a locally connected neural network 204 is such that each neuron in the layer will have the same or similar connectivity pattern, but different values (e.g., 210, 212, 214, and 216). ) may be configured to have connection strengths that may have. Locally connected connectivity patterns may give rise to spatially distinct receptive fields in higher layers, such that higher layer neurons in a given region receive inputs that are tuned through training on the properties of a limited portion of the total input to the network. Because you can.

One example of a locally connected neural network is a convolutional neural network. Figure 2C shows an example of a convolutional neural network 206. Convolutional neural network 206 may be configured such that connection strengths associated with inputs for each neuron in the second layer are shared (e.g., 208). Convolutional neural networks may be well suited to problems where the spatial location of inputs is meaningful. Convolutional neural network 206 may be used to perform one or more aspects of video compression and/or decompression, in accordance with aspects of the present disclosure.

One type of convolutional neural network is a deep convolutional network (DCN). FIG. 2D illustrates a detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capturing device 230, such as a car-mounted camera. DCN 200 in this example may be trained to identify traffic signs and numbers provided on traffic signs. Of course, DCN 200 may be trained for other tasks, such as identifying lane markings or traffic lights.

DCN 200 can also be trained with supervised learning. During training, DCN 200 may be presented with an image, such as an image 226 of a speed limit sign, and then a forward pass may be computed to produce output 222. DCN 200 may include a feature extraction section and a classification section. Upon receiving image 226, convolution layer 232 may apply convolution kernels (not shown) to image 226 to generate a first set of feature maps 218. As an example, the convolution kernel for convolution layer 232 may be a 5x5 kernel that generates 28x28 feature maps. In this example, because four different feature maps are generated from the first set of feature maps 218, four different convolution kernels were applied to image 226 in convolution layer 232. Convolutional kernels may also be referred to as filters or convolutional filters.

The first set of feature maps 218 may be subsampled by a maximum pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220, such as 14x14, is smaller than the size of the first set of feature maps 218, such as 28x28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may further be convolved through one or more subsequent convolution layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

In the example of FIG. 2D , the second set of feature maps 220 are convolved to produce a first feature vector 224. Additionally, first feature vector 224 is further convolved to generate second feature vector 228. Each feature of the second feature vector 228 may include a number corresponding to a possible feature of the image 226, such as “sign”, “60”, and “100”. Softmax function ( (not shown) may convert the numbers in the second feature vector 228 to probabilities.As such, the output 222 of the DCN 200 is the probability of the image 226 containing one or more features.

In this example, the probabilities at output 222 for “sign” and “60” are: “30”, “40”, “50”, “70”, “80”, “90”, and “100”. It is higher than the probabilities of the others in output 222. Before training, the output 222 produced by DCN 200 is likely to be inaccurate. Accordingly, the error may be calculated between output 222 and the target output. The target output is the ground truth of image 226 (e.g., “Sign” and “60”). The weights of DCN 200 may then be adjusted so that the output 222 of DCN 200 is more closely aligned with the target output.

To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient can also indicate the amount by which the error would increase or decrease if the weights were adjusted. In the top layer, the gradient may correspond directly to the values of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted to reduce error. This method of adjusting weights may be referred to as “back propagation” because it involves a “backward pass” through the neural network.

In practice, the error gradient of the weights may be computed over a small number of examples, so that the computed gradient approximates the actual error gradient. This approximation method may also be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the overall system stops decreasing or until the error rate reaches a target level. After training, the DCN may be presented with new images, and a forward pass through the network may yield an output 222 that may be considered the DCN's inference or prediction.

DBN (deep belief network) is a probabilistic model that includes multiple layers of hidden nodes. DBN can also be used to extract hierarchical representations of training data sets. DBN may be obtained by stacking layers of Restricted Boltzmann Machines (RBM). RBM is a type of artificial neural network that can learn probability distributions over a set of inputs. RBMs are often used in unsupervised learning because RBMs can learn probability distributions in the absence of information about the class into which each input should be categorized. Using a hybrid unsupervised and supervised paradigm, the bottom-level RBMs of a DBN can be trained in an unsupervised manner and act as feature extractors, while the top-level RBMs are supervised (with respect to the joint distribution of inputs from the previous layer and target classes). It can be trained as a classifier or it can act as a classifier.

Deep convolutional networks (DCN) are networks of convolutional networks, consisting of additional pooling and normalization layers. DCNs have achieved state-of-the-art performance for many tasks. DCNs can be trained using supervised learning where both input and output targets are known for many examples and are used to modify the network's weights by use of gradient descent methods.

DCN may be a feed-forward network. Additionally, as described above, connections from a neuron in the first layer of the DCN to a group of neurons in the next higher layer are shared across the neurons in the first layer. DCNs' feed-forward and shared connections may be used for fast processing. The computational burden of a DCN may be much less than that of a similarly sized neural network containing, for example, recurrent or feedback connections.

The processing of each layer of a convolutional network may be viewed as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, a convolutional network trained on that input can generate two spatial dimensions along the axes of the image and a third dimension that captures color information. It can also be considered as three-dimensional. The outputs of the convolutional connections may be considered to form a feature map in a subsequent layer, with each element of the feature map (e.g., 220) representing a range of neurons in the previous layer (e.g., feature maps (218) )) and receives input from each of multiple channels. The values in the feature map may be further processed with non-linearity, such as rectification, max(0,x). Values from neighboring neurons may be further pooled, which may correspond to down-sampling and provide additional local invariance and dimensionality reduction.

3 is a block diagram illustrating an example of a deep convolutional network 350. Deep convolutional network 350 may include multiple different types of layers based on connectivity and weight sharing. As shown in Figure 3, deep convolutional network 350 includes convolutional blocks 354A and 354B. Each of the convolution blocks 354A and 354B may be composed of a convolution layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

Convolutional layers 356 may include one or more convolutional filters that may be applied to input data 352 to generate a feature map. Although only two convolutions on blocks 354A and 354B are shown, the present disclosure is not so limited and instead, any number of convolution blocks (e.g., blocks 354A, 354B) may be included in the deep convolutional network 350. Normalization layer 358 may normalize the output of the convolutional filters. For example, normalization layer 358 may provide whitening or lateral suppression. The max pooling layer 360 may provide down-sampling aggregation across space for local invariance and dimensionality reduction.

For example, parallel filter banks of a deep convolutional network may be loaded on CPU 102 or GPU 104 of SOC 100 to achieve high performance and low power consumption. In alternative embodiments, parallel filter banks may be loaded on DSP 106 or ISP 116 of SOC 100. Additionally, deep convolutional network 350 may have access to other processing blocks that may exist on SOC 100, such as sensor processor 114 and navigation module 120, dedicated to sensors and navigation, respectively. .

Deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled “FC1”) and layer 362B (labeled “FC2”). Deep convolutional network 350 may further include a logistic regression (LR) layer 364. There are weights (not shown) to be updated between each layer 356, 358, 360, 362A, 362B, and 364 of the deep convolutional network 350. The output of each of the layers (e.g., 356, 358, 360, 362A, 362B, 364) is input data 352 (e.g., image) supplied from the first convolution block of the convolution blocks 354A. Layers (e.g., 356, 358, 360, 362A, 362B, 364) in deep convolutional network 350 to learn hierarchical feature representations from fields, audio, video, sensor data, and/or other input data. ) can function as one of the following inputs. The output of the deep convolutional network 350 is the classification score 366 for the input data 352. Classification score 366 may be a set of probabilities, where each probability is a probability of the input data containing a feature from the set of features.

As previously mentioned, digital video data can contain large amounts of data, which can place a significant burden on communication networks as well as devices that process and store video data. For example, recording uncompressed video content typically results in large file sizes that increase significantly as the resolution of the recorded video content increases. In one illustrative example, uncompressed 16-bit per channel video recorded at 1080p/24 (e.g., a resolution of 1920 pixels wide and 1080 pixels high, at which 24 frames per second are captured) is a frame It can occupy 12.4 megabytes per second, or 297.6 megabytes per second. Uncompressed 16-bit per channel video recorded at 4K resolution at 24 frames per second can occupy 49.8 megabytes per frame, or 1195.2 megabytes per second.

Network bandwidth is another limitation that can be problematic for large video files. For example, video content is often delivered over wireless networks (e.g., via LTE, LTE-Advanced, New Radio (NR), WiFi™, Bluetooth™, or other wireless networks) and is often transmitted over consumer Internet traffic. It can make up a large part. Despite advances in the amount of bandwidth available in wireless networks, it may still be desirable to reduce the amount of bandwidth used to deliver video content in these networks.

Because uncompressed video content can result in large files that can involve significant memory for physical storage and significant bandwidth for transmission, video coding techniques may be used to compress and then decompress such video content. .

In order to reduce the size of video content - and therefore the amount of storage involved in storing the video content - and the amount of bandwidth involved in delivering the video content, various video coding techniques have been developed, particularly for specific video formats such as HEVC, AVC, MPEG, and VVC. It can be performed according to coding standards. Video coding often uses prediction methods such as inter-prediction or intra-prediction, which exploit redundancies present in video images or sequences. A common goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradations to video quality. As demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

Typically, an encoding device encodes video data according to a video coding standard to produce an encoded video bitstream. In some examples, an encoded video bitstream (or “video bitstream” or “bitstream”) is a series of one or more coded video sequences. An encoding device can generate coded representations of pictures by partitioning each picture into multiple slices. A slice is independent of other slices so that information in that slice is coded without dependence on data from other slices within the same picture. A slice includes one or more slice segments, including an independent slice segment and, if any, one or more dependent slice segments that depend on previous slice segments. In HEVC, slices are partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. The CTB of luma samples and one or more CTBs of chroma samples, along with the syntax for the samples, are referred to as a coding tree unit (CTU). A CTU may also be referred to as a “tree block” or “largest coding unit” (LCU). CTU is the basic processing unit for HEVC encoding. A CTU can be divided into multiple coding units (Cus) of various sizes. A CU contains luma and chroma sample arrays, referred to as coding blocks (Cbs).

Luma and chroma CBs can be further divided into prediction blocks (PBs). A PB is a block of samples of the luma component or chroma component that uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled for use). The luma PB and one or more chroma PBs, together with the associated phrases, form a prediction unit (PU). For inter-prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream for each PU and used for inter-prediction of the luma PB and one or more chroma PBs. do. Motion parameters may also be referred to as motion information. A CB may also be partitioned into one or more transform blocks (Tbs). TB represents a square block of samples of the color component to which a residual transform (eg, in some cases the same two-dimensional transform) is applied to code the prediction residual signal. A transformation unit (TU) represents TBs of luma and chroma samples and corresponding syntax elements. Transform coding is described in more detail below.

According to the HEVC standard, transformations may be performed using TUs. TUs may be sized based on the size of PUs within a given CU. TUs are typically the same size as the PUs or may be smaller than the PUs. In some examples, residual samples corresponding to a CU may be subdivided into smaller units using a quadtree structure, known as a residual quad tree (RQT). Leaf nodes of an RQT may correspond to TUs. Pixel difference values associated with TUs may be transformed to form transform coefficients. The transform coefficients may then be quantized by the encoding device.

Once the pictures of video data are partitioned into CUs, the encoding device predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain the residuals (described below). For each CU, the prediction mode may be signaled within the bitstream using syntax data. The prediction mode may include intra prediction (or intra-picture prediction) or inter prediction (or inter-picture prediction). Intra prediction uses correlation between spatially neighboring samples within a picture. For example, using intra prediction, each PU can be subjected to, for example, a DC prediction to find the average value for the PU, a planar prediction to fit a planar surface for the PU, and extrapolation from neighboring data. is predicted from neighboring image data within the same picture using directional prediction, or any other suitable type of prediction. Inter prediction uses temporal correlation between pictures to derive motion compensated predictions for blocks of image samples. For example, using inter prediction, each PU is predicted using motion compensated prediction from image data in one or more reference pictures (before or after the current picture in output order). The decision whether to code a picture region using inter-picture or intra-picture prediction may be made at the CU level, for example.

After performing prediction using intra and/or inter prediction, the encoding device can perform transformation and quantization. For example, following prediction, the encoding device may calculate residual values corresponding to the PU. Residual values may include pixel difference values between the current block (PU) of pixels being coded and the prediction block used to predict the current block (eg, a predicted version of the current block). For example, after generating a prediction block (e.g., issuing inter-prediction or intra-prediction), the encoding device may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block contains a set of pixel difference values that quantify the difference between the pixel value of the current block and the pixel value of the prediction block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In this example, the residual block is a two-dimensional representation of pixel values.

Any residual data that may remain after the prediction is performed is transformed using a block transform that may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, another suitable transform function, or any combination thereof. . In some cases, one or more block transforms (e.g., sizes 32 x 32, 16 x 16, 8 x 8, 4 x 4, or other suitable sizes) may be applied to the residual data of each CU. In some embodiments, a TU may be used for transformation and quantization processes implemented by an encoding device. A given CU with one or more PUs may also include one or more TUs. As described in more detail below, the residual values may be converted to transform coefficients using a block transform, and then quantized and scanned using a TU to generate serialized transform coefficients for entropy coding.

The encoding device may perform quantization of transform coefficients. Quantization provides additional compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, coefficients with n-bit values may be rounded down to m-bit values during quantization, where n is greater than m.

Once quantization is performed, the coded video bitstream contains any other appropriate data, such as quantized transform coefficients, prediction information (e.g., prediction mode, motion vector, block vector, etc.), partitioning information, and other syntax data. Includes. Different elements of the coded video bitstream may then be entropy encoded by the encoding device. In some examples, the encoding device may scan the quantized transform coefficients utilizing a predefined scan order to generate a serialized vector that can be entropy encoded. In some examples, the encoding device may perform adaptive scan. After scanning the quantized transform coefficients to form a vector (eg, a one-dimensional vector), the encoding device may entropy encode the vector. For example, the encoding device may use context adaptive variable length coding, context adaptive binary arithmetic coding, syntax-based context adaptive binary arithmetic coding, stochastic interval partitioning entropy coding, or other suitable entropy encoding techniques.

The encoding device can store the encoded video bitstream and/or transmit the encoded video bitstream data over a communication link to a receiving device, which can include a decoding device. A decoding device may decode encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more coded video sequences that make up the encoded video data. The decoding device may then rescale and perform inverse transformation on the encoded video bitstream data. The residual data is then passed to the prediction stage of the decoding device. The decoding device then predicts a block of pixels (e.g., PU) using intra-prediction, inter-prediction, IBC, and/or other types of prediction. In some examples, prediction is added to the output of the inverse transform (residual data). The decoding device may output the decoded video to a video destination device, which may include a display or other output device for displaying the decoded video data to a consumer of the content.

Video coding systems and techniques defined by various video coding standards (e.g., the HEVC video coding techniques described above) may be able to retain much of the information in the raw video content, and may require signal processing and information It may also be defined a priori based on theoretical concepts. However, in some cases, machine learning (ML)-based image and/or video systems offer advantages over non-ML-based image and video coding systems, such as deep learning-based end-to-end video coding (DLEC) systems. can be provided. As mentioned above, many deep learning based systems are a combination of an autoencoder sub-network and a second sub-network responsible for learning a probabilistic model for the quantized latencies used for entropy coding. It is designed. This architecture can be considered a combination of transformation, quantization modules (encoder sub-network) and entropy modeling sub-network modules.

FIG. 4 illustrates a system 400 that includes a device 402 configured to perform video encoding and decoding using a deep learning based system 410 . Device 402 is coupled to camera 407 and storage medium 414 (e.g., a data storage device). In some implementations, camera 407 is configured to provide image data 408 (e.g., a video data stream) to processor 404 for encoding by deep learning based system 410. In some implementations, device 402 may be coupled to and/or include multiple cameras (e.g., a dual camera system, three cameras, or another number of cameras). In some cases, device 402 may be coupled to a microphone and/or other input device (e.g., a touch input device such as a keyboard, mouse, touchscreen, and/or touchpad, and/or other input device). there is. In some examples, a camera 407, storage medium 414, microphone, and/or other input device may be part of device 402.

Device 402 is also coupled to a second device 490 via a transmission medium 418, such as one or more wireless networks, one or more wired networks, or a combination thereof. For example, transmission medium 418 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. Transmission medium 418 may form part of a packet-based network, such as a local area network, a wide area network, or a global network, such as the Internet. Transmission medium 418 may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from a source device to a destination device. The wireless network may include any wireless interface or combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, packet-based network, WiFiTM, radio frequency (RF), UWB, WiFi- Direct, cellular, Long-Term Evolution (LTE), WiMaxTM, etc.) A wired network may include any wired interface (e.g., fiber, Ethernet, powerline Ethernet, Ethernet over coaxial cable, Digital Signal Line (DSL), etc.). Wired and/or wireless networks can be implemented using a variety of equipment such as base stations, routers, access points, bridges, gateways, switches, etc. Encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.

Device 402 includes memory 406, one or more processors (herein referred to as "I/F 1") coupled to a first interface ("I/F 1") 412 and a second interface ("I/F 2") 416. (referred to as “processor”) 404. Processor 404 is configured to receive image data 408 from camera 407, memory 406, and/or storage medium 414. Processor 404 is coupled to storage medium 414 via a first interface 412 (e.g., via a memory bus) and a second interface 416 (e.g., a network interface device, wireless transceiver). and an antenna, one or more other network interface devices, or a combination thereof) to a transmission medium 418.

Processor 404 includes deep learning based system 410. Deep learning based system 410 includes an encoder portion 462 and a decoder portion 466. In some implementations, deep learning-based system 410 may include one or more autoencoders. Encoder portion 462 is configured to receive input data 470 and process input data 470 to generate output data 474 based at least in part on input data 470 .

In some implementations, the encoder portion 462 of the deep learning-based system 410 is configured to perform lossy compression of the input data 470 to generate output data 474, such that the output data 474 is It has fewer bits than 470. Encoder portion 462 encodes input data 470 (e.g., images or video frames) without using motion compensation based on any previous representations (e.g., one or more previously reconstructed frames). ) can be trained to compress. For example, encoder portion 462 may compress a video frame using video data only from that video frame and not using any data from previously reconstructed frames. Video frames processed by encoder portion 462 may be referred to herein as intra-predicted frames (I-frames). In some examples, I-frames may be generated using traditional video coding techniques (e.g., according to HEVC, VVC, MPEG-4, or another video coding standard). In these examples, processor 404 may include or be coupled to a video coding device (e.g., an encoding device) configured to perform block-based intra prediction such as described above for the HEVC standard. In these examples, deep learning based system 410 may be excluded from processor 404.

In some implementations, encoder portion 462 of deep learning-based system 410 uses motion compensation based on a previous representation (e.g., one or more previously reconstructed frames) to encode input data 470 (e.g., can be trained to compress video frames). For example, encoder portion 462 may compress a video frame using video data from that video frame and using data from previously reconstructed frames. Video frames processed by encoder portion 462 may be referred to herein as intra-predicted frames (P-frames). Motion compensation can be used to determine data in the current frame by accounting for how pixels from a previously reconstructed frame move to new positions in the current frame along with residual information.

As shown, the encoder portion 462 of the deep learning based system 410 may include a neural network 463 and a quantizer 464. Neural network 463 may include one or more convolutional neural networks (CNNs), one or more fully-connected neural networks, one or more gated recurrent units (GRUs), one or more long-term short-term memory (LSTM) networks, one or more ConvRNNs, and one or more ConvGRUs. , may include one or more ConvLSTMs, one or more GANs, any combination thereof, and/or other types of neural network architectures that generate intermediate data 472. Intermediate data 472 is input to quantizer 464. Examples of components that may be included in encoder portion 462 are illustrated in FIGS. 6-10.

Quantizer 464 is configured to perform quantization and, in some cases, entropy coding of intermediate data 472 to produce output data 474. Output data 474 may include quantized (and in some cases entropy coded) data. Quantization operations performed by quantizer 464 may result in the generation of quantized codes (or data representing quantized codes generated by deep learning based system 410) from intermediate data 472. Quantization codes (or data representing quantized codes) may also be referred to as latent codes or latent (denoted z). The entropy model applied to potential may be referred to herein as a “prior”. In some examples, quantization and/or entropy coding operations may be performed using existing quantization and entropy coding operations performed when encoding and/or decoding video data according to existing video coding standards. In some examples, quantization and/or entropy coding operations may be performed by deep learning based system 410. In one illustrative example, deep learning-based system 410 can be trained using supervised training, where residual data is used as input, and quantized codes and entropy codes are used to produce known outputs (labels) during training. s) are used.

Decoder portion 466 of deep learning based system 410 is configured to receive output data 474 (e.g., directly from quantizer 464 and/or from storage medium 414). Decoder portion 466 may process output data 474 to generate a representation 476 of input data 470 based at least in part on output data 474 . In some examples, the decoder portion 466 of deep learning-based system 410 may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more and a neural network 468, which may include one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and/or other types of neural network architectures. Examples of components that may be included in decoder portion 466 are illustrated in FIGS. 6-10.

Processor 404 is configured to transmit output data 474 to at least one of transmission medium 418 or storage medium 414 . For example, output data 474 may be stored on a storage medium (or decompressed) for later retrieval and decoding (or decompression) by decoder portion 466 to produce a representation 476 of input data 470 as reconstructed data. 414). The reconstructed data can be used for various purposes, such as for playback of encoded/compressed video data to generate output data 474. In some implementations, output data 474 is reconstructed data in another decoder device (e.g., device 402) that matches decoder portion 466 to produce a representation 476 of input data 470. , at the second device 490, or at another device). For example, second device 490 may include a decoder that matches (or substantially matches) decoder portion 466 and output data 474 is transmitted to second device 490 over transmission medium 418. ) may also be transmitted. The second device 490 may process the output data 474 to generate a representation 476 of the input data 470 as reconstructed data.

Components of system 400 include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and /or other suitable electronic circuits), and/or may be implemented using electronic circuits or other electronic hardware, and/or use a computer to perform the various operations described herein. It may include and/or be implemented using software, firmware, or any combination thereof.

Although system 400 is shown as including specific components, those skilled in the art will understand that system 400 may include more or fewer components than those shown in FIG. 4 . For example, system 400 may also include or be part of a computing device that includes input devices and output devices (not shown). In some implementations, system 400 also includes one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components) , database components, and/or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and /or other processing devices), one or more wireless interfaces for performing wireless communications (e.g., including a baseband processor and one or more transceivers for each wireless interface), communicating via one or more hardwired connections one or more wired interfaces (e.g., a serial interface, such as a Universal Serial Bus (USB) input, Lightning connector, and/or other wired interface), and/or other components not shown in Figure 4. It may include, or may be part of, a computing device.

In some implementations, system 400 may be implemented locally by and/or included in a computing device. For example, computing devices include mobile devices, personal computers, tablet computers, virtual reality (VR) devices (e.g., head-mounted displays (HMDs) or other VR devices), augmented reality (AR) devices (e.g., HMD, AR glasses, or other AR devices), wearable devices, servers (e.g., software as a service (SaaS) systems or other server-based systems), televisions, and/or devices for performing the techniques described herein. Can include any other computing device with resource capabilities.

In one example, the deep learning based system 410 includes a memory 406 coupled to the processor 404 and configured to store instructions executable by the processor 404, and an antenna coupled to the processor 404 and output data ( 474) may be incorporated into a portable electronic device including a wireless transceiver operable to transmit to a remote device.

As mentioned above, deep learning based systems are typically designed to process non-subsampled input formats such as RGB or YUV 4:4:4. For examples of image and video coding methods targeting RGB input, see J. Balle, D. Minnen, S. Singh, S. J. Hwang, N. Johnston, "Variational image compression with a scale hyperprior", ICLR, 2018 ("J. Balle Paper") and D. Minnen, J. Balle, G. Toderici, "Joint Autoregressive and Hierarchical Priors for Learned Image Compression", CVPR 2018 (referred to as the "D. Minnen Paper") It is incorporated herein by reference in its entirety and for all purposes.

Figure 5 is a diagram showing an example of a deep learning-based system 500. The g _a and g _s subnetworks in the deep learning-based system of Figure 5 correspond to an encoder subnetwork (e.g., encoder portion 462) and a decoder subnetwork (e.g., decoder portion 466), respectively. . The g _a and g _s sub-networks in Figure 5 are designed for a three-channel RGB input, where all three R, G, and B input channels pass through the same neural network layers (convolutional layers and generalization processed by segmented normalization (GDN) layers). Neural network layers may include convolutional layers (including convolutional layer 510) that perform convolutional operations and GDN and/or inverse-GDN (IGDN) nonlinear layers that implement local division normalization. Local partition normalization is a type of transformation that has been shown to be particularly suitable for density modeling and compression of images. Deep learning based systems (such as the one shown in Figure 5) target input channels with similar statistical properties, such as RGB data (statistical properties of different R, G and B channels are similar).

Although many deep learning-based systems are designed to process RGB input, most image and video coding systems use YUV input formats (e.g., YUV 4:2:0 input format in many cases). The chrominance (U and V) channels of data in YUV format may be subsampled with respect to the luminance (Y) channel. Subsampling results in minimal effect on visual quality (e.g., no significant or noticeable effect on visual quality). Subsampled formats include YUV 4:2:0 format, YUV 4:2:2 format, and/or other YUV formats. Correlation across channels is reduced in YUV format, which may not be the case for other color formats (eg, RGB format). Additionally, the statistics of the luminance (Y) and chrominance (U and V) channels are different. For example, the U and V channels have smaller variance compared to the luminance channel, whereas in RGB formats, for example, the statistical properties of the different R, G, and B channels are more similar. Video coders-decoders (or codecs) are designed according to the input characteristics of the data (eg, a codec can encode and/or decode data according to the input format of the data). For example, if the chrominance channels of a frame are subsampled (e.g., the chrominance channels are half the resolution compared to the luminance channels), then when CODEC predicts blocks of the frame for motion compensation, the luminance blocks are the chrominance blocks. It will be twice as large in both width and height compared to . In another example, CODEC may determine how many pixels to be encoded or decoded for chrominance and luminance, among other things.

To support YUV formats (e.g., YUV 4:2:0 format), deep learning-based architectures must be modified. For example, if RGB input data (which, as mentioned above, most deep learning-based systems are designed to process) is replaced with YUV 4:4:4 input data (all channels have the same dimension), the input data can be The performance of deep learning-based systems processing is reduced due to the different statistical properties of the luminance (Y) and chrominance (U and V) channels. As mentioned above, chrominance (U and V) channels are subsampled in some YUV formats, such as in the case of YUV 4:2:0. For example, for content with the YUV 4:2:0 format, the U and V channel resolution is half the Y channel resolution (one quarter of the Y channel because the U and V channels are half the width and height). has a size that is). This subsampling may make the input data incompatible with the input of deep learning-based systems. The input data is the information that the deep learning-based system is attempting to encode and/or decode (e.g., a YUV frame containing three channels including luminance (Y) and chrominance (U and V) channels).

In some end-to-end video coding deep learning-based systems, autoencoders code intra-frames relative to the original frames, motion vectors (e.g., dense optical flow), and residuals of motion-compensated frames. It is used for. In one example, a flow autoencoder can be used to jointly learn to code the scale-space as well as the optical flow, and a residual autoencoder codes the residual between the warped prediction frame and the original frame both in the RGB domain. .

As described above, a system providing an ML-based system (e.g., comprising one or more deep learning-based architectures) that efficiently supports one or more YUV formats (e.g., YUV 4:2:0 format), and The techniques are described herein. Deep learning-based architecture(s) may encode and/or decode video data containing stand-alone frames (or images) and/or multiple frames. For example, an ML-based system may obtain as input the luma component of the current frame and the luma component of a previously-reconstructed frame, which can be reconstructed by a previous instance of the ML-based system. An ML-based system may process luma components of current and previous frames to estimate motion information (e.g., flow information, such as optical flow information) for the luma component of the current frame. Using the luma component of the current frame, an ML-based system can determine (e.g., estimate) a motion estimate (e.g., flow information such as optical flow information) for one or more chroma components of the current frame. there is. This technique can be performed on all components of the frame. More details will be provided later.

FIG. 6 is a diagram illustrating an example of a neural network architecture of a deep learning-based system 600 configured to perform video coding. The neural network architecture of Figure 6 includes an intra prediction engine 602 and an inter prediction engine 610. Intra prediction engine 602 and inter prediction engine 610 may include autoencoders (e.g., variable autoencoders (VAE)) as shown in FIG. 6, but may include other types of autoencoders in other implementations. May include neural network architectures. As shown, the intra prediction engine 602 generates a latent representation of the input frame 604 ( (shown as) processes the pixel information of the input frame 604. The input frame 604 includes a luma component ( shown as ) and two chroma components ( and shown as ). A latent representation may also be referred to as a bitstream containing a number of bits that are a coded version of the input frame 604. latent expression Based on the latent representation/bitstream received from another device, the decoder sub-network of intra-prediction engine 602 generates a reconstructed frame 606 (or a reconstructed version of the input frame 604). , , shown as , where “hats” for components indicate reconstructed values).

The inter-prediction engine 610 includes a flow engine 618, a residual engine 620, and a warping engine 622. As shown, the flow engine 618 takes as input the luma component (at time t) of the current frame 614: ) and the reconstructed luma component (at previous time t-1) of the previous frame 615 ) to obtain. Luma component ( ) and luma component ( ), the flow engine 618 generates the luma component ( Potential representation of motion information (e.g., flow information) for ) ( ) is created. Motion information may be optical flow information (e.g., a plurality of motion or displacement vectors) indicating movement of pixels of the current frame 614 (at time t) relative to the previous frame 615 (at time t-1). and in some cases a per-pixel or per-sample scale component). Potential expression ( ) can also be referred to as a bitstream, and the luma component for the current frame 614 ( ) may contain a number of bits representing a coded version of The flow engine (618) uses luma components (not chroma components) for the current frame (614). ), the potential expression ( )(bitstream) is reduced in size compared to using all components of the current frame 614 to determine motion information.

Luma component ( )'s potential expression ( ) (or a bitstream received from another device representing a component of the frame), the flow engine 618 generates the luma component ( ), and also determine the motion information (denoted as f ^L ) for the chroma components of the current frame 614 ( , ) determine the motion information (denoted as f ^C ). Details of determining or estimating motion information for the chroma component based on the determined motion information for the luma component are described below with reference to FIGS. 7A and 7B.

The warping engine 622 determines the luma component (at time t) of the current frame 614. ) and chroma components ( , ) is configured to perform warping using the motion information (f ^L and f ^C ) determined for. For example, the warping engine 622 may determine the luma component ( ) and chroma components ( , ) can warp the pixels of the current frame 614 (at time t) by the amount indicated by the motion information (f ^L and f ^C ). In some aspects, warping engine 622 may perform spatial-scale flow (SSF) warping. For example, SSF warping can apply trilinear interpolation to generate inter-frame predictions from learned scale-flow vectors, where the predictors can be formulated as:

, and

thus:

Equation (1)

The above trilinear interpolation uses the luma component ( ) and chroma components ( ) performed on a component-by-component basis (e.g., for each luma component and for each individual U and V chroma component) based on one or more warping parameters determined based on the motion information (f ^L and f ^C ) of It can be. For example, warping parameters represent the horizontal component (in the x-direction) of a motion or displacement vector. , representing the vertical component (in the y-direction) of a motion or displacement vector. , (referred to as the scale field) representing a progressively smoothed version of the reconstructed frames combined with spatial motion/displacement information (vx and vy) It may also include .

The output from the warping engine 622 (after warping is performed by the warping engine 622) is , , Includes the predictions shown in Figure 6 as , where is the luma component ( ) corresponds to the prediction for, is the chroma component ( ) corresponds to the prediction for, is the chroma component of the current frame (614) ( ) corresponds to the prediction for.

Next, the deep learning-based system 600 generates a residual signal for the luma component ( ), chroma component ( ) for the residual signal ( ), and chroma component ( ) for the residual signal ( ), the corresponding luma component of the current frame 614 ( ) and chroma components ( , ) predictions from ( , , ) can be subtracted. The residual engine 620 generates a potential expression for the residual ( ) can be created. Potential expression of residual ( ) (or a potential representation of the residual received from another device), the residual engine 620 generates the reconstructed residual signal for the luma component ( ), chroma component ( ) for the reconstructed residual signal ( ), and chroma component ( ) for the reconstructed residual signal ( ) can be generated for the current frame including the reconstructed residual. The deep learning-based system 600 uses the reconstructed residual ( , , ) to predictions ( , , ) can be added.

7A shows the luma component of the current frame (at time t), collectively shown as luma components 722. ) and the reconstructed luma component of the previous frame (at time t-1) ( ) is a diagram illustrating an example of the flow engine 718 operating as ). As described above, in some cases the flow engine 718 may be implemented as an autoencoder (VAE _flow) . In some cases, the combined deep learning-based architecture, as shown in Figure 7A, allows the flow engine 718 to generate luma motion information (e.g., SSF f ^L ) and chroma motion information (e.g., SSF To estimate f ^C ), the current frame ( ) and previously reconstructed frames ( ) can be designed to use both luma components. For example, as described herein, chroma motion information (eg, f ^C ) can be derived based on luma motion information (eg, f ^L ).

As shown in Figure 7a, the luma component of the current frame ( ), the ^luma component of the current frame ( ) and the reconstructed luma component of the previous frame ( ) are processed by several convolutional layers and activation layers (collectively shown as forward pass 723). The notations “↓2” and “↑2” in Figure 7A refer to stride values, where ↓2 refers to a stride of 2 (for downsampling as indicated by ↓”), and ↑2 It also refers to a stride of 2 (for upsampling as indicated by ↑"). For example, convolutional layer 724 may multiply the input luma components by a factor of 4 by applying a 5x5 convolutional filter in the horizontal and vertical dimensions by a stride value of 2. and ) downsample. The resulting output of the convolutional layer 724 is the luma component of the current frame ( ) are N arrays of feature values (corresponding to N channels) representing luma motion information (f ^L ) for ). The notation “2/N” indicates two input channels and N output channels. A non-linear layer following the convolutional layer 724 may process the feature values output by the convolutional layer 724. Each of the successive convolutional layers and nonlinear layers outputs the features output by the previous layer until the final convolutional layer 725 of the forward pass 723 outputs features to the bottleneck portion 726 of the flow engine 718. Features can be processed.

The output of the forward pass 723 is processed by the bottleneck portion 726 of the flow engine 718 to produce the luma component ( ) Generates a bitstream or rateant representing the luma motion information (f ^L ). The bottleneck portion 726 may include a quantization engine and an entropy encoding engine in the forward pass 723 and an entropy decoding engine and an inverse quantization engine in the reverse pass 728 of the flow engine 718. For example, the quantization engine may perform quantization on the features output by the final convolution layer 725 of the forward pass 723 to generate a quantized output. The entropy encoding engine may generate a bitstream by entropy encoding the quantized output from the quantization engine. In some cases, the entropy encoding engine may use the prior generated by the hyperprior network to perform entropy encoding. A neural network system may output a bitstream to a server device or system for storage, transmission to another device, and/or otherwise output the bitstream.

The reverse pass 728 may, in some cases, be a decoder sub-network of a neural network system of flow engine 718 or a decoder sub-network of a neural network system of another flow engine (on another device). The entropy decoding engine of the flow engine 718 entropy decodes the bitstream output by the entropy encoding engine of the bottleneck 726 (or the entropy encoding engine of the bottleneck of another flow engine), and reverse passes the entropy decoded data to 728. ) can be output to the inverse quantization engine. The entropy decoding engine may use the fryer generated by the hyperprior network to perform entropy decoding. The inverse quantization engine can inverse quantize data.

The convolution layers and inverse activation layers of the backward pass 728 then process the dequantized data from the bottleneck 726 to generate the luma component ( ) can also generate motion information 729 (f ^L ). Motion information (729) (f ^L ) is the luma component of the current frame ( ), such as a motion vector for each sample (e.g., having a magnitude in the horizontal or x-direction and a magnitude in the vertical or y-direction). In some cases, motion information 729(fL) may further include a scale component. For example, as shown in FIG. 7A for illustration purposes, motion information 729 is ingredient, ingredients, and Contains ingredients. As mentioned above, , , and Ingredients are predictions , , may be used in equation (1) by the warping engine 622 to warp the pixels of the current frame 614 (at time t) to generate .

The luma component of the current frame ( ), the flow engine 718 may determine or predict motion information 731 (f ^C ) for chroma components of ^the current frame. For example, the flow engine 718 uses the luma component (f C) to obtain motion information 731 (f ^C ) for the chroma components. ) motion information 729 (f ^L ) may be subsampled. Motion information 731 (f ^C ) for chroma components may include motion vectors (e.g., magnitude in the horizontal or x-direction and vertical or y - has a size in the direction). In some cases, motion information 731(f ^C ) may further include a scale component. For example, as shown in FIG. 7A, motion information 731 (f ^C ) for chroma components of the current frame is ingredient, ingredients and Contains ingredients. Similar to the motion information 729 (f ^L ) for the luma component, the chroma motion information 731 (f ^C ) , and The components are used in equation (1) to warp the pixels of the current frame 614 (at time t) by the warping engine 622 to generate the predictions. , , can be created.

In some aspects, convolutional layer 730 with down-sampling determines the luma component of the current frame ( ) can be trained (e.g., using unsupervised learning or training) to learn motion information 731 (fC) for the chroma components of the current frame based on motion information 729 (fL) for there is. In one illustrative example, a training set that may be used to train flow engine 718 may include luma and chroma motion information (as ground truth). Luma motion information may be input to the neural network of the flow engine 718, and the resulting chroma motion information output from the flow engine 718 may be converted to a loss function (e.g., L1 or sum of absolute differences, L2 norm or square). It can be minimized using the ground truth chroma motion information (sum of differences, or other loss function).

The convolution layer 730 is |3/3|5x5 conv↓2| in FIG. 7A. It is displayed as . The notation "3/3" indicates that there are three input channels resulting in three output channels. As mentioned above, the notations "↓2" and "↑2" refer to stride values, with ↓2 referring to a stride of 2 for downsampling (as indicated by "↓"), and ↑2 Refers to a stride of 2 for upsampling (as indicated by “↑”). For example, the convolution layer 730 may apply a 5x5 convolutional filter in the horizontal and vertical dimensions with a stride value of 2 to obtain the luma component ( ) Downsample the motion information 729 (f ^L ) by a factor of 4 (e.g., for YUV 4:2:0 format). In some examples, convolutional layer 730 may be trained to downsample by different factors for different formats (eg, YUV 4:2:2 format, etc.). The resulting output of the convolutional layer 724 is the luma component ( ) is a 3x3 array of feature values (corresponding to three channels), which is a downsampled version of the motion information 729 (f ^L ) for ).

In other aspects (not shown in FIG. 7A), the motion information 731 (fC) for the chroma component of the current frame is the luma component ( ) can be obtained by directly subsampling the motion information 729 (f ^L ). For example, flow engine 718 can determine chroma flow without using convolutional layer 730 to process luma flow(s). In one illustrative example, instead of the convolutional layer 730, flow engine 718 directly subsamples luma motion information 729 (f ^L ) to obtain chroma motion information 731 (f ^C ). It may include a subsampler (which may be separate from the neural network of the flow engine 718).

7B shows a subsampling engine 735 for subsampling the determined luma motion information for the current frame (e.g., using the flow engine 718 of FIG. 7A) to obtain chroma motion information for the current frame. This is a diagram illustrating an example. For illustrative purposes, a simplified example is provided with each N channel (N = 2) of luma motion information 732 having a resolution of 4x4 (4 rows and 4 columns), for a total of 16 flow motion or displacement It has vectors. The subsampling engine 735 subsamples or downsamples the luma motion information 732 to generate or obtain chroma motion information 738, which is a subsampled/downsampled version of the luma motion information 732.

The illustrative example of FIG. 7B shows chroma motion information 738 that is a quarter of the size of luma motion information 732. For example, as previously explained, for content with the YUV 4:2:0 format, the U and V channel resolution is half the Y channel resolution (since the U and V channels are half the width and height). , has a size that is one quarter of the Y channel). Subsampling engine 735 may be trained or otherwise configured to handle formats other than the 4:2:0 format, in which case subsampling includes generating chroma information with different resolutions than shown in FIG. 7A. can do.

In some aspects, as described above, subsampling engine 735 may be trained (e.g., using unsupervised learning or training) to determine chroma motion information 738 from luma motion information 732. It may include the convolution layer 730 of FIG. 7A. In other aspects, subsampling engine 735 may include a subsampler that directly subsamples luma motion information 732 to obtain chroma motion information 738.

The number of channels in the bottleneck M (denoted N in Figure 7A) as well as in the convolution or transform layers of the forward pass 723 and backward pass 728 may be set to any suitable value. In one illustrative example, the number of channels N may be selected as N=192 and M=128. Successive smoothed versions of the reconstructed frames (associated with the scale field s) can be obtained by using a filtering or smoothing operator. In one example, a Gaussian blurring filter with different widths may be used. In another example, a Gaussian pyramid with successive filtering and interpolation can be used to generate smoothed versions of the reconstructed frames. Additionally, an arbitrarily large number of scales S may be used. In one example, scale S can be set to S=3, and the scale level is can be selected as, where may represent the Gaussian filter width.

The non-linear activation layers in FIG. 7A are shown as PReLUs for illustrative purposes, but other types of non-linear activation layers may be used, such as generalized divisive normalization (GDN) layers, a combination of PReLU and GDN layers.

In some examples, to effectively support one or more YUV formats (e.g., YUV 4:2:0), intra prediction engine 602 and residual engine 620 of FIG. 6 may be configured to support one or more YUV formats (e.g., YUV 4:2:0). It can be designed based on the general neural network architectures shown in FIG. 10. For example, the architectures shown in FIGS. 8A, 9, and 10 may be configured to process input data having the YUV 4:2:0 format. In some examples, a neural network architecture similar to that shown in Figures 8A, 9, or 10 can support other types of YUV content (e.g., content with YUV 4:4:4 format, YUV 4:2:2 format, etc. ) and/or can be used to encode and/or decode content with other input formats. In some cases, each architecture shown in FIGS. 8A, 9, and 10 includes a residual autoencoder operating with a YUV (eg, 4:2:0) residual.

FIG. 8A is a diagram illustrating an example of a front-end neural network system 800 that can be configured to operate directly with 4:2:0 input (Y, U, and V) data. As shown in Figure 8A, in the encoder sub-network (also referred to as the forward pass) of the neural network system, branched luma and chroma channels (luma Y channel 802 and U and V chroma channels 804) ) are combined using a 1x1 convolutional layer 806, and then a non-linear layer 808 (also referred to as a non-linear operator) is applied. Similar operations are performed in the decoder sub-network (also called backward pass) of a neural network system, but in reverse order. For example, as shown in Figure 8A, an inverse non-linear layer 809 (also referred to as an inverse non-linear operator) is applied, the Y and U, V channels are separated using a 1x1 convolution layer 813, and , individual Y and U, V channels are processed using respective inverse nonlinear layers 815, 816 and convolutional layers 817, 818.

The first two neural network layers of the encoder sub-network of the neural network system 800 in FIG. 8A are a first convolutional layer 811 (denoted as Nconv |3x3|↓1), a second convolutional layer 810 (denoted as Nconv |5x5|↓2), and includes a first non-linear layer 814 and a second non-linear layer 812. The last two neural network layers in the decoder sub-network of the front-end neural network architecture of Figure 8A are the first inverse non-linear layer 816, the second inverse non-linear layer 815, and the reconstructed chrominance of the frame (U and V). ) components (denoted as 2conv |3x3|↑1), and a second convolutional layer 817 (denoted as 2conv | (indicated as 5x5|↑2). The "Nconv" notation refers to the number of output channels (N) (corresponding to the number of output features) of a given convolutional layer (the value of N defines the number of output channels). The 3x3 and 5x5 notations indicate the size of each convolution kernel (e.g., 3x3 kernel and 5x5 kernel). The notations "↓1" and "↓2" refer to stride values, where ↓1 refers to a stride of 1 (for downsampling, as indicated by ↓"), and ↓2 refers to (for downsampling) (for upsampling) refers to a stride of 1. The notations "↑1" and "↑2" refer to stride values, where ↑1 refers to a stride (for upsampling as indicated by ↑") of 1 and , ↑2 refers to a stride of 1 (for upsampling).

For example, convolutional layer 810 downsamples input luma channel 802 by a factor of 4 by applying a 5x5 convolutional filter in the horizontal and vertical dimensions by a stride value of 2. The resulting output of the convolutional layer 810 is N arrays of feature values (corresponding to N channels). The convolutional layer 811 processes the input chroma (U and V) channels 804 by applying a 3x3 convolutional filter in the horizontal and vertical dimensions with a stride value of 1. The resulting output of the convolutional layer 811 is N arrays of feature values (corresponding to N channels). The arrays of feature values output by the convolution layer 810 have the same dimensions as the arrays of feature values output by the convolution layer 811. Non-linear layer 812 can then process the feature values output by convolution layer 810, and non-linear layer 814 can process feature values output by convolution layer 811.

The 1x1 convolutional layer 806 may then process the feature values output by the non-linear layers 812 and 814. The 1x1 convolution layer 806 may generate a linear combination of features associated with the luma channel 802 and chroma channels 804. The linear combination operation operates as a per-value cross-channel mixing of the Y and UV components, predicting cross-components (e.g., cross-luminance and chrominance components), which improves coding performance. bring about Each 1x1 convolution filter of the 1x1 convolution layer 806 may include a respective scaling factor applied to the corresponding Nth channel of the luma channel 802 and the corresponding Nth channel of the chroma channels 804. there is.

FIG. 8B is a diagram illustrating example operation of a 1x1 convolution layer 838. As described above, N represents the number of output channels. As shown in Figure 8B, 2N channels, including an N-channel chroma (combined U and V) output 832 and an N-channel luma (Y) output 834, are used for the 1x1 convolution layer 838. Provided as input. In the example of Figure 8B, the value of N is equal to 2, indicating two channels of values for N-channel chroma output 832 and two channels of values for N-channel luma output 834. . Referring to FIG. 8A , N-channel chroma output 832 can be an output from non-linear layer 814 and N-channel luma output 834 can be an output from non-linear layer 812.

The 1x1 convolution layer 838 processes 2N channels and performs a characteristic linear combination of the 2N channels, then outputs an N-channel set of features or coefficients. The 1x1 convolution layer 838 includes two 1x1 convolution filters (based on N=2). The first 1x1 convolution filter is shown with s1 values, and the second 1x1 convolution filter is shown with s2 values. The s1 value represents the first scaling factor and the s2 value represents the second scaling factor. In one illustrative example, the s1 value is equal to 3 and the s2 value is equal to 4. Each of the 1x1 convolution filters in the 1x1 convolution layer 838 has a stride value of 1, indicating that scaling factors s1 and s2 will be applied to the respective values in the UV output 832 and Y output 834. .

For example, the scaling factor s1 of the first 1x1 convolution filter applies to each value in the first channel (C1) of the UV output (832) and to each value in the first channel (C1) of the Y output (834) do. If each value of the first channel C1 of the UV output 832 and each value of the first channel C1 of the Y output 834 are scaled by the scaling factor s1 of the first 1x1 convolution filter, scaling The combined values are combined into the first channel C1 of the output values 839. The scaling factor s2 of the second 1x1 convolution filter is applied to each value in the second channel (C2) of the UV output (832) and to each value in the second channel (C2) of the Y output (834). Each value of the second channel (C2) of the UV output 832 and each value of the second channel (C2) of the Y output 834 is scaled by the scaling factor s2 of the second 1x1 convolution filter, and then the scaling The combined values are coupled to the second channel (C2) of output values (839). As a result, the four Y and UV channels (two Y channels and two combined UV channels) are mixed or combined into two output channels (C1 and C2).

Returning to Figure 8A, the output of the 1x1 convolutional layer 806 is processed by additional nonlinear layers of the encoder sub-network and additional convolutional layers. Bottleneck 820 may include a quantization engine and an entropy encoding engine on the encoder sub-network (or forward pass) and an entropy decoding engine and inverse quantization engine on the decoder sub-network (or backward pass). The quantization engine may perform quantization on the features output by the final neural network layer 819 of the encoder sub-network to generate a quantized output. The entropy encoding engine may generate a bitstream by entropy encoding the quantized output from the quantization engine. In some cases, the entropy encoding engine may use the prior generated by the hyperprior network to perform entropy encoding. A neural network system may output a bitstream to a server device or system for storage, transmission to another device, and/or otherwise output the bitstream.

A decoder sub-network of a neural network system or a decoder sub-network of another neural network system (of another device) may decode the bitstream. The entropy decoding engine of the bottleneck 820 (of the decoder sub-network) may entropy decode the bitstream and output the entropy decoded data to the dequantization engine of the decoder sub-network. The entropy decoding engine may use the fryer generated by the hyperprior network to perform entropy decoding. The inverse quantization engine can inverse quantize data. The dequantized data may be processed by multiple convolutional layers and multiple inverse nonlinear layers of the decoder sub-network.

After being processed by several convolutional and non-linear layers, the 1x1 convolutional layer 813 can process the data output by the final inverse non-linear layer 809. The 1x1 convolution layer 813 may include 2N convolutional filters that may split the data into Y channel features and combined UV channel features. For example, each of the N channels output by the inverse nonlinear layer 809 can be processed (resulting in scaling) using 2N 1x1 convolutions of the 1x1 convolution layer 813. For each scaling factor ni corresponding to an output channel (from a total of 2N output channels) applied to the N input channels, the decoder sub-network performs a summation over the N input channels, resulting in 2N output channels. can result in outputs. In one illustrative example, for scaling factor n1, the decoder sub-network can apply scaling factor n1 to N input channels and sum the results, resulting in one output channel. The decoder sub-network may perform this operation for 2N different scaling factors (e.g., scaling factor n1, scaling factor n2 through scaling factor n2N).

Y channel features output by the 1x1 convolution layer 813 can be processed by the inverse nonlinearity 815. The combined UV channel features output by the 1x1 convolution layer 813 can be processed by the inverse nonlinearity 816. The convolutional layer 817 processes the Y channel features and outputs a reconstructed Y channel per sample or pixel (e.g., luminance samples or pixels) of the reconstructed frame, shown as reconstructed Y component 824. can do. Convolutional layer 818 may process the combined UV channel features and produce pixels or samples of the reconstructed frame, shown as reconstructed U and V components 825 (e.g., chrominance-blue samples or A reconstructed U channel per pixel) and a reconstructed V channel per pixel or sample (e.g., chrominance-red samples or pixels) of the reconstructed frame may be output.

In some examples, different variations of the architecture of FIG. 8A with different non-linearity operators may be used as intra prediction engine 602 and residual engine 620. For example, FIGS. 9 and 10 illustrate the front-end architecture of FIG. 8A configured to process data with YUV format (e.g., YUV 4:2:0 input data with Y, U, and V components). These are drawings for explanation. In the neural network system 900 of Figure 9, on the encoder side, the branched luma and chroma channels are combined using a 1x1 convolutional layer (similar to that in Figure 8A), and then the GDN non-linear operator is applied. In the neural network system 1000 of Figure 10, on the encoder side, the branched luma and chroma channels are combined using a 1x1 convolutional layer (similar to that in Figure 8A), and then the PReLU non-linear operator is applied. In one example, both VAE _res and VAE _intra may use the variant shown in FIG. 9. In another example, both VAE _res and VAE _intra may use a variation of Figure 10. In another example, VAE _res may use a variation of Figure 9 and VAE _intra may use a variation of Figure 10. In another example, VAE _intra may use a variation of Figure 9 and VAE _res may use a variation of Figure 10.

FIG. 11 is a flow diagram illustrating an example of a process 1100 for processing video data. At block 1102, process 1100 includes obtaining, by a machine learning system, input video data. The input video data includes at least one luminance component for the current frame (e.g., the luma component of the current frame (at time t) in Figure 7A ) includes. In some cases, the input video data contains at least one luminance component for a previously reconstructed frame (e.g., (at time t-1 in Figure 7A), which may be referred to as at least one reconstructed luminance component. ) Reconstructed luma component of the previous frame ) includes. In some aspects, the current frame includes a video frame. In some cases, the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component. In some aspects, the current frame has a luminance-chrominance (YUV) format. In some cases, the YUV format is YUV 4:2:0 format.

At block 1104, the process determines, by the machine learning system, motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using the at least one luminance component for the current frame. It includes steps to: In some aspects, process 1100 includes determining motion information for at least one luminance component of the current frame based on the at least one luminance component of the current frame and the at least one reconstructed luma component of the previous frame. You may. In some cases, process 1100 may further include determining motion information for one or more chrominance components of the current frame using motion information determined for at least one luminance component of the current frame. In some cases, motion information for one or more chrominance components of the current frame is determined using a convolutional layer of a machine learning system. For example, referring to Figure 7A as an illustrative example, flow engine 718 may and previously reconstructed frames. Current frame using both luma components Luma motion information (eg, SSF f ^L ) and chroma motion information (eg, SSF f ^C ) may be estimated for . As mentioned above, chroma motion information (e.g., f ^C ) 731 can be derived based on luma motion information (e.g., f ^L ) 729 using a convolutional layer 730. there is. In some cases, motion information for one or more chrominance components of the current frame is determined at least in part by sampling motion information determined for at least one luminance component of the current frame.

In some aspects, process 1100 includes, by a machine learning system, using motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame to and determining a warping parameter for one luminance component and one or more warping parameters for one or more chrominance components of the current frame. In some aspects, the warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame include spatial-scale flow (SSF) warping parameters. In some cases, space-scale flow (SSF) warping parameters include learned scale-flow vectors. Referring to Figure 6 as an illustrative example, warping parameters represent the horizontal component (in the x-direction) of a motion or displacement vector. , representing the vertical component (in the y-direction) of a motion or displacement vector. , s (referred to as the scale field) representing a progressively smoothed version of the reconstructed frames combined with spatial motion/displacement information (vx and vy).

Process 1100 may generate one or more inter-frame predictions (e.g., , the predictors in Fig. 6 , , and ) may further include determining. In some cases, one or more inter-frame predictions are made at least in part by applying an interpolation operation using a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame. It is decided. In one illustrative example, the interpolation operation includes a trilinear interpolation operation.

In some examples, the processes described herein may be performed by a computing device or apparatus, such as a computing device with computing device architecture 1200 shown in FIG. 11. In one example, the process or processes implements the neural network architecture shown in Figure 6 and/or any one or more of the neural network architectures shown in Figures 7A, 7B, 8A, 9, and/or 10. It may be performed by a computing device having computing device architecture 1200. In some examples, a computing device is a mobile device (e.g., a mobile phone, tablet computing device, etc.), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a combination. real-world (MR) devices), personal computers, laptop computers, video servers, televisions, vehicles (or computing devices in vehicles), robotic devices, and/or any other device having resource capabilities to perform the processes described herein. It may include or be part of a computing device.

In some cases, a computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers configured to perform steps of the processes described herein. Various components, such as one or more transmitters, receivers or combined transmitter-receivers (e.g., referred to as transceivers), one or more cameras, one or more sensors, and/or other component(s) may include. In some examples, a computing device may include a display, a network interface configured to communicate and/or receive data, any combination thereof, and/or other component(s). A network interface may be configured to communicate and/or receive Internet Protocol (IP) based data or other types of data.

Components of a computing device may be implemented in circuitry. For example, components may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), may include and/or be implemented using electronic circuits or other electronic hardware, which may include central processing units (CPUs), neural processing units (NPUs), and/or other suitable electronic circuits. may include and/or be implemented using computer software, firmware, or any combination thereof.

The processes described herein are illustrated as logic flow diagrams, the operations of which represent a sequence of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be interpreted as a limitation, and any number of the described operations may be combined in any order and/or in parallel to implement the processes.

Additionally, the processes described herein may be performed under the control of one or more computer systems comprised of executable instructions, collectively code executing on one or more processors (e.g., executable instructions, one or more Computer programs, or one or more applications), may be implemented by hardware, or a combination thereof. As previously mentioned, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. Computer-readable or machine-readable storage media may be non-transitory.

FIG. 12 illustrates an example computing device architecture 1200 of an example computing device that can implement various techniques described herein. In some examples, a computing device is a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, It may include a video server, a vehicle (or a computing device in the vehicle), or another device. For example, computing device architecture 1200 may implement the system of FIG. 6 . Components of the computing device architecture 1200 are shown to communicate electrically with each other using a bus-like connection 1205. The exemplary computing device architecture 1200 includes a processing unit (CPU or processor) 1210 and computing device memory 1215, such as read-only memory (ROM) 1220 and random access memory (RAM) 1225. and a computing device connection 1205 that couples various computing device components to the processor 1210.

Computing device architecture 1200 may include a cache of high-speed memory integrated directly with, proximate to, or as part of processor 1210. Computing device architecture 1200 may copy data from memory 1215 and/or storage device 1230 to cache 1212 for quick access by processor 1210. In this way, the cache can provide a performance boost that avoids processor 1210 delays while waiting for data. These and other modules may control or be configured to control processor 1210 to perform various actions. Other computing device memory 1215 may also be available. Memory 1215 may include multiple different types of memory with different performance characteristics. Processor 1210 may include any general-purpose processor and hardware, such as Service 1 1232, Service 2 1234, and Service 3 1236, stored on storage device 1230, configured to control processor 1210. Alternatively, it may include a special-purpose processor in which software instructions, as well as software services, are integrated into the processor design. Processor 1210 may be multiple cores or a standalone system that includes processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

To enable user interaction with computing device architecture 1200, input devices 1245 may be any number of devices, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, a keyboard, mouse, motion input, speech, etc. It can represent the input mechanism of . Output device 1235 may also be one or more of a number of output mechanisms known to those skilled in the art, such as a display, projector, television, speaker device, etc. In some cases, multimode computing devices may allow a user to provide multiple types of input to communicate with computing device architecture 1200. Communications interface 1240 may generally control and manage user input and computing device output. There are no restrictions on operation with any particular hardware arrangement, so the basic features herein may easily replace improved hardware or firmware arrangements as they are developed.

Storage device 1230 is non-volatile memory, such as a hard disk or magnetic cassette, flash memory card, solid state memory device, digital versatile disk, cartridge, random access memories (RAMs) 1225, read only memory (ROM) 1220 and hybrids thereof may be other types of computer-readable media capable of storing data accessible by a computer. Storage device 1230 may include services 1232, 1234, and 1236 for controlling processor 1210. Other hardware or software modules are considered. Storage device 1230 may be connected to computing device connection 1205. In one aspect, a hardware module performing a particular function is stored on a computer-readable medium associated with necessary hardware components, such as processor 1210, connection 1205, output device 1235, etc., to perform that function. May include software components.

Aspects of the present disclosure include any device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) that includes or is coupled to one or more active depth sensing systems. Applicable to suitable electronic devices. Although described below with respect to a device having or coupled to a single light projector, aspects of the present disclosure are applicable to devices having any number of light projectors and, therefore, are not limited to specific devices.

The term “device” is not limited to one or a specific number of physical objects (such as a smartphone, a controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more parts that may implement at least some portions of the present disclosure. Although the following description and examples use the term “device” to describe various aspects of the present disclosure, the term “device” is not limited to a particular configuration, type, or number of objects. Additionally, the term “system” is not limited to multiple components or specific embodiments. For example, a system may be implemented on one or more printed circuit boards or other substrates and may have movable or static components. Although the following description and examples use the term “system” to describe various aspects of the present disclosure, the term “system” is not limited to a particular configuration, type, or number of objects.

Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by one skilled in the art that the embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the present technology refers to individual functional blocks comprising devices, device components, method steps or routines embodied in software, or combinations of hardware and software. It may also be presented as containing blocks. Additional components other than those shown in the drawings and/or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

Individual embodiments may be described above as a process or method shown as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. Additionally, the order of operations may be rearranged. The process ends when its operations are complete, but may have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination may correspond to a return to that function's calling function or main function.

Processes and methods according to the above-described examples may be implemented using computer-executable instructions stored on or otherwise available from computer-readable media. These instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible via a network. Computer-executable instructions may be intermediate format instructions, such as, for example, binaries, assembly language, firmware, source code, etc.

The term “computer-readable media” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain, or carry instruction(s) and/or data. Computer-readable media may include non-transitory media on which data may be stored and that do not contain transient electronic signals propagated via carrier waves and/or wireless or wired connections. Examples of non-transitory media include, among others, magnetic disks or tapes, optical storage media such as flash memory, memory or memory devices, magnetic or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices. , compact disk (CD), or digital versatile disk (DVD), or any suitable combination thereof. A computer-readable medium may contain code and/or code that may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. Machine executable instructions may be stored. A code segment may be coupled to another code segment or hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be communicated, forwarded, or transmitted through any suitable means, including memory sharing, message passing, token passing, network transport, etc.

In some embodiments, computer-readable storage devices, media, and memories may include a wireless signal or cable containing a bit stream, etc. However, when mentioned, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, a hardware description language, or any combination thereof, and may take any of a variety of form factors. there is. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. Processor(s) may perform the necessary tasks. Common examples of form factors include laptops, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rackmount devices, standalone devices, etc. The functionality described herein can also be implemented in peripherals or add-in cards. Such functionality may also be implemented on a circuit board among different chips or different processes running in a single device, by way of further example.

Instructions, media for carrying these instructions, computing resources for implementing them, and other structures to support such computing resources are example means for providing the functionality described in this disclosure.

In the foregoing description, aspects of the application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the application is not limited thereto. Accordingly, although exemplary embodiments of the present application have been described in detail herein, the inventive concept may be embodied and adapted in various other ways, and the appended claims cover such modifications except as limited by prior art. It is intended to be interpreted as The various features and aspects of the above-described application may be used individually or jointly. Additionally, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. For purposes of illustration, methods have been described in a specific order. It should be appreciated that in alternative embodiments, methods may be performed in a different order than described.

Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein do not depart from the scope of the present disclosure, and the less than ("≤") and greater than ("≥") symbols, respectively. You will see that it can be replaced with .

When components are described as being "configured to perform" a particular operation, such configuration may mean, for example, designing an electronic circuit or other hardware to perform the operation, such as a programmable electronic circuit (e.g., a microprocessor or other suitable electronic device). circuit) to perform an operation, or any combination thereof.

The phrase “coupled to” refers to any component that is physically connected, directly or indirectly, to another component, and/or that communicates directly or indirectly with the other component (e.g., through a wired or wireless connection). , and/or connected to another component via another suitable communication interface.

Claim language or other language reciting “at least one of” a set or “one or more of” a set indicates that one member of that set or multiple members of that set (in any combination) satisfies the claim. For example, claim language reciting “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” or “at least one of A, B, or C” may be defined as A, B, C, or A and B, or A and C, or B and C, or A and B and C. A language set “at least one of” and/or “one or more of” a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” or “at least one of A or B” may mean A, B, or A and B, and not listed in the set of A and B. Additional items may be included.

The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been described above generally in terms of their functionality. Whether this functionality is implemented as hardware or software will depend on the specific application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be construed as causing a departure from the scope of the present application.

The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general purpose computers, wireless communication device handsets, or integrated circuit devices with multiple uses, including applications in wireless communication device handsets and other devices. . Any features described as modules or components may be implemented together in an integrated logic device or separately as separate but interoperable logic devices. If implemented in software, the techniques may be realized, at least in part, by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. Computer-readable media includes memory or data storage media, such as random access memory (RAM), synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and electrically generated EEPROM (EEPROM). It may include erasable programmable read-only memory, FLASH memory, magnetic or optical data storage media, etc. The techniques may, additionally or alternatively, be computer readable, carrying or communicating program code in the form of instructions or data structures, such as propagated signals or waves, and capable of being accessed, read, and/or executed by a computer. It may also be realized at least in part by an enabling communication medium.

Program code may be implemented in one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic. It may be executed by a processor, which may include one or more processors such as circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein, may refer to any of the foregoing structure, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. there is.

Illustrative examples of the present disclosure include:

Aspect 1:

A method of processing video data, comprising: obtaining, by a machine learning system, input video data, wherein the input video data includes at least one luminance component for a current frame; and determining, by the machine learning system, using the at least one luminance component for the current frame, motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame. How to.

Aspect 2:

The method of aspect 1, wherein the machine learning system uses motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame to determine determining a warping parameter and one or more warping parameters for one or more chrominance components of the current frame; and determining one or more inter-frame predictions for the current frame using the warping parameters for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame. method.

Aspect 3:

The method of aspect 2, wherein the one or more inter-frame predictions are interpolated using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. A method determined at least in part by applying an interpolation operation.

Aspect 4:

The method of aspect 3, wherein the interpolation operation comprises a trilinear interpolation operation.

Aspect 5:

The method of any of aspects 2 to 4, wherein the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame are space-scale flow (SSF). Method, including warping parameters.

Aspect 6:

The method of aspect 5, wherein the SSF warping parameters comprise learned scale-flow vectors.

Aspect 7:

The method of any of aspects 1 to 6, further comprising using the at least one luminance component for the current frame to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame. Determining motion information for at least one luminance component of the current frame based on the at least one luminance component of the current frame and the at least one reconstructed luma component of the previous frame; and determining motion information for one or more chrominance components of the current frame using motion information determined for at least one luminance component of the current frame.

Aspect 8:

The method of aspect 7, wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system.

Aspect 9:

The method of aspect 7, wherein the motion information for the one or more chrominance components of the current frame is determined at least in part by sampling the motion information determined for the at least one luminance component of the current frame.

Aspect 10:

The method of any of aspects 1-9, wherein the current frame comprises a video frame.

Aspect 11:

The method of any of Aspects 1 to 10, wherein the one or more color difference components comprise at least one color difference-blue component and at least one color difference-red component.

Aspect 12:

The method of any of aspects 1 to 11, wherein the current frame has a luminance-chrominance (YUV) format.

Aspect 13:

The method of aspect 12, wherein the YUV format is YUV 4:2:0 format.

Aspect 14:

1. An apparatus for processing video data, comprising: at least one memory; and one or more processors coupled to the at least one memory, wherein the one or more processors acquire input video data using a machine learning system, wherein the input video data is configured to obtain at least one memory for the current frame. perform obtaining the input video data, including a luminance component; and using the machine learning system to generate motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using the at least one luminance component for the current frame. A device configured to make a decision.

Aspect 15:

The method of aspect 14, wherein the one or more processors, using a machine learning system, use motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame to: determine a warping parameter for at least one luminance component and one or more warping parameters for one or more chrominance components of the current frame; and determine one or more inter-frame predictions for the current frame using the warping parameters for at least one luminance component of the current frame and the one or more warping parameters for one or more chrominance components of the current frame.

Aspect 16:

The method of aspect 15, wherein the one or more inter-frame predictions are interpolated using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. A device determined at least in part by applying an operation.

Aspect 17:

The apparatus of aspect 16, wherein the interpolation operation comprises a trilinear interpolation operation.

Aspect 18:

The method of any of aspects 15 to 17, wherein the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame are space-scale flow (SSF). A device containing warping parameters.

Aspect 19:

The apparatus of aspect 18, wherein the SSF warping parameters include learned scale-flow vectors.

Aspect 20:

The method of any of aspects 14 to 19, wherein using the at least one luminance component for the current frame to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame. , the one or more processors determine motion information for at least one luminance component of the current frame based on the at least one luminance component of the current frame and the at least one reconstructed luma component of the previous frame; and determine motion information for one or more chrominance components of the current frame using motion information determined for at least one luminance component of the current frame.

Aspect 21:

The apparatus of aspect 20, wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system.

Aspect 22:

The method of aspect 20, wherein to determine the motion information for the one or more chrominance components of the current frame, the one or more processors are configured to sample the motion information determined for the at least one luminance component of the current frame. used device.

Aspect 23:

The apparatus of any of aspects 14-22, wherein the current frame comprises a video frame.

Aspect 24:

The device of any of Aspects 14-23, wherein the one or more chrominance components comprise at least one chrominance-blue component and at least one chrominance-red component.

Aspect 25:

The apparatus of aspects 14-24, wherein the current frame has a luminance-chrominance (YUV) format.

Aspect 26:

The apparatus of aspect 25, wherein the YUV format is a YUV 4:2:0 format.

Aspect 27:

The apparatus of any of aspects 14-26, further comprising at least one camera configured to capture one or more frames.

Aspect 28:

The apparatus of any of aspects 14-27, further comprising at least one display configured to display one or more frames.

Aspect 29:

The apparatus of any of aspects 14-28, wherein the apparatus comprises a mobile device.

Aspect 30: A computer-readable storage medium storing instructions that, when executed, cause one or more processors to perform any of the operations of aspects 1-29.

Aspect 31: An apparatus comprising means for performing any of the operations of aspects 1-29.

Claims (30)

A method of processing video data, comprising:
Obtaining, by a machine learning system, input video data including at least one luminance component for a current frame; and
By the machine learning system, using the at least one luminance component for the current frame, motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame A method comprising determining. The method of claim 1, further
The at least one luminance component of the current frame, by the machine learning system, using motion information for the at least one luminance component of the current frame and motion information for the one or more chrominance components of the current frame. determining a warping parameter for and one or more warping parameters for the one or more chrominance components of the current frame; and
Using the warping parameters for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame, determine one or more inter-frame predictions for the current frame. A method comprising the steps of: 3. The method of claim 2, wherein the one or more inter-frame predictions are performed using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. A method determined at least in part by applying an interpolation operation. 4. The method of claim 3, wherein the interpolation operation comprises a trilinear interpolation operation. 3. The method of claim 2, wherein the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame are spatial-scale flow (SSF) warping parameters. Including, method. 6. The method of claim 5, wherein the SSF warping parameters include learned scale-flow vectors. 2. The method of claim 1, wherein the motion information for the at least one luminance component of the current frame using the at least one luminance component for the current frame and the motion for the one or more chrominance components of the current frame The steps to determine information are:
determining the motion information for the at least one luminance component of the current frame based on the at least one luminance component of the current frame and the at least one reconstructed luma component of a previous frame; and
Determining the motion information for the one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame. The method of claim 7, wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system. 8. The method of claim 7, wherein the motion information for the one or more chrominance components of the current frame is determined at least in part by sampling the motion information determined for the at least one luminance component of the current frame. The method of claim 1, wherein the current frame comprises a video frame. The method of claim 1, wherein the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component. The method of claim 1, wherein the current frame has a luminance-chrominance (YUV) format. 13. The method of claim 12, wherein the YUV format is YUV 4:2:0 format. A device for processing video data, comprising:
at least one memory; and
Comprising one or more processors coupled to the at least one memory, the one or more processors
Using a machine learning system, obtain input video data containing at least one luminance component for the current frame, and
Using the machine learning system, the at least one luminance component for the current frame is used to generate motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame. A device configured to determine information. 15. The method of claim 14, wherein the one or more processors:
Using the machine learning system, the at least one luminance of the current frame is determined based on motion information for the at least one luminance component of the current frame and motion information for the one or more chrominance components of the current frame. Determine a warping parameter for a component and one or more warping parameters for the one or more chrominance components of the current frame, and
Using the warping parameters for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame, determine one or more inter-frame predictions for the current frame. A device configured to: 16. The method of claim 15, wherein the one or more inter-frame predictions are performed using the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame. A device determined at least in part by applying an interpolation operation. 17. The apparatus of claim 16, wherein the interpolation operation comprises a trilinear interpolation operation. 16. The method of claim 15, wherein the warping parameter for the at least one luminance component of the current frame and the one or more warping parameters for the one or more chrominance components of the current frame are spatial-scale flow (SSF) warping parameters. Including device. 19. The apparatus of claim 18, wherein the SSF warping parameters include learned scale-flow vectors. 15. The method of claim 14, wherein the motion information for the at least one luminance component of the current frame using the at least one luminance component for the current frame and the motion for the one or more chrominance components of the current frame To determine information, the one or more processors:
determine the motion information for the at least one luminance component of the current frame based on the at least one luminance component of the current frame and the at least one reconstructed luma component of a previous frame, and
and determine the motion information for the one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame. 21. The apparatus of claim 20, wherein the motion information for the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system. 21. The method of claim 20, wherein to determine the motion information for the one or more chrominance components of the current frame, the one or more processors are configured to sample the motion information determined for the at least one luminance component of the current frame. configured device. 15. The apparatus of claim 14, wherein the current frame comprises a video frame. 15. The device of claim 14, wherein the one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component. 15. The apparatus of claim 14, wherein the current frame has a luminance-chrominance (YUV) format. 26. The device of claim 25, wherein the YUV format is YUV 4:2:0 format. 15. The apparatus of claim 14, further comprising at least one camera configured to capture one or more frames. 15. The apparatus of claim 14, further comprising at least one display configured to display one or more frames. 15. The device of claim 14, wherein the device comprises a mobile device. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to:
Using the machine learning system, obtain input video data containing at least one luminance component for the current frame, and
Using the machine learning system, the at least one luminance component for the current frame is used to generate motion information for the at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame. A non-transitory computer-readable storage medium that stores information.

Images