Masked Modeling for Self-supervised Representation Learning on Vision and Beyond

The notations used in this survey are illustrated in Table I, and we will present a detailed demonstration of the changes and corresponding relationships between the symbols and variables in the table.

In this paper, $x$ denotes a data sequence which can be a sentence in NLP, a patch sequence in CV, and a data sequence in another modality. In CV, $\boldsymbol{x}$ denotes a patch sequence, that $\boldsymbol{x}\in\mathbb{R}^{N\times(P^{2}C)}$ and $N$ denotes the number of the patch, $P^{2}\times C$ denotes the dimension of a patch vector. That means $\boldsymbol{x}=[\boldsymbol{x}_{1},\boldsymbol{x}_{2},\cdots,\boldsymbol{x}_{n}]$, and $\boldsymbol{x}_{i}$ denote a patch, and $\boldsymbol{x}_{i}=[x_{i}^{1},x_{i}^{2},\cdots,x_{i}^{P^{2}\times C}]$. In this paper, we use $\boldsymbol{x}^{k},\boldsymbol{x}_{i}^{k}$ to denote the different sequences and patches, and we use $\boldsymbol{x}^{v_{i}}$ to denote the different views of the patch. In NLP, $\boldsymbol{x}=[\boldsymbol{x_{1}},\boldsymbol{x_{2}},\cdots,\boldsymbol{x_{L}}]$ denotes the original sentence. we use $\boldsymbol{e}=[\boldsymbol{e_{1}},\boldsymbol{e_{2}},\cdots,\boldsymbol{e_{L}}]$ to denote the embedded sequence. Encoder and decoder are denoted by $f_{\theta}(\cdot)$ and $g_{\phi}(\cdot)$, where $\theta$ and $\phi$ are learnable parameters. In masked modeling, some tokens or patches of $\boldsymbol{x}$ are selected to mask, and we use $\mathcal{M}=\{0,1\}^{N}$ to denote the mask set, which means a masked sequence can be denoted as $\boldsymbol{x}\odot\mathcal{M}=[\boldsymbol{x_{1}},\cdots,\boldsymbol{x_{i-1}},0,\boldsymbol{x_{i+1}},\cdots,\boldsymbol{x_{n}}]$. The left visible patch(token) can be denoted as $\tilde{x}=x_{i=1,\mathbb{I}_{\mathcal{M}=1}}^{N}$ or $\tilde{e}=e_{i=1,\mathbb{I}_{\mathcal{M}=1}}^{N}$.

In this subsection, we will give a brief introduction to the methods of SSL. Typically, SSL methods are universally divided into two categories [142], i.e., Generative and Discriminative, as shown in Table 3.

Generative model usually encodes the input $x$ into a latent variable $z$ and decodes the latent variable $z$ to reconstruct the input $x$ with an encoder-decoder architecture. Autoregressive models typically model a series of regressions one by one for one input.

Auto-Regressive models typically model a series of regressions one by one for one input, where the current output depends on the previous inputs or outputs in the sequence. GPT [91] and Transformer [209] are both AR models. The learning object of the AR model can be formulated as:

where each variable is dependent on previous variables [142].

Auto-Encoder typically reconstructs the input from the corrupted input. The learning object of the AE model can be formulated as:

We further divide the AE model into denoising AE and masked AE. The denoising AE model is trained to reconstruct clean data from noisy or corrupted input. By removing noise or corruption, the model learns robust representations. And a masked auto-encoder is trained to predict missing or masked portions of the input data. By reconstructing the missing parts, the model learns contextual representations.

Flow Based model aims to learn densities $p(x)$ from data. Suppose a latent variable $z$ follows a known distribution $p_{Z}(x)$ and define $z=f_{\theta}(x)$. The learning objective is to maximize the likelihood [142]:

GAN-Based model so-called adversarial learning involves training two models in competition with each other, typically a generator $\mathcal{G}$ and discriminator $\mathcal{D}$, which can be formulated as:

Diffusion-based model initially processes images through a series of Gaussian noise treatments, followed by restoration of the image through the model. The diffusion-based model process is divided into forward and reverse processes. The forward process treats the image with cumulative Gaussian noise, which can be modeled as follows:

in which $\beta_{t}$ is mean coefficient. The reverse process of the diffusion-based model, which involves denoising and inference, has a learning objective that can be modeled as follows:

Discriminative model are typically formulated using contrastive learning objectives. The core idea in contrastive learning is to train encoders to produce similar representations for semantically related instances while distinguishing unrelated samples [142]. Contrasting at the context-instance level involves comparing the local feature, which is encoded, with the global representation from the identical sample. In contrast, the instance-instance contrast method is more focused on the representation at the instance level, examining the commonalities across multiple samples [142]. InfoNCE [167] is one of the basic loss functions in contrastive learning. It can be formulated as:

Masked Language Modeling. Masked Language Modeling was first introduced in BERT. The central idea of MLM is to randomly mask tokens within a sentence and replace them with a Mask vector. The encoder then predicts the masked vector. We formally define the problem of MLM as follows:

A sentence $\boldsymbol{x}=[\boldsymbol{x_{1}},\boldsymbol{x_{2}},\cdots,\boldsymbol{x_{L}}]$ is first tokenized as $\boldsymbol{e}=[\boldsymbol{e_{1}},\boldsymbol{e_{2}},\cdots\boldsymbol{e_{L}}]$ through a tokenizer $\boldsymbol{q}_{\phi}(\cdot|\cdot)$, in which $L$ denotes the number of the tokens in this sentence. The masked sequence of the embedded sentence $\boldsymbol{e}\odot\mathcal{M}$ is fed into a Transformers encoder $f_{\theta}(\cdot)$. $m_{i}=f_{\theta}(\tilde{e})$ is the hidden state of the last layer at the masked position and can be regarded as a fusion of contextualized representations of surrounding tokens. And the MLM task [115] can be formulated mathematically as:

Masked Image Modeling. The core concept of Masked Image Modeling (MIM) aligns with that of MLM. It involves masking certain pixel regions of the input image and reconstructing the original image based on the unmasked portions. Given that images lack the tokenizer structure inherent in natural language, the intuitive approach is to reconstruct pixel values directly. However, due to the high redundancy and dimensionality of image pixel information, pixel-level reconstruction is often challenging. This has historically hindered the progress of MIM. It wasn’t until the introduction of the ViT, which segments images into patches that MIM began to emerge as a feasible approach. We formally define the problem of MIM as follows: A image $\mathbf{X}\in\mathbb{R}^{H\times W\times C}$ is partitioned into multiple patches $\boldsymbol{x}\in\mathbb{R}^{N\times(P^{2}C)}$, $\boldsymbol{x}=[\boldsymbol{x_{1}},\boldsymbol{x_{2}},\cdots,\boldsymbol{x_{N}}]$ where $N$ denotes the number of patch. Masked sequence can be denoted as $\boldsymbol{x}\odot\mathcal{M}$. The remaining unmasked patches $\tilde{\boldsymbol{x}}$ is used to reconstruct the original pixel through an encoder$f_{\theta}(\cdot)$ and a decoder $g_{\theta}(\cdot)$. We use $m_{i}$ to denote the hidden layer at the masked portion as NLP and $m_{i}=f_{\theta}(\tilde{x})$, The learning object can be formulated as:

Beyond. Beyond computer vision and natural language processing, masked modeling can also be applied to various data structures and multimodal domains. The core idea is to mask parts of the input vector with mask tokens and then reconstruct the data through an encoder-decoder framework. Masked data modeling can be formally described as: given an input sequence $x$ of any modality, we generate the corrupted sample $x\odot\mathcal{M}$ by replacing elements in $x_{m}$ with mask tokens [MASK]. We use $\mathcal{S}(\cdot,\cdot)$ to denote the similarity between the predicted mask tokens and the original data. The learning object can be formulated as:

in which $\mathcal{S}(\cdot,\cdot)$ can be MSE and other functions which measure the similarities.

Based on the current research on MIM for self-supervised pre-training, this paper conducts an in-depth investigation and proposes a unified research framework and paradigm for MIM, providing a detailed classification of existing studies. The framework mainly consists of four modules, namely: Mask, Target, Encoder, and Head. An overview of our framework is visually presented in Figure 4. In the following subsections, we will elaborate on the specific contents of these four modules.

Mask: Mask module is to generate a mask set $\mathcal{M}$. The masked image can be denoted as $\boldsymbol{x}\odot\mathcal{M}$. Some typical mask strategy consists of Random Mask, Attention Mask, Contextual Mask, and so on.

Target: The Target module’s role is to generate supervisory signals. The target module can be formulated as: $\mathcal{T}(f_{\omega}(\boldsymbol{x}))$, $f_{\omega}(\cdot)$ is a model with parameter $\omega$. Within this module, models like VQ-GAN [50] and dVAE can be utilized as tools to extract these signals, and different supervisory signals can lead to different model preferences.

Encoder: The Encoder $f_{\theta}(\cdot)$ is the target for MIM pre-training and can adopt various network architectures, such as Transformer, CNN, or a hybrid of both. The input for the Encoder can be a visible patch and a combination of visible and masked patches.

Head: The Head module’s purpose is to compare the supervisory signals with the encoded features. The primary task of MIM is to reconstruct the original image, so the most common head is the MIM head, which reconstructs the original image or its features. Additionally, there’s the Contrastive head, which employs contrastive learning to enhance the model’s performance.

Based on the unified perspective we proposed, the MIM problem can be mathematically represented as:

Permuting and combining these four modules, we have meticulously categorized the research on MIM. The detailed classification is elaborated in Figure III.

iGPT [30]: The input image $\mathbf{X}$, when arranged according to pixel values and subsequently downsampled, forms a pixel sequence $\boldsymbol{x}$ that is fed into a Transformer structure identical to GPT-2 [91]. This model predicts the value of the next pixel $\boldsymbol{x}_{t}$ based on the current pixel value $\boldsymbol{x}_{1:t}$. Given that iGPT predicts pixel values in sequence, its masking approach can be considered as “Basic Masking“, with the target being the Token. Based on GPT, the encoder of the iGPT is Transformer and the decoder is a Linear MIM Head. The loss of iGPT can be formulated as Eq. 1.

MAE [78]: The overview of MAE can be seen in Figure 5. The input image $\mathbf{X}\in\mathbb{R}^{H\times W\times C}$ is partitioned into multiple patches $\boldsymbol{x}\in\mathbb{R}^{N\times(P^{2}C)}$, where approximately 75% of the patches are Randomly Masked. The remaining unmasked patches $\tilde{x}$ are then fed into the Transformer Encoder $f_{\theta}(\cdot)$ which generates the features. These features, in conjunction with the masked patches, are input into the Transformer Decoder $g_{\omega}(\cdot)$ for the purpose of reconstructing the Pixels of the original image. The quality of the reconstruction is measured using the MSE loss function. MAE [78] is formulated as:

iGPT [30] and MAE [78] represent two distinct basic frameworks within MIM research: iGPT [30] is based on the Auto-Regressive MIM research paradigm, while MAE is grounded in the Auto-Encoder paradigm. Both are categorized within the classification we proposed. The four modules of iGPT [30] can be classified as: Basic Masking(Auto-Regressive Masking) + Transformer + Tokenizer + MIM Head, whereas MAE can be categorized as Basic Masking (Random) + Transformer + Pixel + MIM Head. Table II summarizes the difference between iGPT and MAE.

This subsection will also spotlight typical masking strategies employed in MIM. For classification purposes, we bifurcate masking strategies into basic and advanced masking. Basic masking, which encompasses pixel-wise predictions based on AR models and the Random Mask introduced by MAE, has been elaborated upon in Sec. 3. Consequently, our ensuing discussion will primarily focus on Advanced Masking techniques. As illustrated in the accompanying figure, Advanced Masking can be further subdivided into four types: Hard Sampling, Mixture, Adversarial Mask, and Contextual Mask.

Remark: Despite the excellent performance, Mixture mask and Adversarial Mask have a more expensive computational cost. It can be concluded that an attention-based mask strategy usually performs better in mining hard samples and costs less.

In this subsection, we will delve into the targets used during MIM training. For classification purposes, we categorize these targets into three main types: tokenizer, pixel, and features. Delving deeper, these categories can be further detailed, with comprehensive explanations provided in the accompanying Figure 6.

Compared to the traditional Transformer, Hierarchical Transformers often have lower computational overhead, faster training speeds, and better generalization capabilities on downstream tasks. As a result, some research focuses on enhancing the encoder structure to improve the operational efficiency of MIM or devising novel masking techniques to make MIM adaptable to various network architectures.

Transfer encoder to Hierarchical vision transformer: GreenMIM [94] (BTPM) inputs the masked image $\mathbf{X}\odot\mathcal{M}$ into a Hierarchical Transformer encoder. At each stage, Group Window Attention is used to perform self-attention computation on the masked image within groups. To reduce unnecessary computations in areas that are masked or do not contain useful information, the sparse convolution is introduced to discard invisible patches and only processes on the visible patches, achieving patch merging, similar to Figure 7. The processed data is then input into the Transformer decoder to reconstruct the original image. In this way, GreenMIM enhances computational efficiency as MAE [78] while maintaining the performance as SimMIM [241]. HiViT [271] (BTPM) removes local inter-unit operations, resulting in structurally simple hierarchical vision transformers. Hiera [190] (BTPM) eliminates the need for many of the complex components found in other hierarchical vision transformers and achieves superior accuracy. ConvMAE [63] (Basic Mask + CNN + Pixel + MIM Head, BCPM) proposes a multi-scale hybrid convolution-transformer, employs a masked convolution to prevent information leakage in the convolution blocks and a block-wise mask to reduce the computational cost. SparseMAE [276] (BCPM) introduces sparse MHSA and FFN blocks for sparse pre-training.

Make MIM Compatible with Convolutional Neural Networks: CIM [53] (Basic Mask + CNN + Tokenizer + MIM Head, BCTM) employs an auxiliary generator equipped with a compact trainable BEiT to corrupt the input images, thereby enhancing the network’s capability to either restore the original image pixels or predict whether each visual token has been replaced by a sample from the generator. Due to CIM’s approach of using an auxiliary generator to corrupt the input, there’s no need for specific input formats or preprocessing, which is compatible with CNNs. Since the objective of the enhancing network is to either restore the original image or predict if each visual token has been replaced by a generator sample, only forward propagation is required, ensuring compatibility with CNN architectures. A2MIM [124] (BCFM) introduces a unified architecture compatible with both Transformers and CNNs. Specifically, A2MIM posits that masking at the block embedding layer aligns well with the attention mechanism of Transformers, offering robustness against occlusion. For CNNs, masking at the network’s input stages leads to low-order interactions, undermining CNN’s context extraction capability. Therefore, the authors suggest masking intermediate features encompassing semantic and spatial information, allowing the mask token to encode interactions with a moderate number of tokens.

Specially designed for Convolutional Neural Network: Spark [203] (BCPM) identified that the inability of convolutional operations to recognize irregularly randomly masked input images and the single-scale nature of BERT pre-training is fundamentally inconsistent with the hierarchical structure of convolutional networks, which is the primary reason MIM cannot be implemented on CNNs. To address this, Spark proposed treating unmasked pixels as 3D point clouds and using sparse convolution for encoding, allowing the model to operate on irregularly masked images, as in Figure 7. To integrate the hierarchical structure of convolutional networks, they introduced a hierarchical decoder to reconstruct images from multi-scale features. The authors validated this approach on traditional convolutional neural network models such as ResNet and ConvNeXt, and its performance showed significant improvements compared to contrastive learning and Transformer-based MIM. ConvNext.v2 [228] (BCPM) introduces a fully convolutional masked auto-encoder. Its core structure is based on ConvNext, where the convolution operation is transformed into sparse convolution. The decoder employs a lightweight ConvNext block, which simultaneously processes encoded pixels and masked tokens for image reconstruction, effectively migrating MIM to the CNN structure. Additionally, ConvNext.v2 proposes a Global Response Normalization layer that normalizes the feature map on each channel, capable of handling batches of any size. The framework of ConvNext.v2 is shown in Figure 7.

The choice of the head in MIM exhibits significant variations. In our classification, we distinguish the heads into three categories: Contrastive Head, MIM Head, and a combination of Both Contrastive Head and MIM Head. It’s essential to highlight that both the MIM Head and Contrastive Head can have diverse internal architectures. The specifics of these structures are visually represented in the provided figure. In the following sections, we will bifurcate our discussion into two primary segments, focusing separately on the MIM Head and the Contrastive Head.

Supervised learning, often referred to as statistical learning methods, typically possesses profound mathematical theoretical guarantees, providing precise mathematical conditions under which learning is assuredly successful. Training and test datasets usually stem from the assumption of independent and identically distributed statistics. As the number of training iterations increases, one can often achieve lower training and test losses. This is because supervised learning is relatively straightforward. In contrast, unsupervised learning lacks the simple and intuitive theoretical guarantees present in supervised learning. Intuitively, we believe that the essence of unsupervised learning is a form of information compression. The compression algorithms learned from the training set represent the universal knowledge and structure inherent within the data. The way to evaluate these compression algorithms is to determine whether they extract all the knowledge from unlabeled data, i.e., whether they provide as much assistance as possible and yield the maximum benefit. We will elucidate and summarize the theoretical foundations of Masked modeling from three perspectives.

From Contrastive Learning: Layer Grafted [100](Random + Transformer + Pixel + Contrastive Head, RTPC) finds that MIM and CL are suitable for lower and higher layers, respectively. The authors design a gradient surgery experiment by computing the cosine similarity between gradients of two tasks following [257] and verify the MIM loss and CL loss have different targets to optimize. The cosine similarity can be defined as:

They propose a ”sequential cascade” approach where early layers are first trained under one MIM loss, and then later layers continue to be trained under another CL loss. That is:

[264] demonstrates that the mask loss exhibits a lower bound compared to the align loss in contrastive learning, making it more effective than aligning within contrastive learning.

Subsequently, a uniform loss, akin to that in contrastive learning, is incorporated into the mask loss.

From Masking: [107] models MIM as a hierarchical latent variable model. The objective of MIM is to recover the latent variable $z$ shared between visible patches and invisible patches based on the lower-level visible patches. This latent variable encapsulates the information shared between the visible patch and the invisible portions. Both a very low mask ratio and an extremely high mask ratio tend to make the model focus on recovering low-level latent variable information, making it challenging to learn higher-level semantic features. Therefore, the mask ratio in MAE, which lies between the two, can assist the model in capturing higher-level latent variable information, enhancing its representation capability.

From Empirical Study: Many studies have extensively explored certain characteristics of masked language modeling through numerous experiments and obtained some valuable conclusions. [240] and [111] verified through extensive experiments that, compared to other self-supervised methods like jigsaw puzzles and image inpainting, masked language models demonstrate better transferability and superior performance on tasks like pose estimation, depth prediction, video object tracking, and object detection. [242] showed that masked models tend to underperform and are prone to overfitting on small datasets. As the dataset grows larger, the performance improvement of masked language models accelerates. [108] suggested that the efficacy of masked language modeling stems largely from the masking operation itself as the key to good performance, while different masking strategies contribute limited improvements.

We summarize some conclusions:

1. 1.

   From Contrastive Learning: Unlike contrastive learning, MIM tends to capture low-level features and exhibits a strong local bias, while contrastive learning leans towards high-level features. This naturally explains why the development of contrastive learning preceded MIM. Before the advent of ViT, the dominant architecture in the visual domain was CNN, which inherently has a strong local bias. This made it complementary to contrastive learning, enhancing each other’s strengths. However, both MIM and CNN share this pronounced local bias, leading to suboptimal performance of MIM on CNN architectures. With the rise of ViT, which emphasizes capturing global information, it pairs better with MIM, propelling MIM to the forefront of SSL algorithms.

2. 2.

   From Masking: Masking is the most fundamental and crucial technique in MIM. Compared to NLP, the visual domain typically employs a higher mask ratio. This is because, in contrast to language, image information is more redundant. A small masking ratio doesn’t significantly impact the overall semantic understanding of an image. Therefore, a larger mask ratio is used to obscure some of the image’s key information, increasing the difficulty of the reconstruction task and enabling the model to learn more robust representations.

3. 3.

   From Empirical Study: Models based on MIM exhibit certain characteristics and preferences. For instance, they rely more on large-scale data for training and tend to learn better representations with larger datasets. Masked modeling performs better on tasks that require more detailed visual information, such as video object tracking and pose estimation. These tasks demand the model’s ability to capture low-level information.

The majority of MIM research is based on Auto-regressive (AR) for generative SSL; however, autoregressive modeling has always been one of the important methods in generative self-supervision. Consequently, a significant body of research combines autoregressive generation with MIM, achieving both representation learning and generative tasks. In this section, we will introduce classic autoregressive generative model architectures and then discuss research paradigms that integrate representation learning with autoregression. In Figure 11, we provide a detailed comparison of the differences between these two research paradigms.

Current research in artificial intelligence and deep learning is increasingly oriented towards the integration of data from multiple modalities. Consequently, multimodal research has emerged as one of the most significant directions in the field of artificial intelligence. We categorize multimodal studies into three broad types. The first category involves the use of multimodal data for pre-training, where the focus is on extending visual network architectures to multimodal contexts and exploring the upper limits of model capabilities through scaling up, as summarized in Table VIII. The second category primarily concentrates on the generation of multimodal data, encompassing tasks such as text-to-image conversion, as summarized in Table IX. The third category represents a vision generalist model, aiming to unify various visual tasks under a single network architecture.

Research applying Masked Modeling to the video domain primarily focuses on adapting models for high-dimensional video data. This category of research can be divided into two parts: one part is based on the AE architecture, adapting video data to the MAE framework, and the other is based on the AR architecture, using autoregressive methods (such as VQ-VAE, VQ-GAN) to predict video data.