👁️ Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment

Xin Xiao^1,2*, Bohong Wu^2*, Jiacong Wang^2,3, Chunyuan Li², Xun Zhou², Haoyuan Guo ²

¹School of Computer Science, Wuhan University, ²ByteDance Inc

³School of Artificial Intelligence, University of Chinese Academy of Sciences

abstract: Existing image-text modality alignment in Vision Language Models (VLMs) treats each text token equally in an autoregressive manner. Despite being simple and effective, this method results in sub-optimal cross-modal alignment by over-emphasizing the text tokens that are less correlated with or even contradictory with the input images. In this paper, we advocate for assigning distinct contributions for each text token based on its visual correlation. Specifically, we present by contrasting image inputs, the difference in prediction logits on each text token provides strong guidance of visual correlation. We therefore introduce Contrastive ALignment (CAL), a simple yet effective re-weighting strategy that prioritizes training visually correlated tokens. Our experimental results demonstrate that CAL consistently improves different types of VLMs across different resolutions and model sizes on various benchmark datasets. Importantly, our method incurs minimal additional computational overhead, rendering it highly efficient compared to alternative data scaling strategies.

• 2024.06 🔥🔥🔥 The code is released.

We provide results comparision for LLaVA-NEXT here.

conda create -n CAL python=3.10 -y
conda activate CAL
bash install.sh

Please follow the instruction in LLaVA to prepare data.

For customized data preparation, please refer to this guide.

You can execute demo bash scripts in this directory to train LLaVA models.

Before training, you need to customize some settings in the following table. Otherwise, the code will use the default paths specified in run.sh. When using multiple data sources, simply concatenate their paths with a space.

For developers who cannot access Hugging Face directly, you can download the checkpoint files (modelscope) and modify the LLM path to the absolute path.

For example, to train LLaVA-NEXT with the Vicuna-13B LLM, run:

bash run_scripts/llava16_7b.sh

Note: The code will dynamically calculate the batch size for each GPU according to the total_batchsize, grad_acumsteps and the number of GPUs. When your resourses are limited, you can reduce total_batchsize or set a larger grad_acumsteps in the settings.

$batchsize_{singleGPU} = batchsize_{total}/(grad_{acumsteps}*GPU_{num})$

For multinode training, you need to prepare a hostfile. We provide an example here. Customize it based on your environment.

We evaluate our model using lmms-eval. This tool is quick and easy to use, requiring no dataset preparation. For more details, please refer to the lmms-eval repository.

The core modifications primarily involve three files:

1. llava/constants.py
2. llava/model/llava_arch.py
3. llava/model/language_model/llava_llama.py

You can run CAL by modifying the corresponding files in LLaVA-style codebase, like MGM.

• LLaVA: the codebase we built upon.
• lmms-eval: the codebase we evaluate our model.

Thanks a lot for their great works.

@misc{xiao2024seeing,
      title={Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment}, 
      author={Xin Xiao and Bohong Wu and Jiacong Wang and Chunyuan Li and Xun Zhou and Haoyuan Guo},
      year={2024},
      eprint={2405.17871},
}