A very small man can cast a very large shadow.

          ——George R.R. Martin, A Clash of Kings

[Technical report (coming soon)]  [Demo]  [Huggingface]

This repository contains the official training/evaluation code of the Imp project, which aims to provide a family of a strong multimodal small language models (MSLMs). Our imp-v1-3b is a strong MSLM with only 3B parameters, which is build upon a small yet powerful SLM Phi-2 (2.7B) and a powerful visual encoder SigLIP (0.4B), and trained on the LLaVA-v1.5 training set.

As shown in the Evaluation, imp-v1-3b significantly outperforms the counterparts of similar model sizes, and even achieves slightly better performance than the strong LLaVA-7B model on various multimodal benchmarks.

We also release the model weights a running example of imp-v1-3b on Huggingface. Technical report will be released soon. We will persistently improve our model and release the next versions to further improve model performance :)

• February 9, 2024: Training and evaluation codes of imp-v1-3b are released.

• Prerequisites
• Usage
• Training
• Evaluation
• License
• Citation

1. Clone this repository and navigate to the folder

git clone https://github.com/MILVLG/imp.git
cd imp

2. Install Package

We recommend using Anaconda to create a new environment for the project, and install the requirements with the following commands:

conda create -n imp python=3.10 -y
conda activate imp
pip install -r requirements.txt
pip install flash-attn==2.4.2 --no-build-isolation

3. (Optional) Manually download the pretrained model repositories, i.e., google/siglip-so400m-patch14-384 and microsoft/phi-2 to your local directories and modify the corresponding paths in the training and evaluation scripts, respectively.

The training pipeline and datasets of imp-v1-3b are directly inherited from LLaVA-v1.5. The training

• Multimodal pretraining: train a projector on a subset of ∼558K image-text pairs to connect a frozen pretrained vision encoder and a frozen LLM.
• Multimodal instruction tuning: fine-tune the projector and LoRA in the LLM with multimodal instruction data and VQA-formatted data to empower the MLSM the ability of multimodal instruction following.

Imp is trained on 8 A100 (40G) GPUs. You can reduce the per_device_train_batch_size and increase the gradient_accumulation_steps to match your resources. .But always keep the global batch size the same: global_batch_size = per_device_train_batch_size $\times$ gradient_accumulation_steps $\times$ num_gpus.

bash scripts/pretrain.sh

datasets
├── llava_v1_5_mix665k.json
└── finetune_images
    ├── coco
    │   └── train2017
    ├── gqa
    │   └── images
    ├── ocr_vqa
    │   └── images
    ├── textvqa
    │   └── train_images
    └── vg
        ├── VG_100K
        └── VG_100K_2

bash scripts/finetune_lora.sh
# bash scripts/finetune.sh # fully finetuning is not recommended

We follow the evaluation of LLaVA-v1.5 and conduct experiments on 9 commonly-used benchmarks, including 5 academic VQA benchmarks and 4 popular MLLM benchmarks. All evaluation scripts are placed in the scripts/eval folder.

Before preparing task-specific data, you should download eval.zip and unzip it to ./playground/data/eval. For more specific instructions, please refer to LLaVA's Evaluation.md.

It is supported to evaluate your own trained model checkpoints or our released imp-v1-3b model at Huggingface Hub.

For more detailed evaluation scripts, please See Evaluation.md.

Using the provided model ckpts, you can reproduce the following results. Our imp-v1-3b model significantly outperforms existing MSLMs of similar model sizes, and is comparable with the strong LLaVA-v1.5-7b model.

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

This project is maintained by the MILVLG@Hangzhou Dianzi University (HDU) led by Prof. Zhou Yu and Jun Yu, and is mainly developed by Zhenwei Shao and Xuecheng Ouyang. We hope our model may serve as a strong baseline to inspire future research on MSLM, as well as its derivative applications on mobile devices and robots.

If you use our model or refer our work in your studies, please cite:

@misc{imp2024,
  author = {Shao, Zhenwei and Ouyang, Xuecheng and Gai, Zhenbiao and Yu, Zhou and Yu, Jun},
  title = {Imp: An emprical study of multimodal small language models},
  year = {2024},
  url = {https://huggingface.co/MILVLG/imp-v1-3b}
}