Visual instruction tuning towards large language and vision models with GPT-4 level capabilities, enhanced with the Qwen2 base model.

For more details on usage, refer to the original LLaVA repository. This custom repository specifically integrates the Qwen2 base model to leverage its advanced capabilities.

LLaVA Dataset + FinVis Dataset

git lfs install
git clone https://www.modelscope.cn/TobyYang7/llava-qwen2-1.5b-instruct-finvis.git

Download the MMMU dataset first and rename it as MMMU_eval\data. For more details, you need to follow the official instructions here.

bash eval.sh

bash pretrain_qwen2.sh

The checkpoint for the pretrain projector is located at checkpoints/Qwen2-1.5B-Instruct-pretrain-FinVis/mm_projector.bin

bash ft_qwen2.sh

bash run_cli.sh

This repository builds upon the original LLaVA project, integrating the Qwen2 base model for improved performance.

If you are not using Linux, do NOT proceed, see instructions for macOS and Windows.

1. Clone this repository and navigate to the custom LLaVA folder

   git clone https://github.com/TobyYang7/Llava_Qwen2.git
   cd Llava_Qwen2

2. Install Package

   conda create -n llava python=3.10 -y
   conda activate llava
   pip install --upgrade pip  # enable PEP 660 support
   pip install -e .

3. Install additional packages for training cases

   pip install -e ".[train]"
   pip install flash-attn --no-build-isolation