[Usage] Not able to fine tune the LLaVA model with llava-v1.5-7b.

[ayushgupta9198]

Describe the issue

Issue: Not able to fine tune the LLaVA model with llava-v1.5-7b.

also I am sharing my arguments below here so when I am running the code it gives me error as

size mismatch for 2.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([0]).
size mismatch for 2.bias: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([0]).

Command:

this are arguments:

#fine tuning code

!deepspeed llava/train/train_mem.py \
    --deepspeed ./scripts/zero3.json \
    --model_name_or_path iuhaotian/llava-v1.5-7b \
    --version v1 \
    --data_path ./blip_laion_cc_sbu_558k_samples.json \
    --image_folder /teamspace/studios/this_studio/llava_train/LLaVA/playground/data/LLaVA-Pretrain/images \
    --vision_tower openai/clip-vit-large-patch14-336 \
    --pretrain_mm_mlp_adapter /teamspace/studios/this_studio/llava_train/LLaVA/checkpoint-huggingface/mm_projector.bin \
    --mm_projector_type mlp2x_gelu \
    --mm_vision_select_layer -2 \
    --mm_use_im_start_end False \
    --mm_use_im_patch_token False \
    --bf16 False \
    --fp16 True \
    --output_dir ./output/llava-v1.5-7b \
    --num_train_epochs 1 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1 \
    --save_total_limit 1 \
    --learning_rate 1e-3 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 True \
    --model_max_length 128 \
    --image_aspect_ratio pad \
    --group_by_modality_length True \
    --gradient_checkpointing True \
    --dataloader_num_workers 4 \
    --lazy_preprocess True \
    --report_to "none"

I have tried with my mm_projector.bin file also tried with hugging face file but still the same error

currenlty I am using lighting AI studio with A10G GPU.

if anyone know how to fix this please let me know.

Note : Here I have kept the minimum numbers just to start with fine tunning so do not add anything about the number added in the arguments.

Thanks.

[kartikey9254]

same issue . have you found any potential solutions ?

[TheRoadQaQ]

model_name_or_path can not be iuhaotian/llava-v1.5-7b, should be llama2 or vicuna.

[ayushgupta9198]

hi

I have solved the issue by changing the names in arguments aling with that I have also made some changes in LLaVA.py file.

now code is working on both pretrain and fine tunning.

Thanks

[hvgupta]

in arguments aling

Hi, can you please show the changes you have made

[pear-blossoms]

in arguments aling

Hi, can you please show the changes you have made
well, if you try to us llava-v1.5-7b, you can just remove --pretrain_mm_mlp_adapter
it already contains the adapter weights