[Question] finetune LLaVA-1.5 with LoRA.: does not appear to have a file named config.json.

[yangzian035210]

Question

Thanks for your great work! I have a question about finetune LLaVA-1.5 with LoRA:
OSError: /checkpoints/llava-v1.5-13b-lora-v2/checkpoint-6000 does not appear to have a file named config.json.

[dhifafaz]

have you figured it out?

[CiaoHe]

It seems like we need add a trigger when saving intermediate checkpoint, like the ones after train finished:

if training_args.lora_enable:
    state_dict = get_peft_state_maybe_zero_3(
        model.named_parameters(), training_args.lora_bias
    )
    non_lora_state_dict = get_peft_state_non_lora_maybe_zero_3(
        model.named_parameters()
    )
    if training_args.local_rank == 0 or training_args.local_rank == -1:
        model.config.save_pretrained(training_args.output_dir)
        model.save_pretrained(training_args.output_dir, state_dict=state_dict)
        torch.save(non_lora_state_dict, os.path.join(training_args.output_dir, 'non_lora_trainables.bin'))
else:
    safe_save_model_for_hf_trainer(trainer=trainer,
                                   output_dir=training_args.output_dir)

[CiaoHe]

I added a custom callback just above the trainer initialize

# save callback
from transformers import TrainerCallback
class SaveCallback(TrainerCallback):
    def on_save(self, args, state, control, **kwargs):
        checkpoint_dir = os.path.join(args.output_dir, 'checkpoint-{}'.format(state.global_step))
        if args.lora_enable:
            state_dict = get_peft_state_maybe_zero_3(
                model.named_parameters(), training_args.lora_bias
            )
            non_lora_state_dict = get_peft_state_non_lora_maybe_zero_3(
                model.named_parameters()
            )
            if args.local_rank in [-1, 0]:
                model.config.save_pretrained(checkpoint_dir)
                model.save_pretrained(checkpoint_dir, state_dict=state_dict)
                torch.save(non_lora_state_dict, os.path.join(checkpoint_dir, 'non_lora_trainables.bin'))

and pass it to the trainer initializing

trainer = LLaVATrainer(model=model,
                tokenizer=tokenizer,
                args=training_args,
                callbacks=[SaveCallback()],
                **data_module)

[wuwu-C]

hey I met the same problem. Did you solve this?