Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support InternVL 1.5/2.0 finetune #737

Open
wants to merge 28 commits into
base: main
Choose a base branch
from

Conversation

hhaAndroid
Copy link
Collaborator

@hhaAndroid hhaAndroid commented May 31, 2024

基本说明

  • ./v2/ 包含着 InterVL 2 训练配置的配置文件
  • 支持了 InternVL 2B/4B/8B/26B 模型全量/LoRA/QLoRA 单图模式的微调,会尽快支持多图和视频的微调。
  • 在训练完成后,可以使用 ./v1_5/convert_to_official.py 脚本将 XTuner 训练的模型转换为官方格式,从而复用官方所支持的所有工具链
  • 目前所有配置都是以 8xA100 80G 显卡为基准,2B/4B 可以使用 ZERO1 训练,8B 模型要 ZERO2 运行,26B 模型必须要 ZERO3,并且没有对参数进行过多的调整,你可以按照你自己的需求进行修改
  • 目前是以 LLaVA SFT 数据进行验证,无法充分反应微调性能,你可以根据自己的需求进行数据自定义,后续我们会提供一个相对公平的微调数据集

Basic Introduction

  • ./v2/ contains the configuration files for training InterVL 2
  • Supported fine-tuning of the InternVL 2B/4B/8B/26B model in full/LoRA/QLoRA single-image mode for now. We will support fine-tuning on multiple images and videos as soon as possible.
  • After training, you can use the ./v1_5/convert_to_official.py script to convert the model trained by XTuner to the official format, so as to reuse all the official supported toolchains
  • All configurations are based on 8xA100 80G graphics cards, 2B/4B can use ZERO1 training, 8B models can use ZERO2, 26B models must run ZERO3, and there is no excessive adjustment of parameters, you can modify them according to your own needs
  • It is verified with LLaVA SFT data, which cannot fully reflect the fine-tuning performance. You can customize the data according to your own needs. We will provide a relatively fair fine-tuning dataset later

@hhaAndroid hhaAndroid changed the title [WIP] Support InternVL finetune Support InternVL finetune Jun 5, 2024
@KooSung
Copy link
Contributor

KooSung commented Jun 21, 2024

@hhaAndroid hello,请问这个进度大概完成了多少?什么时候会merge到主分支呢?

@hhaAndroid
Copy link
Collaborator Author

@hhaAndroid hello,请问这个进度大概完成了多少?什么时候会merge到主分支呢?

已经完成了,你可以直接用,这个就已经是最终形态了

Copy link

@cydiachen cydiachen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

find one error

xtuner/configs/internvl/v1_5/convert_to_official.py Outdated Show resolved Hide resolved
@hhaAndroid hhaAndroid requested a review from pppppM July 4, 2024 03:36
@hhaAndroid hhaAndroid changed the title Support InternVL finetune Support InternVL1.5/2.0 finetune Jul 4, 2024
@hhaAndroid hhaAndroid changed the title Support InternVL1.5/2.0 finetune Support InternVL 1.5/2.0 finetune Jul 4, 2024
@zhoujz10
Copy link

zhoujz10 commented Jul 9, 2024

请问对多图微调的支持预计要多久呢~

@sunzjz
Copy link

sunzjz commented Jul 10, 2024

请问最新VL40B的模型支持吗

@bjzhb666
Copy link

image
没有找到internvl的文件夹呢?请问现在可以用吗?

@BIGBALLON
Copy link

同问,,config下似乎没有internvl的文件夹?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants