- train_sft.sh와 train_dpo.sh 두 가지 실행 스크립트를 만들어놨습니다. 이 두 가지 스크립트를 이용하여 LLM을 튜닝하세요.
- Qwen LLM을 사용하기 위해서는 deepspeed 패키지의 버전이 0.12.5 이상이어야 합니다.
pip install -r requirements.txt
pip install deepspeed==0.12.5
- data 폴더를 보면 json 데이터 형식이 있는데 이를 참고하여 데이터를 만드세요.
- 모든 데이터는 템플릿과 결합되어 전처리가 진행됩니다. 템플릿에 대한 설명은 3. 템플릿을 참고하세요.
- 예를 들어 data 폴더 내의 alpaca_gpt4_data_en.json 형식을 보면 아래와 같습니다.
{
"instruction": "Give three tips for staying healthy.",
"input": "",
"output": "1. Eat a balanced and nutritious diet: Make sure your meals are inclusive of a variety of fruits and vegetables, lean protein, whole grains, and healthy fats. This helps to provide your body with the essential nutrients to function at its best and can help prevent chronic diseases.\n\n2. Engage in regular physical activity: Exercise is crucial for maintaining strong bones, muscles, and cardiovascular health. Aim for at least 150 minutes of moderate aerobic exercise or 75 minutes of vigorous exercise each week.\n\n3. Get enough sleep: Getting enough quality sleep is crucial for physical and mental well-being. It helps to regulate mood, improve cognitive function, and supports healthy growth and immune function. Aim for 7-9 hours of sleep each night."
}
instruction이 현 질문에 해당하며, output이 현 질문에 대한 답변으로 전처리됩니다.
Human: Give three tips for staying healthy.<|im_end|>
Assistant: 1. Eat a balanced and nutritious diet: Make sure your meals are inclusive of a variety of fruits and vegetables, lean protein, whole grains, and healthy fats. This helps to provide your body with the essential nutrients to function at its best and can help prevent chronic diseases.\n\n2. Engage in regular physical activity: Exercise is crucial for maintaining strong bones, muscles, and cardiovascular health. Aim for at least 150 minutes of moderate aerobic exercise or 75 minutes of vigorous exercise each week.\n\n3. Get enough sleep: Getting enough quality sleep is crucial for physical and mental well-being. It helps to regulate mood, improve cognitive function, and supports healthy growth and immune function. Aim for 7-9 hours of sleep each night.<|im_end|>
- 예를 들어 data 폴더 내의 oaast_sft.json 형식을 보면 아래와 같습니다.
{
"instruction": "What can be done at a regulatory level to ensure, in a monopsony, that the power over the employees isn't abused? List a few options and focus on the organs that should act.",
"input": "",
"output": "Here are some potential regulatory options to address monopsony power over employees:\n\n- Antitrust enforcement agencies like the FTC or DOJ could investigate and take action against dominant employers that are abusing their market power to suppress wages or worsen working conditions. They could force divestitures or other structural remedies.\n- Labor standards agencies could set industry-specific minimum wage floors or other regulations around compensation and working conditions. This would reduce the ability of dominant employers to push wages/conditions below certain thresholds.\n- Sector-specific regulators could impose requirements around how companies hire, compensate, and treat workers. For example, banking regulators could prohibit abusive practices by dominant banks with regard to their employees.\n- Legislatures could pass laws",
"history": [
[
"Can you write a short introduction about the relevance of the term \"monopsony\" in economics? Please use examples related to potential monopsonies in the labour market and cite relevant research.",
"Monopsony refers to a market structure where there is only one buyer of a good or service. In the context of labor markets, a monopsony occurs when there is only one employer in a particular geographic area or industry, giving them significant power to set wages and working conditions. Research has shown that the existence of monopsonies can lead to lower wages and reduced job opportunities for workers. For example, a study by the National Bureau of Economic Research found that in industries with high levels of concentration, workers earn lower wages and are less likely to receive benefits such as health insurance."
]
]
}
- History가 앞의 대화로 간주되고 instruction이 현 질문에 해당하며, output이 현 질문에 대한 답변으로 전처리됩니다.
Human: Can you write a short introduction about the relevance of the term "monopsony" in economics? Please use examples related to potential monopsonies in the labour market and cite relevant research.
Assistant: Monopsony refers to a market structure where there is only one buyer of a good or service. In the context of labor markets, a monopsony occurs when there is only one employer in a particular geographic area or industry, giving them significant power to set wages and working conditions. Research has shown that the existence of monopsonies can lead to lower wages and reduced job opportunities for workers. For example, a study by the National Bureau of Economic Research found that in industries with high levels of concentration, workers earn lower wages and are less likely to receive benefits such as health insurance.<|im_end|>
Human: What can be done at a regulatory level to ensure, in a monopsony, that the power over the employees isn't abused? List a few options and focus on the organs that should act.
Assistant: Here are some potential regulatory options to address monopsony power over employees:
- Antitrust enforcement agencies like the FTC or DOJ could investigate and take action against dominant employers that are abusing their market power to suppress wages or worsen working conditions. They could force divestitures or other structural remedies.
- Labor standards agencies could set industry-specific minimum wage floors or other regulations around compensation and working conditions. This would reduce the ability of dominant employers to push wages/conditions below certain thresholds.
- Sector-specific regulators could impose requirements around how companies hire, compensate, and treat workers. For example, banking regulators could prohibit abusive practices by dominant banks with regard to their employees.
- Legislatures could pass laws<|im_end|>
- 예를 들어 data 폴더 내의 comparison_gpt4_data_en.json 형식을 보면 아래와 같습니다.
{
"instruction": "What are the three primary colors?",
"input": "",
"output": [
"The three primary colors are red, blue, and yellow. These colors are called primary because they cannot be created by mixing other colors and all other colors can be made by combining them in various proportions. In the additive color system, used for light, the primary colors are red, green, and blue (RGB).",
"Red, Yellow, and Green."
]
}
Human: What are the three primary colors?
Assistant:
chosen:
The three primary colors are red, blue, and yellow. These colors are called primary because they cannot be created by mixing other colors and all other colors can be made by combining them in various proportions. In the additive color system, used for light, the primary colors are red, green, and blue (RGB).<|im_end|>
rejected:
Red, Yellow, and Green.<|im_end|>
src > llm_tuner > data > template.py
폴더를 보면 각 LLM마다 템플릿이 존재합니다. json과 템플릿이 결합되어 전처리가 진행됩니다.- 예를 들어 default 템플릿은 아래와 같습니다.
_register_template(
name="default",
format_user=StringFormatter(slots=["Human: {{content}}\nAssistant: "]),
format_system=StringFormatter(slots=["{{content}}\n"]),
format_separator=EmptyFormatter(slots=["\n"]),
)
👋 Join our WeChat.
[ English | 中文 ]
Preview LLaMA Board at 🤗 Spaces or ModelScope.
Launch LLaMA Board via CUDA_VISIBLE_DEVICES=0 python src/train_web.py
. (multiple GPUs are not supported yet in this mode)
Here is an example of altering the self-cognition of an instruction-tuned language model within 10 minutes on a single GPU.
tutorial.mp4
- Benchmark
- Changelog
- Supported Models
- Supported Training Approaches
- Provided Datasets
- Requirement
- Getting Started
- Projects using LLaMA Factory
- License
- Citation
- Acknowledgement
Compared to ChatGLM's P-Tuning, LLaMA-Factory's LoRA tuning offers up to 3.7 times faster training speed with a better Rouge score on the advertising text generation task. By leveraging 4-bit quantization technique, LLaMA-Factory's QLoRA further improves the efficiency regarding the GPU memory.
Definitions
- Training Speed: the number of training samples processed per second during the training. (bs=4, cutoff_len=1024)
- Rouge Score: Rouge-2 score on the development set of the advertising text generation task. (bs=4, cutoff_len=1024)
- GPU Memory: Peak GPU memory usage in 4-bit quantized training. (bs=1, cutoff_len=1024)
- We adopt
pre_seq_len=128
for ChatGLM's P-Tuning andlora_rank=32
for LLaMA-Factory's LoRA tuning.
[24/02/05] Qwen1.5 (Qwen2 beta version) series models are supported in LLaMA-Factory. Check this blog post for details.
[24/01/18] We supported agent tuning for most models, equipping model with tool using abilities by fine-tuning with --dataset glaive_toolcall
.
[23/12/23] We supported unsloth's implementation to boost LoRA tuning for the LLaMA, Mistral and Yi models. Try --use_unsloth
argument to activate unsloth patch. It achieves 1.7x speed in our benchmark, check this page for details.
[23/12/12] We supported fine-tuning the latest MoE model Mixtral 8x7B in our framework. See hardware requirement here.
Full Changelog
[23/12/01] We supported downloading pre-trained models and datasets from the ModelScope Hub for Chinese mainland users. See this tutorial for usage.
[23/10/21] We supported NEFTune trick for fine-tuning. Try --neftune_noise_alpha
argument to activate NEFTune, e.g., --neftune_noise_alpha 5
.
[23/09/27] We supported --shift_attn
argument to enable shift short attention.
[23/09/23] We integrated MMLU, C-Eval and CMMLU benchmarks in this repo. See this example to evaluate your models.
[23/09/10] We supported FlashAttention-2. Try --flash_attn
argument to enable FlashAttention-2 if you are using RTX4090, A100 or H100 GPUs.
[23/08/12] We supported RoPE scaling to extend the context length of the LLaMA models. Try --rope_scaling linear
argument in training and --rope_scaling dynamic
argument at inference to extrapolate the position embeddings.
[23/08/11] We supported DPO training for instruction-tuned models. See this example to train your models.
[23/07/31] We supported dataset streaming. Try --streaming
and --max_steps 10000
arguments to load your dataset in streaming mode.
[23/07/29] We released two instruction-tuned 13B models at Hugging Face. See these Hugging Face Repos (LLaMA-2 / Baichuan) for details.
[23/07/18] We developed an all-in-one Web UI for training, evaluation and inference. Try train_web.py
to fine-tune models in your Web browser. Thank @KanadeSiina and @codemayq for their efforts in the development.
[23/07/09] We released FastEdit ⚡🩹, an easy-to-use package for editing the factual knowledge of large language models efficiently. Please follow FastEdit if you are interested.
[23/06/29] We provided a reproducible example of training a chat model using instruction-following datasets, see Baichuan-7B-sft for details.
[23/06/22] We aligned the demo API with the OpenAI's format where you can insert the fine-tuned model in arbitrary ChatGPT-based applications.
[23/06/03] We supported quantized training and inference (aka QLoRA). Try --quantization_bit 4/8
argument to work with quantized models.
Model | Model size | Default module | Template |
---|---|---|---|
Baichuan2 | 7B/13B | W_pack | baichuan2 |
BLOOM | 560M/1.1B/1.7B/3B/7.1B/176B | query_key_value | - |
BLOOMZ | 560M/1.1B/1.7B/3B/7.1B/176B | query_key_value | - |
ChatGLM3 | 6B | query_key_value | chatglm3 |
DeepSeek (MoE) | 7B/16B/67B | q_proj,v_proj | deepseek |
Falcon | 7B/40B/180B | query_key_value | falcon |
InternLM2 | 7B/20B | wqkv | intern2 |
LLaMA | 7B/13B/33B/65B | q_proj,v_proj | - |
LLaMA-2 | 7B/13B/70B | q_proj,v_proj | llama2 |
Mistral | 7B | q_proj,v_proj | mistral |
Mixtral | 8x7B | q_proj,v_proj | mistral |
Phi-1.5/2 | 1.3B/2.7B | q_proj,v_proj | - |
Qwen | 1.8B/7B/14B/72B | c_attn | qwen |
Qwen1.5 | 0.5B/1.8B/4B/7B/14B/72B | q_proj,v_proj | qwen |
XVERSE | 7B/13B/65B | q_proj,v_proj | xverse |
Yi | 6B/34B | q_proj,v_proj | yi |
Yuan | 2B/51B/102B | q_proj,v_proj | yuan |
Note
Default module is used for the --lora_target
argument, you can use --lora_target all
to specify all the available modules.
For the "base" models, the --template
argument can be chosen from default
, alpaca
, vicuna
etc. But make sure to use the corresponding template for the "chat" models.
Please refer to constants.py for a full list of models we supported.
Approach | Full-parameter | Partial-parameter | LoRA | QLoRA |
---|---|---|---|---|
Pre-Training | ✅ | ✅ | ✅ | ✅ |
Supervised Fine-Tuning | ✅ | ✅ | ✅ | ✅ |
Reward Modeling | ✅ | ✅ | ✅ | ✅ |
PPO Training | ✅ | ✅ | ✅ | ✅ |
DPO Training | ✅ | ✅ | ✅ | ✅ |
Note
Use --quantization_bit 4
argument to enable QLoRA.
Pre-training datasets
Supervised fine-tuning datasets
- Stanford Alpaca (en)
- Stanford Alpaca (zh)
- Alpaca GPT4 (en&zh)
- Self Cognition (zh)
- Open Assistant (multilingual)
- ShareGPT (zh)
- Guanaco Dataset (multilingual)
- BELLE 2M (zh)
- BELLE 1M (zh)
- BELLE 0.5M (zh)
- BELLE Dialogue 0.4M (zh)
- BELLE School Math 0.25M (zh)
- BELLE Multiturn Chat 0.8M (zh)
- UltraChat (en)
- LIMA (en)
- OpenPlatypus (en)
- CodeAlpaca 20k (en)
- Alpaca CoT (multilingual)
- OpenOrca (en)
- SlimOrca (en)
- MathInstruct (en)
- Firefly 1.1M (zh)
- Wiki QA (en)
- Web QA (zh)
- WebNovel (zh)
- Nectar (en)
- deepctrl (en&zh)
- Ad Gen (zh)
- ShareGPT Hyperfiltered (en)
- ShareGPT4 (en&zh)
- UltraChat 200k (en)
- AgentInstruct (en)
- LMSYS Chat 1M (en)
- Evol Instruct V2 (en)
- Glaive Function Calling V2 (en)
- Open Assistant (de)
- Dolly 15k (de)
- Alpaca GPT4 (de)
- OpenSchnabeltier (de)
- Evol Instruct (de)
- Dolphin (de)
- Booksum (de)
- Airoboros (de)
- Ultrachat (de)
Preference datasets
Please refer to data/README.md for details.
Some datasets require confirmation before using them, so we recommend logging in with your Hugging Face account using these commands.
pip install --upgrade huggingface_hub
huggingface-cli login
- Python 3.8+ and PyTorch 1.13.1+
- 🤗Transformers, Datasets, Accelerate, PEFT and TRL
- sentencepiece, protobuf and tiktoken
- jieba, rouge-chinese and nltk (used at evaluation and predict)
- gradio and matplotlib (used in web UI)
- uvicorn, fastapi and sse-starlette (used in API)
Method | Bits | 7B | 13B | 30B | 65B | 8x7B |
---|---|---|---|---|---|---|
Full | 16 | 160GB | 320GB | 600GB | 1200GB | 900GB |
Freeze | 16 | 20GB | 40GB | 120GB | 240GB | 200GB |
LoRA | 16 | 16GB | 32GB | 80GB | 160GB | 120GB |
QLoRA | 8 | 10GB | 16GB | 40GB | 80GB | 80GB |
QLoRA | 4 | 6GB | 12GB | 24GB | 48GB | 32GB |
Please refer to data/README.md for checking the details about the format of dataset files. You can either use a single .json
file or a dataset loading script with multiple files to create a custom dataset.
Note
Please update data/dataset_info.json
to use your custom dataset. About the format of this file, please refer to data/README.md
.
git clone https://github.com/hiyouga/LLaMA-Factory.git
conda create -n llama_factory python=3.10
conda activate llama_factory
cd LLaMA-Factory
pip install -r requirements.txt
If you want to enable the quantized LoRA (QLoRA) on the Windows platform, you will be required to install a pre-built version of bitsandbytes
library, which supports CUDA 11.1 to 12.1.
pip install https://github.com/jllllll/bitsandbytes-windows-webui/releases/download/wheels/bitsandbytes-0.39.1-py3-none-win_amd64.whl
If you have trouble with downloading models and datasets from Hugging Face, you can use LLaMA-Factory together with ModelScope in the following manner.
export USE_MODELSCOPE_HUB=1 # `set USE_MODELSCOPE_HUB=1` for Windows
Then you can train the corresponding model by specifying a model ID of the ModelScope Hub. (find a full list of model IDs at ModelScope Hub)
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
--model_name_or_path modelscope/Llama-2-7b-ms \
... # arguments (same as above)
LLaMA Board also supports using the models and datasets on the ModelScope Hub.
CUDA_VISIBLE_DEVICES=0 USE_MODELSCOPE_HUB=1 python src/train_web.py
Important
If you want to train models on multiple GPUs, please refer to Distributed Training.
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
--stage pt \
--do_train \
--model_name_or_path path_to_llama_model \
--dataset wiki_demo \
--finetuning_type lora \
--lora_target q_proj,v_proj \
--output_dir path_to_pt_checkpoint \
--overwrite_cache \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--save_steps 1000 \
--learning_rate 5e-5 \
--num_train_epochs 3.0 \
--plot_loss \
--fp16
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
--stage sft \
--do_train \
--model_name_or_path path_to_llama_model \
--dataset alpaca_gpt4_en \
--template default \
--finetuning_type lora \
--lora_target q_proj,v_proj \
--output_dir path_to_sft_checkpoint \
--overwrite_cache \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--save_steps 1000 \
--learning_rate 5e-5 \
--num_train_epochs 3.0 \
--plot_loss \
--fp16
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
--stage rm \
--do_train \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_sft_checkpoint \
--create_new_adapter \
--dataset comparison_gpt4_en \
--template default \
--finetuning_type lora \
--lora_target q_proj,v_proj \
--output_dir path_to_rm_checkpoint \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 4 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--save_steps 1000 \
--learning_rate 1e-6 \
--num_train_epochs 1.0 \
--plot_loss \
--fp16
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
--stage ppo \
--do_train \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_sft_checkpoint \
--create_new_adapter \
--dataset alpaca_gpt4_en \
--template default \
--finetuning_type lora \
--lora_target q_proj,v_proj \
--reward_model path_to_rm_checkpoint \
--output_dir path_to_ppo_checkpoint \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 4 \
--lr_scheduler_type cosine \
--top_k 0 \
--top_p 0.9 \
--logging_steps 10 \
--save_steps 1000 \
--learning_rate 1e-5 \
--num_train_epochs 1.0 \
--plot_loss \
--fp16
Warning
Use --per_device_train_batch_size=1
for LLaMA-2 models in fp16 PPO training.
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
--stage dpo \
--do_train \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_sft_checkpoint \
--create_new_adapter \
--dataset comparison_gpt4_en \
--template default \
--finetuning_type lora \
--lora_target q_proj,v_proj \
--output_dir path_to_dpo_checkpoint \
--per_device_train_batch_size 2 \
--gradient_accumulation_steps 4 \
--lr_scheduler_type cosine \
--logging_steps 10 \
--save_steps 1000 \
--learning_rate 1e-5 \
--num_train_epochs 1.0 \
--plot_loss \
--fp16
accelerate config # configure the environment
accelerate launch src/train_bash.py # arguments (same as above)
Example config for LoRA training
compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
downcast_bf16: 'no'
gpu_ids: all
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
deepspeed --num_gpus 8 --master_port=9901 src/train_bash.py \
--deepspeed ds_config.json \
... # arguments (same as above)
Example config for full-parameter training with DeepSpeed ZeRO-2
{
"train_batch_size": "auto",
"train_micro_batch_size_per_gpu": "auto",
"gradient_accumulation_steps": "auto",
"gradient_clipping": "auto",
"zero_allow_untested_optimizer": true,
"fp16": {
"enabled": "auto",
"loss_scale": 0,
"initial_scale_power": 16,
"loss_scale_window": 1000,
"hysteresis": 2,
"min_loss_scale": 1
},
"zero_optimization": {
"stage": 2,
"allgather_partitions": true,
"allgather_bucket_size": 5e8,
"reduce_scatter": true,
"reduce_bucket_size": 5e8,
"overlap_comm": false,
"contiguous_gradients": true
}
}
python src/export_model.py \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_checkpoint \
--template default \
--finetuning_type lora \
--export_dir path_to_export \
--export_size 2 \
--export_legacy_format False
Warning
Merging LoRA weights into a quantized model is not supported.
Tip
Use --export_quantization_bit 4
and --export_quantization_dataset data/c4_demo.json
to quantize the model after merging the LoRA weights.
python src/api_demo.py \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_checkpoint \
--template default \
--finetuning_type lora
Tip
Visit https://localhost:8000/docs
for API documentation.
python src/cli_demo.py \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_checkpoint \
--template default \
--finetuning_type lora
python src/web_demo.py \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_checkpoint \
--template default \
--finetuning_type lora
CUDA_VISIBLE_DEVICES=0 python src/evaluate.py \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_checkpoint \
--template vanilla \
--finetuning_type lora \
--task mmlu \
--split test \
--lang en \
--n_shot 5 \
--batch_size 4
CUDA_VISIBLE_DEVICES=0 python src/train_bash.py \
--stage sft \
--do_predict \
--model_name_or_path path_to_llama_model \
--adapter_name_or_path path_to_checkpoint \
--dataset alpaca_gpt4_en \
--template default \
--finetuning_type lora \
--output_dir path_to_predict_result \
--per_device_eval_batch_size 8 \
--max_samples 100 \
--predict_with_generate \
--fp16
Warning
Use --per_device_train_batch_size=1
for LLaMA-2 models in fp16 predict.
Tip
We recommend using --per_device_eval_batch_size=1
and --max_target_length 128
at 4/8-bit predict.
- StarWhisper: A large language model for Astronomy, based on ChatGLM2-6B and Qwen-14B.
- DISC-LawLLM: A large language model specialized in Chinese legal domain, based on Baichuan-13B, is capable of retrieving and reasoning on legal knowledge.
- Sunsimiao: A large language model specialized in Chinese medical domain, based on Baichuan-7B and ChatGLM-6B.
- CareGPT: A series of large language models for Chinese medical domain, based on LLaMA2-7B and Baichuan-13B.
- MachineMindset: A series of MBTI Personality large language models, capable of giving any LLM 16 different personality types based on different datasets and training methods.
Tip
If you have a project that should be incorporated, please contact via email or create a pull request.
This repository is licensed under the Apache-2.0 License.
Please follow the model licenses to use the corresponding model weights: Baichuan2 / BLOOM / ChatGLM3 / DeepSeek / Falcon / InternLM2 / LLaMA / LLaMA-2 / Mistral / Phi-1.5/2 / Qwen / XVERSE / Yi / Yuan
If this work is helpful, please kindly cite as:
@Misc{llama-factory,
title = {LLaMA Factory},
author = {hiyouga},
howpublished = {\url{https://github.com/hiyouga/LLaMA-Factory}},
year = {2023}
}
This repo benefits from PEFT, QLoRA and FastChat. Thanks for their wonderful works.