Lion (generated by stable diffusion 2.1)

Usage and License Notices: Like Alpaca, Lion is intended and licensed for research use only.

arXiv preprint: https://arxiv.org/abs/2305.12870

• 🙏 Since our team members are perparing for the PhD Qualifying Exam, we apologize for any possible delay in responding to your questions. We warmly welcome all inquiries and appreciate your constructive feedback :)
• [2023/05/23] We released the code for training and inference.

1. Overview

2. Recovering Lion weights

3. Inference

4. Training Process

5. Evaluation

6. Citation

7. Disclaimer

The high-level overview of our adversarial distillation framework, where we craft a compact Student LLM based on a superior closed-source LLM that serves three roles: the Teacher, the Referee, and the Generator. From left to right, there are three stages in an iteration:

1. an imitation stage to align the student’s response with the teacher’s response;
2. a discrimination stage to identify hard samples;
3. a generation stage to produce new hard samples for escalating the challenges presented to the student model.

We release Lion weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the Lion weights. Instructions:

1. Get the original LLaMA weights in the huggingface format by following the instructions here
2. Please download our delta model at the following link(The model weights will come soon)
3. Use the following scripts to get Lion weights by applying our delta:

python src/weight_diff.py recover --path_raw <path_to_step_1_dir> --path_diff <path_to_step_2_dir> --path_tuned <path_to_store_recovered_weights>

For inference and training of Lion, please first install the requirements:

pip install -r requirements.txt

We provide the decoding script for Lion, which reads a input file and generates corresponding responses for each sample, and finally consolidates them into an output file.

python src/lion_inference.py \
    --model_dir <path_to_hf_converted_lion_ckpt_and_tokenizer> \
    --data_dir <path_to_input_json_file> \
    --output_dir <path_to_output_json_file> \
    --num_gpus 8

Below shows one iteration of our adversarial distillation framework.

python src/chatgpt_inference.py \
    -q <path_to_json_file_for_the_Train_Pool> \
    -o <path_to_chatgpt_inference_for_the_Train_Pool> \
    --api_key <your_openai_api_key>

Fine-tuning was conducted on on a machine with 8 A100 80G GPUs.

torchrun --nproc_per_node=8 --master_port=<your_random_port> src/train.py \
    --model_name_or_path <path_to_hf_converted_ckpt_and_tokenizer> \
    --data_path <path_to_chatgpt_inference_for_the_Train_Pool> \
    --bf16 True \
    --output_dir result \
    --num_train_epochs 3 \
    --model_max_length 1024 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --gradient_accumulation_steps 8 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 500 \
    --save_total_limit 1 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --fsdp "full_shard auto_wrap" \
    --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
    --tf32 True

python src/chatgpt_inference.py \
    -q <path_to_json_file_for_the_Cache_Pool> \
    -o <path_to_chatgpt_inference_for_the_Cache_Pool> \
    --api_key <your_openai_api_key>

python src/lion_inference.py \
    --model_dir <path_to_hf_converted_lion_ckpt_and_tokenizer> \
    --data_dir <path_to_json_file_for_the_Cache_Pool> \
    --output_dir <path_to_lion_inference_for_the_Cache_Pool> \
    --num_gpus 8

python src/chatgpt_referee.py \
    -a <path_to_chatgpt_inference_for_the_Cache_Pool> <path_to_lion_inference_for_the_Cache_Pool> \
    -o <path_to_output_review_file> \
    --api_key <your_openai_api_key>

python src/discrimination.py \
    --review_path <path_to_output_review_file> \
    --chatgpt_inference_path <path_to_chatgpt_inference_for_the_Cache_Pool> \
    --lion_inference_path path_to_lion_inference_for_the_Cache_Pool \
    --hard_save_path <path_to_identified_hard_instructions> \
    --easy_save_path <path_to_identified_easy_instructions>

Fill the openai.api_key = "<you_openai_api_key>" in src/utils.py.

python -m src/generate_hard_instruction generate_instruction_following_data \
    --seed_tasks_path <path_to_identified_hard_instructions> \
    --output_dir <path_to_generated_hard_instructions> \
    --num_instructions_to_generate 3000

python -m src/generate_easy_instruction generate_instruction_following_data \
    --seed_tasks_path <path_to_identified_easy_instructions> \
    --output_dir <path_to_generated_easy_instructions> \
    --num_instructions_to_generate 3000

we leverage GPT-4 to automatically rate the response quality (with scores from 1 to 10) between two models on 80 unseen Vicuna-Instructions. ChatGPT has been chosen as the reference model to estimate the relative capability of diverse LLMs against it. The relative score is reported in percentage, computed as the ratio of the sum of scores.

Relative Overall Response Quality:

Relative Response Quality of Diverse Task Categories:

We employ the alignment criteria proposed by Askell et al. (2021), which define that an assistant is considered aligned if it is characterized by being helpful, honest, and harmless (HHH). We performed a human evaluation on 252 UserOriented-Instructions. To estimate the won rate, we compare the frequency of won, tie, and lost between each pair of models below.

Please cite our paper if you use the code in this repo.

@article{DBLP:journals/corr/abs-2305-12870,
  author       = {Yuxin Jiang and
                  Chunkit Chan and
                  Mingyang Chen and
                  Wei Wang},
  title        = {Lion: Adversarial Distillation of Closed-Source Large Language Model},
  journal      = {CoRR},
  volume       = {abs/2305.12870},
  year         = {2023},
  url          = {https://doi.org/10.48550/arXiv.2305.12870},
  doi          = {10.48550/arXiv.2305.12870},
  eprinttype   = {arXiv},
  eprint       = {2305.12870},
  timestamp    = {Mon, 22 May 2023 09:49:16 UTC},
  biburl       = {https://dblp.org/rec/journals/corr/abs-2305-12870.bib},
  bibsource    = {dblp computer science bibliography, https://dblp.org}
}

The resources, including code, data, and model weights, associated with this project are restricted for academic research purposes only and cannot be used for commercial purposes. The content produced by any version of Lion is influenced by uncontrollable variables such as randomness, and therefore, the accuracy of the output cannot be guaranteed by this project. This project does not accept any legal liability for the content of the model output, nor does it assume responsibility for any losses incurred due to the use of associated resources and output results.