Add command to train Alpaca-LoRA 7B on Slurm

README.md

@@ -1,3 +1,52 @@
+# Note for the Fork
+
+This fork is a cleaned up version of the original [Stanford Alpaca](https://github.com/tatsu-lab/stanford_alpaca) code and [Alpaca-LoRA](https://github.com/tloen/alpaca-lora).
+
+## Commands for Models
+### Alpaca-LoRA 7B
+```bash
+#!/bin/bash
+
+#SBATCH --partition=spgpu
+#SBATCH --time=04-00:00:00
+
+### request 1 node with 1 gpus, total of 1 gpus (WORLD_SIZE==1)
+### Based on https://gist.github.com/TengdaHan/1dd10d335c7ca6f13810fff41e809904
+#SBATCH --nodes=1
+#SBATCH --ntasks-per-node=1
+#SBATCH --gpus-per-node=1
+#SBATCH --cpus-per-task=4
+#SBATCH --mem-per-cpu=32GB
+
+# set up job
+module load python/3.10.4 cuda
+pushd <path_to_repo>
+source .venv/bin/activate
+
+# run job
+# Hyperparameters closely resemble those of Alpaca-LoRA.
+# https://github.com/tloen/alpaca-lora#official-weights
+python train.py \
+ --model_name_or_path <path_to_converted_llama_weights> \
+ --output_dir <path_to_output_dir> \
+ --num_train_epochs 10 \
+ --learning_rate 3e-4 \
+ --per_device_train_batch_size 32 \
+ --gradient_accumulation_steps 4 \
+ --fp16 True \
+ --use_lora True \
+ --train_in_8bit True \
+ --warmup_steps 100 \
+ --evaluation_strategy "steps" \
+ --eval_steps 200 \
+ --save_strategy "steps" \
+ --save_steps 200 \
+ --save_total_limit 3 \
+ --load_best_model_at_end True \
+ --group_by_length True \
+ --logging_steps 10
+```
+
 
 <p align="center" width="100%">
 <a href="https://crfm.stanford.edu/alpaca/" target="_blank"><img src="assets/logo.png" alt="Stanford-Alpaca" style="width: 50%; min-width: 300px; display: block; margin: auto;"></a>
@@ -52,24 +101,24 @@ We used the following prompts for fine-tuning the Alpaca model:
 
  ```
  Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
- 
+
  ### Instruction:
  {instruction}
- 
+
  ### Input:
  {input}
- 
+
  ### Response:
  ```
 
 - for examples with an empty input field:
 
  ```
  Below is an instruction that describes a task. Write a response that appropriately completes the request.
- 
+
  ### Instruction:
  {instruction}
- 
+
  ### Response:
  ```
 
@@ -108,7 +157,7 @@ We fine-tune our models using standard Hugging Face training code.
 We fine-tune LLaMA-7B and LLaMA-13B with the following hyperparameters:
 
 | Hyperparameter | LLaMA-7B | LLaMA-13B |
-|----------------|----------|-----------|
+| -------------- | -------- | --------- |
 | Batch size | 128 | 128 |
 | Learning rate | 2e-5 | 1e-5 |
 | Epochs | 3 | 5 |