Methods for controlling hallucinations for LLM in Summarization
| MODEL | Strategy | Consistency Rate | Answer Rate | Average Length |
|---|---|---|---|---|
| Mistral-7B-Instruct-v0.1 | Greedy | 93.2 | 100.0 | 93.5 |
| Mistral-7B-Instruct-v0.1 | num_beam = 10 | 95.3 | 100.0 | 127.7 |
| Mistral-7B-Instruct-v0.1 | Greedy + DoLA | 93.7 | 100.0 | 93.6 |
| Mistral-7B-Instruct-v0.1 | Greedy + DPO(LoRA) | 95.8 | 100.0 | 97.0 |
| Mistral-7B-Instruct-v0.1 | Greedy + Fava | 93.7 | 100.0 | 93.3 |
| Mistral-7B-Instruct-v0.1 | DPO(LoRA) + num_beam=10 | 96.9 | 100.0 | 123.7 |
| Mistral-7B-Instruct-v0.1 | Best_of_N + Temperature=0.7 + n=10 | 99.3 | 100.0 | 89.6 |
Note: Prompt slightly different from the orginal HHEM benchmark, causing different numbers.
- Download the leaderboard dataset (https://huggingface.co/spaces/vectara/leaderboard/raw/main/src/datasets/leaderboard_dataset.csv)
- Generate the model response
generated.csv, see methods below - Run evaluation on the reposnse file
python -c "from leaderboard import run_eval;run_eval('generated.csv')"- Notebook: 1_decoding.ipynb
- Notebook: 5_best_of_n.ipynb
- Paper Link
- Notebook: 2_dola.ipynb
- Paper Link
- Notebook: 3_dpo.ipynb
- Training code: dpo_training.py
- Note: our setting is different from the original paper, we used CNN/Dailymail+XSum+VitaminC as the source dataset and HHEM model as the reference metric for factuality.
- Web Link
- Notebook: 4_fava.ipynb