Skip to content

Commit

Permalink
Updated formatting
Browse files Browse the repository at this point in the history
  • Loading branch information
Yixin Wan [email protected] committed Nov 12, 2023
1 parent e236e72 commit 526f0ff
Showing 1 changed file with 12 additions and 4 deletions.
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,9 @@ Arxiv version available at: https://arxiv.org/abs/2310.09219
Refer to the following steps to generate recommendation letters using ChatGPT and other LLMs.

### Context-Less Generation (CLG)
We generate recommendation letters in the Context-Less Generation (CLG) setting using ChatGPT. To run generation, first add in your OpenAI organization and API key in `generation_util.py`. Then, use the following command to generate using ChatGPT:
We generate recommendation letters in the Context-Less Generation (CLG) setting using ChatGPT.

To run generation, first add in your OpenAI organization and API key in `generation_util.py`. Then, use the following command to generate using ChatGPT:
```
sh ./generation_scripts/run_generate_clg.sh
```
Expand Down Expand Up @@ -50,7 +52,9 @@ sh ./evaluation_scripts/run_bias_clg.sh
```

### Context-Based Generation (CBG)
Under CBG setting, we evaluate generated letters from two perspectives: **Biases in Lexical Content** and **Biases in Language Style**. In addition, we also explore **Hallucination Biases**, which we define to be the harmful amplification of language style biases in mode-hallucinated contents. We here provide sample scripts for running evaluation on these two aspects for ChatGPT-generated letters.
Under CBG setting, we evaluate generated letters from two perspectives: **Biases in Lexical Content** and **Biases in Language Style**. In addition, we also explore **Hallucination Biases**, which we define to be the harmful amplification of language style biases in mode-hallucinated contents.

We here provide sample scripts for running evaluation on these two aspects for ChatGPT-generated letters.

#### Biases in Lexical Content
To evaluate Biases in Lexical Content for ChatGPT-generated letters, run:
Expand All @@ -69,7 +73,9 @@ Next, to evaluate Biases in Language Style for ChatGPT-generated letters, run:
sh ./evaluation_scripts/run_language_style_eval.sh
```

Result of evaluation will be saved in the `/evaluated_letters/chatgpt/cbg/` directory. Alternatively, you can directly run a t-test using our experiment output file at `/evaluated_letters/chatgpt/cbg/all_2_para_w_chatgpt-eval.csv` by running:
Result of evaluation will be saved in the `/evaluated_letters/chatgpt/cbg/` directory.

Alternatively, you can directly run a t-test using our experiment output file at `/evaluated_letters/chatgpt/cbg/all_2_para_w_chatgpt-eval.csv` by running:
```
python ttest.py -if ./evaluated_letters/chatgpt/cbg/all_2_para_w_chatgpt-eval.csv
```
Expand All @@ -89,7 +95,9 @@ Result of hallucination detectrion will be saved in the `/evaluated_letters/chat
sh ./evaluation_scripts/run_language_style_eval_hallucination.sh
```

Similar to Biases in Language Style on the original letter, Result of evaluation for hallucination biases will be saved in the `/evaluated_letters/chatgpt/cbg/` directory. Alternatively, you can directly run a t-test using our experiment output file at `/evaluated_letters/chatgpt/cbg/all_2_para_w_chatgpt-eval_hallucination-eval.csv` by running:
Similar to Biases in Language Style on the original letter, Result of evaluation for hallucination biases will be saved in the `/evaluated_letters/chatgpt/cbg/` directory.

Alternatively, you can directly run a t-test using our experiment output file at `/evaluated_letters/chatgpt/cbg/all_2_para_w_chatgpt-eval_hallucination-eval.csv` by running:
```
python ttest.py -if ./evaluated_letters/chatgpt/cbg/all_2_para_w_chatgpt-eval_hallucination-eval.csv --eval_hallucination_part
```

0 comments on commit 526f0ff

Please sign in to comment.