Skip to content

Commit

Permalink
update README with prompt invariance argument
Browse files Browse the repository at this point in the history
  • Loading branch information
ChristyKoh committed Apr 12, 2023
1 parent 34bc364 commit 2da069a
Showing 1 changed file with 5 additions and 0 deletions.
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,11 @@ The following will generate a CCS (Contrast Consistent Search) reporter instead
elk elicit microsoft/deberta-v2-xxlarge-mnli imdb --net ccs
```

To test prompt invariance across multiple datasets, use the `--combined_template_path` command line argument, which will create a new `templates.yaml` file with templates from all the datasets.
```bash
elk elicit bigscience/bloomz-560m christykoh/ag_news_pt ag_news --combined_template_path=spar_w/ag_news
```

The following command will evaluate the probe from the run naughty-northcutt on the hidden states extracted from the model deberta-v2-xxlarge-mnli for the imdb dataset. It will result in an `eval.csv` and `cfg.yaml` file, which are stored under a subfolder in `elk-reporters/naughty-northcutt/transfer_eval`.

```bash
Expand Down

0 comments on commit 2da069a

Please sign in to comment.