Add skill acquisition eval (openai#1497)

# Thank you for contributing an eval! ♥️

🚨 Please make sure your PR follows these guidelines, **failure to follow
the guidelines below will result in the PR being closed automatically**.
Note that even if the criteria are met, that does not guarantee the PR
will be merged nor GPT-4 access be granted. 🚨

**PLEASE READ THIS**:

In order for a PR to be merged, it must fail on GPT-4. We are aware that
right now, users do not have access, so you will not be able to tell if
the eval fails or not. Please run your eval with GPT-3.5-Turbo, but keep
in mind as we run the eval, if GPT-4 gets higher than 90% on the eval,
we will likely reject it since GPT-4 is already capable of completing
the task.

We plan to roll out a way for users submitting evals to see the eval
performance on GPT-4 soon. Stay tuned! Until then, you will not be able
to see the eval performance on GPT-4. **Starting April 10, the minimum
eval count is 15 samples, we hope this makes it easier to create and
contribute evals.**

Also, please note that we're using **Git LFS** for storing the JSON
files, so please make sure that you move the JSON file to Git LFS before
submitting a PR. Details on how to use Git LFS are available
[here](https://git-lfs.com).

## Eval details 📑

### Eval name

Skill acquisition

### Eval description

This eval tests models' ability to learn a skill with minimal human
involvement. In the initial release, models are evaluated on questions
related to the [Miskito
language](https://en.wikipedia.org/wiki/Miskito_language). Some samples
are translation and others are language manipulation exercises.

### What makes this a useful eval?

-

## Criteria for a good eval ✅

Below are some of the criteria we look for in a good eval. In general,
we are seeking cases where the model does not do a good job despite
being capable of generating a good response (note that there are some
things large language models cannot do, so those would not make good
evals).

Your eval should be:

- [x] Thematically consistent: The eval should be thematically
consistent. We'd like to see a number of prompts all demonstrating some
particular failure mode. For example, we can create an eval on cases
where the model fails to reason about the physical world.
- [x] Contains failures where a human can do the task, but either GPT-4
or GPT-3.5-Turbo could not.
- [x] Includes good signal around what is the right behavior. This means
either a correct answer for `Basic` evals or the `Fact` Model-graded
eval, or an exhaustive rubric for evaluating answers for the `Criteria`
Model-graded eval.
- [x] **Include at least 15 high-quality examples.**

If there is anything else that makes your eval worth including, please
document it below.

### Unique eval value

> Insert what makes your eval high quality that was not mentioned above.
(Not required)

## Eval structure 🏗️

Your eval should

- [x] Check that your data is in `evals/registry/data/{name}`
- [x] Check that your YAML is registered at
`evals/registry/evals/{name}.yaml`
- [x] Ensure you have the right to use the data you submit via this eval

(For now, we will only be approving evals that use one of the existing
eval classes. You may still write custom eval classes for your own
cases, and we may consider merging them in the future.)

## Final checklist 👀

### Submission agreement

By contributing to Evals, you are agreeing to make your evaluation logic
and data under the same MIT license as this repository. You must have
adequate rights to upload any data used in an Eval. OpenAI reserves the
right to use this data in future service improvements to our product.
Contributions to OpenAI Evals will be subject to our usual Usage
Policies (<https://platform.openai.com/docs/usage-policies>).

- [x] I agree that my submission will be made available under an MIT
license and complies with OpenAI's usage policies.

### Email address validation

If your submission is accepted, we will be granting GPT-4 access to a
limited number of contributors. Access will be given to the email
address associated with the commits on the merged pull request.

- [x] I acknowledge that GPT-4 access will only be granted, if
applicable, to the email address used for my merged pull request.

### Limited availability acknowledgment

We know that you might be excited to contribute to OpenAI's mission,
help improve our models, and gain access to GPT-4. However, due to the
requirements mentioned above and the high volume of submissions, we will
not be able to accept all submissions and thus not grant everyone who
opens a PR GPT-4 access. We know this is disappointing, but we hope to
set the right expectation before you open this PR.

- [x] I understand that opening a PR, even if it meets the requirements
above, does not guarantee the PR will be merged nor GPT-4 access be
granted.

### Submit eval

- [x] I have filled out all required fields of this form
- [x] I have used **Git LFS** for the Eval JSON data
- [x] (Ignore if not submitting code) I have run `pip install
pre-commit; pre-commit install` and have verified that `mypy`, `black`,
`isort`, `autoflake` and `ruff` are running when I commit and push

Failure to fill out all required fields will result in the PR being
closed.

### Eval JSON data

Since we are using Git LFS, we are asking eval submitters to add in as
many Eval Samples (at least 5) from their contribution here:

<details>
  <summary>View evals in JSON</summary>

  ### Eval
  ```jsonl
  INSERT_EVAL_HERE
  ```
</details>

evals/elsuite/skill_acquisition/readme.md

@@ -0,0 +1,64 @@
+# Skill acquisition
+
+This eval tests models' ability to learn a skill with minimal human involvement. In the initial release, models are evaluated on questions related to the [Miskito language](https://en.wikipedia.org/wiki/Miskito_language). Some samples are translation and others are language manipulation exercises. 
+
+## Usage
+Run with:
+```bash
+oaieval <solver> skill_acquisition.miskito
+```
+
+Where the solver can be any generation solver in `evals/registry/solvers/defaults.yaml`, eg. `generation/cot/gpt-3.5-turbo-16k`.
+
+## Evaluation process
+Every time the eval is run, the model is evaluated twice. The first time, it answers the question directly using whatever prompting technique is executed by the solver you choose. The second time the model runs in a loop, interacting with an interface which gives it access to a knowledge base. The knowledge base contains text files, some of which are relevant for answering the question, while others are unrelated. If models can use this interface to increase their performance on the task, we can say that they've improved or acquired their language translation and manipulation skills.
+
+## Prompts
+See `skill_acquisition/utils.py` to review/adjust the prompts used in this eval.
+
+## Datasets
+
+The dataset is generated from [this language course](https://en.wikibooks.org/wiki/Miskito), which comprises 229 questions. We further split this into manipulation-only (`miskito_test_manipulation.jsonl`) and translation-only (`miskito_test_translation.jsonl`) subsets.
+
+## Variants
+
+We test zero-shot and few-shot prompting techniques on the dataset:
+
+| Dataset | Zero-shot | Few-shot |
+| --------- | -------- | -------- |
+| Miskito | `skill_acquisition.miskito.zero-shot.full`|`skill_acquisition.miskito.few-shot.full`|
+
+The `full` in this case refers to the size of the dataset – there are also variants for testing where only 5 examples are considered, called `dev5`. For full details, look at `evals/registry/skill_acquisition/skill_acquisition.yaml`.
+
+For the few-shot setting, use the eval-specific solvers in `evals/registry/solvers/skill_acquisition.yaml` to avoid train/test leakage.
+
+## Token Usage Estimates
+
+Below is a rough estimate of the total number of tokens consumed by some variations the eval, including both input and output tokens:
+
+| Model | Solver | Prompt tokens | Completion tokens | Total tokens
+| --- | --- | --- | --- | --- |
+| gpt-3.5-turbo | direct | 1,000,000 | 23,000 | 1,050,000 |
+| gpt-3.5-turbo | cot | 930,000 | 120,000 | 1,050,000 |
+| gpt-3.5-turbo | fewshot | 450,000 | 9,600 | 460,000 |
+| gpt-3.5-turbo-16k | direct | 1,400,000 | 24,000 | 1,500,000 |
+| gpt-3.5-turbo-16k | cot | 2,000,000 | 120,000 | 2,100,000 |
+| gpt-3.5-turbo-16k | fewshot | 610,000 | 10,000 | 620,000 |
+| gpt-4-base | direct | 1,800,000 | 420,000 | 2,200,000 |
+| gpt-4-base | cot | 4,700,000 | 890,000 | 5,600,000 |
+| gpt-4-base | fewshot | 1,400,000 | 320,000 | 1,700,000 |
+| gpt-4-1106-preview | direct | 1,700,000 | 100,000 | 1,800,000 |
+| gpt-4-1106-preview | cot | 1,600,000 | 99,000 | 1,700,000 |
+| gpt-4-1106-preview | fewshot | 1,700,000 | 95,000 | 1,800,000 |
+| gpt-4-32k | direct | 1,800,000 | 80,000 | 1,900,000 |
+| gpt-4-32k | cot | 2,700,000 | 180,000 | 2,900,000 |
+| gpt-4-32k | fewshot | 190,000 | 6,000 | 190,000 |
+
+## Version History
+v0: Initial version released
+
+
+## Contribution statement
+
+Eval design, implementation, and results evaluation were primarily conducted by Andrei Alexandru. Giulio Starace was responsible for code reviews throughout the implementation process, along with fine-grained feedback on the project in general. Additional guidance was provided by (alphabetically by last-name) Steven Adler, James Aung and Chan Jun Shern, who scoped and managed the broader research project, including input on evaluation design, results analysis, and interpretation.
+

evals/elsuite/skill_acquisition/scraping/scrape_distractor_articles.py

@@ -0,0 +1,96 @@
+# %%
+import json
+import re
+
+import requests
+from bs4 import BeautifulSoup
+from markdownify import markdownify as md
+
+articles_to_scrape = [
+ "https://en.wikipedia.org/wiki/Mosquito",
+ "https://en.wikipedia.org/wiki/Mosquito_Coast",
+ "https://en.wikipedia.org/wiki/Nicaragua",
+ "https://en.wikipedia.org/wiki/Honduras",
+ "https://en.wikipedia.org/wiki/Miskito_language",
+ "https://en.wikipedia.org/wiki/Miskito_people",
+]
+dirpath = "evals/registry/data/skill_acquisition/distractor_articles/"
+
+
+def clean_soup(content):
+ for infobox_tag in content.find_all("table", class_="infobox"):
+ infobox_tag.decompose()
+ for figure_tag in content.find_all("figure"):
+ figure_tag.decompose()
+ for style_tags in content.find_all("style"):
+ style_tags.decompose()
+ reflist_div = '<div class="reflist"'
+ if reflist_div in str(content):
+ reflist_index = str(content).index(reflist_div)
+ content = BeautifulSoup(str(content)[:reflist_index], "html.parser")
+
+ return content
+
+
+def clean_heading_text(
+ heading_text,
+ strip_list=["a"],
+ css_selector_pattern=r"\.\w+[^{}]*\{[^}]*\}",
+ whitespace_pattern=r"\s\s+",
+):
+ text = md(str(heading_text), strip=strip_list)
+ text = (
+ # text.replace("\n", "")
+ text.replace("\u3000", " ")
+ .replace("\xa0", " ")
+ .replace("| --- |", "")
+ .replace("|", "")
+ )
+ text = re.sub(whitespace_pattern, "", re.sub(css_selector_pattern, "", text))
+ return text
+
+
+for article in articles_to_scrape:
+ response = requests.get(article)
+ soup = BeautifulSoup(response.text, "html.parser")
+
+ content = soup.find("div", class_="mw-content-ltr mw-parser-output")
+ content = clean_soup(content)
+ headings = str(content).split("<h2>")
+
+ sections = {}
+ for heading_text in headings:
+ if "</h2>" not in heading_text:
+ sections["Introduction"] = clean_heading_text(heading_text)
+ continue
+ span = heading_text[: heading_text.index("</h2>")]
+ heading_title = BeautifulSoup(span, "html.parser").contents[0].contents[0]
+ text = heading_text[heading_text.index("</h2>") + 5 :]
+ if heading_title not in ["References", "See also", "External links", "Footnotes"]:
+ sections[heading_title] = clean_heading_text(text)
+
+ article_title = article.split("/")[-1]
+
+ print(f"Scraped {article_title} successfully. Headings: {sections.keys()}\n")
+ filename = f"{article_title.lower()}.jsonl"
+
+ with open(dirpath + filename, "w") as f:
+ for k, v in sections.items():
+ f.write(json.dumps({"title": k, "content": v}, ensure_ascii=False) + "\n")
+
+# Separate code to scrape human rights article, as it's in a different format.
+with open("human_rights.html", "r") as f:
+ html = f.read()
+
+soup = BeautifulSoup(html, "html.parser")
+content = soup.find("div", class_="migrated-content")
+md_content = md(str(content)).replace("\xa0", " ").replace("\u3000", " ")
+
+with open(dirpath + "human_rights_miskito.jsonl", "w") as f:
+ f.write(
+ json.dumps(
+ {"title": "Declaration of Human Rights in Miskito", "content": md_content},
+ ensure_ascii=False,
+ )
+ + "\n"
+ )

evals/elsuite/skill_acquisition/scraping/scrape_miskito.py

@@ -0,0 +1,135 @@
+# %%
+import json
+
+import bs4
+import requests
+from bs4 import BeautifulSoup
+from markdownify import markdownify as md
+
+# TODO: make sure italicised text is crawled properly and that hints are excluded from answers. 
+# TODO: Split any multi-part questions into individual questions.
+
+miskito_base_url = "https://en.wikibooks.org/wiki/Miskito/Lesson_{idx}"
+
+
+def process_practice_section_div(practice_div: bs4.element.Tag):
+ tds = practice_div.find_all("td")
+ instructions = (
+ md(str(tds[1]))
+ .replace("*", "")
+ .replace("|", "")
+ .strip()
+ .replace("What do these mean?", "Translate to English:")
+ .replace("What do these sentences mean?", "Translate to English:")
+ )
+ question_text = tds[2]
+ questions = question_text.find_all("li")
+ questions = [str(q.contents[0]) for q in questions]
+ answer_text = tds[3]
+ answers = answer_text.find_all("li")
+ answers = [str(a.contents[0]) for a in answers]
+ return instructions, questions, answers
+
+
+def extract_toc_sections(content: bs4.element.Tag):
+ toc = content.find_all("div", class_="toc")[0]
+ lis = toc.find_all("li", class_="toclevel-1")
+ lis = [li.find_all("span", class_="toctext")[0].contents[0] for li in lis]
+
+ lis = [md(str(li)).strip().replace("*", "") for li in lis]
+ return lis
+
+
+def process_miskito_page():
+ qa_pairs_by_lesson = {}
+ articles_without_qa_pairs = []
+ for idx in range(1, 11):
+ response = requests.get(miskito_base_url.format(idx=idx))
+ soup = BeautifulSoup(response.text, "html.parser")
+ content = soup.find("div", class_="mw-content-ltr mw-parser-output")
+
+ # Extract the question-answer pairs.
+ divs_with_specific_style = content.find_all(
+ "div", style=lambda value: value and "width:300px; float:right;" in value
+ )
+ lesson_qa_pairs = []
+ for i, div in enumerate(divs_with_specific_style):
+ if i == 0 and idx == 1: # First section of first lesson is not in the same format.
+ instructions = "Translate to English:"
+ questions = div.find_all("ul")[0].find_all("li")
+ questions = [str(q.contents[0]) for q in questions]
+ answers = div.find_all("ul")[1].find_all("li")
+ answers = [str(a.contents[0]) for a in answers]
+ lesson_qa_pairs += [
+ {"question": q, "answer": a, "instructions": instructions}
+ for q, a in zip(questions, answers)
+ ]
+ continue
+ instructions, questions, answers = process_practice_section_div(div)
+ for q, a in zip(questions, answers):
+ lesson_qa_pairs += [{"question": q, "answer": a, "instructions": instructions}]
+ qa_pairs_by_lesson[f"lesson_{idx}"] = lesson_qa_pairs
+
+ # Remove them from the page and store the page contents.
+ for div in divs_with_specific_style:
+ div.decompose()
+
+ articles_without_qa_pairs += [content]
+
+ return qa_pairs_by_lesson, articles_without_qa_pairs
+
+
+# %%
+# Write to file: all questions by lesson, and all questions in evallib format.
+qa_pairs_by_lesson, clean_articles = process_miskito_page()
+qa_by_lesson_file = "miskito_qa_pairs_by_lesson.jsonl"
+
+with open(qa_by_lesson_file, "w") as f:
+ for lesson, qa_pairs in qa_pairs_by_lesson.items():
+ f.write(json.dumps({"lesson": lesson, "qa_pairs": qa_pairs}) + "\n")
+
+miskito_qa = "miskito_qa.jsonl"
+with open(miskito_qa, "w") as f:
+ for lesson, qa_list in qa_pairs_by_lesson.items():
+ for qa_dict in qa_list:
+ instructions = qa_dict["instructions"][:-1] + ": "
+ f.write(
+ json.dumps(
+ {
+ "input": [{"role": "user", "content": instructions + qa_dict["question"]}],
+ "ideal": qa_dict["answer"],
+ },
+ ensure_ascii=False,
+ )
+ + "\n"
+ )
+# %%
+as_text = [str(a).split("<h2>")[1:] for a in clean_articles]
+sections_by_heading = {}
+for article in as_text:
+ for heading in article:
+ hsoup = BeautifulSoup(heading, "html.parser")
+ heading_name = (
+ md(str(hsoup.find("span", class_="mw-headline").contents[0])).replace("*", "").strip()
+ )
+ hsoup.find("span", class_="mw-editsection").decompose()
+ content = (
+ md(str(hsoup))
+ .strip()
+ .replace("*", "")
+ .replace("|", "")
+ .replace("What do they mean?", "")
+ .replace(" --- ", "")
+ .replace("\u2003", " ")
+ .replace(" ", " ")
+ )
+ content = content.split(" Study ")[1] if "Study " in content else content
+ sections_by_heading[heading_name] = content.strip()
+
+sections_by_heading
+# %%
+file = "lessons_no_exercises.jsonl"
+with open(file, "w") as f:
+ for heading, content in sections_by_heading.items():
+ f.write(json.dumps({"title": heading, "content": content}, ensure_ascii=False) + "\n")
+# %%