Skip to content
This repository has been archived by the owner on Oct 20, 2022. It is now read-only.

Docs V0.10 #164

Merged
merged 7 commits into from
Sep 23, 2021
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Change order of sections
  • Loading branch information
brandenchan committed Sep 21, 2021
commit 81876a4efe4ba8794231604ba0c2d9351881f6b6
26 changes: 13 additions & 13 deletions docs/latest/guides/evaluation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,6 @@ To get started using Haystack for evaluation, we recommend having a look at our

</div>

## Datasets

Annotated datasets are crucial for evaluating the retrieval as well as the question answering capabilities of your system.
Haystack is designed to work with question answering datasets that follow SQuAD format.
Please check out our [annotation tool](/guides/annotation) if you're interested in creating your own dataset.

<div className="max-w-xl bg-yellow-light-theme border-l-8 border-yellow-dark-theme px-6 pt-6 pb-4 my-4 rounded-md dark:bg-yellow-900">

**Data Tool:** have a look at our `SquadData` object in `haystack/squad_data.py` if you'd like to manipulate SQuAD style data using Pandas dataframes.

</div>

## Open vs Closed Domain

There are two evaluation modes known as **open domain** and **closed domain.**
Expand Down Expand Up @@ -89,4 +77,16 @@ While F1 and EM would both score “one hundred percent” as sharing zero simil

SAS is particularly useful to seek out cases where F1 doesn't give a good indication of the validity of a predicted answer.

You can read more about SAS in [this paper](https://arxiv.org/abs/2108.06130).
You can read more about SAS in [this paper](https://arxiv.org/abs/2108.06130).

## Datasets

Annotated datasets are crucial for evaluating the retrieval as well as the question answering capabilities of your system.
Haystack is designed to work with question answering datasets that follow SQuAD format.
Please check out our [annotation tool](/guides/annotation) if you're interested in creating your own dataset.

<div className="max-w-xl bg-yellow-light-theme border-l-8 border-yellow-dark-theme px-6 pt-6 pb-4 my-4 rounded-md dark:bg-yellow-900">

**Data Tool:** have a look at our `SquadData` object in `haystack/squad_data.py` if you'd like to manipulate SQuAD style data using Pandas dataframes.

</div>