Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jiangjin-add-evaluation-part.v2 #2

Merged
merged 1 commit into from
Dec 17, 2023
Merged

jiangjin-add-evaluation-part.v2 #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,14 +100,31 @@ The dataset is saved as a list of dicts, each of which represents a data instanc


## Evaluation
We evaluate the model's ability to edit temporal knowledge in terms of two dimensions: the time at which the model is questioned (historical, current) and the form of the time at which it is questioned (relative, explicit).

| | Historical | Current |
| -------- | --------------------------------------------- | ------------------------------------------ |
| Relative | Historical Relative time Question Score (**HRS**) | Current Relative time Question Score (**CRS**) |
| Explicit | Historical Explicit time Question Score (**HES**) | Current Explicit time Question Score (**CES**) |

In addition, we eliminate the effect of question description by paraphrasing the question. We measure this as the **Paraphrase Question Score**.

Examples are as follows:
- time:
- historical: "Billy Wrigglesworth used to play for"
- current: "Billy Wrigglesworth's team is"
- time form:
- relative: "Billy Wrigglesworth's team is"
- explicit: "From 1947 to 1948, Billy Wrigglesworth is a player of"
- paraphrase: "From 1948 to 1953, Billy Wrigglesworth plays for"


## METO



## Issues or Questions?
If you come across any issues while using the datasets or have any questions regarding the repository or the paper, please don't hesitate to reach out. You can contact Xunjian Yin at ([email protected]) or create an issue.

## Citation
If you use our code in your research, please cite our work:
Expand Down