Skip to content

amandacurry/convabuse

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ConvAbuse

Data from the paper ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI by Amanda Cercas Curry, Gavin Abercrombie, and Verena Rieser.

Link to paper

Please cite as:

@inproceedings{cercas-curry-etal-2021-convabuse,
title = "{C}onv{A}buse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational {AI}",
author = "Cercas Curry, Amanda and
Abercrombie, Gavin and
Rieser, Verena",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2021",
address = "Online and Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2021.emnlp-main.587",
doi = "10.18653/v1/2021.emnlp-main.587",
pages = "7388--7403"
}

File descriptions

We provide two versions of the dataset:

  1. ConvAbuseEMNLPfull.csv: The full dataset, including repeat annotations used to calculate intra-annotator agreement. Each row represents the response of an individual annotator
  2. ConvAbuseEMNLPtrain.csv, ConvAbuseEMNLPvalid.csv, ConvAbuseEMNLPtest.csv: The train, validation, and test splits used for the experiments in the paper. Each row includes the annotations of all the annotators who labelled that example.

All annotations are presented in binary format (1/0). Annotators only labelled secondary tasks (e.g. abuse type, target etc.) in cases they had considered to be abusive (i.e. -1/-2/-3).

The columns are:

Column Column header Explanation
0. example_no
1. annotator_id Annotator ID
2. conv_id Conversation ID
3. prev_agent Agent's previous utterance
4. prev_user User's previous utterance
5. agent Agent utterance
6. user User (target) utterance
7. bot Agent name (CarbonBot/Eliza)
8. is_abuse.1 Not abusive
9. is_abuse.0 Ambiguous
10. is_abuse.-1 Mildly abusive
11. is_abuse.-2 Strongly abusive
12. is_abuse.-3 Very strongly abusive
13. type.ableism Type: Ableism
14. type.homophobic Type: Homophobic
15. type.intellectual Type: Intellectual
16. type.racist Type: Racist
17. type.sexism Type: Sexist
18. type.sex_harassment Type: Sexual harassment
19. type.transphobic Type: Transphobic
20. target.generalised Target: General
21. target.individual Target: Individual
22. target.system Target:system/agent
23. directness.explicit Directness: Explicit
24. directness.implicit Directness: Implicit

In (2), the dataset is divided into train, vailidation, and test splits. (2) contains the same fields as (1), but each row includes all the annotators responses, with the annotator response column headers preceded by the annotator ID, e.g. Annotator1.is_abuse.-1.

Column Column header
0. example_id
1. conv_id
2. prev_agent
3. prev_user
4. agent
5. user
6. bot
7. Annotator1_is_abuse.1
8. Annotator1_is_abuse.0
9. Annotator1_is_abuse.-1
10. Annotator1_is_abuse.-2
11. Annotator1_is_abuse.-3
12. Annotator1_type.ableist
13. Annotator1_type.homophobic
14. Annotator1_intellectual
15. Annotator1_racist
16. Annotator1_sexism
17. Annotator1_sex_harassment
18. Annotator1_transphobic
19. Annotator1_target.generalised
20. Annotator1_target.individual
21. Annotator1_target.system
22. Annotator1_explicit
23. Annotator1_implicit
24. Annotator2_is_abuse.1
... ...
142. Annotator8_implicit

Note that for privacy reasons, we provide data from CarbonBot and E.L.I.Z.A only. We are unable to release the examples from Alana used in the paper.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published