Skip to content

A list of ethics related resources for researchers and practitioners of Natural Language Processing and Computational Linguistics

Notifications You must be signed in to change notification settings

acl-org/ethics-reading-list

Repository files navigation

Ethics Union Bibliography

PRs Welcome GitHub last commit GitHub contributors

A list of ethics related resources for researchers and practitioners of Natural Language Processing and Computational Linguistics. This is a public list moderated by the current ACL Ethics Committee. Please issue a pull request against the repository to have your suggestions discussed before they are approved for integration with the list. Thanks!

This list is intentionally kept with simple formatting in Markdown to allow machine-readable processing of the resource.

Guidelines

  • Add your name to the contributors section as part of your PR. Include an affiliation and a weblink if you'd like.
  • References follow a two-tier organization: by year, then by first-author surname. #tag papers with topics so that they can be found on a per topic basis.
  • Use APA style where possible. Confine references to a single line.
  • Add minimally a paper link to direct readers directly to the .pdf or metadata page (ACL Anthology for example) of the paper.
  • Papers are organized by tags. We accept PRs to add or re-organize tags. Please help tag your own papers!
    • Tags for topics: starting with t
    • Tags for bibliographic type: starting with type
    • Simply copy the tags from the below #tags section to tag, ordering tags alphabetically and putting topic tags before type ones.
    • Tags are provided using the shields.io service
  • Prefer peer-reviewed conference or journal reference to link to ArXiv whenever possible.

Contributed by

(put in alpha order by surname)

Contents

Tags

We have tagged papers with several topic tags and bibliographic type. You can click on these images to get to per-topic or per-type filtered versions of this list (automatically produced on new pushes to the repository). These are indicative tags and not comprehensive. We accept pull requests to change them!

By Topic

General Resources Biases Crowdsourcing Issues Data Dual Use Environmental Impact Evaluation Language Diversity Model Issues Uncategorized

By Bibliographic Type

published preprint post report

2023

[Contents]

  • Kirk, H. R., Vidgen, B., Röttger, P., Thrush, T., and Hale, S. A. (2023). Hatemoji: A test suite and adversarially-generated dataset for benchmarking and detecting emoji-based hate. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. (NAACL '23') 10.18653/v1/2022.naacl-main.97 [paper] Biases Evaluation published
  • McMillan-Major, Angelina, Emily M. Bender and Batya Friedman. (2023). Data Statements: From Technical Concept to Community Practice, ACM Journal on Responsible Computing. [paper] Data published
  • Nejadgholi, I., Kiritchenko, S., Fraser, K.C., Balkir, E. (2023) Concept-Based Explanations to Test for False Causal Relationships Learned by Abusive Language Classifiers. In Proceedings of the 7th Workshop on Online Abuse and Harms (WOAH), pages 138–149, Toronto, Canada. Association for Computational Linguistics. [paper] Biases Model Issues published
  • Pyatkin, V., Yung, F., Scholman, M. C., Tsarfaty, R., Dagan, I., and Demberg, V. (2023). Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design. Transaction of Association for Computational Linguistics (TACL '23). [paper] Crowdsourcing Issues published

2022

[Contents]

  • Alorwu, A., Savage, S., van Berkel, N., Ustalov, D., Drutsa, A., Oppenlaender, J., Bates, O., Hettiachchi, D., Gadiraju, U., Goncalves, J., and Hosio, S. (2022). REGROW: Reimagining Global Crowdsourcing for Better Human-AI Collaboration. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA '22). Association for Computing Machinery, New York, NY, USA, Article 88, 1–7 [paper] Crowdsourcing Issues published

  • Balkir, E., Kiritchenko, S., Nejadgholi, I., Fraser, K.C. (2022) Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models. In Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022), pages 80–92, Seattle, U.S.A. Association for Computational Linguistics. [paper] Biases published

  • Balkir, E., Nejadgholi, I., Fraser, K.C., Kiritchenko, S. (2022). Necessity and Sufficiency for Explaining Text Classifiers: A Case Study in Hate Speech Detection. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2672–2686, Seattle, United States. Association for Computational Linguistics. [paper] Biases published

  • Chalkidis I., Pasini T., Zhang S., Tomada L., Schwemer S., and Søgaard A. (2022). FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4389–4406, Dublin, Ireland. Association for Computational Linguistics. [paper] Biases published

  • Fraser, K.C., Kiritchenko, S., Balkir, E. (2022) Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophy. In Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022), pages 26–42, Seattle, U.S.A. Association for Computational Linguistics. [paper] General Resources published

  • Fraser, K.C., Kiritchenko, S., Nejadgholi, I. (2022). Computational Modelling of Stereotype Content in Text. Frontiers in Artificial Intelligence, 5, 2022. doi:10.3389/frai.2022.826207. [paper] Biases published

  • Meade N., Poole-Dayan E., and Reddy S. (2022). An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1878–1898, Dublin, Ireland. Association for Computational Linguistics. [paper] Biases published

  • Meehan C., Mrini K., and Chaudhuri K. (2022). Sentence-level Privacy for Document Embeddings. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3367–3380, Dublin, Ireland. Association for Computational Linguistics. [paper] Uncategorized published

  • Mohammad S. (2022). Ethics Sheets for AI Tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8368–8379, Dublin, Ireland. Association for Computational Linguistics. [paper] General Resources published

  • Nejadgholi, I., Balkir, E., Fraser, K.C., Kiritchenko, S. (2022) Towards Procedural Fairness: Uncovering Biases in How a Toxic Language Classifier Uses Sentiment Information.In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 225–237, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. [paper] Biases Model Issues published

  • Névéol A., Dupont Y., Bezançon J., and Fort K..(2022). French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8521–8531, Dublin, Ireland. Association for Computational Linguistics. [paper] Biases published

  • Przybyła P., and Shardlow M. (2022). Using NLP to quantify the environmental cost and diversity benefits of in-person NLP conferences. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3853–3863, Dublin, Ireland. Association for Computational Linguistics. [paper] Environmental Impact published

2021

[Contents]

  • Abdalla, M. & Abdalla, M. (2021). The Grey Hoodie Project: Big Tobacco, Big Tech, and the Threat on Academic Integrity Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, Association for Computing Machinery, 2021, 287-297. [paper] General Resources published

  • Aka, O., Burke, K., Bäuerle, A., Greer, C., & Mitchell, M. (2021). Measuring Model Biases in the Absence of Ground Truth. DOI:10.1145/3461702.3462557. AIES '21: AAAI/ACM Conference on AI, Ethics, and Society. [paper] Biases published

  • Bannour, N., Ghannay, S., Névéol, A. and Ligozat, A.-L. 2021. Evaluating the carbon footprint of NLP methods: a survey and analysis of existing tools. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pages 11–21, Virtual. Association for Computational Linguistics. [paper] Environmental Impact published

  • Bender, Emily M., Friedman, B. and McMillan-Major, A. (2021). A Guide for Writing Data Statements for Natural Language Processing [paper] Data report

  • Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021, March). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?🦜. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 610-623). doi:10.1145/3442188.3445922 [paper] Model Issues Biases published

  • Birhane, A., Prabhu, V. U., & Kahembwe, E. (2021). Multimodal datasets: misogyny, pornography, and malignant stereotypes. arXiv preprint arXiv:2110.01963. [paper] Data preprint

  • Dodge, J., Sap, M., Marasovic, A., Agnew, W., Ilharco, G., Groeneveld, D., ... & Face, H. (2021, September). Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286–1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [paper] Data published

  • Field, A., Blodgett, S. L., Talat, Z., & Tsvetkov, Y. (2021, August). A Survey of Race, Racism, and Anti-Racism in NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1905–1925, Online. Association for Computational Linguistics. doi:10.18653/v1/2021.acl-long.149 [paper] Model Issues Biases published

  • Fraser K. C., Nejadgholi, I. and Kiritchenko, S. (2021). Understanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 600–616, Online. Association for Computational Linguistics. [paper] Biases published

  • Kiritchenko, S., Nejadgholi, I., and Fraser, K. C. (2021). Confronting Abusive Language Online: A Survey from the Ethical and Human Rights Perspective. Journal of Artificial Intelligence Research, 71: 431-478, July 2021. doi:10.1613/jair.1.12590. [paper] General Resources published

  • Kreutzer, J., Caswell, I., Wang, L., Wahab, A., van Esch, D., Ulzii-Orshikh, N., ... & Adeyemi, M. (2021). Quality at a glance: An audit of web-crawled multilingual datasets.Transactions of the Association for Computational Linguistics, The MIT Press, 2022, 10, pp.50-72. [paper] Crowdsourcing Issues published

  • Kummerfeld, J. K. (2021). Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 343–349, Online. Association for Computational Linguistics. [paper] Crowdsourcing Issues published

  • Lannelongue, L., Grealey, J., & Inouye, M. (2021). Green algorithms: Quantifying the carbon footprint of computation. Advanced Science, 2100707. doi:10.1002/advs.202100707. [paper] Environmental Impact published

  • Markl, N., & Lai, C. (2021, April). Context-sensitive evaluation of automatic speech recognition: considering user experience & . In Proceedings of the First Workshop on Bridging Human–Computer Interaction and Natural Language Processing (pp. 34-40). [paper] Language Diversity published

  • Moss, E., Watkins, E. A., Singh, R., Elish, M. C., & Metcalf, J. (2021). Assembling Accountability: Algorithmic Impact Assessment for the Public Interest. Available at SSRN 3877437. [paper] Data report

  • Shmueli, B., Fell, J., Ray, S., & Ku, L. W. (2021). Beyond fair pay: Ethical implications of NLP crowdsourcing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3758–3769, Online. Association for Computational Linguistics. [paper] Crowdsourcing Issues published

  • Tan, S., & Joty, S. (2021). Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglots. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. doi:10.18653/v1/2021.naacl-main.282 [paper Language Diversity published

  • Tan, S., Joty, S., Baxter, K., Taeihagh, A., Bennett, G. A., & Kan, M. Y. (2021). Reliability Testing for Natural Language Processing Systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4153–4169, Online. Association for Computational Linguistics. doi:10.18653/v1/2021.acl-long.321 [paper] Evaluation published

2020

[Contents]

  • Anthony, L. F. W., Kanding, B., & Selvan, R. (2020). Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. arXiv preprint arXiv:2007.03051. [paper] Environmental Impact preprint

  • Bird, S. (2020, December). Decolonising speech and language technology. In Proceedings of the 28th International Conference on Computational Linguistics (pp. 3504-3519). doi:10.18653/v1/2020.coling-main.313 [paper] Language Diversity Data published

  • Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020). Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454–5476, Online. Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.485. [paper] Biases published

  • Bonastre, J. F. (2020). 1990-2020: retours sur 30 ans d’échanges autour de l’identification de voix en milieu judiciaire. In 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). 2e atelier Éthique et TRaitemeNt Automatique des Langues (ETeRNAL) (pp. 38-47). ATALA; AFCP. [paper] Dual Use published

  • Caglayan, O., Madhyastha, P., & Specia, L. (2020). Curious case of language generation evaluation metrics: A cautionary tale. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2322–2328, Barcelona, Spain (Online). International Committee on Computational Linguistics. doi:10.18653/v1/2020.coling-main.210. [paper] Evaluation published

  • Ethayarajh, K., & Jurafsky, D. (2020, November). Utility is in the eye of the user: A critique of NLP leaderboards. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) doi:10.18653/v1/2020.emnlp-main.393. [paper] Evaluation published

  • Floridi, L., Chiriatti, M. GPT-3: Its Nature, Scope, Limits, and Consequences. Minds & Machines 30, 681–694 (2020). https://doi.org/10.1007/s11023-020-09548-1 [paper] Model Issues published

  • Garnerin, M., Rossato, S., & Besacier, L. (2020). Pratiques d’évaluation en ASR et biais de performance (Evaluation methodology in ASR and performance bias). In Actes de la 6e conférence conjointe Journées d'Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). 2e atelier Éthique et TRaitemeNt Automatique des Langues (ETeRNAL) (pp. 1-9). [paper] Evaluation published

  • Goldfarb-Tarrant, S., Marchant, R., Sanchez, R. M., Pandya, M., & Lopez, A. (2020). Intrinsic bias metrics do not correlate with application bias. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) [paper]. Evaluation published

  • Hagendorff, T. The Ethics of AI Ethics: An Evaluation of Guidelines Minds & Machines, 2020, 30, 99-120. [paper] General Resources published

  • Henderson, P., Hu, J., Romoff, J., Brunskill, E., Jurafsky, D., & Pineau, J. (2020). Towards the systematic reporting of the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248), 1-43. [paper] Environmental Impact published

  • Jo, E. S., & Gebru, T. (2020, January). Lessons from archives: Strategies for collecting sociocultural data in machine learning. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 306-316). [paper] Data published

  • Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2020, 6282-6293. doi:10.18653/v1/2020.acl-main.560 [paper] Data Language Diversity published

  • Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., ... & Goel, S. (2020). Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14), 7684-7689. [paper] Language Diversity published

  • Bogdan Kulynych, Rebekah Overdorf, Carmela Troncoso, and Seda Gürses. 2020. POTs: protective optimization technologies. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* '20). Association for Computing Machinery, New York, NY, USA, 177–188. DOI:https://doi.org/10.1145/3351095.3372853. [paper] General Resources published

  • Linzen, T. (2020, July). How can we accelerate progress towards human-like linguistic generalization?. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.465 [paper] Evaluation published

  • Mathur, N., Baldwin, T., & Cohn, T. (2020, July). Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.448 [paper] Evaluation published

  • Mohammad, S. M. (2020, July). Gender gap in natural language processing research: Disparities in authorship and citations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. doi:10.18653/v1/2020.acl-main.702 [paper] Biases published

  • Nangia, N., Vania, C., Bhalerao, R., & Bowman, S. R. (2020, November). CrowS-pairs: A challenge dataset for measuring social biases in masked language models. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). doi:10.18653/v1/2020.emnlp-main.154 [paper] Evaluation published

  • Nissim, M., van Noord, R., & van der Goot, R. (2020). Fair is better than sensational: Man is to doctor as woman is to doctor. Computational Linguistics, 46(2), 487-497. doi:10.1162/coli_a_00379 [paper] Biases published

  • Paullada, A., Raji, I. D., Bender, E. M., Denton, E., & Hanna, A. (2020). Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, Volume 2, Issue 11, 12 November 2021, Pages 100388. [paper] Data published

  • Schneider, J. M., Rehm, G., Montiel-Ponsoda, E., Doncel, V. R., Revenko, A., Karampatakis, S., ... & Maganza, F. (2020, May). Orchestrating NLP Services for the Legal Domain. In Proceedings of the 12th Language Resources and Evaluation Conference (pp. 2332-2340). [paper] Uncategorized published

  • Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green AI. Communications of the ACM, 63(12), 54-63. [paper] Environmental Impact published

  • Tan, S., Joty, S., Kan, M. Y., & Socher, R. (2020, July). It's Morphin'Time! Combating Linguistic Discrimination with Inflectional Perturbations. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [paper] Language Diversity published

  • Tan, S., Joty, S., Varshney, L. R., & Kan, M. Y. (2020, November). Mind your inflections! Improving NLP for non-standard Englishes with Base-Inflection Encoding. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [paper] Language Diversity published

  • Trebaol, M. J. T., Hartley, M.-A., & Ghadikolaei, H. S. (2020). A tool to quantify and report the carbon footprint of machine learning computations and communication in academia and healthcare. Infoscience EPFL: record 278189. [report] Environmental Impact report

  • Vidgen, B., & Derczynski, L. (2020). Directions in abusive language training data, a systematic review: Garbage in, garbage out. PloS one, 15(12), e0243300. [paper] Data published

2019

[Contents]

  • Bender, E. M. (2019). The # benderrule: On naming the languages we study and why it matters. The Gradient, 14. [paper] Language Diversity post

  • Bregeon, D., Antoine, J. Y., Villaneau, J., & Lefeuvre-Halftermeyer, A. (2019). Redonner du sens à l’accord interannotateurs: vers une interprétation des mesures d’accord en termes de reproductibilité de l’annotation. Traitement Automatique des Langues, 60(2), 23. [paper] Evaluation published

  • Garimella, A., Banea, C., Hovy, D., & Mihalcea, R. (2019, July). Women’s syntactic resilience and men’s grammatical luck: Gender-Bias in Part-of-Speech Tagging and Dependency Parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3493-3498). [paper] Biases published

  • Hara, K.; Adams, A.; Milland, K.; Savage, S.; Hanrahan, B. V.; Bigham, J. P. & Callison-Burch, C. Worker Demographics and Earnings on Amazon Mechanical Turk: An Exploratory Analysis Association for Computing Machinery, 2019, 1-6. [paper] Crowdsourcing Issues published

  • Huang, X., & Paul, M. (2019, June). Neural user factor adaptation for text classification: Learning to generalize across author demographics. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (* SEM 2019) (pp. 136-146). [paper] Language Diversity published

  • Kann, K., Cho, K., & Bowman, S. R. (2019). Towards realistic practices in low-resource natural language processing: the development set. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3342–3349, Hong Kong, China. Association for Computational Linguistics. doi:10.18653/v1/D19-1329 [paper] Data published

  • Lacoste A., Luccioni A., Schmidt V., & Dandres T. (2019). Quantifying the carbon emissions of machine learning. In Climate Change workshop, NeurIPS 2019. [paper] Environmental Impact published

  • Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., ... & Gebru, T. (2019, January). Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency (pp. 220-229). [paper] Data published

  • Monteiro, M. (2019). Ruined by design: How designers destroyed the world, and what we can do to fix it. Mule Design. General Resources published

  • Raji, I. D., & Yang, J. (2019). About ML: Annotation and benchmarking on understanding and transparency of machine learning lifecycles. arXiv preprint arXiv:1912.06166. [paper] Data preprint

  • Sap, M., Gabriel, S., Qin, L., Jurafsky, D., Smith, N. A., & Choi, Y. (2019). Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490, Online. Association for Computational Linguistics. [paper] Biases published

  • Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy. Association for Computational Linguistics. doi:10.18653/v1/P19-1355. [paper] Environmental Impact published

  • Zmigrod, R., Mielke, S. J., Wallach, H., & Cotterell, R. (2019). Counterfactual data augmentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1651–1661, Florence, Italy. Association for Computational Linguistics. [paper] Language Diversity published

2018

[Contents]

  • Bender, E. M., & Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6, 587-604 doi:10.1162/tacl_a_00041 [paper] Data published
  • Curry, A. C., & Rieser, V. (2018, June). # MeToo Alexa: How conversational systems respond to sexual harassment. In Proceedings of the second ACL workshop on ethics in natural language processing (pp. 7-14). [paper] Biases published

  • Fort, K., & Névéol, A. (2018, January). Présence et représentation des femmes dans le traitement automatique des langues en France. In Penser la Recherche en Informatique comme pouvant être Située, Multidisciplinaire Et Genrée (PRISME-G). [paper] Biases published

  • Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé III, H., & Crawford, K. (2018). Datasheets for datasets. Commun. ACM 64, 12 (December 2021), 86–92. DOI:https://doi.org/10.1145/3458723. [paper] Data published

  • Hara, K.; Adams, A.; Milland, K.; Savage, S.; Callison-Burch, C. & Bigham, J. P. A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk CHI 2018, 2018. [paper] Crowdsourcing Issues published

  • Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018). The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677. [paper] Data preprint

  • Kiritchenko S. and Mohammad S. 2018. Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 43–53, New Orleans, Louisiana. Association for Computational Linguistics. [paper] Biases published

  • Schluter, N. (2018). The glass ceiling in NLP. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 2793-2798). doi:10.18653/v1/D18-1301 [paper] Biases published

2017

[Contents]

  • Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183-186. Evaluation published

  • Jurgens, D., Tsvetkov, Y., & Jurafsky, D. (2017, July). Incorporating dialectal variability for socially equitable language identification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 51-57). [paper] Language Diversity published

  • Koolen, C. & van Cranenburgh, A. These are not the Stereotypes You are Looking For: Bias and Fairness in Authorial Gender Attribution. In Proceedings of the first ACL workshop on ethics in natural language processing (pp. 12-22). [paper] Biases published

  • Leidner, J. L. & Plachouras, V. Ethical by Design: Ethics Best Practices for Natural Language Processing. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, Association for Computational Linguistics, 2017, 30-40. [paper] General Resources published

  • Mieskes, M. (2017, April). A quantitative study of data in the NLP community. In Proceedings of the first ACL workshop on ethics in natural language processing (pp. 23-29). [paper] Data published

  • Parra Escartin, C.; Reijers, W.; Lynn, T.; Moorkens, J.; Way, A. & Liu, C.-H. Ethical Considerations in NLP Shared Tasks. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, Association for Computational Linguistics, 2017, 66-73. [paper] General Resources published

  • Rudinger, R., May, C., & Van Durme, B. (2017, April). Social bias in elicited natural language inferences. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing (pp. 74-79). [paper] Biases published

  • Šuster, S., Tulkens, S., & Daelemans, W. (2017). A short review of ethical challenges in clinical natural language processing. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing. [paper] General Resources published

  • Tatman, R. (2017, April). Gender and dialect bias in YouTube’s automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing (pp. 53-59). [paper] Language Diversity published

2016

[Contents]

  • Amblard, M. (2016). Pour un TAL responsable. Traitement Automatique des Langues, 57(2), 21-45. [paper] General Resources published
  • Cohen, K. B.; Fort, K.; Adda, G.; Zhou, S. & Farri, D. Ethical Issues in Corpus Linguistics And Annotation: Pay Per Hit Does Not Affect Effective Hourly Rate For Linguistic Resource Development On Amazon Mechanical Turk ETHics In Corpus collection, Annotation and Application workshop, 2016. [paper] Crowdsourcing Issues published

  • Fort, K., & Couillault, A. (2016, May). Yes, we care! results of the ethics and natural language processing surveys. In international Language Resources and Evaluation Conference (LREC) 2016. [paper] General Resources published

  • Hovy, D., & Spruit, S. L. (2016, August). The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 591-598) doi:10.18653/v1/P16-2096. [paper] General Resources published

  • Larson, J., Angwin, J., & Parris, T. (2016). Breaking the black box: How machines learn to be racist. ProPublica. [paper] Biases post

  • Lefeuvre-Halftermeyer, A., Govaere, V., Antoine, J. Y., Allegre, W., Pouplin, S., Departe, J. P., ... & Spagnulo, A. (2016). Typologie des risques pour une analyse éthique de l'impact des technologies du TAL. Traitement Automatique des Langues, 57(2), 47-71. [paper] General Resources published

  • Mathet, Y., & Widlöcher, A. (2016). Évaluation des annotations: ses principes et ses pièges. Traitement Automatique des Langues, 57(2), 73-98. [paper] Evaluation published

2015

[Contents]

  • Bretonnel Cohen, K.; Pestian, J. P. & Fort, K. Annotating suicide notes : ethical issues at a glance. In Proc. of ETeRNAL (Ethique et Traitement Automatique des Langues), June 2015, Caen, France. [paper] Data published

  • Ferraro, F., Mostafazadeh, N., Vanderwende, L., Devlin, J., Galley, M., & Mitchell, M. (2015). A survey of current datasets for vision and language research. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 207–213, Lisbon, Portugal. Association for Computational Linguistics. doi:10.18653/v1/D15-1021 [paper] General Resources published

  • Hovy, D., & Søgaard, A. (2015, July). Tagging performance correlates with author age. In Proceedings of the 53rd annual meeting of the Association for Computational Linguistics and the 7th international joint conference on natural language processing (volume 2: Short papers) (pp. 483-488). [paper] Language Diversity published

  • Jørgensen, A., Hovy, D., & Søgaard, A. (2015, July). Challenges of studying and processing dialects in social media. In Proceedings of the workshop on noisy user-generated text (pp. 9-18). [paper] Language Diversity published

  • Lefeuvre A., Antoine J-Y., Allegre W.. Ethique conséquentialiste et traitement automatique des langues : une typologie de facteurs de risques adaptée aux technologies langagières. Atelier Ethique et TRaitemeNt Automatique des Langues (ETeRNAL'2015), conférence TALN'2015, Jun 2015, Caen, France. pp.53-66. ⟨hal-01170630⟩ [paper] General Resources published

2014

[Contents]

  • Callison-Burch, C. (2014, September). Crowd-workers: Aggregating information across turkers to help them find higher paying work. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 2, No. 1). [paper] Crowdsourcing Issues published

  • Couillault, A., Fort, K., Adda, G., & De Mazancourt, H. (2014, May). Evaluating corpora documentation with regards to the ethics and big data charter. In International Conference on Language Resources and Evaluation (LREC). [paper] Data published

  • Fort K., Adda G., Sagot B., Mariani J., Couillault A.. Crowdsourcing for Language Resource Development: Criticisms About Amazon Mechanical Turk Overpowering Use. Vetulani, Zygmunt and Mariani, Joseph. Human Language Technology Challenges for Computer Science and Linguistics, 8387, Springer International Publishing, pp.303-314, 2014, Lecture Notes in Computer Science, 978-3-319-08957-7. [paper] Crowdsourcing Issues published

2013

[Contents]

  • Irani, L. C., & Silberman, M. S. (2013, April). Turkopticon: Interrupting worker invisibility in amazon mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 611-620). [paper] Crowdsourcing Issues published

2011

[Contents]

  • Bederson, B. B., & Quinn, A. J. (2011). Web workers unite! addressing challenges of online laborers. In CHI'11 Extended Abstracts on Human Factors in Computing Systems (pp. 97-106). Crowdsourcing Issues published

  • Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon Mechanical Turk: Gold mine or coal mine?. Computational Linguistics, 37(2), 413-420. doi:10.1162/COLI_a_00057 [paper] Crowdsourcing Issues published

  • Kenny, D. The ethics of machine translation. New Zealand Society of Translators and Interpreters Annual Conference 2011, 2011. [paper] General Resources published

2010

[Contents]

  • Adda, G. & Mariani, J. Language resources and Amazon Mechanical Turk: legal, ethical and other issues. Proceedings of Legal Issues for Sharing Language Resources workshop in International Conference on Language Resources and Evaluation (LREC), European Language Resources Association (ELRA), 2010. [paper] Crowdsourcing Issues published

  • Drugan, J. & Babych, B. Shared Resources, Shared Values? Ethical Implications of Sharing Translation Resources. Proceedings of the Second Joint EM+/CNGL Workshop: Bringing MT to the User: Research on Integrating MT in the Translation Industry, Association for Machine Translation in the Americas, 2010, 3-10. [paper] Data published

  • Snyder, J. (2010). Exploitation and sweatshop labor: Perspectives and issues. Business Ethics Quarterly, 20(2), 187-213. [paper] Crowdsourcing Issues published

2006

  • Kacmarcik, G., & Gamon, M. (2006, July). Obfuscating document stylometry to preserve author anonymity. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions (pp. 444-451). [paper] Dual Use published

About

A list of ethics related resources for researchers and practitioners of Natural Language Processing and Computational Linguistics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published