Skip to content

Latest commit

 

History

History
77 lines (59 loc) · 8.24 KB

named_entity_recognition.md

File metadata and controls

77 lines (59 loc) · 8.24 KB

Named entity recognition

Named entity recognition (NER) is the task of tagging entities in text with their corresponding type. Approaches typically use BIO notation, which differentiates the beginning (B) and the inside (I) of entities. O is used for non-entity tokens.

Example:

Mark Watney visited Mars
B-PER I-PER O B-LOC

CoNLL 2003 (English)

The CoNLL 2003 NER task consists of newswire text from the Reuters RCV1 corpus tagged with four different entity types (PER, LOC, ORG, MISC). Models are evaluated based on span-based F1 on the test set. ♦ used both the train and development splits for training.

Model F1 Paper / Source Code
CNN Large + fine-tune (Baevski et al., 2019) 93.5 Cloze-driven Pretraining of Self-attention Networks
Flair embeddings (Akbik et al., 2018)♦ 93.09 Contextual String Embeddings for Sequence Labeling Flair framework
BERT Large (Devlin et al., 2018) 92.8 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
CVT + Multi-Task (Clark et al., 2018) 92.61 Semi-Supervised Sequence Modeling with Cross-View Training Official
BERT Base (Devlin et al., 2018) 92.4 BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
BiLSTM-CRF+ELMo (Peters et al., 2018) 92.22 Deep contextualized word representations AllenNLP Project AllenNLP GitHub
Peters et al. (2017) ♦ 91.93 Semi-supervised sequence tagging with bidirectional language models
CRF + AutoEncoder (Wu et al., 2018) 91.87 Evaluating the Utility of Hand-crafted Features in Sequence Labelling Official
Bi-LSTM-CRF + Lexical Features (Ghaddar and Langlais 2018) 91.73 Robust Lexical Features for Improved Neural Network Named-Entity Recognition Official
Chiu and Nichols (2016) ♦ 91.62 Named entity recognition with bidirectional LSTM-CNNs
HSCRF (Ye and Ling, 2018) 91.38 Hybrid semi-Markov CRF for Neural Sequence Labeling HSCRF
IXA pipes (Agerri and Rigau 2016) 91.36 Robust multilingual Named Entity Recognition with shallow semi-supervised features Official
NCRF++ (Yang and Zhang, 2018) 91.35 NCRF++: An Open-source Neural Sequence Labeling Toolkit NCRF++
LM-LSTM-CRF (Liu et al., 2018) 91.24 Empowering Character-aware Sequence Labeling with Task-Aware Neural Language Model LM-LSTM-CRF
Yang et al. (2017) ♦ 91.26 Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
Ma and Hovy (2016) 91.21 End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
LSTM-CRF (Lample et al., 2016) 90.94 Neural Architectures for Named Entity Recognition

Long-tail emerging entities

The WNUT 2017 Emerging Entities task operates over a wide range of English text and focuses on generalisation beyond memorisation in high-variance environments. Scores are given both over entity chunk instances, and unique entity surface forms, to normalise the biasing impact of entities that occur frequently.

Feature Train Dev Test
Posts 3,395 1,009 1,287
Tokens 62,729 15,733 23,394
NE tokens 3,160 1,250 1,589

The data is annotated for six classes - person, location, group, creative work, product and corporation.

Links: WNUT 2017 Emerging Entity task page (including direct download links for data and scoring script)

Model F1 F1 (surface form) Paper / Source
Flair embeddings (Akbik et al., 2018) 49.59 Pooled Contextualized Embeddings for Named Entity Recognition / Flair framework
Aguilar et al. (2018) 45.55 Modeling Noisiness to Recognize Named Entities using Multitask Neural Networks on Social Media
SpinningBytes 40.78 39.33 Transfer Learning and Sentence Level Features for Named Entity Recognition on Tweets

Ontonotes v5 (English)

The Ontonotes corpus v5 is a richly annotated corpus with several layers of annotation, including named entities, coreference, part of speech, word sense, propositions, and syntactic parse trees. These annotations are over a large number of tokens, a broad cross-section of domains, and 3 languages (English, Arabic, and Chinese). The NER dataset (of interest here) includes 18 tags, consisting of 11 types (PERSON, ORGANIZATION, etc) and 7 values (DATE, PERCENT, etc), and contains 2 million tokens. The common datasplit used in NER is defined in Pradhan et al 2013 and can be found here.

Model F1 Paper / Source Code
Flair embeddings (Akbik et al., 2018) 89.71 Contextual String Embeddings for Sequence Labeling Official
CVT + Multi-Task (Clark et al., 2018) 88.81 Semi-Supervised Sequence Modeling with Cross-View Training Official
Bi-LSTM-CRF + Lexical Features (Ghaddar and Langlais 2018) 87.95 Robust Lexical Features for Improved Neural Network Named-Entity Recognition Official
BiLSTM-CRF (Strubell et al, 2017) 86.99 Fast and Accurate Entity Recognition with Iterated Dilated Convolutions Official
Iterated Dilated CNN (Strubell et al, 2017) 86.84 Fast and Accurate Entity Recognition with Iterated Dilated Convolutions Official
Chiu and Nichols (2016) 86.28 Named entity recognition with bidirectional LSTM-CNNs
Joint Model (Durrett and Klein 2014) 84.04 A Joint Model for Entity Analysis: Coreference, Typing, and Linking
Averaged Perceptron (Ratinov and Roth 2009) 83.45 Design Challenges and Misconceptions in Named Entity Recognition (These scores reported in (Durrett and Klein 2014)) Official

Go back to the README