Must-read Papers on knowledge-enhanced pretrained language models.

• senseBERT

paper: [1908.05646] SenseBERT: Driving Some Sense into BERT (arxiv.org)

code: None

• SentiLARE

paper: [1911.02493] SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge (arxiv.org)

code: https://github.com/thu-coai/SentiLARE

• LIBERT

paper: [1909.02339] Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (arxiv.org)

code: https://github.com/anlausch/LIBERT

• KnowBERT

paper: [1909.04164] Knowledge Enhanced Contextual Word Representations (arxiv.org)

code: https://github.com/allenai/kb

• BERT-MK

paper: BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models - ACL Anthology

code: None

• KT-NET

paper: Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension (aclanthology.org)

code: None

• KGLM

paper: [1906.07241] Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling (arxiv.org)

code: https://github.com/rloganiv/kglm-model

• Guan et al(2020)

paper: A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation - ACL Anthology

code: https://github.com/thu-coai/CommonsenseStoryGen

• K-BERT

paper: [1909.07606] K-BERT: Enabling Language Representation with Knowledge Graph (arxiv.org)

code: https://github.com/autoliuweijie/K-BERT (not official)

• CoLAKE

paper: [2010.00309] CoLAKE: Contextualized Language and Knowledge Embedding (arxiv.org)

code: https://github.com/txsun1997/CoLAKE

• SpanBERT

paper: https://arxiv.org/abs/1907.10529

code: https://github.com/facebookresearch/SpanBERT

• KEPLER

paper: [1911.06136] KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (arxiv.org)

code: https://github.com/THU-KEG/KEPLER

• FaE

paper: [2007.00849] Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge (arxiv.org)

code: None

• K-ADAPTER

paper: [2002.01808] K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (arxiv.org)

code: https://github.com/microsoft/K-Adapter

• REALM

paper: gltpc.2020.pdf (kentonl.com)

code: None

• Synatx-BERT

paper: [[2103.04350v1] Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees (arxiv.org)](https://arxiv.org/abs/2103.04350v1#:~:text=However%2C how to incorporate the syntax trees effectively,an arbitrary pre-trained checkpoint based on Transformer architecture.)

code: https://github.com/nkh2235/SyntaxBERT

• WKLM

paper: [1912.09637] Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model (arxiv.org)

code: None

• kgPLM

paper: [2012.03551] KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative Learning (arxiv.org)

code: None

• LUKE

paper: [2010.01057] LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (arxiv.org)

code: https://github.com/studio-ousia/luke

• ERICA

paper: [2012.15022v1] ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (arxiv.org)

code: https://github.com/thunlp/ERICA

• GRF

paper: [2009.11692v1] Language Generation with Multi-Hop Reasoning on Commonsense Knowledge Graph (arxiv.org)

code: https://github.com/cdjhz/multigen

• KG-BART

paper: [2009.12677v2] KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoning (arxiv.org)

code: https://github.com/yeliu918/KG-BART

• COMET

paper: [1906.05317v2] COMET: Commonsense Transformers for Automatic Knowledge Graph Construction (arxiv.org)

code1: atcbosselut/comet-commonsense

code2: Saner3/pytorch-transformers-comet

• SKEP

paper: [2005.05635] SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis (arxiv.org)

code: https://github.com/baidu/Senta

• Ecommerce-BERT

paper: [2009.02835] E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce (arxiv.org)

code: None

• RAG

paper: [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arxiv.org)

code: https://github.com/huggingface/transformers ??

• KIF

paper: [2004.12744] Augmenting Transformers with KNN-Based Composite Memory for Dialogue (arxiv.org)

code: None

• KALM

paper: [2007.00655] Knowledge-Aware Language Model Pretraining (arxiv.org)

code: None

• LRLM

paper: [[1908.07690] Latent Relation Language Models (arxiv.org)](https://arxiv.org/abs/1908.07690#:~:text=In this paper%2C we propose Latent Relation Language,able to annotate the posterior probability of )

code: None

• EaE

paper: [[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision (arxiv.org)](https://arxiv.org/abs/2004.07202#:~:text=We introduce a new model - Entities as,EAE's entity representations are learned directly from text.)

code: erolm-a/colla-framework: Conversational Framework for Linguistic Agents (github.com)

• AMS

paper: [1908.06725] Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models (arxiv.org)

code: None

• GLM

paper: [2004.14224] Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (arxiv.org)

code: None

• SMedBERT

paper: [2108.08983] SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining (arxiv.org)

code: https://github.com/matnlp/smedbert

• KG-BERT

paper: [1909.03193] KG-BERT: BERT for Knowledge Graph Completion (arxiv.org)

code: https://github.com/yao8839836/kg-bert

• ERNIE

paper: [1904.09223] ERNIE: Enhanced Representation through Knowledge Integration (arxiv.org)

code: https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/ernie-1.0

• ERINE(THU)

paper: [1905.07129] ERNIE: Enhanced Language Representation with Informative Entities (arxiv.org)

code: https://github.com/thunlp/ERNIE

• ERINE2.0

paper: https://arxiv.org/abs/1907.12412

code: PaddlePaddle/ERNIE: Official implementations for various pre-training models of ERNIE-family, covering topics of Language Understanding & Generation, Multimodal Understanding & Generation, and beyond. (github.com)

• ERINE3.0

paper: [2107.02137] ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (arxiv.org)

demo: HERE

• A Survey of Knowledge Enhanced Pre-trained Models [paper]

• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey [paper]