Must-read Papers on knowledge-enhanced pretrained language models.
- senseBERT
paper: [1908.05646] SenseBERT: Driving Some Sense into BERT (arxiv.org)
code: None
- SentiLARE
code: https://github.com/thu-coai/SentiLARE
- LIBERT
code: https://github.com/anlausch/LIBERT
- KnowBERT
paper: [1909.04164] Knowledge Enhanced Contextual Word Representations (arxiv.org)
code: https://github.com/allenai/kb
- BERT-MK
paper: BERT-MK: Integrating Graph Contextualized Knowledge into Pre-trained Language Models - ACL Anthology
code: None
- KT-NET
code: None
- KGLM
code: https://github.com/rloganiv/kglm-model
- Guan et al(2020)
paper: A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation - ACL Anthology
code: https://github.com/thu-coai/CommonsenseStoryGen
- K-BERT
paper: [1909.07606] K-BERT: Enabling Language Representation with Knowledge Graph (arxiv.org)
code: https://github.com/autoliuweijie/K-BERT (not official)
- CoLAKE
paper: [2010.00309] CoLAKE: Contextualized Language and Knowledge Embedding (arxiv.org)
code: https://github.com/txsun1997/CoLAKE
- SpanBERT
paper: https://arxiv.org/abs/1907.10529
code: https://github.com/facebookresearch/SpanBERT
- KEPLER
code: https://github.com/THU-KEG/KEPLER
- FaE
code: None
- K-ADAPTER
paper: [2002.01808] K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (arxiv.org)
code: https://github.com/microsoft/K-Adapter
- REALM
paper: gltpc.2020.pdf (kentonl.com)
code: None
- Synatx-BERT
paper: [[2103.04350v1] Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees (arxiv.org)](https://arxiv.org/abs/2103.04350v1#:~:text=However%2C how to incorporate the syntax trees effectively,an arbitrary pre-trained checkpoint based on Transformer architecture.)
code: https://github.com/nkh2235/SyntaxBERT
- WKLM
code: None
- kgPLM
code: None
- LUKE
code: https://github.com/studio-ousia/luke
- ERICA
code: https://github.com/thunlp/ERICA
- GRF
code: https://github.com/cdjhz/multigen
- KG-BART
code: https://github.com/yeliu918/KG-BART
- COMET
paper: [1906.05317v2] COMET: Commonsense Transformers for Automatic Knowledge Graph Construction (arxiv.org)
code1: atcbosselut/comet-commonsense
code2: Saner3/pytorch-transformers-comet
- SKEP
paper: [2005.05635] SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis (arxiv.org)
code: https://github.com/baidu/Senta
- Ecommerce-BERT
code: None
- RAG
paper: [2005.11401] Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arxiv.org)
code: https://github.com/huggingface/transformers ??
- KIF
paper: [2004.12744] Augmenting Transformers with KNN-Based Composite Memory for Dialogue (arxiv.org)
code: None
- KALM
paper: [2007.00655] Knowledge-Aware Language Model Pretraining (arxiv.org)
code: None
- LRLM
paper: [[1908.07690] Latent Relation Language Models (arxiv.org)](https://arxiv.org/abs/1908.07690#:~:text=In this paper%2C we propose Latent Relation Language,able to annotate the posterior probability of )
code: None
- EaE
paper: [[2004.07202] Entities as Experts: Sparse Memory Access with Entity Supervision (arxiv.org)](https://arxiv.org/abs/2004.07202#:~:text=We introduce a new model - Entities as,EAE's entity representations are learned directly from text.)
code: erolm-a/colla-framework: Conversational Framework for Linguistic Agents (github.com)
- AMS
code: None
- GLM
code: None
- SMedBERT
code: https://github.com/matnlp/smedbert
- KG-BERT
paper: [1909.03193] KG-BERT: BERT for Knowledge Graph Completion (arxiv.org)
code: https://github.com/yao8839836/kg-bert
- ERNIE
paper: [1904.09223] ERNIE: Enhanced Representation through Knowledge Integration (arxiv.org)
code: https://github.com/PaddlePaddle/PaddleNLP/tree/develop/examples/language_model/ernie-1.0
- ERINE(THU)
paper: [1905.07129] ERNIE: Enhanced Language Representation with Informative Entities (arxiv.org)
code: https://github.com/thunlp/ERNIE
- ERINE2.0
paper: https://arxiv.org/abs/1907.12412
- ERINE3.0
demo: HERE