Skip to content

Code for our EMNLP 2020 Paper "AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network"

License

Notifications You must be signed in to change notification settings

Alibaba-NLP/AIN

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network

The code is mainly for our EMNLP 2020 paper: AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network. The code is mainly based on MultilangStructureKD with a lot of modifications.

AINs approximate the CRF inference steps. Instead of Forward-backward algorithm and Viterbi algorithm that requires sequential compuation over the input sequence, we use Mean-Field Variational Inference algorithm to approximate the CRF inference which can be paralleled over the sequence. In the paper, we show that AIN on the first-order CRF is about 10.2 and 4.4 times faster than the traditional CRF approach in training and prediction respectively with long sentences. Moreover, AINs achieves competitive accuracy with the tranditional CRF approach. Please refer to the paper for more details.

Note that the speed comparison in the paper is based on the Viterbi decoder of NCRF++ for fairness, which can be parallelized over batch. However, the Viterbi decoder in this code is from Flair. As a result, the prediction speed improvement in this code maybe higher than the reported value in the paper.


Performance

Here is a comparison of speed and accuracy of CRF and AINs on CoNLL 2003 English NER dataset. Following NCRF++, we report max test F1 score (we report averaged F1 score in the paper). For the speed comparison, we run on a single Nvidia GTX-1080 GPU with a batch size of 32.

MODEL F1 Speed (Sents/s)
Word + CharCNN + Softmax 91.11 5783
Word + CharCNN + CRF 91.32 297
Word + CharCNN + CRF (NCRF++) 91.35 1900~
Word + CharCNN + AIN-1O 91.34 4278
Word + CharCNN + AIN-2O 91.41 3697

Requirements

The project is based on PyTorch 1.1+ and Python 3.6+.

pip install -r requirements.txt

Training

To train AIN on the first-order CRF, run:

python -u train.py --config config/3iter_word_char_charcnn_300epoch_32batch_0.1lr_1window_256hidden_en_lample_monolingual_mfvi_sentloss_10patience_baseline_fast_2nd_startend_sentbatch_norelearn_nodev_ner35.yaml

To train AIN on the factorized second-order CRF, run:

python -u train.py --config config/3iter_word_char_charcnn_300epoch_32batch_0.1lr_2window_256hidden_en_lample_monolingual_mfvi_sentloss_10patience_baseline_fast_2nd_startend_sentbatch_norelearn_nodev_ner35.yaml

Reference

AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network

Contact

Please email your questions or comments to Xinyu Wang.

About

Code for our EMNLP 2020 Paper "AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network"

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages