This is the official PyTorch implementation for the paper: "Perception and Semantic Aware Regularization for Sequential Confidence Calibration". For more details, please refer to our paper.

• This work was tested with PyTorch 1.9.0, CUDA 11.1, python 3.8
  
• Requirements : fastai==1.0.60 lmdb pillow opencv-python tensorboardX editdistance nltk natsort numpy

We use lmdb dataset for training and evaluation. The training datasets can be downloaded in clovaai and the ensemble of evalutaion datasets can be downloaded here.

• Training datasets
  • MJSynth (MJ)
  • SynthText (ST)
• Evaluation datasets
  • Regular datasets
    • IIIT5K (IIIT)
    • Street View Text (SVT)
    • ICDAR2013 (IC13)
    • ICDAR2003 (IC03)
  • Irregular dataset
    • ICDAR2015 (IC15)
    • Street View Text Perspective (SVTP)
    • CUTE80 (CUTE)
• Tree structure of data_lmdb_release directory

  data_lmdb_release
  ├── charset_36.txt
  ├── charset_94.txt
  ├── test
  └── training
      ├── MJ
      │   ├── MJ_test
      │   ├── MJ_train
      │   └── MJ_valid
      └── ST
  

Perception Similarity Sequence

bash get_perc_sim_seq.sh

Semantic Correlation Sequence

CUDA_VISIBLE_DEVICES=0 python abinet/get_sem_cor_seq.py --checkpoint /path/to/checkpoint

• --checkpoint: path to pretrained model of BCN in ABINet.

CUDA_VISIBLE_DEVICES=0 python train.py \
--train_data data_lmdb_release/training --valid_data data_lmdb_release/test \
--select_data MJ-ST --batch_ratio 0.5-0.5 --lr 0.1 \
--Transformation TPS --FeatureExtraction ResNet --SequenceModeling BiLSTM --Prediction Attn \
--saved_model /path/to/checkpoint --calibrator PSSR --alpha 0.05 --with_vis

• --saved_model: path to model to be calibrated.
• --calibrator: select calibration method.
• --alpha: weight factor for balancing in the Eq. (2) in our paper.
  Note: Detailed argument settings can be found in clovaai.

CUDA_VISIBLE_DEVICES=0 python test.py \
--eval_data data_lmdb_release/test \
--Transformation TPS --FeatureExtraction ResNet --SequenceModeling BiLSTM --Prediction Attn \
--saved_model pretrained_models/TPS-ResNet-BiLSTM-Attn.pth

• --saved_model: path to calibrated model.

Download calibrated model by PSSR from this link.

This implementation has been based on clovaai and ABINet.

If you find our work useful in your research, please cite the following:

@inproceedings{peng2023perception,
  title={Perception and Semantic Aware Regularization for Sequential Confidence Calibration},
  author={Peng, Zhenghua and Luo, Yu and Chen, Tianshui and Xu, Keke and Huang, Shuangping},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={10658--10668},
  year={2023}
}