This repository includes the official implementation of SAPA, a universal feature upsampling operator, presented in our paper:

SAPA: Similarity-Aware Point Affiliation for Feature Upsampling

Proc. Annual Conference on Neural Information Processing Systems (NeurIPS 2022) Spotlight

Hao Lu, Wenze Liu, Zixuan Ye, Hongtao Fu, Yuliang Liu, Zhiguo Cao

Huazhong University of Science and Technology, China

• 17 July 2023: The code of the extended version On Point Affiliation in Feature Upsampling is available here.
• 11 Nov 2022: Our paper is picked as Spotlight presentation (5%)!
• 27 Sep 2022: Code is released, and camera ready is avaliable on arXiv!

• Simple and effective: SAPA defines a novel class of similarity-aware upsampling kernels, which can simultaenously encourage semantic smoothness and boundrary sharpness;
• Generic: SAPA can benefit a number of dense prediction tasks. In our paper, we validate semantic segmentation, object detection, depth estimation, and image matting;
• Plug-and-play: SAPA is applicable to any encoder-decoder architectures or feature pyramid networks;
• Lightweight: SAPA only introduces negligible extra computational overhead and number of parameters.
• Memory-efficient: We have provided a CUDA implementation to reduce memory cost and to improve efficiency.

Our codes are tested on Python 3.8.8 and PyTorch 1.9.0.

cd sapa
python setup.py develop

Here is a summary of our reported results on different dense prediction tasks:

-Notes: i) For SegFormer, we use their official code rather than the version in mmsegmentation, and conduct experiments on a single A6000 GPU. ii) We observe that MaskFormer and Mask2Former are somewhat unstable. Due to limited resources, we only report one-run results.

If you find this work or code useful for your research, please cite:

@inproceedings{lu2022sapa,
  title={SAPA: Similarity-Aware Point Affiliation for Feature Upsampling},
  author={Lu, Hao and Liu, Wenze and Ye, Zixuan and Fu, Hongtao and Liu, Yuliang and Cao, Zhiguo},
  booktitle={Proc. Annual Conference on Neural Information Processing Systems (NeurIPS)},
  year={2022}
}

This code is for academic purposes only. Contact: Hao Lu ([email protected])