Zhuoyan Luo*, Yinghao Wu*, Yong Liu, Yicheng Xiao, Xiao-Ping Zhang, Yujiu Yang

Tsinghua University

• [2024/05/27] The code is coming soon.

The newly proposed Generalized Referring Expression Segmentation (GRES) amplifies the formulation of classic RES by involving multiple/non-target scenarios. Recent approaches focus on optimizing the last modality-fused feature which is directly utilized for segmentation and object-existence identification. However, the attempt to integrate all-grained information into a single joint representation is impractical in GRES due to the increased complexity of the spatial relationships among instances and deceptive text descriptions. Furthermore, the subsequent binary target justification across all referent scenarios fails to specify their inherent differences, leading to ambiguity in object understanding. To address the weakness, we propose a Hierarchical Semantic Decoding with Counting Assistance framework (HDC). It hierarchically transfers complementary modality information across granularities, and then aggregates each well-aligned semantic correspondence for multi-level decoding. Moreover, with complete semantic context modeling, we endow HDC with explicit counting capability to facilitate comprehensive object perception in multiple/single/non-target settings. Experimental results on gRefCOCO, Ref-ZOM, R-RefCOCO, and RefCOCO benchmarks demonstrate the effectiveness and rationality of HDC which outperforms the state-of-the-art GRES methods by a remarkable margin.

Code in this repository is built upon several public repositories. Thanks for the wonderful work ReLA! !

if you find it helpful, please cite

@article{luo2024hdc,
  title={HDC: Hierarchical Semantic Decoding with Counting Assistance for Generalized Referring Expression Segmentation},
  author={Luo, Zhuoyan and Wu, Yinghao and Liu, Yong and Xiao, Yicheng and Zhang, Xiao-Ping and Yang, Yujiu},
  journal={arXiv preprint arXiv:2405.15658},
  year={2024}
}