News | Abstract | Usage | Results | Statement

2023.07.04

• The paper is post on arxiv!

This is the official repository of the paper RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation

In this study, we present the RefSAM model, which for the first time explores the potential of SAM for RVOS by incorporating multi-view information from diverse modalities and successive frames at different timestamps. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-Modal MLP that projects the text embedding of the referring expression into sparse and dense embeddings, serving as user-interactive prompts. Subsequently, a parameter-efficient tuning strategy is employed to effectively align and fuse the language and vision features. Through comprehensive ablation studies, we demonstrate the practical and effective design choices of our strategy. Extensive experiments conducted on Ref-Youtu-VOS and Ref-DAVIS17 datasets validate the superiority and effectiveness of our RefSAM model over existing methods.

The code will be released soon.

A comprehensive comparison between RefSAM and existing methods.

We show the visualization results of our RefSAM model. It can be seen that RefSAM is capable of effectively segmenting and tracking the referred object even in challenging scenarios, such as variations in person poses, and occlusions between instances.

Furthermore, we present the results of differnt models. It is clear that our RefSAM demonstrates significantly enhanced cross-modal understanding capability.

This project is for research purpose only. For any other questions please contact [email protected].