Qi Yang, Xing Nie, Tong Li, Pengfei Gao, Ying Guo, Cheng Zhen, Pengfei Yan and Shiming Xiang

This repository provides the PyTorch implementation for the paper "Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation" accepted by CVPR 2024 (Highlight).

• (2024. 4.06) Our paper(COMBO) is marked as Highlight Paper! 😮
• (2024. 3.19) Our checkpoints are available to the public, looking from YannQi/COMBO-AVS-checkpoints · Hugging Face!
• (2024. 3.14) Our code is available to the public in $\pi$ day!
• (2024. 3.12) Our code is ready to share for the public 🌲🌲🌲!
• (2024. 2.27) Our paper(COMBO) is accepted by CVPR 2024!
• (2023.11.17) We completed the implemention of COMBO and push the code.

• Upload the pre-masks and the checkpoints at the YannQi/COMBO-AVS-checkpoints · Hugging Face!

• Linux or macOS with Python ≥ 3.6

# recommended
pip install -r requirements.txt
pip install soundfile
# build MSDeformAttention
cd models/modeling/pixel_decoder/ops
sh make.sh

• Preprocessing for detectron2

  For using Siam-Encoder Module (SEM), we refine 1-line code of the detectron2.

  The refined file that requires attention is located at:

  conda_envs/xxx/lib/python3.xx/site-packages/detectron2/checkpoint/c2_model_loading.py (refine the xxx to your own environment)

  Commenting out the following code in L287 will allow the code to run without errors:

# raise ValueError("Cannot match one checkpoint key to multiple keys in the model.")  

• Install Semantic-SAM (Optional)

# Semantic-SAM
pip install git+https://github.com/cocodataset/panopticapi.git
git clone https://github.com/UX-Decoder/Semantic-SAM
cd Semantic-SAM
python -m pip install -r requirements.txt

Find out more at Semantic-SAM

Please refer to the link AVSBenchmark to download the datasets. You can put the data under data folder or rename your own folder. Remember to modify the path in config files. The data directory is as bellow:

|--AVS_dataset
   |--AVSBench_semantic/
   |--AVSBench_object/Multi-sources/
   |--AVSBench_object/Single-source/

Preprocess the AVSS dataset for efficient training.

python3 avs_tools/preprocess_avss_audio.py
python3 avs_tools/process_avssimg2fixsize.py

• The pretrained backbone is available from benchmark AVSBench pretrained backbones YannQi/COMBO-AVS-checkpoints · Hugging Face.

|--pretrained
   |--detectron2/R-50.pkl
   |--detectron2/d2_pvt_v2_b5.pkl
   |--vggish-10086976.pth
   |--vggish_pca_params-970ea276.pth

• Generate class-agnostic masks (Optional)

sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh train # or ms3, avss
sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh val 
sh avs_tools/pre_mask/pre_mask_semantic_sam_s4.sh test

• Generate Maskiges (Optional)

python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split train # or ms3, avss
python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split val
python3 avs_tools/pre_mask2rgb/mask_precess_s4.py --split test

• Move Maskiges to the following folder Note: For convenience, we provide pre-generated Maskiges for S4\MS3\AVSS subset on the YannQi/COMBO-AVS-checkpoints · Hugging Face.

|--AVS_dataset
    |--AVSBench_semantic/pre_SAM_mask/
    |--AVSBench_object/Multi-sources/ms3_data/pre_SAM_mask/
    |--AVSBench_object/Single-source/s4_data/pre_SAM_mask/

# ResNet-50
sh scripts/res_train_avs4.sh # or ms3, avss

# PVTv2
sh scripts/pvt_train_avs4.sh # or ms3, avss

# ResNet-50
sh scripts/res_test_avs4.sh # or ms3, avss

# PVTv2
sh scripts/pvt_test_avs4.sh # or ms3, avss

We provide the checkpoints of the S4 Subset at YannQi/COMBO-AVS-checkpoints · Hugging Face.

@misc{yang2023cooperation,
      title={Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation},
      author={Qi Yang and Xing Nie and Tong Li and Pengfei Gao and Ying Guo and Cheng Zhen and Pengfei Yan and Shiming Xiang},
      year={2023},
      eprint={2312.06462},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}