This repository is an official implementation of the paper Referring Multi-Object Tracking. More project details can be found in the website.

❗ ❗ ❗ The bootstrapped version of this repository is released. Please see TempRMOT.

Abstract. Existing referring understanding tasks tend to involve the detection of a single text-referred object. In this paper, we propose a new and general referring understanding task, termed referring multi-object tracking (RMOT). Its core idea is to employ a language expression as a semantic cue to guide the prediction of multi-object tracking. To the best of our knowledge, it is the first work to achieve an arbitrary number of referent object predictions in videos. To push forward RMOT, we construct one benchmark with scalable expressions based on KITTI, named Refer-KITTI. Specifically, it provides 18 videos with 818 expressions, and each expression in a video is annotated with an average of 10.7 objects. Further, we develop a transformer-based architecture TransRMOT to tackle the new task in an online manner, which achieves impressive detection performance.

• (2023/03/19) RMOT dataset and code are released.
• (2023/03/07) RMOT paper is available on arxiv.
• (2023/02/28) RMOT is accepted by CVPR2023! The dataset and code is coming soon!

The basic environment setup is on top of MOTR, including conda environment, pytorch version and other requirements.

You can download our created expression and labels_with_ids. The KITTI images are from official website, which are unzipped into ./KITTI/training. The Refer-KITTI is organized as follows:

.
├── refer-kitti
│   ├── KITTI
│           ├── training
│           ├── labels_with_ids
│   └── expression

Note:

• Our expression (.json) contains corresponding object ids, and the corresponding boxes can be found in 'labels_with_ids' using these ids.
• The 'label_with_ids' is generated from a script from folder tools. But we strongly recommend not using it because the generated track_id may not correspond the track_id of our expression files.

You can download COCO pretrained weights from Deformable DETR ''+ iterative bounding box refinement''. Then training TransRMOT on 8 GPUs as following:

sh configs/r50_rmot_train.sh

Note:

• If the RoBERTa is not working well, please download the RoBERTa weights from Hugging Face for local using.

You can download the pretrained model of TransRMOT (the link is in "Main Results" session), then run following command to generate and save prediction boxes:

sh configs/r50_rmot_test.sh

You can get the main results by runing the evaluation part. You can also use our prediction and gt file.

cd TrackEval/script
sh evaluate_rmot.sh

The main results of TransRMOT:

We also provide FairMOT results as references.

If you find RMOT useful in your research, please consider citing:

@inproceedings{wu2023referring,
  title={Referring Multi-Object Tracking},
  author={Wu, Dongming and Han, Wencheng and Wang, Tiancai and Dong, Xingping and Zhang, Xiangyu and Shen, Jianbing},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={14633--14642},
  year={2023}
}

• MOTR
• Deformable DETR
• TrackEval