This is the official repository for the paper "Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World" (Accepted by ICCV 2023)
Here we provide sample code for CaCao boosting SGG dataset in standard setting and open-world setting.
Download the enhanced dataset for VG training, you can use this Google drive link.
the VG dataset is required and put all the images in a same folder VG_100K
download
put above in ./datasets/vg/ folder
put coco2014_train in ./datasets/coco folder
put Vit in ./vit-base-patch32-224-in21k folder
put bert pytorch.bin in ./bert-base-uncased folder
# creat imdb_512.h5
python vg_to_imdb.py
# obtain initialized clusters for CaCao
python adaptive_cluster.py
# establish the mapping from open-world boosted data to target predicates for enhancement
python fine_grained_mapping.py
# obtain cross-modal prompt tuning models for better predicate boosting
python cross_modal_tuning.py --mode 50
python cross_modal_tuning.py --mode all
# enhance the existing SGG dataset with our CaCao model in <pre_trained_visually_prompted_model>
python fine_grained_predicate_boosting_data_prepare.py --mode 50
python fine_grained_predicate_boosting_data_prepare.py --mode all
python fine_grained_predicate_boosting.py --mode 50
python fine_grained_predicate_boosting.py --mode all
The SGG part code is implemented based on Scene-Graph-Benchmark.pytorch, FGPL, and SSRCNN(One-Stage). Thanks for their great works!
If you find this work useful for your research, please cite our paper and star our git repo:
@article{yu2023visually,
title={Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World},
author={Yu, Qifan and Li, Juncheng and Wu, Yu and Tang, Siliang and Ji, Wei and Zhuang, Yueting},
journal={arXiv preprint arXiv:2303.13233},
year={2023}
}
or
@inproceedings{yu2023visually,
title={Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World},
author={Yu, Qifan and Li, Juncheng and Wu, Yu and Tang, Siliang and Ji, Wei and Zhuang, Yueting},
booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
year={2023}
}