Most existing cross-modal language-to-video retrieval (VR) research focuses on single-modal input from video, i.e., visual representation, while text is omnipresent in human environments and frequently critical to understand video. To study how to retrieve video with both modal inputs, i.e., visual and text semantic representations, we firstly introduce a large-scale and cross-modal Video Retrieval dataset with text reading comprehension, TextVR, which contains 42.2k sentence queries for 10.5k videos of 8 scenario domains, i.e., Street View (indoor), Street View (outdoor), Game, Sports, Driving, Activity, TV Show, and Cooking. The proposed TextVR requires one unified cross-modal model to recognize and comprehend texts, relate them to the visual context, and decide what text semantic information is vital for the video retrieval task.
The baseline model of TextVR is based on the video retrieval model frozen-in-time. To setup the environment of TextVR, we use conda
to manage our dependencies. Our developers use CUDA 11.1
to do experiments. You can specify the appropriate cudatoolkit
version to install on your machine in the requirements.txt
file, and then run the following commands to install FlowText:
conda create -n textvr python=3.8
conda activate textvr
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
git clone https://github.com/callsys/TextVR
cd TextVR
pip install -r requirements.txt
To run TextVR, you need to download some files, which mainly contain the resized videos and annotation files(5.4G), the original videos(85G, google drive, baidu drive, nenm) and the trained model(2.1G).
The unscrambled annotation file of the test set is released, replace TextVR_test_rand.json
with TextVR_test.json
for offline testing.
Once you have downloaded the files, link them to the TextVR directory:
ln -s path/to/TextVR_data TextVR/data/TextVR
ln -s path/to/TextVR_ckpt TextVR/ckpt
The format of the download files are as follows:
TextVR
|
└─── data
| |
| └─── TextVR
| └─── TextVR_train.json
| └─── TextVR_test_rand.json
| └─── Videos
| | └─── Activaty
| | └─── Cooking
| | └─── ......
| | └─── Technology
| |
| └─── Kwai_VideoOCR
| └─── Activaty
| └─── Cooking
| └─── ......
| └─── Technology
|
└─── ckpt
└─── config.json
└─── textvr.pth
where Kwai_VideoOCR
are the text in the video spotted by the Kuaishou OCR api.
Training TextVR with given config file configs/TextVR_fusion.json
:
python train.py -c configs/TextVR_fusion.json
Inference TextVR with given weights ckpt/textvr.pth
and save the similarity matrix sim_matrix.npy
:
python test.py -c configs/TextVR_fusion.json -r ckpt/textvr.pth --sim_path sim_matrix.npy
sim_matrix.npy
is a ndarray S of shape 2727 x 2727 (2727 is the size of the test set), where S(x, y) denotes the similarity score between the x-th caption and the y-th video. Here is an example of sim_matrix.npy
, its format should look like this:
S(1,1),S(1,2),...,S(1,2727)
S(2,1),S(2,2),...,S(2,2727)
...........................
S(2726,1),....,S(2726,2727)
S(2727,1),....,S[2727,2727)
Note that the caption of the test set has been scrambled so that the model cannot be validated offline. If you want to verify a model's performance on the test set, submit the similarity matrix sim_matrix.npy
to the competition website.
Code is largely based on frozen-in-time.
Work is fully supported by MMU of Kuaishou Technology.