This repository contains a PyTorch implementation of the PVSE network and the MRW dataset proposed in our paper Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval (CVPR 2019). The code and data are free to use for academic purposes only.

Please also visit our project page

1. MRW Dataset
2. Setting up an environment
3. Download and prepare data
4. Evaluate pretrained models
5. Train your own model

Our My Reaction When (MRW) dataset contains 50,107 video-sentence pairs crawled from social media, where videos display physical or emotional reactions to the situations described in sentences. This subreddit /r/reactiongifs contains several examples; below shows some representative examples pairs;

(a) Physical Reaction	(b) Emotional Reaction	(c) Animal Reaction	(d) Lexical Reaction
MRW a witty comment I wanted to make was already said	MFW I see a cute girl on Facebook change her status to single	MFW I cant remember if I've locked my front door	MRW a family member askes me why his computer isn't working

Below shows the descriptive statistics of the datset. The word vocabulary size is 34,835. The dataset can be used for evaluting cross-modal retrieval systems under ambiguous/weak-association between vision and language.

We provide detailed analysis of the dataset in the supplementary material of the main paper.

Follow the instruction below to download the dataset.

We recommend creating a virtual environment and install packages there. Note, you must install the Cython package first.

python3 -m venv <your virtual environment name>
source <your virtual environment name>/bin/activate
pip3 install Cython
pip3 install -r requirements.txt

cd data
bash prepare_mrw_dataset.sh

This will download the dataset (without videos) in a JSON format, a vocabulary file, and train/val/test splits. It will then prompt an option:

Do you wish to download video data and gulp them? [y/n]

We provide two ways to obtain the data. A recommended option is to download pre-compiled data in a GulpIO binary storage format, which contains video frames sampled at 8 FPS. For this, simpliy hit n (this will terminate the script) and download our pre-compiled GulpIO data in this link (54 GB). After finish downloading, extract the tarball under data/mrw/gulp to train and/or test our models.

If you wish to download raw video clips and gulp them on your own, hit y once prompted with the message above. This will start downloading videos and, once finished, start gulping the video files at 8 FPS (you can change this in download_gulp_mrw.py). If you encounter any problem downloading the video files, you may also download them directly from this link (19 GB), and then continue gulping them using the script download_gulp_mrw.py.

cd data
bash prepare_tgif_dataset.sh

This will download the dataset (without videos) in a TSV format, a vocabulary file, and train/val/test splits. Please note, we use a slightly modified version of the TGIF dataset because of invalid video files; the script will automatically download the modified version.

It will then prompt an option:

Do you wish to gulp the data? [y/n]

Similar to the MRW data, we provide two options to obtain the data: (1) download pre-compiled GulpIO data, or (2) download raw video clips and gulp them on your own, and we recommend the first option for an easy start. For this, simply hit n and download our pre-compiled GulpIO data in this link (89 GB). After finish downloading tgif-gulp.tar.gz, extract the tarball under data/tgif/gulp.

If you wish to gulp your own dataset, hit y and follow the prompt. Note that you must first download a tarball containing the videos before gulping. You can download the file tgif.tar.gz (124 GB) from this link and place it under ./data/tgif. Once you have the video data, the script will start gulping the video files.

cd data
bash prepare_coco_dataset.sh

Download all six pretrained models in a tarball at this link. You can also download each individual files using the links below.

Using the pretrained models you should be able to reproduce the results in the table below

You can train your own model using train.py; check option.py for all available options.

For example, you can train our PVSE model (k=2) on COCO using the command below. It uses ResNet152 as a backbone CNN, GloVe word embedding, MMD loss weight 0.01 and DIV loss weight 0.1, and bacth size of 256:

python3 train.py --data_name coco --cnn_type resnet152 --wemb_type glove --margin 0.1 --max_violation --num_embeds 2 --img_attention --txt_attention --mmd_weight 0.01 --div_weight 0.1 --batch_size 256

For video models, you should set the parameter --max_video_length; otherwise it defaults to 1 (single frame). Here's an example command:

python3 train.py --data_name mrw --max_video_length 4 --cnn_type resnet18 --wemb_type glove --margin 0.1 --num_embeds 4 --img_attention --txt_attention --mmd_weight 0.01 --div_weight 0.1 --batch_size 128

If you use any of the material in this repository we ask you to cite:

@inproceedings{song-pvse-cvpr19,
  author    = {Yale Song and Mohammad Soleymani},
  title     = {Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval},
  booktitle = {CVPR},
  year      = 2019

Our code is based on the implementation by Faghri et al.

Last edit: Tuesday July 16, 2019