Webpage • Paper

This repository provides the code for our FrozenBiLM paper (NeurIPS 2022), including:

• Environment setup
• Data downloading instructions
• Data preprocessing and visual feature extraction scripts, as well as preprocessed data and features
• Pretrained checkpoints
• Training and evaluation scripts for cross-modal training, downstream fully-supervised, few-shot and zero-shot VideoQA, including various baselines

To install requirements, run:

conda create -n frozenbilm_env python=3.8 
conda activate frozenbilm_env
conda install pytorch==1.8.1 torchvision==0.9.1 cudatoolkit=11.1 -c pytorch -c nvidia 
pip install requirements.txt

You may fill the global paths in args.py.
To use a given text-pretrained language model, you should download the corresponding weights from the Hugging Face Hub and put them in TRANSFORMERS_CACHE.

If you wish to start VideoQA training or inference quickly.

To download pretrained checkpoints, pre-processed data, ASR and visual features, run:

bash download/download_checkpoints.sh <MODEL_DIR>
bash download/download_downstream.sh <DATA_DIR>

It requires about 8GB for the models, and 12GB for the data.
Note that most pretrained checkpoints only contain updated parameters due to storage limitations (and not the frozen parameters).
This means you have to make sure that you have properly downloaded weights from Hugging Face for the language model of your choice when using a provided checkpoint.
For completeness, frozenbilm.pth, frozenbilm_bertbase_noadapter.pth and frozenbilm_bertlarge_noadapter.pth contain all parameters.
Also note that due to storage issue, we do not host publicly visual features for the WebVid10M dataset.

python extract/extract.py --csv <csv_path>

python extract/merge_features.py --folder <features_path> \
--output_path <DEFAULT_DATASET_DIR>/clipvitl14.pth --dataset <dataset>

Note that checkpoints finetuned on 10% or 1% of downstream datasets (few-shot setting) are also made accessible here.
Variants using a BERT-Base or BERT-Large language model (without adapters) instead of DeBERTa are also present in this folder.

To train FrozenBiLM on WebVid10M, run:

python -m torch.distributed.launch --nproc_per_node 8 --use_env main.py \
--combine_datasets webvid --combine_datasets_val webvid --save_dir=trainwebvid \
--lr=3e-5 --ds_factor_ff=8 --ds_factor_attn=8 \
--batch_size=16 --batch_size_val=16 --epochs=2 \

To evaluate the cross-modal trained FrozenBiLM on LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA or TGIF-QA FrameQA, run:

python -m torch.distributed.launch --nproc_per_node 8 --use_env videoqa.py --test --eval \
--combine_datasets <dataset> --combine_datasets_val <dataset> --save_dir=zs<dataset> \
--ds_factor_ff=8 --ds_factor_attn=8 --suffix="." \
--batch_size_val=32 --max_tokens=256 --load=<CKPT_PATH> --<dataset>_vocab_path=$DATA_DIR/<dataset>/vocab1000.json

To evaluate the cross-modal trained FrozenBiLM on How2QA or TVQA, run:

python -m torch.distributed.launch --nproc_per_node 8 --use_env mc.py --eval \ 
--combine_datasets <dataset> --combine_datasets_val <dataset> --save_dir=zs<dataset> \ 
--ds_factor_ff=8 --ds_factor_attn=8 --suffix="." \
--batch_size_val=32 --max_tokens=512 --load=<CKPT_PATH>

To finetune the cross-modal trained FrozenBiLM on LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA or TGIF-QA FrameQA, run:

python -m torch.distributed.launch --nproc_per_node 4 --use_env videoqa.py \
--combine_datasets <dataset> --combine_datasets_val <dataset> --save_dir=ft<dataset> \
--lr=5e-5 --schedule=linear_with_warmup --load=<CKPT_PATH> \
--ds_factor_ff=8 --ds_factor_attn=8 --suffix="." \
--batch_size=8 --batch_size_val=32 --max_tokens 256 --epochs=20

Pass --ft_lm --ft_mlm --ds_factor_ff=0 --ds_factor_attn=0 for the UnFrozenBiLM variant.

To finetune the cross-modal trained FrozenBiLM on How2QA or TVQA, run:

python -m torch.distributed.launch --nproc_per_node 8 --use_env mc.py \
--combine_datasets <dataset> --combine_datasets_val <dataset> --save_dir=ft<dataset> \
--lr=5e-5 --schedule=linear_with_warmup --load=<CKPT_PATH> \
--ds_factor_ff=8 --ds_factor_attn=8 --suffix="." \
--batch_size=2 --batch_size_val=8 --max_tokens=256 --epochs=20 

Pass --ft_lm --ft_mlm --ds_factor_ff=0 --ds_factor_attn=0 --batch_size=1 for the UnFrozenBiLM variant.

For few-shot VideoQA, we sample a subpart of the train dataframe file and change --<dataset>_train_csv_path.
The random subsets used in the paper are released here.

The transformer models implementation is inspired by Hugging Face's transformers library.
The feature extraction code is inspired by Just Ask.

This code is released under the Apache License 2.0.
The licenses for datasets used in the paper are available at the following links: iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, How2QA and TVQA.

If you found this work useful, consider giving this repository a star and citing our paper as followed:

@article{yang2022frozenbilm,
title = {Zero-Shot Video Question Answering via Frozen Bidirectional Language Models},
author = {Yang, Antoine and Miech, Antoine and Sivic, Josef and Laptev, Ivan and Schmid, Cordelia},
journal={arXiv preprint arXiv:2206.08155}
year = {2022},
}