Gated Temporal Diffusion is an approach for stochastic long-term dense action anticipation, that models uncertainty in the observed and future frames jointly while preserving distinction between these two intrinsically different parts using gating.

Here is the overview of our proposed model:

If you find this code or our model useful, please cite our paper:

@inproceedings{gtda2024zatsarynna,
    author    = {Olga Zatsarynna and Emad Bahrami and Yazan Abu Farha and Gianpiero Francesca and Juergen Gall},
    title     = {Gated Temporal Diffusion for Stochastic Long-Term Dense Anticipation},
    booktitle={European Conference on Computer Vision},
    year={2024},
}

To create the conda environment run the following command:

conda env create --name gtda --file docker/env.yml
source activate gtda

The features and annotations of the Breakfast dataset can be downloaded from link 1 or link 2.

Follow the instructions at Assembly101-Download-Scripts to download the TSM features. We converted the .lmdb features to numpy for faster loading. The coarse-annotations can be downloaded from assembly101-annotations.

To train the stochastic anticipation model, run:

bash scripts/prob/train_<dataset>_prob.sh

To evaluate the stochastic anticipation model, run:

bash scripts/prob/predict_<dataset>_prob.sh

This will show the evaluation results of the final model, as well as save final predictions into the ./diff_results directory. With the final results saved, you can run the stochastic evaluation faster using the following script:

bash scripts/prob/predict_precomputed_<dataset>_prob.sh

Make sure to update the paths (features and annotations) in the above scripts to match your system. For changing the training and evaluation splits (for Breakfast dataset), as well as values of other hyper-parameters, modify the scripts accordingly.

Coming soon!

To train the determinstic anticipation model, run the corresponding script:

bash scripts/determ/train_<dataset>_determ.sh

To evaluate the deterministic anticipation model, run the corresponding script:

bash scripts/determ/predict_<dataset>_determ.sh

Coming soon!

In our code we made use of the following repositories: PDPP, MS-TCN and LTC. We sincerely thank the authors for their codebases!

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.