This repository is the official implementation of the AutoBots architectures. We include support for the following datasets:
- nuScenes (ego-agent, multi-agent)
- Argoverse (ego-agent)
- TrajNet++ Synthetic dataset (multi-agent)
- Interaction-Dataset (multi-agent)
Visit our webpage for more information.
- Create a python 3.7 environment. I use Miniconda3 and create with
conda create --name AutoBots python=3.7 - Run
pip install -r requirements.txt
That should be it!
Follow the instructions in the READMEs of each dataset (found in datasets).
All experiments were performed locally on a single GTX 1080Ti.
The trained models will be saved in results/{Dataset}/{exp_name}.
Training AutoBot-Ego on nuScenes while using the raw road segments in the map:
python train.py --exp-id test --seed 1 --dataset Nuscenes --model-type Autobot-Ego --num-modes 10 --hidden-size 128 --num-encoder-layers 2 --num-decoder-layers 2 --dropout 0.1 --entropy-weight 40.0 --kl-weight 20.0 --use-FDEADE-aux-loss True --use-map-lanes True --tx-hidden-size 384 --batch-size 64 --learning-rate 0.00075 --learning-rate-sched 10 20 30 40 50 --dataset-path /path/to/root/of/nuscenes_h5_files
Training AutoBot-Ego on nuScenes while using the Birds-eye-view image of the road network:
python train.py --exp-id test --seed 1 --dataset Nuscenes --model-type Autobot-Ego --num-modes 10 --hidden-size 128 --num-encoder-layers 2 --num-decoder-layers 2 --dropout 0.1 --entropy-weight 40.0 --kl-weight 20.0 --use-FDEADE-aux-loss True --use-map-image True --tx-hidden-size 384 --batch-size 64 --learning-rate 0.00075 --learning-rate-sched 10 20 30 40 50 --dataset-path /path/to/root/of/nuscenes_h5_files
Training AutoBot-Joint on nuScenes while using the raw road segments in the map:
python train.py --exp-id test --seed 1 --dataset Nuscenes --model-type Autobot-Joint --num-modes 10 --hidden-size 128 --num-encoder-layers 2 --num-decoder-layers 2 --dropout 0.1 --entropy-weight 40.0 --kl-weight 20.0 --use-FDEADE-aux-loss True --use-map-lanes True --tx-hidden-size 384 --batch-size 64 --learning-rate 0.00075 --learning-rate-sched 10 20 30 40 50 --dataset-path /path/to/root/of/nuscenes_h5_files
Training AutoBot-Ego on Argoverse while using the raw road segments in the map:
python train.py --exp-id test --seed 1 --dataset Argoverse --model-type Autobot-Ego --num-modes 6 --hidden-size 128 --num-encoder-layers 2 --num-decoder-layers 2 --dropout 0.1 --entropy-weight 40.0 --kl-weight 20.0 --use-FDEADE-aux-loss True --use-map-lanes True --tx-hidden-size 384 --batch-size 64 --learning-rate 0.00075 --learning-rate-sched 10 20 30 40 50 --dataset-path /path/to/root/of/argoverse_h5_files
Training AutoBot-Joint on TrajNet++:
python train.py --exp-id test --seed 1 --dataset trajnet++ --model-type Autobot-Joint --num-modes 6 --hidden-size 128 --num-encoder-layers 2 --num-decoder-layers 2 --dropout 0.1 --entropy-weight 40.0 --kl-weight 20.0 --use-FDEADE-aux-loss True --tx-hidden-size 384 --batch-size 64 --learning-rate 0.00075 --learning-rate-sched 10 20 30 40 50 --dataset-path /path/to/root/of/npy_files
Training AutoBot-Joint on the Interaction-Dataset while using the raw road segments in the map:
python train.py --exp-id test --seed 1 --dataset interaction-dataset --model-type Autobot-Joint --num-modes 6 --hidden-size 128 --num-encoder-layers 2 --num-decoder-layers 2 --dropout 0.1 --entropy-weight 40.0 --kl-weight 20.0 --use-FDEADE-aux-loss True --tx-hidden-size 384 --batch-size 64 --learning-rate 0.00075 --learning-rate-sched 10 20 30 40 50 --dataset-path /path/to/root/of/interaction_dataset_h5_files
For all experiments, you can evaluate the trained model on the validation dataset by running:
python evaluate.py --dataset-path /path/to/root/of/interaction_dataset_h5_files --models-path results/{Dataset}/{exp_name}/{model_epoch}.pth --batch-size 64
Note that the batch-size may need to be reduced for the Interaction-dataset since evaluation is performed on all agent scenes.
We also provide extra scripts that can be used for submitting to the nuScenes, Argoverse and Interaction-Dataset Evaluation server.
For nuScenes:
python useful_scripts/generate_nuscene_results.py --dataset-path /path/to/root/of/nuscenes_h5_files --models-path results/Nuscenes/{exp_name}/{model_epoch}.pth
For Argoverse:
python useful_scripts/generate_argoverse_test.py --dataset-path /path/to/root/of/argoverse_h5_files --models-path results/Argoverse/{exp_name}/{model_epoch}.pth
For the Interaction-Dataset:
python useful_scripts/generate_indst_test.py --dataset-path /path/to/root/of/interaction_dataset_h5_files --models-path results/interaction-dataset/{exp_name}/{model_epoch}.pth
If you use this repository, please cite our work:
@inproceedings{
girgis2022latent,
title={Latent Variable Sequential Set Transformers for Joint Multi-Agent Motion Prediction},
author={Roger Girgis and Florian Golemo and Felipe Codevilla and Martin Weiss and Jim Aldon D'Souza and Samira Ebrahimi Kahou and Felix Heide and Christopher Pal},
booktitle={International Conference on Learning Representations},
year={2022},
url={https://openreview.net/forum?id=Dup_dDqkZC5}
}