Daniel Joska, Liam Clark, Naoya Muramatsu, Ricardo Jericevich, Fred Nicolls, Alexander Mathis, Mackenzie W. Mathis, Amir Patel
AcinoSet is a dataset of 13 free-running cheetahs in the wild that contains 119,490 frames of multi-view synchronized high-speed video footage, camera calibration files and 8,522 human-annotated frames. We utilize markerless animal pose estimation with DeepLabCut to provide 2P keypoints. Then we use three methods that can serve as strong baselines for 3D pose estimation tool development: traditional sparse bundle adjustment, an Extended Kalman Filter, and a trajectory optimization-based method we call Full Trajectory Estimation. We believe this dataset will be useful benchmark for a diverse range of fields such as ecology, robotics, biomechanics as well as computer vision.
- Python3, anaconda, code dependencies are within conda env files.
- 8,522 ground truth 2D frames
- 119,490 processed frames with 2D keypoint estimation outputs (H5 files as in the DLC format, and raw video)
- this is currently organized by date > animal ID > "run/attempt"
- 3D files that are processed using our FTE baseline model. These can be used for 3D GT.
- these files are called
traj_opt.pickle
, have a relatedscene_sba.json
file, and can be loaded in the GUI.
- these files are called
- A GUI to inspect the 3D dataset, which can be found here
The following sections document how this was created by the code within this repo:
- You can use the
full_cheetah
model provided in the DLC Model Zoo To re-create the H5 files (or on new videos). - Here, we also already provide the videos and H5 outputs of all frames, here.
If you want to label more cheetah data, you can also do so within the DeepLabCut framework. We provide a conda file for an easy-install, but please see the repo for installation and instructions for use.
$ conda env create -f conda_envs/dDLC.yml -n DLC
You can manually define points on each video with Argus. Documentation is here.
Build the environment:
$ conda env create -f conda_envs/argus.yml -n argus
Launch Argus/Clicker:
$ python
>>> import argus_gui as ag; ag.ClickerGUI()
Keyboard Shortcuts:
G
... to a specific frameX
... to switch the sync mode setting the windows to the same frameA
... to use the auto-tracker7
,Y
,U
,I
... growing the view finder at the bottom rightO
... to bring up the options dialogS
... to bring up a save dialog
Then you must convert the output data from Argus to work with the rest of the pipeline (here is an example):
$ python converter_argus.py \
--data_dir ../data/2019_03_07/extrinsic_calib/videos
Build the environment.
$ conda env create --file conda_envs/cv.yml
Launch Jupyter Lab:
$ jupyter lab
Run calib_with_gui.ipynb
, and follow the instructions.
Prepare the environment.
$ pyenv local anaconda3-5.2.0/envs/cv
Run full_traj_opt.py
, or use the supplied Jupyter Notebook:
$ python full_traj_opt.py \
--n_camera 6 \
--logs_dir ../logs \
--configs_dir ../configs \
--data_dir ../data/2019_03_09/jules/flick1 \
--scene_file ../data/2019_03_09/extrinsic_calib/scene_sba.json
If you want to view the 3D animation, run full_traj_optimisation.ipynb
and follow the instructions!