AiOS performs human localization and SMPL-X estimation in a progressive manner. It is composed of (1) the body localization stage that predicts coarse human location; (2) the Body refinement stage that refines body features and produces face and hand locations; (3) the Whole-body Refinement stage that refines whole-body features and regress SMPL-X parameters.
- download datasets for evaluation
- download SMPL-X body models.
- download SMPL body models
SMPL_FEMALE.pkl,SMPL_MALE.pkl,SMPL_NEUTRAL.pklprovided by SMPLer-X. - download other SMPL-X dependent files:
SMPLX_to_J14.pkl,MANO_SMPLX_vertex_ids.pkl,SMPL-X__FLAME_vertex_ids.npy,SMPLX_NEUTRAL.pklprovided by SMPLer-X. - download AiOS checkpoint
- download AGORA validation set Humandata Organize them according to this datastructure:
AiOS/
├── config/
└── data
├── body_models
└── smplx
| ├──MANO_SMPLX_vertex_ids.pkl
| ├──SMPL-X__FLAME_vertex_ids.npy
| ├──SMPLX_NEUTRAL.pkl
| ├──SMPLX_to_J14.pkl
| ├──SMPLX_NEUTRAL.npz
| ├──SMPLX_MALE.npz
| └──SMPLX_FEMALE.npz
└── smpl
| ├──SMPL_FEMALE.pkl
| ├──SMPL_MALE.pkl
| └──SMPL_NEUTRAL.pkl
├── cache
├── checkpoint
│ └── aios_checkpoint.pth
├── datasets
│ ├── agora
│ └── bedlam
└── multihuman_data
└── agora_validation_multi_3840_1010.npz
# Create a conda virtual environment and activate it.
conda create -n aios python=3.8 -y
conda activate aios
# Install PyTorch and torchvision.
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
# Install Pytorch3D
git clone -b v0.6.1 https://github.com/facebookresearch/pytorch3d.git
cd pytorch3d
pip install -v -e .
cd ..
# Install MMCV, build from source
git clone -b v1.6.1 https://github.com/open-mmlab/mmcv.git
cd mmcv
export MMCV_WITH_OPS=1
export FORCE_MLU=1
pip install -v -e .
cd ..
# Install other dependencies
conda install -c conda-forge ffmpeg
pip install -r requirements.txt
# Build deformable detr
cd models/aios/ops
python setup.py build install
cd ../../..- Place the mp4 video for inference under
AiOS/demo/ - Prepare the pretrained models to be used for inference under
AiOS/data/checkpoint - Inference output will be saved in
AiOS/demo/{INPUT_VIDEO}_out
cd main
sh scripts/inference.sh {INPUT_VIDEO} {OUTPUT_DIR}
# For inferencing short_video.mp4 with output directory of demo/short_video_out
sh scripts/inference.sh short_video demo| NMVE | NMJE | MVE | MPJPE | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DATASETS | FB | B | FB | B | FB | B | F | LH/RH | FB | B | F | LH/RH |
| BEDLAM | 87.6 | 57.7 | 85.8 | 57.7 | 83.2 | 54.8 | 26.2 | 28.1/30.8 | 81.5 | 54.8 | 26.2 | 25.9/28.0 |
| AGORA-Test | 102.9 | 63.4 | 100.7 | 62.5 | 98.8 | 60.9 | 27.7 | 42.5/43.4 | 96.7 | 60.0 | 29.2 | 40.1/41.0 |
| AGORA-Val | 105.1 | 60.9 | 102.2 | 61.4 | 100.9 | 60.9 | 30.6 | 43.9/45.6 | 98.1 | 58.9 | 32.7 | 41.5/43.4 |
a. Make test_result dir
mkdir test_resultb. AGORA Validatoin
Run the following command and it will generate a 'predictions/' result folder which can evaluate with the agora evaluation tool
sh scripts/test_agora_val.sh data/checkpoint/aios_checkpoint.pth agora_valb. AGORA Test Leaderboard
Run the following command and it will generate a 'predictions.zip' which can be submitted to AGORA Leaderborad
sh scripts/test_agora.sh data/checkpoint/aios_checkpoint.pth agora_testc. BEDLAM
Run the following command and it will generate a 'predictions.zip' which can be submitted to BEDLAM Leaderborad
sh scripts/test_bedlam.sh data/checkpoint/aios_checkpoint.pth bedlam_testSome of the codes are based on MMHuman3D, ED-Pose and SMPLer-X.
