Benchmark your method against several other methods on the popular FLIC and LSP datasets.
To benchmark the algorithms simply run the scripts files benchmark_flic.m
and benchmark_lsp.m
to evaluate the algorithms on the FLIC and LSP datasets, respectively.
Adding predictions of a new algorithm is fairly simple:
- Create a folder with the name of the algorithm in
algorithms/
. - Create a file called
algorithm.txt
and assign a label/alias name for the algorithm to be used to identify the algorithms name in the plot's legend. - Add the predictions files with the keypoints/sticks coordinates with the names
pred_keypoints_lsp_oc.mat
,pred_keypoints_lsp_pc.mat
,pred_sticks_lsp_oc.mat
andpred_sticks_lsp_pc.mat
.
Note: The scripts will skip the missing files when benchmarking a method.
Several options are available for configuration. These, however, require the user to change the file manually.
Plot options:
list
: specify which algorithms to plot. If empty, plots all algorithms.bSave
: save plot images toplots/
folder (if set totrue
).printLegend
: prints a legend in every plot (if set totrue
).pcp_threshold
: PCP evaluation threshold.pck_threshold
: PCK evaluation threshold.
For now, the available datasets for PCK and PCP evaluation are the FLIC and LSP. Other datasets may be introduced if it is justifiable for inclusion.
@inproceedings{modec13,
title={MODEC: Multimodal Decomposable Models for Human Pose Estimation},
author={Sapp, Benjamin and Taskar, Ben},
booktitle={In Proc. CVPR},
year={2013},
}
@inproceedings{Johnson11,
title = {Learning Effective Human Pose Estimation from Inaccurate Annotation},
author = {Johnson, Sam and Everingham, Mark},
year = {2011},
booktitle = {IEEE Proc. CVPR}
}
All available methods for the LSP benchmark were downloaded from MPII's website.
FLIC methods were gather from some authors's predictions available online.
Results of the algorithms are shown bellow.
Method | Elbow | Wrist |
---|---|---|
Sapp et al., CVPR'13 | 72.5 | 54.5 |
Yang et al., CVPR'16 | 91.6 | 88.8 |
Chen et al., NIPS'14 | 89.8 | 86.8 |
Wei et al., CVPR'16 | 92.5 | 90.0 |
Newell et al., arXiv'16 | 98.0 | 95.5 |
Method | Torso | Upper leg | Lower leg | Upper arm | Forearm | Head | PCP |
---|---|---|---|---|---|---|---|
Wang et al., CVPR'13 | 87.5 | 56.0 | 55.8 | 43.1 | 32.1 | 79.1 | 54.1 |
Pishchulin et al., ICCV' 13 | 88.7 | 63.6 | 58.4 | 46.0 | 35.2 | 85.1 | 58.0 |
Tompson et al., NIPS'14 | 90.3 | 70.4 | 61.1 | 63.0 | 51.2 | 83.7 | 66.6 |
Fan et al., CVPR'15 | 95.4 | 77.7 | 69.8 | 62.8 | 49.1 | 86.6 | 70.1 |
Chen et al., NIPS'14 | 96.0 | 77.2 | 72.2 | 69.7 | 58.1 | 85.6 | 73.6 |
Yang et al., CVPR'16 | 95.6 | 78.5 | 71.8 | 72.2 | 61.8 | 83.9 | 74.8 |
Rafi et al., BMVC'16 | 97.6 | 87.3 | 80.2 | 76.8 | 66.2 | 93.3 | 81.2 |
Belagiannis et al., arXiv'16 | 96.0 | 86.7 | 82.2 | 79.4 | 69.4 | 89.4 | 82.1 |
Lifshitz et al., ECCV'16 | 97.3 | 88.8 | 84.4 | 80.6 | 71.4 | 94.8 | 84.3 |
Pishchulin et al., CVPR'16 | 97.0 | 88.8 | 82.0 | 82.4 | 71.8 | 95.8 | 84.3 |
Yu et al., ECCV'16 | 98.0 | 93.1 | 88.1 | 82.9 | 72.6 | 83.0 | 85.4 |
Insafutdinov et al., ECCV'16 | 97.0 | 90.6 | 86.9 | 86.1 | 79.5 | 95.4 | 87.8 |
Wei et al., CVPR'16 | 98.0 | 92.2 | 89.1 | 85.8 | 77.9 | 95.0 | 88.3 |
Bulat et al., ECCV'16 | 97.7 | 92.4 | 89.3 | 86.7 | 79.7 | 95.2 | 88.9 |
Method | Head | Shoulder | Elbow | Wrist | Hip | Knee | Ankle | Total |
---|---|---|---|---|---|---|---|---|
Wang et al., CVPR'13 | 84.7 | 57.1 | 43.7 | 36.7 | 56.7 | 52.4 | 50.8 | 54.6 |
Pishchulin et al., ICCV' 13 | 87.2 | 56.7 | 46.7 | 38.0 | 61.0 | 57.5 | 52.7 | 57.1 |
Tompson et al., NIPS'14 | 90.6 | 79.2 | 67.9 | 63.4 | 69.5 | 71.0 | 64.2 | 72.3 |
Fan et al., CVPR'15 | 92.4 | 75.2 | 65.3 | 64.0 | 75.7 | 68.3 | 70.4 | 73.0 |
Chen et al., NIPS'14 | 91.8 | 78.2 | 71.8 | 65.5 | 73.3 | 70.2 | 63.4 | 73.4 |
Yang et al., CVPR'16 | 90.6 | 78.1 | 73.8 | 68.8 | 74.8 | 69.9 | 58.9 | 73.6 |
Rafi et al., BMVC'16 | 95.8 | 86.2 | 79.3 | 75.0 | 86.6 | 83.8 | 79.8 | 83.8 |
Yu et al., ECCV'16 | 87.2 | 88.2 | 82.4 | 76.3 | 91.4 | 85.8 | 78.7 | 84.3 |
Belagiannis et al., arXiv'16 | 95.2 | 89.0 | 81.5 | 77.0 | 83.7 | 87.0 | 82.8 | 85.2 |
Lifshitz et al., ECCV'16 | 96.8 | 89.0 | 82.7 | 79.1 | 90.9 | 86.0 | 82.5 | 86.7 |
Pishchulin et al., CVPR'16 | 97.0 | 91.0 | 83.8 | 78.1 | 91.0 | 86.7 | 82.0 | 87.1 |
Insafutdinov et al., ECCV'16 | 97.4 | 92.7 | 87.5 | 84.4 | 91.5 | 89.9 | 87.2 | 90.1 |
Wei et al., CVPR'16 | 97.8 | 92.5 | 87.0 | 83.9 | 91.5 | 90.8 | 89.9 | 90.5 |
Bulat et al., ECCV'16 | 97.2 | 92.1 | 88.1 | 85.2 | 92.2 | 91.4 | 88.7 | 90.7 |
This code is a modified version of the original code made available by MPII.
The available code is released under the MIT license.