Skip to content

Evaluation tool for benchmarking human pose prediction algorithms on the FLIC and LSP datasets.

License

Notifications You must be signed in to change notification settings

farrajota/human-pose-benchmark

Repository files navigation

Articulated Human pose evaluation/benchmark

Benchmark your method against several other methods on the popular FLIC and LSP datasets.

How to use

To benchmark the algorithms simply run the scripts files benchmark_flic.m and benchmark_lsp.m to evaluate the algorithms on the FLIC and LSP datasets, respectively.

Adding a new algorithm to the list

Adding predictions of a new algorithm is fairly simple:

  1. Create a folder with the name of the algorithm in algorithms/.
  2. Create a file called algorithm.txt and assign a label/alias name for the algorithm to be used to identify the algorithms name in the plot's legend.
  3. Add the predictions files with the keypoints/sticks coordinates with the names pred_keypoints_lsp_oc.mat, pred_keypoints_lsp_pc.mat, pred_sticks_lsp_oc.mat and pred_sticks_lsp_pc.mat.

Note: The scripts will skip the missing files when benchmarking a method.

Options

Several options are available for configuration. These, however, require the user to change the file manually.

Plot options:

  • list: specify which algorithms to plot. If empty, plots all algorithms.
  • bSave: save plot images to plots/ folder (if set to true).
  • printLegend: prints a legend in every plot (if set to true).
  • pcp_threshold: PCP evaluation threshold.
  • pck_threshold: PCK evaluation threshold.

Available datasets

For now, the available datasets for PCK and PCP evaluation are the FLIC and LSP. Other datasets may be introduced if it is justifiable for inclusion.

@inproceedings{modec13,
    title={MODEC: Multimodal Decomposable Models for Human Pose Estimation},
    author={Sapp, Benjamin and Taskar, Ben},
    booktitle={In Proc. CVPR},
    year={2013},
}
@inproceedings{Johnson11,
	title = {Learning Effective Human Pose Estimation from Inaccurate Annotation},
	author = {Johnson, Sam and Everingham, Mark},
	year = {2011},
	booktitle = {IEEE Proc. CVPR}
}

Benchmark results

All available methods for the LSP benchmark were downloaded from MPII's website.

FLIC methods were gather from some authors's predictions available online.

Results of the algorithms are shown bellow.

Frames Labeled In Cinema (FLIC)

PCK(0.2) - Observer Centric

Method Elbow Wrist
Sapp et al., CVPR'13 72.5 54.5
Yang et al., CVPR'16 91.6 88.8
Chen et al., NIPS'14 89.8 86.8
Wei et al., CVPR'16 92.5 90.0
Newell et al., arXiv'16 98.0 95.5
legends hip
wrist elbow shoulder

Leeds Sport Pose (LSP)

PCP(0.5) - Person Centric

Method Torso Upper leg Lower leg Upper arm Forearm Head PCP
Wang et al., CVPR'13 87.5 56.0 55.8 43.1 32.1 79.1 54.1
Pishchulin et al., ICCV' 13 88.7 63.6 58.4 46.0 35.2 85.1 58.0
Tompson et al., NIPS'14 90.3 70.4 61.1 63.0 51.2 83.7 66.6
Fan et al., CVPR'15 95.4 77.7 69.8 62.8 49.1 86.6 70.1
Chen et al., NIPS'14 96.0 77.2 72.2 69.7 58.1 85.6 73.6
Yang et al., CVPR'16 95.6 78.5 71.8 72.2 61.8 83.9 74.8
Rafi et al., BMVC'16 97.6 87.3 80.2 76.8 66.2 93.3 81.2
Belagiannis et al., arXiv'16 96.0 86.7 82.2 79.4 69.4 89.4 82.1
Lifshitz et al., ECCV'16 97.3 88.8 84.4 80.6 71.4 94.8 84.3
Pishchulin et al., CVPR'16 97.0 88.8 82.0 82.4 71.8 95.8 84.3
Yu et al., ECCV'16 98.0 93.1 88.1 82.9 72.6 83.0 85.4
Insafutdinov et al., ECCV'16 97.0 90.6 86.9 86.1 79.5 95.4 87.8
Wei et al., CVPR'16 98.0 92.2 89.1 85.8 77.9 95.0 88.3
Bulat et al., ECCV'16 97.7 92.4 89.3 86.7 79.7 95.2 88.9
total torso head
upper leg lower leg
upper arm forearm

PCK(0.2) - Person Centric

Method Head Shoulder Elbow Wrist Hip Knee Ankle Total
Wang et al., CVPR'13 84.7 57.1 43.7 36.7 56.7 52.4 50.8 54.6
Pishchulin et al., ICCV' 13 87.2 56.7 46.7 38.0 61.0 57.5 52.7 57.1
Tompson et al., NIPS'14 90.6 79.2 67.9 63.4 69.5 71.0 64.2 72.3
Fan et al., CVPR'15 92.4 75.2 65.3 64.0 75.7 68.3 70.4 73.0
Chen et al., NIPS'14 91.8 78.2 71.8 65.5 73.3 70.2 63.4 73.4
Yang et al., CVPR'16 90.6 78.1 73.8 68.8 74.8 69.9 58.9 73.6
Rafi et al., BMVC'16 95.8 86.2 79.3 75.0 86.6 83.8 79.8 83.8
Yu et al., ECCV'16 87.2 88.2 82.4 76.3 91.4 85.8 78.7 84.3
Belagiannis et al., arXiv'16 95.2 89.0 81.5 77.0 83.7 87.0 82.8 85.2
Lifshitz et al., ECCV'16 96.8 89.0 82.7 79.1 90.9 86.0 82.5 86.7
Pishchulin et al., CVPR'16 97.0 91.0 83.8 78.1 91.0 86.7 82.0 87.1
Insafutdinov et al., ECCV'16 97.4 92.7 87.5 84.4 91.5 89.9 87.2 90.1
Wei et al., CVPR'16 97.8 92.5 87.0 83.9 91.5 90.8 89.9 90.5
Bulat et al., ECCV'16 97.2 92.1 88.1 85.2 92.2 91.4 88.7 90.7
legends head
ankle knee hip
wrist elbow shoulder

Acknowledgements

This code is a modified version of the original code made available by MPII.

License

The available code is released under the MIT license.

About

Evaluation tool for benchmarking human pose prediction algorithms on the FLIC and LSP datasets.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published