Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with testing using distributed and mpii #89

Closed
pablovela5620 opened this issue Aug 27, 2020 · 5 comments
Closed

Issue with testing using distributed and mpii #89

pablovela5620 opened this issue Aug 27, 2020 · 5 comments
Assignees
Labels
kind/bug something isn't working

Comments

@pablovela5620
Copy link

So I was trying to test the mpii dataset using both test.py and ./tool/dist_test.sh and came across this small issue. It looks like mpii uses the evaluation metric PCKh rather than mPA so I changed the args as follows

For test.py

 python tools/test.py configs/top_down/hrnet/mpii/hrnet_w32_mpii_256x256.py ~/0Dev/data/models/hrnet_w32_mpii_256x256-6c4f923f_20200812.pth --eval PCKh

for ./tools/dist_test.sh

./tools/dist_test.sh configs/top_down/hrnet/mpii/hrnet_w32_mpii_256x256.py ~/0Dev/data/models/hrnet_w32_mpii_256x256-6c4f923f_20200812.pth 2 --eval PCKh

I get the assertion error saying that metric != PCKh and modified the error so I could figure out what was going on and saw

assert metric == 'PCKh', f"got {metric}"
AssertionError: got ['PCKh']

I'm fairly new to this repo so I don't know the ins and outs enough to make a pull request, so I wanted to make you guys aware of this issue!

@jin-s13 jin-s13 self-assigned this Aug 28, 2020
@jin-s13 jin-s13 added the kind/bug something isn't working label Aug 28, 2020
@jin-s13
Copy link
Collaborator

jin-s13 commented Aug 28, 2020

Thanks for reporting. I will check this.

@jin-s13
Copy link
Collaborator

jin-s13 commented Aug 28, 2020

Will be fixed in this commit.

@jin-s13 jin-s13 closed this as completed Aug 28, 2020
@pablovela5620
Copy link
Author

I also just noticed that it looks like most mpii configs have their evaluation = dict(interval=1, metric='mPA') rather than 'PCKh` which will cause errors when evaluating

@jin-s13
Copy link
Collaborator

jin-s13 commented Aug 28, 2020

Thanks! Yes, It should be PCKh.
We will fix the typos.

@jin-s13
Copy link
Collaborator

jin-s13 commented Aug 29, 2020

Fixed in #93

rollingman1 pushed a commit to rollingman1/mmpose that referenced this issue Nov 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants