More details on training dataset

[haiphamcse]

Hi there, loved your work! I want to ask a bit more detail about your training dataset. From the paper it is seen that Multi-HMR uses AGORA and BEDLAM. However, how did you train Multi-HMR with the 30FPS BEDLAM or only 6FPS variant?

[fabienbaradel]

Hi @haiphamcse, that's a good question. We use the 6FPS variant which already leads to 286'228 images.

[haiphamcse]

hi there, thanks for the details. I want to follow-up with a question about Human depth estimation benchmark
From my opinion, one of the highlights of Multi-HMR is the ability to generate consistent human placement in 3D, which is quantitatively evaluated in Table 6a. However, I want to ask why you choose MRPE and PCOD as the metrics for this? (and why isn't PCOD evaluated on 3PDW and AGORA). As I'm aware that Relative Human dataset from BEV also measure this, why didn't multi-hmr do the same? Thanks for the support!

[fabienbaradel]

You are welcome. That's a good question. We will provide PCOD on 3DPW and AGORA in an updated version of our paper on arXiv. Thanks for pointing this out.
We believe that reporting this 3D metric was a better way to say that we can place humans in the 3D scene instead of reporting metrics from the Relative Human dataset where you do not have 3D GT. But this is indeed a good dataset to look at for our future experiments. Thanks!

[haiphamcse]

Hi there, thank you for the quick response. May I ask is the code for reproducing Multi-HMR quantitative results (especially human depth estimation) coming out soon? Our team is currently developing methods based on Multi-HMR but are having a hard time reproducing the results.

[fabienbaradel]

Hi @haiphamcse , we do not plan to release the evaluation code for the moment. It is possible that you get bad performance because you do not give the ground-truth camera parameters (focal length, principal point). In our demo code we set these parameters to the default value, please update them here.
What kind of depth estimation error do you get for example?
Thanks for your time,

[haiphamcse]

Hi there, sorry for the late reply. We are still currently (struggling and) adapting Multi-HMR to existing benchmarks (3DPW, AGORA, etc). One of the problems we encountered was how to obtain head box GT to get human queries. Here we tried to benchmark Multi-HMR with the provided ckpt on 3DPW and use GT primary keypoints. We obtain the following results (which is far from the reported results). P/S: we did not input the focal length into Multi-HMR

[fabienbaradel]

Hi, since Multi-HMR is taking as input the entire image you do not need to use the GT primary keypoints compared to single-erson methods. Just feed the image to Multi-HMR and it will output a list of detected persons with their associated vertices.
Run a greedy matching algorithm to match detected persons and GT persons similar to what you can find here and then compute MPJPE metrics.
Hope it helps.

[nguyenquivinhquang]

Hi, how can you evaluate the 3DPW SMPL pose? Since the multi-hmr model predicts the SMPL-X pose, I guess that you will convert the SMPL-X mesh to SMPL mesh like this here, or using any other method?

Thank you

[fabienbaradel]

Good question @nguyenquivinhquang indeed we followed this strategy.
At inference time, we use the regressor from from BEDLAM to regress the SMPL vertices from the SMPL-X mesh.
Please have a look to their repo and data.

[nguyenquivinhquang]

Hi @fabienbaradel, I noticed that you're using this code matching the predicted and ground truth poses. However, I observed that this function sets missing_punish to the ground truth pose error, which doesn't correspond to any predicted pose. Have you utilized missing_punish? If so, could you please explain why it's set to 150 and if there's any related paper that mentions this approach?

[fabienbaradel]

Hi @nguyenquivinhquang , yes we followed this evaluation protocol for CMU Panoptic similar to what ROMP/BEV are doing. I think that this way of taking missing detection into account comes from Coherent reconstruction of multiple humans from a single image, similar to what Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes - The Importance of Multiple Scene Constraints is doing in the initial paper.
Hope it helps.

[haiphamcse]

Hi there, thank you for the quick reply on all of our problems! I was wondering about section 3.1, as only AGORA has facial expression GT (alpha), how do you supervised this parameter for other dataset while training? Thanks!

[nguyenquivinhquang]

Hi @fabienbaradel, I tried implementing the evaluation code based on your suggestion. However, the result on the 3DPW test set I reproduced is as follows:

I also observed that the person's height from the model is lower than the person's height from the ground truth.

I hope that you can release the validation code for 3DPW or provide some suggestions for improving the accuracy.

[fabienbaradel]

Hi @nguyenquivinhquang , thanks for sharing the table. Please take into account that results in the paper are obtained after fine-tuning on the 3DPW training data. The first line that you share (Multi-HMR (pretrained) wo.punish) is already giving decent results, the PA-MPJPE is good and corresponds to what we get w/o finetuning on 3DPW-train. However the MPJPE and PVE are a bit higher than what we get.
Do you take into account the gender (male/female) for generating the human mesh ground-truth in 3DPW? It is impacting a lot the GT compared to a Neutral SMPL model.
Moreover you say that 2366 humans are missing (missing objects), this number seems to be high. What is the recall that you get on 3DPW? It should be around 99%.
Hope it helps,

[fabienbaradel]

Hi @nguyenquivinhquang @haiphamcse , I just want to let you know that today we are releasing the training and evaluation code of Multi-HMR 😄
We are providing the training code for training on BEDLAM-train and evaluating on 3DPW-test, BEDLAM-validation and EHF.
It think that this code release is solving the issue that you encounter in this issue.
I am closing this issue but feel free to re-open it if needed.

[nguyenquivinhquang]

Thank you for releasing the training and evaluation code. Also, congratulations on having your paper accepted to ECCV 😄

[fabienbaradel]

Thank you @nguyenquivinhquang 😃

[nguyenquivinhquang]

Hi there,

Could you release the dataloader for the MuPoT and CMU datasets? Moreover, I tried to run the evaluation code with checkpoint 672L, and the result was quite different from the number you reported in the paper, with PA-MPJPE being 61.9. Therefore, could you release the PA-MPJPE evaluation code as well?

Thank you.

[fabienbaradel]

Hi @nguyenquivinhquang , we do not plan to release the dataloader for MuPoTS and CMU for the moment, maybe in a near future. I will let you know.

Which joints are you using for computing these metrics? You should not take the SMPL joints but rather take the H36M joints maybe the error come from here. I will try to add this metrics and the associated PA soon.

We will update the paper soon on arXiv with updated number specially about the universal model that we are releasing, stay tuned.

[fabienbaradel]

Hi @nguyenquivinhquang , I did few edits to add MPJPE and PA-MPJPE computed on H36M-14 joints.
Please have look here for downloading the appropriate regressor.

[nguyenquivinhquang]

Sorry for the late response; I have rerun and got the result as close as in the paper. Thank you very much 😄

[fabienbaradel]

great news! 😄

[DavidBlack-cmu]

Hi there, I tried to run the evaluation code with your checkpoint multiHMR_896_L, but the result was different from what I expected. Comparing it to the universal model reported in the paper, the MPJPE metric seems to differ. Could you help me understand why this is happening??
.

[fabienbaradel]

Hi @DavidBlack-cmu , PA-MPJPE and PVE are on the same range but indeed the MPJPE is different. Sorry about that, I will update the paper with new numbers.
Thanks for pointing this issue 😄