Issues training with CMU_Panoptic

[mkhoshle]

Hello,

1. I am trying to train the model starting from pretrained resent on the cmu_panoptic dataset. However, I get the following error:

Traceback (most recent call last):
  File "HumanObj_videos_ResNet/train.py", line 277, in <module>
    main()
  File "HumanObj_videos_ResNet/train.py", line 273, in main
    trainer.train()
  File "HumanObj_videos_ResNet/train.py", line 77, in train
    self.train_epoch(epoch)
  File "HumanObj_videos_ResNet/train.py", line 192, in train_epoch
    for iter_index, meta_data in enumerate(self.loader):
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/z/home/mkhoshle/env/romp2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/z/home/mkhoshle/Human_object_transform/HumanObj_videos_ResNet/lib/dataset/mixed_dataset.py", line 79, in __getitem__
    annots = self.datasets[dataset_id][index_sample]
  File "/z/home/mkhoshle/Human_object_transform/HumanObj_videos_ResNet/lib/dataset/image_base.py", line 375, in __getitem__
    return self.get_item_single_frame(index)
  File "/z/home/mkhoshle/Human_object_transform/HumanObj_videos_ResNet/lib/dataset/image_base.py", line 123, in get_item_single_frame
    kp3d, valid_masks[:,1] = self.process_kp3ds(info['kp3ds'], used_person_inds, \
  File "/z/home/mkhoshle/Human_object_transform/HumanObj_videos_ResNet/lib/dataset/image_base.py", line 284, in process_kp3ds
    kp3d_processed[inds] = kp3d
ValueError: could not broadcast input array from shape (17,3) into shape (54,3)

Do you know what I need to do to avoid this error?

2. Also, does the cmu_panoptic have the 2d pose annotation for all the people appearing in every image?

I would appreciate it if you could help me with this,
Thanks,

[Arthur151]

Hi，@mkhoshle,
To prepare a dataloader for a dataset, please refer to

ROMP/romp/lib/dataset/MuPoTS.py

Line 48 in f3c7672

kp3d = self.map_kps(kp3d/1000.,maps=self.kp3d_mapper)

Please note that, via function self.map_kps, we tranfer the different skeleton defination of each dataset (e.g. 17 joints in CMU P) to a unified format (our 54 joints).
The mapping is achieved via:

ROMP/romp/lib/dataset/MuPoTS.py

Line 22 in f3c7672

self.kp3d_mapper = constants.joint_mapping(constants.MuPoTS_17, constants.SMPL_ALL_54)

The skeleton of each dataset is defined in
https://github.com/Arthur151/ROMP/blob/master/romp/lib/constants.py

You can define the skeleton of CMU P as CMUP_17, for example,

CMUP_17 = {'R_Ankle':0 ....}

Our 54 joints are defined as SMPL_ALL_54,

ROMP/romp/lib/constants.py

Line 45 in f3c7672

SMPL_ALL_54 = {**SMPL_24, **SMPL_EXTRA_30}

Then the mapping would be
self.kp3d_mapper = constants.joint_mapping(constants.CMUP_17, constants.SMPL_ALL_54)

[mkhoshle]

@Arthur151 Thanks so much for your feedback. I looked at the cmu_panoptic website. The skeleton has 19 key_points. How do I know which points need to be removed? And also how do I know their connection? I am unsure where to get this information and would appreciate your feedback.
I looked into the annotation file and it has the (['filename', 'width', 'height', 'bboxes', 'kpts2d', 'kpts3d']) . I could not see any info on the keypoints.

Thanks,

[Arthur151]

Hi, @mkhoshle ,
You just need to clear the name of each joint in their order in 'kpts3d', at constants.py like

ROMP/romp/lib/constants.py

Line 133 in f3c7672

Panoptic_19 = {

Panoptic_19 = {
'Neck':0, 'Nose':1, 'Pelvis':2, 'L_Shoulder':3, 'L_Elbow':4, 'L_Wrist':5, 'L_Hip':6,
'L_Knee':7, 'L_Ankle':8, 'R_Shoulder': 9, 'R_Elbow':10, 'R_Wrist':11, 'R_Hip':12, 'R_Knee':13, 'R_Ankle':14,
'L_Eye':15, 'L_Ear':16, 'R_Eye':17, 'R_Ear':18}

Then you can map these joints into our unfied 54 format via
mapping = constants.joint_mapping(constants.Panoptic_19, constants.SMPL_ALL_54)
kps = joint_org[mapping].copy()
kps[mapping==-1] = -2.
The missing joints would be filled with -2. During training, the missing joints would be overlooked.

[mkhoshle]

Hi, @Arthur151 thanks so much for the Feedback. I think you are misunderstood.

• The Kp3d has 17 joints in the annotation file for cmu panoptic. The constant.py has Panoptic_19 and Panoptic_15. I think I need to define Panoptic_17 so that I can do:

self.kp3d_mapper = constants.joint_mapping(constants.Panoptic_17, constants.SMPL_ALL_54) 

My question is that what joints need to be removed from Panoptic_19? I am not sure which joints can be removed.

• Also in the paper you mentioned that not every image has annotation for 2d pose and this is almost the case for any 2d pose dataset. Is this the case for 3d pose datasets as well, such as cmu_panoptic?

Thanks,

[Arthur151]

I am not sure. Maybe this function

ROMP/romp/lib/visualization/visualization.py

Line 310 in f3c7672

def draw_skeleton(image, pts, bones=None, cm=None, label_kp_order=False,r=3):

setting label_kp_order=True, would be helpful.
Using to set up the dataset loader, you can use debug function at

ROMP/romp/lib/dataset/image_base.py

Line 700 in f3c7672

image_kp2d_projection_smpl24 = draw_skeleton_multiperson(render_img.copy(), pred_keypoints_2d, bones=bones[:23], cm=cm[:23], label_kp_order=True)

to check the keypoints.

Yes, all multi-person datasets have this problem, more or less, including cmu_panoptic.

[mkhoshle]

@Arthur151 I am doing the following:

Panoptic_14 = {
    'Neck':0, 'Pelvis':1, 'L_Shoulder':2, 'L_Elbow':3, 'L_Wrist':4, 'L_Hip':5, \
    'L_Knee':6, 'L_Ankle':7, 'R_Shoulder':8, 'R_Elbow':9, 'R_Wrist':10, 'R_Hip':11, 'R_Knee':12, 'R_Ankle':13 \
    }
joint_mapper = constants.joint_mapping(constants.Panoptic_14,constants.SMPL_ALL_54)

J24_TO_H36M = np.array([14, 3, 4, 5, 2, 1, 0, 16, 12, 17, 18, 9, 10, 11, 8, 7, 6])
H36M_TO_LSP = J24_TO_H36M[np.array([6, 5, 4, 1, 2, 3, 16, 15, 14, 11, 12, 13, 8, 10])]
print(H36M_TO_LSP.shape, J24_TO_H36M.shape)

full_kp2d = []
for inds, (kp2d, kp3d) in enumerate(zip(kp2ds, kp3ds)):
    invis_kps = kp2d[:,-1]<0.1
    kp2d *= 1920./832.
    kp2d[invis_kps] = -2.
    kp2d = map_kps(kp2d[H36M_TO_LSP], maps=joint_mapper)
    kp2d[constants.SMPL_ALL_54['Head_top']] = -2.
    full_kp2d.append(kp2d)

bones, cm = constants.All54_connMat, constants.cm_All54

img_path = path+'/images/'+img_name
img = cv2.imread(img_path)
image_kp2d = draw_skeleton_multiperson(img.copy(), full_kp2d[:1], bones=bones, cm=cm, label_kp_order=True)

Here is how the skeleton looks like:

I am a bit confused and am not sure what is it that I am doing wrong.
Also in visible_person_id, kp2ds = determine_visible_person(annots[img_name]['kpts2d'], annots[img_name]['width'], annots[img_name]['height']), visible_person_id = array([0, 1, 2]) meaning there are three people in the image while there are two. Could you please help me understand this? Is not this wrong?

Thanks,

[Arthur151]

@mkhoshle , sorry for the late reply. I was caught by two deadlines.

If you are processing the original ground-truth 3D human pose of CMU Panoptic, then you might use the wrong order of keypoints, this is want I used:
Panoptic_19 = {
'Neck':0, 'Nose':1, 'Pelvis':2, 'L_Shoulder':3, 'L_Elbow':4, 'L_Wrist':5, 'L_Hip':6,
'L_Knee':7, 'L_Ankle':8, 'R_Shoulder': 9, 'R_Elbow':10, 'R_Wrist':11, 'R_Hip':12, 'R_Knee':13, 'R_Ankle':14,
'L_Eye':15, 'L_Ear':16, 'R_Eye':17, 'R_Ear':18}

With it, you can just:
joint_mapper = constants.joint_mapping(constants.Panoptic_19,constants.SMPL_ALL_54)
map_kps(kp2d, maps=joint_mapper)

[mkhoshle]

@Arthur151 I am using the same. But there are some issues. First, when I plot 2d key points taken from annots in your google drive and plot it on top of the image I see the following. Have you happened to see this before?

As you see there is an extra key point that does not make sense. As a result, when I plot the skeleton it looks like the following:

Here is my code:

joint_mapper = constants.joint_mapping(constants.Panoptic_19,constants.SMPL_ALL_54)

full_kp2d = []
for inds, (kp2d, kp3d) in enumerate(zip(kp2ds, kp3ds)):
    invis_kps = kp2d[:,-1]<0.1
    kp2d *= 1920./832.
    kp2d[invis_kps] = -2.
    kp2d = map_kps(kp2d, maps=joint_mapper)
    kp2d[constants.SMPL_ALL_54['Head_top']] = -2.
    full_kp2d.append(kp2d)

Any thoughts?