How to get aligned vertex coordinates?

[Lotayou]

Hi @akanazawa thanks for the work. I would also like to get aligned coordinates of all vertices (on 2D image plane), how can I achieve that?

I've tried a simple affine matching strategy but it won't work. The aligned verts deviate from the actual person, as shown below:

Here are the codes I use for alignment, I first save the network output in a single pickle file.

joints, verts, cams, joints3d, theta = model.predict(
            input_img, get_theta=True)

cam_for_render, vert_shifted, joints_orig = vis_util.get_original(
            proc_param, verts[0], cams[0], joints[0], img_size=img.shape[:2])
        
        pic = {
            'joints': joints_orig,
            'verts': vert_shifted,
            'cams': cam_for_render,
            'joints3d': joints3d[0],
            'theta_cam': theta[0][:3],
            'pose': theta[0][3:-10],
            'shape': theta[0][-10:]
        }
        name = _IMG.split('/')[-1].split('.')[0] + '.pickle'
        with open(os.path.join(result_path, name), 'wb') as file:
            pickle.dump(pic, file)

Then I create an affine mapping to get joints3d onto joints2d:

BLACK = [0.,0.,0.]
RED = [0., 0., 1.]

def main(img_name, pickle_name, out_name):
    im = imread(img_name)
    h,w,c = im.shape
    print(im.shape)
    with open(pickle_name, 'rb') as f:
        if sys.version_info.major == 3:
            data = pickle.load(f, encoding='latin1') # python3 read python2-created pickles
        else:
            data = pickle.load(f) # python3 read python2-created pickles
            
    '''
    for k,v in data.items():
        print(k, v.shape, v.min(), v.max())
    '''
    j3dxy = data['joints3d'][:,:2]
    j2d = data['joints']
    vertsxy = data['verts'][:,:2]
    
    # scaling and translation
    s = (j2d.max(axis=0, keepdims=True) - j2d.min(axis=0, keepdims=True)) \
        / (j3dxy.max(axis=0, keepdims=True) - j3dxy.min(axis=0, keepdims=True))
    j3dxy *= s
    vertsxy *= s
    t = j2d.max(axis=0, keepdims=True) - j3dxy.max(axis=0, keepdims=True)
    j3dxy += t
    vertsxy += t
    
    for v in vertsxy:
        x, y = int(v[0]), int(v[1])
        if x < h and y < w:
            im[y,x] = BLACK
    
    for v in j2d:
        x, y = int(v[0]), int(v[1])
        im[y-3:y+3, x-3:x+3] = RED
    
    imwrite(out_name, im)

Is there a quick workaround to get aligned vertex coordinates from renderer output? Thanks?

[akanazawa]

Hi,

I think you're almost there but off by just a small amount for not accounting for the crop/scale preprocessing or not projecting the points with the same projection function. The 2D joints that the model/demo returns is projected from the 3D joints and they align to the rendered mesh, so you can simply do the same thing, except project all of the vertices instead of just the joint3d points. This happens here:

hmr/src/RunModel.py

Line 119 in 5534c9b

pred_kp = self.proj_fn(Js, cams, name='proj_2d_stage%d' % i)

This can be tricky, since the model operates on the preprocessed image (scaled and cropped). So first, make sure that you can project the vertices just like the joints2d (pred_kp) there. Becareful bc this is originally in the normalized [-1, 1] coord system (this is where it comes back to the preprocessed image coordinate system

hmr/src/RunModel.py

Line 170 in 5534c9b

results['joints'] = ((joints + 1) * 0.5) * self.img_size

).
Then all you have to do is undo the scale and cropping.

Best,

A

[jszgz]

Hello, do you know how to use mpi_inf_3dhp_to_tfrecords.py to convert mpi_inf_3dhp dataset? I failed because the code use jpg as input but the dataset I downloaded is consisting of videos. Do I need to use ffmpeg and write code to convert avi to jpg?