Real time 3D hand pose estimation using MediaPipe

This is a demo on how to obtain 3D coordinates of hand keypoints using MediaPipe and two calibrated cameras. Two cameras are required as there is no way to obtain 3D coordinates from a single camera. Check here: stereo calibrate for a calibration package. Also my blog post on how to stereo calibrate two cameras: link. Alternatively, follow the camera calibration at Opencv documentations: link. If you want to know some details on how this code works, take a look at my accompanying blog post here: link.

MediaPipe
Install mediapipe in your virtual environment using:

pip install mediapipe

Requirements

Mediapipe
Python3.8
Opencv
matplotlib

Usage: Getting real time 3D coordinates
As a demo, I've included two short video clips and corresponding camera calibration parameters. Simply run as:

python handpose3d.py

If you want to use webcam, call the program with camera ids. For example, cameras registered to 0 and 1:

python handpose3d.py 0 1

Make sure the corresponding camera parameters are also updated for your cameras.

The 3D coordinate in each video frame is recorded in frame_p3ds parameter. Use this for real time application. If keypoints are not found, then the keypoints are recorded as (-1, -1, -1). Warning: The code also saves keypoints for all previous frames. If you run the code for long periods, then you will run out of memory. To fix this, remove append calls to: kpts_3d, kpts_cam0. kpts_cam1. When you press the ESC key, hand keypoints detection will stop and three files will be saved to disk. These contain recorded 2D and 3D coordinates.

Usage: Viewing 3D coordinates
The handpose3d.py program creates a 3D coordinates file: kpts_3d.dat. To view the recorded 3D coordinates, simply call:

python show_3d_hands.py