BlendArMocap is a Blender add-on to preform Hand, Face and Pose Detection using Googles Mediapipe requiring just a Webcam.
The detected data can be easily transferred to rifigy rigs.
Blender has to be started with elevated permissions in order to install the required packages opencv and mediapipe via the add-ons preferences.
Internet connection is required to install the required packages. It's recommended to disable VPN's during the installation processes.
Also Blender may has to be restarted during the installation process. To access the webcam feed blender usually has to be started with elevated permissions.
Windows
Right-click the blender application and choose: "Run as administrator"
Mac
Start Blender as admin by using the terminal:
Navigate to Blender: cd /Applications/Blender/Contents/MacOS
Run Blender as admin: sudo ./Blender
Linux
Start Blender as admin using the terminal:
Navigate to Blender: cd /usr/bin
Run Blender as admin: sudo ./blender
When running Blender as admin using sudo in the terminal, it's required to enter the admin password.
Once the add-on packages are installed and your terminal has the permission to access your camera, you can start Blender with just ./Blender.
BlendArMocap uses opencv to access the users webcam and mediapipe by google to preform hand, face and pose detection in blender.
The detected data can be used to drive a rigify rig.
Hands
- Hand rotation
- Finger x-angles
- Finger y-angles
Face
- Head rotation
- Open and close Mouth
- Relative Mouth Corners
- Open and close eyes
- Eyebrow Movement
Pose
- Hand position
- Hand orientation
- Elbow position
- Shoulder position
- Shoulder rotation
- Hip rotation
- Knee position
- Ankle position
Webcam Device Slot
If you have multiple webcam devices you may have to change the integer value until you find the device you want to use.
Defaults the Webcam Device Slot should be 0.
Key Step
The Key Step determines the frequency of Keyframes made in Blender.
Adjust the Keyframe Step so the detection results in Blender match the recording speed.
Target
Select the detection target:
- Hands
- Face
- Pose
Start Detection
When pressing the Start Detection button a window will open which contains the webcam feed and detection results.
The detection results are recorded in Blender at runtime. You can modify the recording starting point by changing the keyframe start in Blender.
May deactivate the rig while detecting if you have transferred animation results previously.
To finish the recording press 'Q', 'ESC' or the 'RIGHT MOUSE BUTTON'.
The detection results can be transferred to a generated rigify rig.
The new rigify face is not supported yet.
Drivers
Select the driver collection you want to transfer.
You can select the parent, or just the collection containing the drivers of your choice.
May not change the collection names nor the empty objects names.
Rig
Select the generated rigify rig you want to transfer to.
May not change the bone names of the rigify rig.
Start Transfer
Transfers detection results from the selected collection to the rigify rig.
Once the transfer has taken place, new recordings will be applied instantly to the rig.
So there is no need to transfer twice.
| Driver Source | Constraint type | Rigify Pose Bone |
|---|---|---|
| hip_center | copy rotation | torso |
| shoulder_center | copy rotation | chest |
| left_hand_ik | copy location | hand_ik.R |
| right_hand_ik | copy location | hand_ik.L |
| left_forearm_ik | copy location | forearm_tweak.L |
| right_forearm_ik | copy location | forearm_tweak.R |
| left_index_ik | copy location | hand_ik.R |
| right_shin_ik | copy location | shin_tweak.L |
| left_shin_ik | copy location | shin_tweak.R |
| left_foot_ik | copy location | foot_ik.R |
| right_foot_ik | copy location | foot_ik.L |
| Hand Driver Source | Constraint type | Rigify Hand Bone |
|---|---|---|
| wrist | copy rotation | hand_ik |
| thumb_cmc | copy rotation | thumb.01 |
| thumb_mcp | copy rotation | thumb.02 |
| thumb_ip | copy rotation | thumb.03 |
| thumb_tip | copy rotation | thumb.01 |
| index_finger_mcp | copy rotation | f_index.01 |
| index_finger_pip | copy rotation | f_index.02 |
| index_finger_dip | copy rotation | f_index.03 |
| index_finger_tip | copy rotation | f_index.01 |
| middle_finger_mcp | copy rotation | f_middle.01 |
| middle_finger_pip | copy rotation | f_middle.02 |
| middle_finger_dip | copy rotation | f_middle.03 |
| middle_finger_tip | copy rotation | f_middle.01 |
| ring_finger_mcp | copy rotation | f_ring.01 |
| ring_finger_pip | copy rotation | f_ring.02 |
| ring_finger_dip | copy rotation | f_ring.03 |
| ring_finger_tip | copy rotation | f_ring.01 |
| pinky_mcp | copy rotation | f_pinky.01 |
| pinky_pip | copy rotation | f_pinky.02 |
| pinky_dip | copy rotation | f_pinky.03 |
| pinky_tip | copy rotation | f_pinky.01 |
| Face Driver Source | Constraint type | Rigify Face Bone |
|---|---|---|
| head | copy rotation | head |
| chin | copy rotation | jaw_master |
| right_eye_t | copy location | lid.T.R.002 |
| right_eye_b | copy location | lid.B.R.002 |
| left_eye_t | copy location | lid.T.L.002 |
| left_eye_b | copy location | lid.B.L.002 |
| mouth_t | copy location | lip.T |
| mouth_b | copy location | lip.B |
| mouth_l | copy location | lips.R |
| mouth_r | copy location | lips.L |
| eyebrow_in_l | copy location | brow.T.L.001 |
| eyebrow_out_l | copy location | brow.T.L.003 |
| eyebrow_in_r | copy location | brow.T.R.001 |
| eyebrow_out_r | copy location | brow.T.R.003 |
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Copyright (C) cgtinker, cgtinker.com, [email protected]
If you want to support the development you can either donate at Gumroad or become a Patreon.
Would be lovely, thanks!