Digihuman is a project which aims to automatically generate whole body pose animation + facial animation on 3D Character models based on the camera input.
This project is my B.Sc thesis of Computer Engineering at Amikabir University of Technology(AUT).

DigiHuman is a system for bringing automation in animation generation on 3D virtual characters. It uses Pose estimation and facial landmarks generator models to create entire body and face animation on 3D characters.
This project is done with MediaPipe and Unity3D. MediaPipe generates 3D landmarks for the human whole body and face, and Unity3D is used to render the final animation after processing the generated landmarks from MediaPipe.

Follow the instruction to run the program!

1. Install MediaPipe and its dependencies based on your Operating System: link
2. Go to backend directory and install GauGan requirements:

 pip install -r requirements.txt

3. You'll need to download the pretrained generator model for the COCO dataset into backend/checkpoints/coco_pretrained/.

Install Unity3D and its requirements by the following guide lines. (Skip if you have Unity3D Installed)

1. Download and install UnityHub
2. Add a new licence in UnityHub and register it on Unity website
3. Install a Unity Editor inside UnityHub(Better to be LTS and a version higher than 2020.3.25f1.
4. Download default 3D character and place it under Assets\Models sdirectory.(build this directory first, if it does not exist)
5. Download and import the following packages into your project for making recording option available with FFmpeg.(Download .unitypackage files and drag them to your project)

• FFmpegOut package (MIT license)
• FFmpegOutBinaries package (GPL)

• Run backend server at backend directory with the following command:

   python server.py
  

• Run Unity Project and open the main scene at Assets\Scenes\Main.unity
• Test the program by uploading videos to backend from Unity project.

You can add your own characters to the project! Characters should have standard Humanoid rig to show kinematic animations. For rendering face animations, hcaracters should have blendmesh for their face. Follow these steps to add your character:

1. Find a 3D character model from Unity asset store or download a free one.(You can download them from websites like Mixamo)
2. Open the character setting and set the rig to humanoid
3. Drag and drop your 3D charcter model to CharacterChooser/CharacterSlideshow/Parent object in Unity main Scene like the image below
4. Add BlendShapeController and QualityData components to character object in scene(which is dragged inside Parent object in last step).
5. Open CharacterSlideshow Object at CharacterChooser/CharacterSlideshow path inside scene hierachy, then add new dragged character to the nodes property(all characters should be refrenced inside nodes).
6. Set BlendShapeController values

• Set character SkinnedMeshRenderer component to BlendShapeController component.
• Find each blnedshape weight number under SkinnedMeshRenderer and set those numbers in BlendShapes field inside BlendShapeController (for specifying each blendshape value to the BlendShapeController component so animation would be shown on character face by modification on these blendhape values)

7. Run the application and you can now select your character for showing animation!

• FFmpeg is licensed under the GNU Lesser General Public License (LGPL) version 2.1 or later. However, FFmpeg incorporates several optional parts and optimizations that are covered by the GNU General Public License (GPL) version 2 or later. If those parts get used the GPL applies to all of FFmpeg.
• Unity FFmpeg packages are licenced under Keijiro Takahashi MIT

• Used SPADE repository developed by NVIDIA

@inproceedings{park2019SPADE,
  title={Semantic Image Synthesis with Spatially-Adaptive Normalization},
  author={Park, Taesung and Liu, Ming-Yu and Wang, Ting-Chun and Zhu, Jun-Yan},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2019}
}