This is a Unity (>= 2021.3) Native Plugin to use MediaPipe (0.10.14).
The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.
With this plugin, you can
- Write MediaPipe code in C#.
- Run MediaPipe's official solution on Unity.
- Run your custom
Calculator
andCalculatorGraph
on Unity.⚠️ Depending on the type of input/output, you may need to write C++ code.
Here is a Hello World! example.
Compare it with the official code!
using Mediapipe;
using UnityEngine;
public sealed class HelloWorld : MonoBehaviour
{
private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
calculator: ""PassThroughCalculator""
input_stream: ""in""
output_stream: ""out1""
}
node {
calculator: ""PassThroughCalculator""
input_stream: ""out1""
output_stream: ""out""
}
";
private void Start()
{
using var graph = new CalculatorGraph(_ConfigText);
using var poller = graph.AddOutputStreamPoller<string>("out");
graph.StartRun();
for (var i = 0; i < 10; i++)
{
graph.AddPacketToInputStream("in", Packet.CreateStringAt("Hello World!", i));
}
graph.CloseInputStream("in");
var packet = new Packet<string>();
while (poller.Next(packet))
{
Debug.Log(packet.Get());
}
graph.WaitUntilDone();
}
}
For more detailed usage, see the API Overview page or the tutorial on the Getting Started page.
This repository does not contain required libraries (e.g. libmediapipe_c.so
, Google.Protobuf.dll
, etc).
You can download them from the release page instead.
file | contents |
---|---|
MediaPipeUnityPlugin-all.zip |
All the source code with required libraries. If you need to run sample scenes on your mobile devices, prefer this. |
com.github.homuler.mediapipe-*.tgz |
A tarball package |
MediaPipeUnityPlugin.*.unitypackage |
A .unitypackage file |
If you want to customize the package or minify the package size, you need to build them by yourself.
For a step-by-step guide, please refer to the Installation Guide on Wiki.
You can also make use of the Package Workflow on Github Actions after forking this repository.
⚠️ libraries that can be built differ depending on your environment.
⚠️ GPU mode is not supported on macOS and Windows.
Editor | Linux (x86_64) | macOS (x86_64) | macOS (ARM64) | Windows (x86_64) | Android | iOS | WebGL | |
---|---|---|---|---|---|---|---|---|
Linux (AMD64) 1 | ✔️ | ✔️ | ✔️ | |||||
Intel Mac | ✔️ | ✔️ | ✔️ | ✔️ | ||||
M1 Mac | ✔️ | ✔️ | ✔️ | ✔️ | ||||
Windows 10/11 (AMD64) 2 | ✔️ | ✔️ | ✔️ |
This plugin implements the following MediaPipe Tasks C# APIs.
cf. The official available solutions
Solution | Android | iOS | Linux | macOS | Windows |
---|---|---|---|---|---|
LLM Inference API | |||||
Object detection | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Image classification | |||||
Image segmentation | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Interactive segmentation | |||||
Hand landmark detection | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Gesture recognition | |||||
Image embedding | |||||
Face detection | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Face landmark detection | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Face stylization | |||||
Pose landmark detection | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
Image generation | |||||
Text classification | |||||
Text embedding | |||||
Language detector | |||||
Audio classification | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
You can also use MediaPipe Framework, which allows you to run Legacy Solutions. However, please note that support for these solutions has ended.
Some solutions (including Legacy solutions) can be tested using the sample app.
Please check Assets/MediaPipeUnity/Samples/Scenes
to see which solutions have samples.
Select any scenes under Assets/MediaPipeUnity/Samples/Scenes
and play.
Select proper Inference Mode and Asset Loader Type from the Inspector Window.
If the native libraries are built for CPU (i.e. --desktop cpu
), select CPU
for inference mode.
For the libraries distributed on the release page, only the CPU is available for use on Windows and macOS.
The default Asset Loader Type is set to Local
, which only works on UnityEditor.
To run it on your devices, switch it to StreamingAssets
and copy the required resources under StreamingAssets
(if you're using MediaPipeUnityPlugin-all.zip
, the StreamingAssets
directory already contains them).
See the tutorial for more details.
https://github.com/homuler/MediaPipeUnityPlugin/wiki
Note that some files are distributed under other licenses.
- MediaPipe (Apache Licence 2.0)
- emscripten (MIT)
third_party/mediapipe_emscripten_patch.diff
contains code copied from emscripten
- FontAwesome (LICENSE)
- Sample scenes use Font Awesome fonts
See also Third Party Notices.md.
Footnotes
-
Tested on Arch Linux. ↩
-
Running MediaPipe on Windows is experimental. ↩