This project is a real-time gesture recognition system to recognize hand gestures and position then convert them into MIDI signals. This project uses Mediapipe and openCV to detect hand gestures and Mido to convert the gestures into MIDI signals.

Note: This project is only tested on Windows using Python 3.11

• Install LoopMIDI from Tobias Erichsen and create a virtual MIDI port. Recommended to use the default name "LoopMIDI Port 1" or change the port name in the main.py.

• Install some MIDI software or VST plugin to test the MIDI signals. You can use simple one like Virtual MIDI Piano Keyboard.

• Set the virtual MIDI port as the input in the MIDI software. For VMPK, Edit > Midi Connections

• Clone this repository using the following command:

git clone https://github.com/jo0707/midiGesture

To install the required packages, you can run the following command:

pip install -r requirements.txt

Or, using virtual environment:

python -m venv venv
Set-ExecutionPolicy Unrestricted -Scope Process # for Windows (Powershell)
venv\Scripts\activate
pip install -r requirements.txt

To use the system, make sure your MIDI software is running and the virtual MIDI port is set as the input. Then, run the following command:

python main.py

The system will then start detecting index finger and thumb gestures. The gestures are as follows:

• Index finger pointing up : Play note
• Index finger touching thumb : Stop note
• Index finger bent : modulate pitch

The system is still in development (and always) and may not work as expected.