- This project demonstrates a real-time hand gesture recognition system using Python, OpenCV, and Gemini AI by Google.
- The system allows users to solve math problems by drawing them in the air.
- The repository contains all the code and resources needed to implement this project.
- Real-time hand detection and tracking using OpenCV and cvzone.
- Recognition of specific hand gestures to perform actions like drawing, clearing canvas, and requesting math problem solutions.
- Integration with Gemini AI for solving math problems based on gestures.
- Python 3.10.12
- cvzone
pip install cvzone
- OpenCV
pip install opencv-python
- Gemini AI
pip install google-generativeai
- Python packages
- numpy
pip install numpy
- pillow
pip install pillow
- numpy
- C++ compiler - install Visual Studio
-
Clone the repository:
git clone https://github.com/LasithaAmarasinghe/Hand-Gesture-Math-Solver.git cd Hand-Gesture-Math-Solver
-
Create and activate a virtual environment (optional but recommended):
python -m venv venv source venv/bin/activate # On Windows use `venv\Scripts\activate`
-
Install the required packages:
pip install -r requirements.txt
-
Get a Gemini API key
You can get an API key from here
-
Replace API key in the code with the obtained API key
-
Execute the main script
python math_solver.py
-
Interact with the Application
- Live Feed
- The application will display the live feed from your webcam.
- Perform Hand Gestures
- Draw: Use one finger to draw on the canvas.
- Clear: Use one finger to clear the canvas.
- Solve Math Problem: Use all fingers except the thumb to submit a math problem for solving.
- Live Feed
-
getHandInfo(img)
- Purpose: Detects and tracks the user's hand in each frame of the webcam feed.
- Implementation: Uses the HandTrackingModule to identify and monitor hand movements.
-
draw(info, prev_pos, canvas)
- Purpose: Draws on the canvas based on the detected hand gestures.
- Parameters:
info
: Information about the detected hand gestures.prev_pos
: Previous position of the drawing.canvas
: Canvas to draw on.
-
sendToAI(model, canvas, fingers)
- Purpose: Sends the drawn content to Google's Generative AI for math problem solving based on recognized gestures.
- Parameters:
model
: Instance of the Generative AI model.canvas
: Canvas containing the drawn content.fingers
: Array indicating the state of fingers (e.g., for gesture recognition).
![](https://private-user-images.githubusercontent.com/106037441/340177476-196a14ec-6067-494c-8e06-69873fa418f3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3MjA3NzIsIm5iZiI6MTcyMDcyMDQ3MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTc3NDc2LTE5NmExNGVjLTYwNjctNDk0Yy04ZTA2LTY5ODczZmE0MThmMy5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcxMVQxNzU0MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0yODUyNzIwNzI0Yzg2ZmNlNGNlOWZmYWZmY2M5MTcwYjQxZjhkNmE3YzNhODU3MTBlYTNlOWJhMjM0NjJjMWU3JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.FsOnpkGUnplb61-hzXVZFtYZAyOcYy0wmmRJ8An022I)
Left-Right Hand Detection
![](https://private-user-images.githubusercontent.com/106037441/340176842-c0eaf562-bb89-4b47-b9f8-4bb96ec3e442.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3MjA3NzIsIm5iZiI6MTcyMDcyMDQ3MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTc2ODQyLWMwZWFmNTYyLWJiODktNGI0Ny1iOWY4LTRiYjk2ZWMzZTQ0Mi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcxMVQxNzU0MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04MDM4NmE2NTFjOWMwOGE0NWEyN2UyZTBjMzVmMTFiMTM1MjM1NzZkOTg0OGI0ZWM5NjdkZWYzNTY3ZmYxYTg1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.lW153gaQDga3ElWe57li0djPI9aHbpbLu5ju4dmFSnk)
![](https://private-user-images.githubusercontent.com/106037441/340176849-a7c1a542-8b85-42a2-87fa-bb20ff280a1e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3MjA3NzIsIm5iZiI6MTcyMDcyMDQ3MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTc2ODQ5LWE3YzFhNTQyLThiODUtNDJhMi04N2ZhLWJiMjBmZjI4MGExZS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcxMVQxNzU0MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lMWVlNjBkMzQ4Y2U3MzZkNDBmMjUwYmNhOTU2NGU2ZmRkMTc5NmMxODQ4ZDM5Y2Q0MWNlOGYxNmQ5ODBkMmNkJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.NbIsMjOCcjRSo_KA_4e1zzmS8TlQLdEw_iHMsmQT-pQ)
Hand Gesture Detection
![](https://private-user-images.githubusercontent.com/106037441/340180522-56434be0-8338-41e2-b060-5b14e69f6325.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3MjA3NzIsIm5iZiI6MTcyMDcyMDQ3MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTgwNTIyLTU2NDM0YmUwLTgzMzgtNDFlMi1iMDYwLTViMTRlNjlmNjMyNS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcxMVQxNzU0MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01NDg5MDVjMWQwOGQwOWVjZDk2OTZmNzEyNWU5ZTNhZWEzZGJkODU5NGFmZDgzOTFkZDk1Y2FkYmU5ZjkxNDA0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.3vVoTABVcYh7HwwZr5CPJaiQ3sC9LicaApdFdh2_9VI)
![](https://private-user-images.githubusercontent.com/106037441/340181207-b66a3d35-f7f7-4b75-b7db-c93ee40d6841.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3MjA3NzIsIm5iZiI6MTcyMDcyMDQ3MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTgxMjA3LWI2NmEzZDM1LWY3ZjctNGI3NS1iN2RiLWM5M2VlNDBkNjg0MS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcxMVQxNzU0MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0zN2MzOTYzZWVmMDZjMTBhNzI5NDc3ZTBkMzBmZmI5Mzg4MDJjN2M3NTQ0NGI2ZDYyMzI4MDE3N2UzNDVmMTA2JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.dJdGW7bQcmFB2aKTr_KYP6AKm8BpTFf92l3zWcZRPJQ)
Math Expression Detection & Canvas
![](https://private-user-images.githubusercontent.com/106037441/340166001-e9558800-a17f-432d-a4f5-d00d0a352cfb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3MjA3NzIsIm5iZiI6MTcyMDcyMDQ3MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTY2MDAxLWU5NTU4ODAwLWExN2YtNDMyZC1hNGY1LWQwMGQwYTM1MmNmYi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcxMVQxNzU0MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1hNDA4YzBhODNmMTlhOGNjNzMzYTc2YTRiY2VjNjcxZjk2YTc5ODU0ODk3ODc3ZTFhZTQwM2U0YzNmOTY1NTk5JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.oj7WrpbrgZ7wEd_a0vOtaNmDgYHpos4RtM7zxcDj8uc)
![](https://private-user-images.githubusercontent.com/106037441/340166035-64664179-a2cc-43a2-af2b-409a6beb4325.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjA3MjA3NzIsIm5iZiI6MTcyMDcyMDQ3MiwicGF0aCI6Ii8xMDYwMzc0NDEvMzQwMTY2MDM1LTY0NjY0MTc5LWEyY2MtNDNhMi1hZjJiLTQwOWE2YmViNDMyNS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNzExJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDcxMVQxNzU0MzJaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT04NDcwZjI2NWJjODFmYTIzMjIxZTAzMjE4ZWFkYzhjOGEzOTVkZjFmOGY1ZmJkNmVlM2U1MWE5MDVkYzlkYTU0JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.n8skSqYaa5oV5pBAgSzPwZzEz1I-CVgWk5hx_Hgkx3M)
- This project is licensed under the MIT License. See the LICENSE file for details.