Use your webcam to control a cartoon version of your face. Open your mouth to hurl cartoon objects in the direction you are facing. Do it alone, or enjoy hours of fun with the whole family.

I built this project as a learning excercise to understand how to work with video and images containing faces.

Key ingredients include:

• face_recognition: an excellent Python library for all things faces. Built on top of the powerful dlib toolkit.
• OpenCV: a vision programmer's best friend for working with images and video.
• Google's Quickdraw dataset: the source of images used for the eyes, nose, and mouth-spewed objects.

While the program is running, click to give it focus, and then press these keys for functionality:

• 'v': toggle video background, vs. white background.
• 'p': toggle showing head pose estimate as a blue line, as well as the 6 face features used to compute it.
• 's': render eyes and nose as a cartoon sketch, or not.
• 'q': quit and cleanup.

Type python sketch_face.py to see command line options for the same as above.

In a nutshell:

1. In a loop, read frames from the camera, scale them down for speed, and process them as follows:
2. Detect points on a face by calling face_recognition.get_landmarks. This returns 68 points categorized by left_eye, nose_tip, etc.
3. Find the centers of the eyes and nose, and replace them with random Quickdraw cartoons of eyes and noses. Render the rest as lines.
4. If the mouth appears to be open (crudely determined by looking at the ratio of height to width), then estimate head pose using the given landmarks, and create random Quickdraw sprites moving in that direction.
5. Check for key presses and toggle various rendering options as described above.

The provided Dockerfile makes installation and running easy (but doesn't work on Mac, read notes below):

git clone https://github.com/goberoi/sketch_face.git
docker build -t goberoi/sketch_face .
./run.sh

Notes:

• Mac users, warning: Docker for Mac does not pass through USB devices to containers so you'll have to find another way (workarounds do exist but they are hairy).
• The above has been tested on Ubuntu 16.04, but not Windows, or Mac.
• The run.sh bash script calles docker run with several parameters for sharing webcam, display, etc.

I relied on lots of quality advice and code to build this:

• This video on Instagram inspired the project idea.
• The face_recognition library's demo code is a great starting point.
• Very useful info on how to increase FPS by putting camera reading IO in another thread.
• Head pose estimation code that I shamelessly 'borrowed'.