Code written for OpenCV during GSoC 2019 related to Facial Landmark Detection

• Description: Facial feature detection and tracking is a high value area of computer vision since humans are interested in what humans are paying attention to, feeling, and enhancing face pictures in selfies etc. At this point, we feel this should just become a standard "built in" ability that computer vision users can just call/rely on. OpenCV already has some code available on for Facial Landmark detection, see Tutorial on Facial Landmark Detector API and Tutorials for face module, but much progress has been made that we want to make available. The task is to create a Facial Landmark detector model with the following requirements:
• Expected Outcomes:
  • For mobile, make available a smaller model with lesser points ( e.g. 5 instead of 68 )
  • Run at better Accuracy than Dlib
  • Face Stabilization ( filtering and optical flow )
  • Add it to the Python API
  • Resources
  • https://docs.opencv.org/4.0.0/d5/d47/tutorial_table_of_content_facemark.html
  • https://docs.opencv.org/4.0.0/de/d27/tutorial_table_of_content_face.html
  • https://github.com/opencv/opencv_contrib/tree/master/modules/face
• Skills Required: Mastery level C++ and Python, college course work in Computer Vision that includes work on face features. Best if you have worked with deep networks before.
• Mentors: Satya Mallick
• Difficulty: Medium

• [06/04/19] Submitted the Proposal for the GSoC 2019 to OpenCV in Facial Landmark Detection.
• [07/05/19] Received the acceptance letter for the GSoC 2019 in Facial Landmark Detection project.
• [27/06/19] End of the Community Bonding Phase and Start of the Coding Phase.
• [02/06/19] Completed training the AAM model and added links to the model and the sample C++ code has been provided.
• [07/06/19] Trained the Kazemi model but the model seemes to not work as efficiently as the LBF and AAM models. May be more training is required but due to RAM limitations could not verify.
• [08/06/19] Added sample C++ code for LBF model and LBF seems to dominate the AAM model in both speed and accuracy.
• [11/06/19] Went through the implementation of the 2D-and-3D face alignment and checked if the model can be added to openCV model zoo. [Pushed to second priority as fixing the python bindings to the working LBF and AAM models is more important.]
• [15/06/19] Went through all the issues and pull requests related to the python bindings on facemark API and these are few which need to be highlighted.
  • The discussion for the issue in the python bindings started with this issue and many work arounds have been tried but many of them failed.
  • However Gusi Te proposed a method to fix the bindings and his pull request to opencv_contrib has been merged but his pull request to opencv failed as it was affecting a test file. This issue still persists as the pull to opencv failed.
  • The latest build of opencv_contrib has the same issue in the fit function and is still open to solve.
• [17/06/19] There also have been some failed attempts to expose the training code of the Facemark API to the users in python. So I am working on these two things and hope it would get completed before the first evaluation. I was able to fix the fit fucntion but drawLandmarks gave a similar error (channels() == CV_MAT_CN(dtype)) may be of because of the fix, so I am looking into it.
• [22/06/19] The python bindings for the fit function have been fixed and the changes have been merged into opencv_contrib:3.4 and soon will be merged into opencv_contrib:master as well. Exposing the training code is under process, owing to the inheritance of Kazemi from facemark and not from facemarkTrain, it is taking longer than expected time. However work is in place and proper code refactoring needs to be done before raising a PR. Expect a PR soon...