Note: This is still Work In Progress!
Java
and Tensorflow
implementation of the MTCNN Face Detector. Based on David Sandberg's FaceNet's MTCNN
python
implementation and the original Zhang, K et al. (2016) ZHANG2016 paper and Matlab implementation.
It reuses the
PNet
, RNet
and ONet
Tensorflow models build in FaceNet's MTCNN and
initialized with the original weights. Here
you can find how to freeze the TF models.
Note that the required Tensorflow models are already pre-bundled with this project! No need to download or freeze those by yourself.
The MTCNN technique involves significant amount of linear algebra operations, such as multi-dimensional array computations and so.
Therefore the ND4J library is leveraged for implementing all (pre)processing steps
required for flowing the data through the PNet
, RNet
and ONet
Tensorflow networks. Furthermore JavaCV is
leveraged for image manipulation and the ND4J-Tensorflow
GraphRunner is used to inferring the pre-trained tensorflow models.
The combination of those libraries allows to exchange processing states between the ND4J
, JavaCV
and Tensorflow
with little data churn.
It also provides off-heap memory management and native support for GPU and BLAS processor features.
NOTE: You can find the original NumPy/Python code snippets as inline comments in front of the equivalent ND4J java implementations.
The FaceDetectionSample1.java demonstrates how to use MtcnnService
for detecting faces in images.
Here is the essence this sample:
// 1. Create face detection service.
MtcnnService mtcnnService = new MtcnnService(30, 0.709, new double[] { 0.6, 0.7, 0.7 });
try (InputStream imageInputStream = new DefaultResourceLoader() .getResource("classpath:/pivotal-ipo-nyse.jpg").getInputStream()) {
// 2. Load the input image (you can use http:/, file:/ or classpath:/ URIs to resolve the input image
BufferedImage inputImage = ImageIO.read(imageInputStream);
// 3. Run face detection
FaceAnnotation[] faceAnnotations = mtcnnService.faceDetection(inputImage);
// 4. Augment the input image with the detected faces
BufferedImage annotatedImage = MtcnnUtil.drawFaceAnnotations(inputImage, faceAnnotations);
// 5. Store face-annotated image
ImageIO.write(annotatedImage, "png", new File("./AnnotatedImage.png"));
// 6. Print the face annotations as JSON
System.out.println("Face Annotations (JSON): " + new ObjectMapper().writeValueAsString(faceAnnotations));
}
It takes an input image detect the faces, produces json annotations and augments the image with the faces.
The face annotation json format looks like this:
[ {
"bbox" : { "x" : 331, "y" : 92, "w" : 58, "h" : 71 }, "confidence" : 0.9999871253967285,
"landmarks" : [ {
"type" : "LEFT_EYE", "position" : { "x" : 346, "y" : 120 } }, {
"type" : "RIGHT_EYE", "position" : { "x" : 374, "y" : 119 } }, {
"type" : "NOSE", "position" : { "x" : 359, "y" : 133 } }, {
"type" : "MOUTH_LEFT", "position" : { "x" : 347, "y" : 147 } }, {
"type" : "MOUTH_RIGHT", "position" : { "x" : 371, "y" : 147 },
} ]
}, { ...
The spring-cloud-starter-stream-processor-face-detection-mtcnn is Spring Cloud Data Flow processor that allows detecting and faces in real time from input image or video streams.
Use the following dependency to add the mtcnn
utility to your project
<dependency>
<groupId>net.tzolov.cv</groupId>
<artifactId>mtcnn</artifactId>
<version>0.0.4</version>
</dependency>
Also register jcentral
to your list of maven repository (it is available out of the box for Gradle).
<repositories>
<repository>
<id>jcenter</id>
<url>https://jcenter.bintray.com/</url>
</repository>
</repositories>
Use this MtcnnServiceBenchmark to perform some basic benchmarking. You can change the image URI to test the performance with different images.
The ND4J, DataVec, ND4J-Tensorflow and JavaCV all rely on core libraries written in C++ (for performance reasons) and wrapped with JNI (e.g JavaCPP) wrapper. This approach has many great advantages such as support for GPU and BLAS math features, off-heap data management, low latency minimal data churn. Still the approach apparently can induce significant jar footprint. By default if you bundle support for OS platforms (e.g. linux, android, windows, macos ..) the target Spring Boot jar can grow to the insane 1GB jar footprint!
If you know what your target platform is going to be then you can remedy this situation by setting the -Djavacpp.platform=
property. For example -Djavacpp.platform=macosx-x86_64
for MacOS target platform.
Another possible solution might be to try to substitute the ND4J, DataVec and JavaCV stack using the newly released []Tensorflow Java Ops API (ver. 1.10+)](https://github.com/tensorflow/tensorflow/releases/tag/v1.10.0) later appear to have the superset of what above stack provides but in a single C++ library with much small footprint.