JP7385416B2 - Image processing device, image processing system, image processing method, and image processing program - Google Patents

Description

The present invention relates to an image processing device, an image processing system, an image processing method, and an image processing program.

BACKGROUND ART Conventionally, a method has been proposed in which an image of a person is analyzed by an image processing device and the face of the person is detected from the image.

For example, Patent Document 1 discloses a face detection device that reduces an input image in stages and performs face detection on the reduced image by template matching.

Further, Non-Patent Document 1 discloses one of the methods of detecting an object from an image using a neural network.

JP2008-257321A

Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, "Feature Pyramid Networks for Object Detection," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2117-2125

However, in the face detection device described in Patent Document 1, face detection processing must be performed at all set stages, resulting in a long processing time. Furthermore, there is a possibility that many faces of a size that the user of the device considers unnecessary to be detected may be detected.

Furthermore, although it is possible to detect faces using the method described in Non-Patent Document 1, if the range of detectable face sizes is expanded too much, a complex network may be required to ensure detection accuracy. This increases the time cost for both learning and detection, and increases the required resources, leading to an increase in economic costs.

The present invention has been made in view of the above-mentioned current situation, and provides an image processing device, an image processing system, an image processing method, and an image processing program that can efficiently detect a face of a size requested by a user. The purpose is to

In order to solve the above-mentioned problems and achieve the objects, the present invention provides an image processing device, the image processing device comprising: a face detection section capable of detecting a face having a size within a predetermined range from an image; a storage unit that stores a set range set as a size range of a face to be detected from an input image; and a scaled-down image that is an image in which the resolution of the input image is changed based on the predetermined range and the set range. and a resolution changing unit that generates a resolution change unit, and the face detection unit detects a face in the enlarged/reduced image.

Further, in the above invention, the resolution changing unit detects at least one of a face having a size corresponding to the maximum value of the setting range and a face having a size corresponding to the minimum value of the setting range. The enlarged/reduced image is generated by changing the resolution of the input image so that the portion can be detected.

Further, in the above invention, the present invention provides that the resolution changing section lowers the resolution of the input image so that the face detecting section can detect a face having a size related to at least the maximum value of the setting range. The enlarged/reduced image is generated by doing the following.

Further, the present invention is characterized in that, in the above-mentioned invention, the face detection section further detects a face in the input image depending on the minimum value of the setting range.

Further, in the above invention, when the enlargement/reduction ratio of the enlarged/reduced image with respect to the input image is X%, the minimum value of the predetermined range is Min1, and the minimum value of the set range is Min2, If the relationship (Min2×X/100)<Min1 is satisfied, the face detection unit further detects a face in the input image.

Further, in the above invention, the present invention provides a first region in the input image corresponding to a face region detected in the enlarged/reduced image is a face region detected in the input image. If the first area and the second area overlap, the face detection unit outputs information related to one of the faces as a detection result depending on the degree of overlap between the first area and the second area. .

Further, in the present invention, in the above invention, the face detection unit further outputs a score related to the face-likeness of the detected face, and the first area overlaps the second area, and the face detection unit further outputs a score related to the face-likeness of the detected face, When outputting information related to one face, information related to the face with a higher face-likeness score is output as a detection result.

Further, in the present invention, in the above invention, the image processing device further includes a face authentication unit that performs authentication processing based on the detection result of the face detection unit, and the first area overlaps the second area, When the face detection section outputs information related to the face detected in the enlarged/reduced image as a detection result, the face authentication section outputs information related to the face detected in the enlarged/reduced image. The present invention is characterized in that authentication processing is performed based on an image area of the input image specified from the coordinates of the input image.

Further, the present invention is characterized in that, in the above invention, the image processing device further includes a face authentication section that performs authentication processing based on the detection result of the face detection section.

Further, in the above invention, when at least a part of the predetermined range exceeds the set range and the face detection unit detects a face having a size outside the set range, the face detection unit The section is characterized in that it does not output information related to a face having a size outside the set range as a detection result.

Further, in the above invention, the present invention is characterized in that the face detection unit detects a plurality of feature points of a face included in the image, and calculates the size of the face based on the plurality of feature points. shall be.

Further, in the present invention, in the above invention, the face detection unit includes a plurality of face detectors having different memory capacities, and the memory capacities of the plurality of face detectors are set according to the plurality of image sizes. It is characterized by being

Further, in the above invention, the present invention provides that the image processing device adjusts at least one part of the periphery of the enlarged/reduced image so that the size of the enlarged/reduced image matches the closest image size among the plurality of image sizes. The image size adjusting section adds a background image to the image size adjusting section, and the face detecting section detects a face on the enlarged/reduced image to which the background image is added.

The present invention also provides an image processing system, the image processing system comprising: a face detection unit capable of detecting a face within a predetermined size range from an image; and a face size range to be detected from an input image. a storage unit that stores a setting range set as , and a resolution changing unit that generates an enlarged/reduced image that is an image with the resolution of the input image changed based on the predetermined range and the setting range, The face detection unit is characterized in that it detects a face in the enlarged/reduced image.

The present invention also provides an image processing method for detecting a face from an image, which includes the steps of: storing a setting range set as a range of the size of a face to be detected from an input image; and a step of generating a scaled-down image that is an image with a changed resolution of the input image based on a predetermined range that is a range of It is characterized by containing.

The present invention also provides an image processing program, wherein the image processing program detects a face from an input image by causing a computer to detect a face within a predetermined range of detectable face sizes and from an input image. means for generating an enlarged/reduced image that is an image obtained by changing the resolution of the input image based on a setting range set as a range of the desired face size; and means for detecting a face in the enlarged/reduced image; It is characterized by functioning as

According to the image processing device, image processing system, image processing method, and image processing program of the present invention, it is possible to efficiently detect a face of a size requested by a user.

2 is a schematic diagram for explaining an overview of a face detection method in Embodiment 1. FIG. 1 is a block diagram illustrating the overall configuration of a system including an image processing device according to a first embodiment. FIG. 1 is a block diagram illustrating the configuration of an image processing apparatus according to a first embodiment. FIG. 3A and 3B are schematic diagrams for explaining an example of a setting range setting method by a setting unit, in which (a) shows a method of capturing an image of a user with a camera, and (b) shows a method of capturing an image of a user displayed on a monitor. Shows how to set the setting range from the face. FIG. 7 is a schematic diagram for explaining another example of a method of setting a setting range by a setting unit, and shows a method of setting a setting range from a face model displayed on a monitor. FIG. 2 is a schematic diagram showing feature points and bounding boxes of a face detected by a face detection unit. FIG. 2 is a schematic diagram showing the correspondence between a face area (bounding box) detected by a face detection unit in a scaled-down image and a face area (bounding box) detected by the face detection unit in an input image. . FIG. 2 is a schematic diagram showing an example of a procedure of processing, mainly face detection processing, performed by the image processing apparatus according to the first embodiment. 2 is a flowchart illustrating an example of a process performed by the image processing apparatus according to the first embodiment, mainly a procedure of face detection processing. FIG. 2 is a schematic diagram illustrating the overall configuration of an image processing system according to a modified embodiment.

Hereinafter, preferred embodiments of an image processing device, an image processing system, an image processing method, and an image processing program according to the present invention will be described with reference to the drawings. In the present invention, specific parameters indicating the size of the face (parameters for specifying the predetermined range and setting range) are not particularly limited, and for example, the distance between the left and right eyes (hereinafter also simply referred to as the distance between the eyes) is not particularly limited. ), the distance from the center of the left and right eyes to the mouth, the distance from the tip of the chin to the top of the head, the distance between the left and right ears, etc. Below, we will discuss examples using the distance between the eyes. explain.

<Overview of this embodiment>
First, an overview of the face detection method in the first embodiment will be explained. In this embodiment, a predetermined range of face sizes (for example, distance between eyes of 8 to 200 pixels) that can be detected by a face detector (face detection engine) composed of a machine-learned inference model and a face that is desired to be detected Change the size of the input image (enlarge or reduce, for example, change the size to 200/400 pixels) according to the setting range set by the user as the size of the face (any setting value, for example, the distance between the eyes is 10 to 400 pixels). A scaled image that is a compressed image is generated, and a face is detected based on the scaled image. This makes it possible to detect a face of a size (inter-eye distance) that cannot be handled by a face detector alone from an enlarged/reduced image. A more detailed explanation will be given below using FIG. 1.

First, as a preliminary process, a face detector is created by machine learning (preferably deep learning) that can detect faces within a predetermined size range (eg, distance between eyes of 8 to 200 pixels) from an input image. Then, in initialization processing during operation, etc., the created face detector is read into the storage unit as a plurality of face detectors having different memory capacities (sizes). Here, the memory capacity of each face detector is set according to a standard size image such as VGA, Full HD, 4K, or any other arbitrary image size.

In the face detection process, as shown in Figure 1, the face detector can detect the face according to the setting range that indicates the size of the face that the user wants to detect (for example, the distance between the eyes). Determine the appropriate image resolution (size) and change the input image resolution (size). That is, the input image is enlarged or reduced so that the face size can be detected by the face detector, and an enlarged/reduced image is generated.

More specifically, for example, the predetermined range of the eye distance that each face detector can detect is 8 to 200 pixels, and the eye distance that the user has set as the size of the face to be detected from the input image. The setting range is 8 to 400 pixels. In this case, each face detector cannot detect a face with an inter-eye distance of 200 to 400 pixels from the input image. Therefore, an enlarged/reduced image is generated by reducing the input image to 50% of the size while maintaining its aspect ratio. Even in this enlarged/reduced image, the distance between the eyes that each face detector can detect is 8 to 200 pixels, but since this enlarged/reduced image is half the size of the input image, the size of the input image is twice that. Faces with an eye-to-eye distance of 16 to 400 pixels will be detected. Therefore, each face detector can detect a face whose inter-eye distance is 200 to 400 pixels when converted to the size of the input image from the enlarged/reduced image.

However, in this case, with the enlarged/reduced image, it is not possible to detect faces whose distance between the eyes is less than 16 pixels when converted to the size of the input image, and it is difficult to detect faces whose size is around the minimum value of the setting range. I can't. Therefore, in this embodiment, face detection is performed not only on enlarged/reduced images but also on input images. Each face detector can detect faces with an inter-eye distance of 8 to 200 pixels from the input image.

In this way, faces with an inter-eye distance of 16 to 400 pixels in terms of the size of the input image are detected from the enlarged/reduced image, and faces with an inter-eye distance of 8 to 200 pixels are detected from the input image. This makes it possible to detect faces of all sizes within the set range (inter-eye distance of 8 to 400 pixels).

At this time, which face detector is used for each of the enlarged/reduced image and the input image is determined according to the size of the image to be detected. Select the one whose memory capacity is closest to the size of the image to be detected (but larger than the image size).

In addition, as shown in Figure 1, if there are multiple detection results, that is, if a face is detected in an overlapping area by multiple face detectors, the detection results are integrated according to the degree of overlap to obtain the final result. .

According to the present embodiment, the resolution (size) of the input image is changed based on a predetermined range that can be handled by the face detector and a setting range desired by the user, and a scaled-down image is generated. Since faces are detected in images, it is possible to efficiently detect faces of the size requested by the user.

<Overall system configuration including image processing device>
Next, the overall configuration of a system including the image processing apparatus according to the first embodiment will be described using FIG. 2. As shown in Fig. 2, this system is constructed to target a predetermined area indoors or outdoors (for example, an outdoor aisle or inside a store), and includes a camera 1 that functions as an image acquisition unit and an image recording unit. The device includes a recorder (image recording device) 2 that functions as a face detection unit and an image processing device 3 that functions as a face detection unit and a face authentication unit. The predetermined area can be set as appropriate depending on the purpose.

The camera 1 is installed at a desired location indoors or outdoors. A camera 1 photographs a predetermined area indoors and outdoors. The photographed video (each frame) is stored in the recorder 2.

The image processing device 3 includes a monitor (display device) 4 as a display unit that displays captured images, etc., and an input device 5 (for example, a keyboard, a mouse, etc.) as an input unit through which an operator performs various input operations. are connected for communication. Note that the monitor 4 and the input device 5 may be configured from a display device with an input function such as a touch panel display.

The image processing device 3 is configured so that the user can view the video captured by the camera 1 in real time on the monitor 4, and is also configured to allow the user to view past video recorded on the recorder 2. There is.

<Configuration of image processing device>
Next, the configuration of the image processing device 3 will be explained using FIG. 3. The image processing device 3 has functions equivalent to a general personal computer, and includes a control section 10 and a storage section 20, as shown in FIG.

The control section 10 has the functions of a video input section 11 , a setting section 12 , a resolution change section 13 , an image size adjustment section 14 , a face detection section 15 , and a face recognition section 16 . The control unit 10 includes, for example, a software program for implementing various processes, a CPU (Central Processing Unit) that executes the software program, and various hardware controlled by the CPU. Software programs and data necessary for the operation of the control unit 10 are stored in the storage unit 20.

Note that each part of the control unit 10 shown in FIG. 3 is realized by causing the CPU of the control unit 10 to execute the image processing program according to the present embodiment. The image processing program according to the present embodiment may be installed in the image processing device 3 in advance, or may be recorded on a computer-readable recording medium as an application program that can run on a general-purpose OS, or may be stored via a network. may be provided to the user.

The storage unit 20 is composed of a storage device such as a hard disk device or a nonvolatile memory, and stores an inference model 21.

The video input unit 11 performs a process of acquiring images from the camera 1 or the recorder 2. When performing face detection in real time, an image is acquired from the camera 1, and when performing face detection from past video, an image is acquired from the recorder 2. The image acquired by the video input section 11 is output to the resolution changing section 13 as an input image. Note that the input image may be a still image or a moving image (video).

The setting unit 12 performs a process of setting a setting range as a range of the size of the face (inter-eye distance) to be detected from the input image according to the user's input operation. The setting range is set by specifying the maximum and minimum values of the size (inter-eye distance) of the face to be detected, and is stored in the storage unit 20. The maximum value and minimum value of the setting range are both arbitrary values, preferably the number of pixels (an arbitrary natural number). The setting range may be set by the user directly inputting the maximum and minimum values of the inter-eye distance using the input device 5, and by the setting unit 12 based on the input values, or may be set by the method described below. Good too.

For example, as shown in FIG. 4A, a user is asked to walk back and forth in front of the camera 1 while taking pictures, and the captured video is displayed on the monitor 4. As shown in FIG. 4(b), the monitor 4 displays a user whose face size gradually changes. Next, while having the user actually check the change in the size of the face on the monitor 4, the user selects an image of the size of the face they want to detect, that is, an image of the smallest face they want to detect, and an image of the largest face they want to detect. have them choose. Then, for each image, the resolution changing unit 13 performs scaling image generation processing and the face detection unit 15 performs face detection processing by changing the scaling ratio variously until the face detection unit 15 detects those faces. Repeat while doing so. Then, from the smallest face detected by the face detection unit 15 and the distance between the eyes of the smallest face, a range (maximum value and minimum value) of the distance between the eyes to be detected is determined. Also, different from this, the user is asked to select the smallest face image they want to detect and the largest face image they want to detect on the monitor 4, and then the user inputs the positions of the eyes of each face. The range (maximum value and minimum value) of the inter-eye distance to be detected may be determined by having the user specify it using the device 5.

Alternatively, as shown in FIG. 5, an illustration image of a face or the like may be displayed as an overlay on the image of the camera 1, and the user may use a mouse or the like to enlarge or reduce the size of the face desired to be detected. In this case, the setting unit 12 determines the maximum and minimum inter-eye distances to be detected from the most reduced face (the smallest face you want to detect) and the most enlarged face (the largest face you want to detect). and set the setting range.

Further, the setting unit 12 may automatically set the size of the face to be detected, that is, the setting range, from the resolution of the input image. This is suitable for face authentication (login) using a smartphone or pass cap camera. In this application, since the face should appear large in the image, the setting range may be, for example, the number of pixels equal to or more than half the resolution of the input image.

The resolution changing unit 13 has a predetermined range of face sizes (inter-eye distance) that can be detected by the face detection unit 15 and the above-mentioned setting range (a range of face sizes (inter-eye distance) to be detected from the input image). ), an enlarged/reduced image, which is an image with a changed resolution of the input image, is generated based on the range of ). The enlarged/reduced image is an image obtained by increasing or reducing the resolution of the input image, that is, an image obtained by enlarging or reducing the input image. Note that normally, the resolution changing unit 13 generates an enlarged/reduced image while maintaining the aspect ratio of the input image.

More specifically, the resolution changing unit 13 enables the face detecting unit 15 to detect at least one of a face with a size corresponding to the maximum value of the setting range and a face with a size corresponding to the minimum value of the setting range. Then, the resolution of the input image is changed to generate a scaled image. As a result, faces whose size exceeds the predetermined detectable range (too large faces that exceed the detectable range of the face detection unit 15) and faces whose size falls below the predetermined detectable range (faces that are too large that exceed the detectable range of the face detection unit 15) are detected. The face detection unit 15 can efficiently detect at least one of the face (too small, below the possible range).

More preferably, the resolution changing unit 13 generates the enlarged/reduced image by lowering the resolution of the input image so that the face detection unit 15 can detect a face having a size corresponding to at least the maximum value of the setting range. . Thereby, it is possible to efficiently detect at least a face whose size exceeds the predetermined detectable range (a face that is too large and exceeds the detectable range of the face detection unit 15). Furthermore, since faces can be detected on enlarged/reduced images obtained by lowering the resolution of the input image, that is, reduced images, the burden of face detection processing can be reduced.

Specifically, for example, if the predetermined range of the eye distance is 8 to 200 pixels and the set range of the eye distance is 8 to 400 pixels, then the maximum eye distance of the set range of 400 pixels is The resolution of the input image is lowered to match the maximum value of a predetermined range. That is, an enlarged/reduced image is generated by enlarging/reducing the input image at an enlargement/reduction ratio (%) calculated from the formula (maximum value of predetermined range)/(maximum value of set range)×100.

Note that the enlargement/reduction ratio means the magnification ratio of the enlarged/reduced image to the input image, and here, it refers to the ratio of the width and height of the enlarged/reduced image to the input image, rather than the area ratio of the enlarged/reduced image to the input image. represent. Specifically, as described above, it is calculated, for example, from the maximum value of a predetermined range and the maximum value of a set range.

The face detection unit 15 detects a face from a detection target image (image data) such as an input image or a scaled image, that is, executes a face detection process. However, the face detection unit 15 is not configured to be able to detect faces of all sizes (inter-eye distance) in an image, but only to be able to detect faces within a predetermined range of sizes (inter-eye distance). There is.

Therefore, the face detection unit 15 performs face detection on at least the enlarged/reduced image. As a result, even if the inter-eye distance (set range) to be detected is outside the predetermined detectable range, it becomes possible to detect faces within the set range from the enlarged/reduced image.

Specifically, for example, if the predetermined range of the inter-eye distance is 8 to 200 pixels, the set range of the inter-eye distance is 50 to 400 pixels, and the input image is scaled down and enlarged at an enlargement/reduction ratio of 50%. When generating a reduced image, the face detection unit 15 detects a face with a size within the set range, since in an enlarged/reduced image, it is possible to detect faces with an inter-eye distance of 16 to 400 pixels when converted to the size of the input image. It will be possible.

However, as mentioned above, for example, if the predetermined range of the inter-eye distance is 8 to 200 pixels, the set range of the inter-eye distance is 8 to 400 pixels, and the input image is reduced at a magnification/reduction ratio of 50%. When generating an enlarged/reduced image using the enlarged/reduced image, the face detection unit 15 cannot detect a face whose inter-eye distance is less than 16 pixels in terms of the size of the input image.

Therefore, the face detection unit 15 further performs face detection on the input image depending on the minimum value of the setting range. As a result, even if the size of the face to be detected (setting range) is smaller than the range that can be detected by the face detection unit 15 in the enlarged/reduced image, face detection is performed on the input image. This increases the possibility of detecting small faces that cannot be detected in enlarged/reduced images.

Specifically, as described above, the predetermined range of the inter-eye distance is 8 to 200 pixels, the set range of the inter-eye distance is 8 to 400 pixels, and the input image is reduced at an enlargement/reduction ratio of 50%. Even in the case where a scaled image is generated using the input image, the face detection unit 15 can detect a face with an inter-eye distance of 8 to 200 pixels in the input image. In other words, it is possible to detect faces with an inter-eye distance of 8 to 16 pixels, which cannot be detected in enlarged/reduced images, from the input image.

The following formula (1) is suitable as a criterion for determining whether or not to further perform face detection on the input image. That is, when the enlargement/reduction ratio of the enlarged/reduced image with respect to the input image is set to X% (where X is a positive number excluding 0), the minimum value of the predetermined range is set to Min1, and the minimum value of the set range is set to Min2, When the relationship expressed by the following formula (1) is satisfied, it is preferable that the face detection unit 15 further performs face detection on the input image.
(Min2×X/100)<Min1 (1)

On the other hand, if the relationship expressed by the above formula (1) is not satisfied, that is, if (Min2×X/100)≧Min1 is satisfied, the face detection unit 15 does not need to perform face detection on the input image. .

As shown in FIG. 6, the face detection unit 15 (each face detector) at least detects the coordinates E1 and E2 of the left and right eyes of the face included in the image to be detected, and a rectangular bounding box indicating the area of the face. This outputs (estimates) BB and a score related to the face-likeness of the face. Note that the score related to face-likeness is a parameter that indicates how much the detected face resembles a face, and is output as a continuous value from 0 to 1, and the closer it is to 1, the more the face is like a face. do. The face detection unit 15 also performs a process of calculating the size of the face (in this embodiment, the distance between the eyes) based on the coordinates E1 and E2 of the left and right eyes. That is, the inter-eye distance corresponds to the distance between the detected coordinates E1 and E2 of the left and right eyes.

The detection result of the face detection unit 15 includes information related to the detected face, including the coordinates E1 and E2 of the left and right eyes, the coordinates of the bounding box BB (coordinates and size of the upper left corner), the score related to face-likeness, This includes the distance between the eyes as the size of the face. The detection results by the face detection unit 15, such as symbols indicating the positions of the left and right eyes and bounding boxes BB, are displayed on the monitor 4 to notify the user.

As shown in FIG. 7, a first region R1 in the input image corresponding to region r1 of the face F1 detected by the face detection section 15 in the enlarged/reduced image is detected by the face detection section 15 in the input image. If the area overlaps with the second area R2 which is the area of the face F2, the face detection unit 15 detects the area of the face F1 or F2 according to the degree of overlap between the first area R1 and the second area R2. Output information as detection results. Thereby, the number of faces output to the face authentication section 16 can be reduced, so the processing cost of the image processing device 3 can be reduced.

Here, specifically, cases where the first region R1 overlaps with the second region R2 include, for example, a case where the same person's face is detected in both the enlarged/reduced image and the input image, and It can be assumed that faces of different people are detected close to each other in each of the reduced image and the input image. In the former case, only the detection results for either the scaled image or the input image should be adopted, and in the latter case, the detection results for both the scaled image and the input image should be adopted. This is the case. Therefore, in order to be able to deal with these cases, as described above, the face detection unit 15 detects either one of the faces F1 or F2 depending on the degree of overlap between the first region R1 and the second region R2. Information related to this is output as a detection result. Here, the overlapping degree of the first region R1 and the second region R2 is specifically, for example, IoU (Intersection over Union), and if IoU is equal to or greater than a predetermined reference value, the first region Since the overlap between region R1 and second region R2 is large, faces related to face F1 or F2 are regarded as faces related to the same person, information related to either face F1 or F2 is output as a detection result, and IoU is If it is less than the reference value, since the overlap between the first region R1 and the second region R2 is small, the faces related to the face F1 or F2 are regarded as the faces of different people, and the information related to both the faces F1 and F2 is generated. are output as detection results. Note that IoU is calculated from the formula (common part of regions)/(union of regions).

Further, when the first region R1 overlaps the second region R2 and outputs information related to either one of the faces F1 or F2, the face detection unit 15 selects the face with a higher face-likeness score. Information related to F1 or F2 is output as a detection result. As a result, information related to the face with a higher face-likeness score among the two faces F1 and F2 detected in the enlarged/reduced image and the input image can be used, so the accuracy of the recognition of the face by the face authentication unit 16 can be improved.

Note that the area r1 related to the face F1 and the area R2 related to the face F2 both correspond to the bounding box BB detected by the face detection unit 15. Further, the relative position of the region r1 within the enlarged/reduced image matches the relative position of the first region R1 within the input image.

In addition, when at least a part of the predetermined range exceeds the set range and the face detection unit 15 detects a face whose size is outside the set range, the face detection unit 15 detects the face that is outside the set range. Information related to a face of size is processed according to setting conditions that can be set by the user. Specifically, it is preferable that the face detection unit 15 not output information regarding a face having a size outside the set range as a detection result. As a result, when a face of a size that should not be detected, that is, a face of a size that the user does not desire, is detected, the fact that the face has been detected is not displayed on the monitor 4, and the user It is possible to easily confirm only the detection results.

In addition, when at least a part of the predetermined range exceeds the set range, more specifically, when the maximum value of the predetermined range exceeds the maximum value of the set range, the minimum value of the predetermined range exceeds the minimum value of the set range. In the case where the maximum value of the predetermined range exceeds the maximum value of the set range, and the minimum value of the predetermined range is lower than the minimum value of the set range.

Also, depending on the setting conditions, if a face with a size outside the setting range is detected, the face may be displayed on the monitor 4 in a different manner from the face to be detected (face with a size within the setting range). You may also do so. For example, the bounding box may be displayed in a different manner, for example, in a different color, for a face whose size is outside the set range and a face whose size is within the set range.

Furthermore, as described above, the face detection unit 15 includes a plurality of face detectors having different memory capacities. Note that the plurality of face detectors differ only in memory capacity, and the predetermined range indicating the size of a detectable face is common to all face detectors.

The memory capacity of the face detector is secured to store calculation results during face detection processing, and is set according to a plurality of image sizes (hereinafter also referred to as reference image sizes). In this way, by setting the memory capacity of the face detector according to multiple reference image sizes, the memory capacity of the detector can be adjusted each time according to the image size of the image to be detected (scaled image or input image). The processing time can be shortened compared to the case where the data is secured. As described above, the memory capacity of the face detector is set according to a plurality of image sizes such as VGA, Full HD, 4K, and any image size.

The face detection unit 15 searches the whole image of the detection target and finds (estimates) a face anywhere in the image. Further, the face detection unit 15 can detect faces that are not facing forward, that is, faces facing in various directions.

More specifically, the face detection unit 15 includes a plurality of face detectors as described above, and each face detector executes the inference model 21 stored in the storage unit 20 by the control unit 10. By doing so, the function, namely the face detection function, is realized.

Here, the inference model 21 will be explained. The inference model 21 is created by supervised machine learning of a data set (teacher data) to which label information (correct data) is attached. More specifically, the inference model 21 uses as input data an image of a person's face and an object other than the person's face (for example, a two-dimensional still image), and uses feature points attached to the face (for example, left and right Convolutional neural network (CNN) It is created by performing deep learning using a learning program using Neural Network.

The inference model 21 created by supervised machine learning functions as an inference program (learned model) in which learned parameters are incorporated. Note that the learned parameters are parameters (coefficients) obtained as a result of learning using a data set. Also. The inference program applies the learned parameters obtained as a result of learning to the input image (still image or each still image forming a video (moving image)) given as input, and applies the learned parameters to the input image. This is a program that defines a series of calculation procedures for outputting results (specifically, the positions of each feature point and bounding box as described above).

Note that the inference model 21 (each face detector) may be additionally trained. That is, by applying a different data set to the inference model 21 and performing further learning, new learned parameters are generated, and the inference program incorporating the new learned parameters is applied to the inference model 21 (each face detection It may also be used as a container.

The image size adjustment unit 14 performs a process of adding a background image to at least a portion of the periphery of the enlarged/reduced image so that the size of the enlarged/reduced image matches the closest image size among the plurality of reference image sizes. As a result, the size of the enlarged/reduced image corresponds to the memory capacity of one of the face detectors. Face detection is performed on the enlarged/reduced image to which the background image has been added by one of the multiple face detectors that has a memory capacity that corresponds to the size of the enlarged/reduced image to which the background image has been added. be exposed. As a result, as mentioned above, face detection processing can be performed without reserving the memory capacity of the face detector each time according to the image size of the image to be detected (scaled/reduced image or input image), which reduces the processing time. Can be shortened.

Further, as described above, when performing face detection not only on enlarged/reduced images but also on input images, the image size adjustment unit 14 similarly determines whether the size of the input image is the largest among the plurality of reference image sizes. Processing is performed to add a background image to at least a portion of the periphery of the input image so as to match a similar image size. As a result, the size of the input image corresponds to the memory capacity of either face detector. Face detection is then performed on the input image to which the background image has been added, by one of the plurality of face detectors that has a memory capacity that corresponds to the size of the input image to which the background image has been added.

Note that the background image is not particularly limited as long as it is an image in which a face cannot be detected by the face detection unit 15 (each face detector), but a monochromatic image such as black is preferable. Furthermore, there are no particular limitations on where to add the background image to the original image (scaled or reduced image or input image). It may be added to only two sides, only three sides, or all four sides. In either case, the background image is usually added as a band-shaped area with a constant width relative to the sides of the original image.

Specifically, for example, the input image size (resolution) is 800 x 800, the scaled image size (resolution) is 400 x 400, VGA (640 x 480), Full HD (1920 x 1080), When there are three face detectors each corresponding to a 4K image size, the image size adjustment unit 14 adds a black background image to each of the input image and the enlarged/reduced image, and adjusts the sizes of the input image and the enlarged/reduced image, respectively. Change to Full HD (1920x1080) and VGA (640x480). Then, face detection is performed on the input image after resizing using a face detector compatible with the image size of Full HD (1920 x 1080), and a VGA (640 x 480) image is detected on the enlarged/reduced image after resizing. Face detection is performed using a face detector that corresponds to the size.

The face authentication section 16 performs authentication processing based on the detection result of the face detection section 15. The specific content of authentication processing is not particularly limited, and includes, for example, identification of the person, estimation of attributes (age, gender, etc.), detection of accessories (presence of glasses, sunglasses, etc.); Although only one process or two or more processes may be performed, it is preferable that the face authentication section 16 at least performs identification. The detection result of the face detection unit 15 referred to by the face authentication unit 16 is not particularly limited as long as it is information that can specify the image area of the face detected by the face detection unit 15, and for example, the coordinates of the bounding box BB, the characteristics, etc. Examples include the position of a point (for example, the coordinates of the left and right eyes).

As explained in FIG. 7, the first region R1 in the input image corresponding to the region r1 of the face F1 detected by the face detection section 15 in the enlarged/reduced image is detected by the face detection section 15 in the input image. When outputting information related to the face F1 that overlaps the second region R2, which is the region of the face F2 that has been enlarged and is detected by the face detection unit 15 in the enlarged/reduced image, as a detection result, the face authentication unit 16 Authentication processing is performed based on the image area of the input image specified from the coordinates of the input image corresponding to the coordinates of the face F1 detected in the image. That is, the face authentication unit 16 performs a process of cutting out the image area from the input image, and performs an authentication process on the image area. Thereby, it is possible to prevent the authentication accuracy by the face authentication section 16 from decreasing. When an image is reduced, pixels are thinned out or pixel values are weighted and averaged, which reduces the amount of information compared to the input image, which may reduce authentication accuracy. In the case of enlarging, interpolation processing is involved and the amount of information is less than the input image, so the authentication accuracy is equal to or less than the input image. Therefore, as described above, by cutting out an image for authentication from an input image, it is possible to prevent a decrease in authentication accuracy.

Here, the coordinates of the face F1 detected in the enlarged/reduced image are preferably the coordinates (coordinates and size of the upper left corner) of the region r1 (bounding box BB) of the face F1, and The second region R2 of the face F2 detected in this manner corresponds to the bounding box BB of the face F2, as described above. Therefore, the face authentication unit 16 determines that the first region R1 in the input image corresponding to the region r1 (bounding box BB) of the face F1 detected in the enlarged/reduced image is the same as that of the face F2 detected in the input image. When information related to the face F1 that overlaps with the second region R2 (bounding box BB) and is detected in the enlarged/reduced image is output as the detection result of the face detection unit 15, the first region R1 in the input image It is more preferable to perform authentication processing based on.

The result of the authentication process by the face authentication section 16 is displayed on the monitor 4 to notify the user.

<Face detection processing procedure>
Next, the procedure of image processing performed by the image processing device 3, mainly the procedure of processing related to face detection, will be explained using FIGS. 8 and 9. Here, we will explain the case where the predetermined range of the distance between the eyes is 8 to 200 pixels, that is, the case where the face detection unit 15 (each face detector) can detect a face with a size of the distance between the eyes of 8 to 200 pixels. do.

First, based on the user's input operation, the setting unit 12 sets a setting range as the range of the size of the face (inter-eye distance) to be detected from the input image (step S11). Here, the setting range of the inter-eye distance is 8 to 400 pixels.

Next, an input image (still image or moving image) is input to the video input unit 11 from the camera 1 or the recorder 2 (step S12). For example, the input image includes a face with an inter-eye distance of 400 pixels.

Next, the resolution changing unit 13 generates an enlarged/reduced image that is an image in which the resolution of the input image is changed based on the predetermined range and the set range (step S13). Specifically, the input image is enlarged/or The image is reduced (reduced in this case) and a scaled image is generated. As a result, the face detection unit 15 can detect faces with an inter-eye distance of 16 to 400 pixels in terms of the resolution of the input image in the enlarged/reduced image. In this way, since a face with an inter-eye distance of less than 16 cannot be detected in an enlarged/reduced image, the face detection unit 15 also performs face detection on the input image.

Next, the image size adjustment unit 14 adds a background image to each of the enlarged/reduced image and the input image, and adjusts the sizes to, for example, VGA (640×480) and Full HD (1920×1080), respectively (step S14).

Next, the face detection unit 15 performs face detection processing on the enlarged/reduced image and the input image (step S15). The face detection unit 15 has three face detectors whose memory capacities correspond to VGA, Full HD, and 4K image sizes, and which face detector to use depends on the image size of the input image and the enlarged/reduced image. Determined accordingly. Here, a face detector whose memory capacity corresponds to VGA performs face detection on enlarged/reduced images, and a face detector whose memory capacity corresponds to Full HD performs face detection on input images.

As a result, detection results by the face detection unit 15 are obtained for each of the enlarged/reduced image and the input image.

If detection results are obtained for each of the enlarged/reduced image and the input image, they are integrated based on their degree of overlap (step S16). Specifically, the face detection unit 15 detects a first region R1 in the input image corresponding to region r1 (bounding box BB) of the face F1 detected in the enlarged/reduced image, and a face F2 detected in the input image. The IoU with the second region R2 (bounding box BB) of If the IoU is less than the reference value, the information regarding both faces F1 and F2 is adopted.

Then, the face detection unit 15 calculates the distance between the eyes from the coordinates of the left and right eyes of the face adopted in step S16, and if the calculated distance between the eyes is not outside the set range, outputs the detection result related to the face. However, if the calculated inter-eye distance is outside the set range, the information related to the face is deleted without outputting the detection result related to the face (step S17).

Thereafter, the face authentication section 16 performs authentication processing based on the detection result of the face detection section 15 (step S18), and the processing ends.

As explained above, in this embodiment, a predetermined range indicating the range of detectable face sizes (inter-eye distance) and a range of face sizes (inter-eye distance) to be detected from the input image are set. The resolution of the input image is changed, that is, scaled based on the set range, and the face detection unit 15 performs face detection on the scaled image. range) is outside the predetermined detectable range, the face detection unit 15 can generate a scaled image of an appropriate size that can be detected, and perform face detection on the scaled image. be able to. Furthermore, since the enlarged/reduced image is generated based on the predetermined range and the set range, face processing can be efficiently performed on the minimum necessary enlarged/reduced image. As described above, it is possible to efficiently detect a face of a size requested by the user.

Note that in the above embodiment, the case where the input image is enlarged/reduced by the enlargement/reduction ratio calculated from the maximum value of the predetermined range and the maximum value of the set range has been specifically described. The reduction rate may be calculated from the minimum value of the predetermined range and the minimum value of the set range, or may be calculated from the median value of the predetermined range and the median value of the set range.

Further, in the above embodiment, a case has been described in which the inference model 21 is constructed by deep learning using a convolutional neural network, but the inference model 21 is not particularly limited as long as it is created by machine learning, and The model 21 may be created by machine learning other than deep learning.

Further, in the above embodiment, a case has been described in which the image processing device 3 is configured as one device, but each function of the image processing device 3 may be realized by a distributed processing system that is appropriately distributed among a plurality of devices.

Specifically, as shown in FIG. 10, for example, a camera 101 as an image acquisition unit and an image recording unit, a personal computer 102 equipped with a monitor and an input device communicably connected to the camera 101, and a personal computer 102 An image processing system may be configured from a cloud server 103 that is communicably connected to the cloud server 103. Then, the personal computer 102 is provided with the functions of the above-mentioned video input section 11, setting section 12, resolution change section 13, image size adjustment section 14, and face detection section 15, and the cloud server 103 is provided with the functions of the above-mentioned face recognition section 16. The monitor and input device of the personal computer 102 are used as a display unit and an input unit, respectively, and the image taken by the camera 101, the detection result by the face detection unit 15, and the authentication result by the face recognition unit 16 are displayed on the monitor. You may. Note that the personal computer 102 or the cloud server 103 may have the function of the image recording unit.

Although the embodiments of the present invention have been described above with reference to the drawings, the present invention is not limited to the above embodiments. Furthermore, the configurations of each embodiment may be combined or modified as appropriate without departing from the gist of the present invention.

As described above, the present invention is a useful technique for efficiently detecting a face of a size requested by a user from an image.

1: Camera 2: Recorder 3: Image processing device 4: Monitor 5: Input device 10: Control section 11: Video input section 12: Setting section 13: Resolution change section 14: Image size adjustment section 15: Face detection section 16: Face Authentication unit 20: Storage unit 21: Inference model 101: Camera 102: Personal computer 103: Cloud server

Claims (16)

a face detection unit capable of detecting a face within a predetermined size range from an image;
a storage unit that stores a setting range set by a user as a size range of a face to be detected from an input image;
a resolution changing unit that determines a resolution of an image that can be detected by the face detection unit based on the predetermined range and the setting range, and generates an enlarged/reduced image that is an image in which the resolution of the input image is changed to the determined resolution; and,
Equipped with
The image processing device is characterized in that the face detection unit detects a face on the enlarged/reduced image. The resolution changing section is configured to detect the face detecting section so that the face detecting section can detect at least one of a face having a size corresponding to the maximum value of the setting range and a face having a size corresponding to the minimum value of the setting range. The image processing apparatus according to claim 1, wherein the enlarged/reduced image is generated by changing the resolution of the input image. The resolution changing unit generates the enlarged/reduced image by lowering the resolution of the input image so that the face detection unit can detect a face having a size corresponding to at least a maximum value of the setting range. The image processing apparatus according to claim 2, characterized in that: The image processing apparatus according to claim 1, wherein the face detection unit further detects a face in the input image depending on a minimum value of the setting range. When the enlargement/reduction ratio of the enlarged/reduced image with respect to the input image is X%, the minimum value of the predetermined range is Min1, and the minimum value of the set range is Min2, (Min2×X/100)<Min1. 5. The image processing apparatus according to claim 4, wherein when the relationship is satisfied, the face detection unit further performs face detection on the input image. If a first area in the input image corresponding to a face area detected in the enlarged/reduced image overlaps a second area that is a face area detected in the input image, the face detection The image processing device according to claim 4 or 5, wherein the unit outputs information related to one of the faces as a detection result depending on the degree of overlap between the first region and the second region. . The face detection unit further outputs a score related to the face-likeness of the detected face, and when the first region overlaps the second region and outputs information related to either one of the faces. 7. The image processing apparatus according to claim 6, wherein information regarding a face having a higher face-likeness score is output as a detection result. further comprising a face authentication unit that performs authentication processing based on the detection result of the face detection unit,
When the first area overlaps the second area and the face detection unit outputs information related to the face detected in the enlarged/reduced image as a detection result, the face authentication unit The image processing device according to claim 6 or 7, wherein the image processing device performs authentication processing based on an image area of the input image specified from coordinates of the input image corresponding to coordinates of the face detected in the image. . The image processing apparatus according to claim 1, further comprising a face authentication section that performs authentication processing based on a detection result of the face detection section. If at least a portion of the predetermined range exceeds the set range and the face detection unit detects a face with a size outside the set range, the face detection unit detects a face with a size outside the set range. 10. The image processing apparatus according to claim 1, wherein the image processing apparatus does not output information regarding the face of the person as a detection result. Any one of claims 1 to 10, wherein the face detection unit detects a plurality of feature points of a face included in an image, and calculates the size of the face based on the plurality of feature points. The image processing device described in . The face detection unit includes a plurality of face detectors having different memory capacities,
The image processing apparatus according to claim 1, wherein the memory capacity of the plurality of face detectors is set according to the plurality of image sizes. further comprising an image size adjustment unit that adds a background image to at least a portion of the periphery of the enlarged/reduced image so that the size of the enlarged/reduced image matches the closest image size among the plurality of image sizes;
13. The image processing apparatus according to claim 12, wherein the face detection unit performs face detection on the scaled image to which the background image is added. a face detection unit capable of detecting a face within a predetermined size range from an image;
a storage unit that stores a setting range set by a user as a size range of a face to be detected from an input image;
a resolution changing unit that determines a resolution of an image that can be detected by the face detection unit based on the predetermined range and the setting range, and generates an enlarged/reduced image that is an image in which the resolution of the input image is changed to the determined resolution; and,
Equipped with
The image processing system is characterized in that the face detection unit detects a face on the enlarged/reduced image. An image processing method for detecting a face from an image, the method comprising:
storing a setting range set by the user as a size range of the face to be detected from the input image;
An image obtained by determining the resolution of an image that can be detected by a face detection unit based on a predetermined range of detectable face sizes and the setting range, and changing the resolution of the input image to the determined resolution. generating a scaled image that is
Detecting a face on the enlarged/reduced image;
An image processing method characterized by comprising: computer to detect faces from images,
Resolution of an image that can be detected by the face detection unit based on a predetermined range that is the range of detectable face sizes and a setting range set by the user as the range of face sizes that should be detected from the input image. means for determining a scaled-down image, which is an image obtained by changing the resolution of the input image to the determined resolution; and means for detecting a face in the scaled-down image;
An image processing program characterized by functioning as an image processing program.

Images