JP2020101927A - Image identification device, identifier learning method, image identification method and program - Google Patents

Abstract

To improve the accuracy of image identification.SOLUTION: An image identification device 100 includes: a crop image generation part 11 for generating a crop image undergoing crop processing being processing for segmenting an image including an identification object area from an input image; a non-crop image generation part 12 for generating a non-crop image which does not undergo clop processing from the input image; a crop image identifier 13 for identifying the crop image; a non-crop image identifier 14 for identifying the non-crop image; and an identification result determination part 15 for acquiring a final identification result by using both an identification result by the crop image identifier 13 and an identification result by the non-crop image identifier 14.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image identifying device, a discriminator learning method, an image identifying method, and a program.

As a technique for identifying an image, a technique for learning a deep neural network (DNN) by using image data whose identification result is known as learning data and using the learned DNN is known. Has been. In the image identification technology, there is no particular limitation on the image to be identified and any image data can be identified, but in some cases, a scale or the like may be included in the image data. For example, when an affected area such as a skin disease is imaged, a scale or the like is often imaged together to clarify the size of the affected area. Also, not only the image of the affected area, but for example, in the case of an image such as a flower image in which it is desired to clarify the size of the object, it is easier to understand the size by photographing the scales together. In many cases, scales and the like are also included in the image data for the image.

In this case, it is conceivable that the scales and the like reflected together may adversely affect the image identification. Therefore, in Non-Patent Document 1, for example, the foreground image that is a disease region is cropped and resized to classify the image. The technology to do is proposed.

NCF Codella ;QB Nguyen ;S. Pankanti ;DA Gutman ;B. Helba ;AC Halpern ;Jr Smith, "Deep learning ensembles for melanoma recognition in dermoscopy images", IBM Journal of Research and Development, Volume.61 Issue 4/5, July-Sept. 2017, Pages:5:1-5:15

According to the technique described in Non-Patent Document 1, the influence of the scale and the like reflected in the image can be reduced by cropping (cutting) and resizing the diseased region, but the aspect ratio of the original image However, since the information such as the size is lost, there is a problem that the accuracy of image identification is lowered.

The present invention has been made to solve the above problems, and an object of the present invention is to provide an image identification device, an identifier learning method, an image identification method, and a program that can improve the accuracy of image identification. ..

In order to achieve the above object, the image identifying apparatus of the present invention,
A cropped image generation unit that generates a cropped image that has been cropped, which is a process of cutting out an image including an identification target region from an input image,
A non-cropped image generation unit that generates a non-cropped image from which the crop processing has not been performed from the input image,
A cropped image identifier for identifying the cropped image,
A non-cropped image discriminator that identifies the non-cropped image,
A discrimination result by the cropped image discriminator, a discrimination result by the non-cropped image discriminator, and a discrimination result determination unit that obtains a final discrimination result using both of them.
Equipped with.

According to the present invention, the accuracy of image identification can be improved.

It is a figure which shows the function structure of the image identification apparatus which concerns on Embodiment 1 of this invention. 6 is a flowchart of a cropped image learning process according to the first embodiment. 6 is a flowchart of a cropped image generation process according to the first embodiment. It is a figure explaining crop image generation processing. 6 is a flowchart of a non-cropped image learning process according to the first embodiment. 6 is a flowchart of non-cropped image generation processing according to the first embodiment. It is a figure explaining non-cropped image generation processing. 6 is a flowchart of an identification process according to the first embodiment. FIG. 6 is a diagram illustrating activation states of a cropped image discriminator and a non-cropped image discriminator when a lesion area is input to the image discrimination device according to the first embodiment. It is a figure explaining the identification result determination part which concerns on the modification 2 of this invention. 9 is a flowchart of overall learning processing according to Modification 2. 9 is a flowchart of an identification process according to modification 2.

Hereinafter, an image identification device and the like according to the embodiments of the present invention will be described with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals.

(Embodiment 1)
The image identification apparatus 100 according to the first embodiment of the present invention uses the DNN discriminator trained with the image obtained by cropping the region to be discriminated and the discriminator based on the DNN trained with the non-cropped image to obtain an unknown image. Identify. The image identifying apparatus 100 can improve the accuracy of image identification by using both the outputs of the discriminators based on such two types of DNNs. Such an image identification device 100 will be described below.

As shown in FIG. 1, the image identifying apparatus 100 according to the first embodiment includes a control unit 10, a storage unit 20, an image input unit 31, an output unit 32, a communication unit 33, and an operation input unit 34.

The control unit 10 is configured by a CPU (Central Processing Unit) or the like, and executes a program stored in the storage unit 20 so that each unit (cropped image generation unit 11, non-cropped image generation unit 12, cropped image identification) described below is executed. The functions of the device 13, the non-cropped image classifier 14, and the classification result determination unit 15) are realized.

The storage unit 20 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and stores a program executed by the CPU of the control unit 10 and necessary data.

The image input unit 31 is a device for inputting image data for learning or image data to be identified (unknown). The control unit 10 acquires image data via the image input unit 31. As the image input unit 31, any device can be used as long as the control unit 10 can acquire image data. For example, when the image data is stored in the storage unit 20 and the image data is acquired by the control unit 10 reading the storage unit 20, the storage unit 20 also serves as the image input unit 31. When the control unit 10 acquires image data from an external server or the like via the communication unit 33, the communication unit 33 also serves as the image input unit 31.

The output unit 32 is a device for the control unit 10 to output the result of identifying the image input from the image input unit 31, and the like. For example, the output unit 32 is a liquid crystal display or an organic EL (Electro-Luminescence) display. In this case, the output unit 32 functions as a display unit. However, the image identifying apparatus 100 may include such a display (display unit) as the output unit 32, or may include the output unit 32 as an interface for connecting an external display. When the image identification device 100 includes the output unit 32 as an interface, the image identification device 100 displays the identification result and the like on an external display connected via the output unit 32. The output unit 32 functions as an output unit.

The communication unit 33 is a device (network interface or the like) for transmitting/receiving data to/from another external device (for example, a server in which a database of image data is stored). The control unit 10 can acquire image data via the communication unit 33.

The operation input unit 34 is a device that receives a user operation input to the image identification apparatus 100, and is, for example, a keyboard, a mouse, a touch panel, or the like. The image identifying apparatus 100 receives an instruction or the like from the user via the operation input unit 34. The operation input unit 34 functions as operation input means.

Next, the function of the control unit 10 will be described. The control unit 10 realizes the functions of the cropped image generation unit 11, the non-cropped image generation unit 12, the cropped image discriminator 13, the non-cropped image discriminator 14, and the discrimination result determination unit 15.

The cropped image generation unit 11 acquires an identification target area from the input image input via the image input unit 31, cuts (crops) a square image obtained by adding a margin area to the acquired identification target area, and An image (cropped image) resized to the input size of the cropped image discriminator 13 is generated. A process in which the crop image generation unit 11 cuts out an image including the identification target area from the input image is called a crop process.

The non-cropped image generation unit 12 generates an image with emphasized edges (non-cropped image) by performing HPF (High Pass Filter) processing on the input image input via the image input unit 31. The non-cropped image generation unit 12 may generate an image in which edges are emphasized by a process other than the HPF process.

The cropped image classifier 13 and the non-cropped image classifier 14 are both classifiers of images by a convolutional neural network (CNN), which is a type of DNN. When the control unit 10 executes the program that realizes the CNN classifier, the control unit 10 functions as the cropped image classifier 13 and also as the non-cropped image classifier 14.

The cropped image classifier 13 includes an input layer to which the cropped image generated by the cropped image generation unit 11 is input, an output layer to which a classification result of the input cropped image is output, and a layer other than the input layer and the output layer. And outputting a result of identifying the cropped image from the output layer.

The non-cropped image discriminator 14 includes an input layer to which the non-cropped image generated by the non-cropped image generation unit 12 is input, an output layer to which a discrimination result of the input non-cropped image is output, and an input layer and an output layer. And an intermediate layer which is a layer other than the above, and outputs the result of identifying the non-cropped image from the output layer.

The discrimination result determination unit 15 finally uses both the output from the cropped image discriminator 13 (the discrimination result of the cropped image) and the output from the non-cropped image discriminator 14 (the discrimination result of the non-cropped image). To obtain a proper identification result. Basically, the discrimination result determination unit 15 simply adds and averages the output from the cropped image discriminator 13 and the output from the non-cropped image discriminator 14 to obtain a final discrimination result. However, it is not limited to this. When obtaining the final discrimination result, the discrimination result determination unit 15 multiplies the output from the cropped image discriminator 13 and the output from the non-cropped image discriminator 14 by predetermined weights, respectively, to obtain a weighted average. You may take. The identification result determination unit 15 functions as an identification result determination unit.

The functional configuration of the image identifying apparatus 100 has been described above. Next, a process of learning the CNN (the cropped image discriminator 13 and the non-cropped image discriminator 14) using the cropped image and the non-cropped image will be described.

First, a cropped image learning process for learning the cropped image discriminator 13 using a cropped image will be described with reference to FIG. This process is executed when the crop image classifier 13 is learned. Further, before executing this processing, it is necessary to prepare learning image data with a correct answer label. The correct answer label indicates what kind of image the image with the correct answer label is. For example, if you want to prepare a CNN that outputs the disease name when a disease image is input, the correct answer label is “disease name”. It is necessary to attach to each learning image data as. As another example, if it is desired to prepare a CNN that outputs the name of a flower when the image of the flower is input, it is necessary to attach "flower name" as a correct answer label to each learning image data.

First, the control unit 10 acquires learning image data via the image input unit 31 (step S101). Then, the cropped image generation unit 11 performs a cropped image generation process described below to generate a cropped image from the learning image data acquired in step S101 (step S102). Step S102 is also called a cropped image generation step.

Next, the control unit 10 inputs the cropped image generated by the cropped image generation unit 11 to the cropped image discriminator 13 and learns the cropped image discriminator 13 based on the correct label attached to the learning image data. (Step S103). Step S103 is also called a cropped image learning step.

Then, the control unit 10 determines whether to end the learning (step S104). For example, after learning a predetermined number (for example, M) of learning image data, the learning ends. If the learning is not completed (step S104; No), such as when there is still learning image data that has not been learned, the process returns to step S101. If learning is ended (step S104; Yes), the cropped image learning process is ended.

Next, the cropped image generation process performed in step S102 will be described with reference to the flowchart of FIG. 3 and the specific example of FIG. This process is a process of generating a cropped image from a given input image.

First, the cropped image generation unit 11 acquires a region to be identified from the given input image (step S111). As an example of the given input image, the input image 1010 in which the identification target 1011 (for example, a disease image) and the scale 1012 are shown is shown in the upper left diagram of FIG. In the upper right part of FIG. 4, a region 1020 that is the identification target acquired by the cropped image generation unit 11 is shown. The method of acquiring the area to be identified is arbitrary, but for example, the cropped image generation unit 11 automatically extracts (and acquires) the area 1020 by the identification object area determiner obtained by machine learning in advance. Such an identification target area determiner prepares, for example, a large amount of image data and a foreground map corresponding to the image (one in which the correct answer data of the identification target area is created by a human) as training data for learning. , Can be created by inputting and learning the teacher data into the CNN.

The acquisition of the area to be identified is not limited to the method by machine learning as described above. The user may specify the area 1020 to be identified in the input image 1010 via the operation input unit 34, and the crop image generation unit 11 may acquire the area 1020 specified by the user. Further, the area 1020 automatically extracted by machine learning or the like may be made modifiable by the user via the operation input unit 34, and the cropped image generation unit 11 may acquire the identification target area corrected by the user. ..

Next, the cropped image generation unit 11 extracts a rectangular area circumscribing the area to be identified acquired in step S111 (step S112). In the middle diagram on the left side of FIG. 4, a rectangular area 1030 circumscribing the identification target 1011 is extracted. The rectangular area 1030 does not have to be a square.

Then, the cropped image generation unit 11 crops (cuts) the square area obtained by adding the margin area to the rectangular area extracted in step S112 (step S113). When the rectangular area 1030 described above is a rectangle, the area to be cropped is made square by adjusting the vertical and horizontal amounts of the margin. In the right middle diagram of FIG. 4, a square area 1040 in which a marginal area 1041 represented by diagonal lines is added to the rectangular area 1030 is shown, but this square area 1040 is cropped (cut out).

The length of the margin 1042 that defines the size of the margin area 1041 may be changed according to the size of the rectangular area 1030, or may be set to a fixed length. When changing according to the size of the rectangular area 1030, for example, “10% of the length of one side of the rectangular area 1030” may be set. When the rectangular area 1030 is a rectangular area having long sides and short sides, for example, “the long side has a margin of 10% of the length of the long side of the rectangular area 1030 added to both sides and the short side. Is added to both sides so as to match the length of the long side with a margin added."

Then, the cropped image generation unit 11 resizes the square area cropped in step S113 according to the input size of the CNN (cropped image classifier 13) (step S114), and ends the process. The image of this resized square area is a clip image. In the lower left diagram of FIG. 4, a resized square area 1050 is shown, but this is a resized and enlarged square area 1040 cut out in step S113. Since the identification target 1011 included in the square area 1050 is also resized at the same magnification as the square area 1050, it is in an enlarged state.

By the above crop image generation processing, a crop image in which the identification target included in the input image is close-up is generated. Then, the crop image discriminator 13 is learned by the above-described crop image learning process using the generated crop image.

Next, a non-cropped image learning process for learning the non-cropped image discriminator 14 using a non-cropped image will be described with reference to FIG. This process is executed when the non-cropped image classifier 14 is learned. Similar to the above cropped image learning process, it is necessary to prepare learning image data with correct answer labels before executing the non-cropped image learning process. It may be the same data as the learning data used in.

First, the control unit 10 acquires learning image data via the image input unit 31 (step S201). Then, the non-cropped image generation unit 12 performs a non-cropped image generation process described below to generate a non-cropped image from the learning image data acquired in step S201 (step S202). Step S202 is also called a non-cropped image generation step.

Next, the control unit 10 inputs the non-cropped image generated by the non-cropped image generation unit 12 to the non-cropped image discriminator 14, and based on the correct label attached to the learning image data, the non-cropped image discrimination is performed. The device 14 is learned (step S203). Step S203 is also called a non-cropped image learning step.

Then, the control unit 10 determines whether to end the learning (step S204). For example, after learning a predetermined number (for example, M) of learning image data, the learning ends. If learning is not to be ended (step S204; No), such as when there is still learning image data that has not been learned, the process returns to step S201. If the learning is ended (step S204; Yes), the non-cropped image learning process is ended.

Next, the non-cropped image generation processing performed in step S202 will be described with reference to the flowchart of FIG. 6 and the specific example of FIG. 7. This process is a process of generating a non-cropped image from a given input image.

First, the non-cropped image generation unit 12 performs the HPF process (edge emphasis process) on the given input image (step S211). As an example of the given input image, the input image 1010 in which the identification target 1011 (for example, a disease image) and the scale 1012 are shown is shown in the upper diagram of FIG. 7. Then, in the middle diagram of FIG. 7, it is shown that the input image 1010 is subjected to the HPF process, resulting in an image 1060 including the edge-enhanced identification target 1061 and the edge-enhanced scale 1062. There is.

Next, the non-cropped image generation unit 12 cuts both sides of the HPF-processed input image (this is referred to as side cut processing), cuts out a square area (step S212), and ends the processing. In the lower diagram of FIG. 7, a square area 1070 is shown, which is a dotted line 1063 cut on both sides of the image 1060 in the middle diagram of FIG. 7.

By the above non-cropped image generation processing, the non-cropped image in which the edge of the input image is emphasized is generated. Then, the non-cropped image discriminator 14 is learned by the above-described non-cropped image learning process using the generated non-cropped image.

By learning the cropped image classifier 13 and the non-cropped image classifier 14 as described above, the image classifying apparatus 100 can classify an unknown input image. Next, an identification process for identifying an unknown input image will be described with reference to FIG. This process is executed when identifying an unknown image.

First, the control unit 10 acquires an unknown image to be identified by the image identifying apparatus 100 via the image input unit 31 (step S301).

Next, the crop image generation unit 11 generates a crop image from the unknown image acquired in step S301 by the above-described crop image generation processing (FIG. 3) (step S302). Then, the control unit 10 inputs the generated cropped image to the cropped image discriminator 13 and acquires the output value of the cropped image discriminator 13 (step S303). Step S303 is also called a cropped image identification step.

Next, the non-cropped image generation unit 12 generates a non-cropped image from the unknown image acquired in step S301 by the above-described non-cropped image generation processing (FIG. 6) (step S304). Then, the control unit 10 inputs the generated non-cropped image into the non-cropped image discriminator 14 and acquires the output value of the non-cropped image discriminator 14 (step S305). Step S305 is also called a non-cropped image identification step. Note that the processes of steps S302 to S303 and the processes of steps S304 to S305 may proceed in parallel, or the processes of steps S304 to S305 and steps S302 to S302 may be performed in reverse to FIG. It may be performed prior to the process of S303.

Then, the discrimination result determination unit 15 averages the output value of the cropped image discriminator 13 acquired in step S303 and the output value of the non-cropped image discriminator 14 acquired in step S305 to obtain the final discrimination result. It is determined (step S306). Step S306 is also called an identification result determination step.

Then, the control unit 10 outputs the final identification result determined by the identification result determination unit 15 to the output unit 32 (step S307), and ends the process. In step S307, the control unit 10 may output not only the final identification result but also the output of the cropped image discriminator 13 and the output of the non-cropped image discriminator 14 to the output unit 32.

By the identification processing described above, the image identification apparatus 100 can improve the identification accuracy by using both the cropped image and the non-cropped image. In addition, since the cropped image generation unit 11 cuts out the identification target area with a margin area added, it is possible to greatly reduce the risk of losing the end of the identification target area (for example, the edge portion of the lesion area boundary). .. Moreover, since the cropped image generation unit 11 cuts out a square when the cropped image is generated, the aspect ratio of the image is maintained, and the cropped image discriminator 13 can also use the aspect ratio as information for discrimination.

As a specific example, as shown in FIG. 9, an input image 1100 including a lesion area 1101 of seborrheic keratoses is input to each of the cropped image discriminator 13 and the non-cropped image discriminator 14 to perform CAM (Class Activation). When an activation map is created by (Mapping), it can be confirmed that the cropped image discriminator 13 shows an active reaction mainly in the central region, and the non-cropped image discriminator 14 shows an activated reaction in the entire lesion region. This is because in the case of the cropped image discriminator 13, it is expected that the lesion region always exists in the central region, whereas in the non-cropped image discriminator 14, it is not necessarily expected that the lesion region exists in the central region. Therefore, it is considered that this is due to the difference that the non-cropped image classifier 14 must perform the processing including the lesion area determination.

In addition, the activation map 1110 shown in FIG. 9 shows each case (melanoma (MM), seborrheic keratosis (SK), pigmented nevus (NCN) when the input image 1100 is input to the crop image discriminator 13. )) It is the figure which showed the activation area|region by CAM. The activation map 1120 shown in FIG. 9 is a diagram showing the CAM activation area of each case when the input image 1100 is input to the non-cropped image discriminator 14. In both figures, the higher activity areas are shown in black. Further, FIG. 9 also shows the score (output value) of each case by each classifier.

As shown in FIG. 9, the score of each case by the crop image discriminator 13 is 0.0890 for melanoma (MM), 0.8681 for seborrheic keratosis (SK), and pigmented nevus ( NCN)) is 0.0429, and the score of each case by the non-crop image discriminator 14 is 0.3145 for melanoma (MM), 0.5713 for seborrheic keratosis (SK), and pigmented nevus. (NCN)) is 0.1142.

In the case of a case image with a large lesion area as in the case shown in FIG. 9, the score of the non-cropped image classifier 14 is the malignant side (MM side) (since the size of the lesion area is generally larger in malignant cases). Also in FIG. 9, the melanoma (MM) score (0.3145) by the non-cropped image classifier 14 has a relatively high value in FIG. However, since the size of the lesion area is normalized in the crop image classifier 13, the structure inside the lesion area is emphasized and classification is performed without depending on the size of the lesion area. Therefore, in the example shown in FIG. 9, the crop image discriminator 13 detects white dots (branchoma-like tumor) and black dots (comedo-like dilated) that are findings of seborrheic keratosis (SK). As a result of the detection, it is speculated that the seborrheic keratosis (SK) score (0.8681) was increased.

As described above, in the image identifying apparatus 100, it is considered that the cropped image discriminator 13 performs the discrimination with emphasis on the internal structure of the discrimination target region, and the non-cropped image discriminator 14 performs the discrimination based on the features captured from the entire input image. Since the final identification result is determined using both of the identification results, the identification accuracy can be improved.

(Modification 1)
In the first embodiment, the cropped image generation unit 11 automatically extracts the area to be identified by the identification object area determiner machine-learned using the foreground map created manually. However, as a method of automatically extracting the area to be identified, there is also a method of using the activation map as shown in FIG. This modification 1 will be described.

The functional configuration of the image identifying apparatus 100 according to the first modification is the same as that of the first embodiment, as shown in FIG. However, in the first modification, the control unit 10 first performs the non-cropped image learning process (FIG. 5) to make the non-cropped image classifier 14 learned before performing the cropped image learning process. Then, in step S111 of the cropped image generation process (FIG. 3 ), the control unit 10 first inputs the input image to the non-cropped image discriminator 14 and acquires the activation map of the non-cropped image discriminator 14. Then, in the activation map, an area activated above a predetermined reference value is extracted as an identification target area.

There are several types of activation map generation methods, but in the case of an activation map by CAM as shown in FIG. 9 or an activation map by Grad-CAM (Gradient-weighted Class Activation Mapping), identification is performed. Since the activation map for each class can be obtained, the activation map obtained by adding all the activation maps obtained for each identification class and taking the average (referred to as "total activation map" here) The identification target area is extracted by using. For example, the average value of all the elements of the total activation map is obtained, and the portion on the total activation map whose value is equal to or more than the average value is set as the identification target area.

In generating the activation map, instead of using CAM or Grad-CAM, simply averaging each element (feature map) of the intermediate layer immediately before the fully connected layer immediately before the CNN output layer in the channel direction is used. There is also an activation map generation method that uses an activation map. When such an activation map is used, this activation map can be treated in the same manner as the above-mentioned comprehensive activation map. For example, the average value of all the elements of the activation map is obtained, and the portion on the activation map whose value is equal to or more than the average value is set as the identification target area.

The image identifying apparatus 100 according to the first modification is the same as the image identifying apparatus 100 according to the first embodiment except that the identification target area is extracted using the activation map in step S111 of the cropped image generation process (FIG. 3 ). Is the same. The image identifying apparatus 100 according to the modified example 1 does not need to separately perform the learning of the determiner for extracting the identification target area in addition to the effect provided in the image identifying apparatus 100 according to the first embodiment (non-cropped image identifyr). Learning 14 makes it possible for the non-cropped image discriminator 14 to be used as a discriminator for extracting a discrimination target region).

(Modification 2)
In the first embodiment, the classification result determination unit 15 obtains the final classification result by taking the average (additional average or weighted average) of the output from the cropped image classifier 13 and the output from the non-cropped image classifier 14. I was getting. However, it is not limited to this. The discrimination result determination unit 15 connects the output layer of the cropped image discriminator 13 and the output layer of the non-cropped image discriminator 14 and inputs them to the fully connected layer, and a new output layer through the fully connected layer. The final identification result may be obtained by Such a second modification will be described.

The functional configuration of the image identifying apparatus 100 according to the modified example 2 is the same as that of the first embodiment and is shown in FIG. As shown in FIG. 10, the discrimination result determination unit 15 according to the modified example 2 connects the output layer 1301 of the cropped image discriminator 13 and the output layer 1401 of the non-cropped image discriminator 14 as a fully connected layer. , And connect to a new output layer 1501 to obtain the final identification result.

In the second modification, the output layer 1501 of the identification result determination unit 15 also needs to learn the neural network. The overall learning process for this will be described with reference to FIG. If this process is performed after the cropped image learning process (FIG. 2) and the non-cropped image learning process (FIG. 5) are completed, the learning time can be shortened. However, it is also possible to perform the overall learning process before performing the cropped image learning process (FIG. 2) or the non-cropped image learning process (FIG. 5). In this case, when the entire learning process is completed, the cropped image is processed. Since the learning of the classifier 13 and the non-cropped image classifier 14 has been completed, it is not necessary to separately perform the cropped image learning process (FIG. 2) and the non-cropped image learning process (FIG. 5).

First, the control unit 10 acquires learning image data via the image input unit 31 (step S401). Then, the crop image generation unit 11 performs a crop image generation process (FIG. 3) and generates a crop image from the learning image data acquired in step S401 (step S402). Further, the non-cropped image generation unit 12 performs non-cropped image generation processing (FIG. 6) to generate a non-cropped image from the learning image data acquired in step S401 (step S403). Note that steps S402 and S403 may proceed in parallel, or conversely to step S403, the processing of step S403 may be performed before the processing of step S402.

The control unit 10 then crops the cropped image (square area 1050) generated by the cropped image generation unit 11 at the CNN in which the cropped image classifier 13 and the non-cropped image classifier 14 are connected as shown in FIG. The non-cropped image (square area 1070) generated by the non-cropped image generation unit 12 is input to the non-cropped image discriminator 14, and based on the correct label attached to the learning image data, The weight of the total connection between the total connection layer connecting the output layers 1301 and 1401 and the output layer 1501 of the identification result determination unit 15 is learned (step S404).

Then, the control unit 10 determines whether to end the learning (step S405). For example, after learning a predetermined number (for example, M) of learning image data, the learning ends. If learning is not completed (step S405; No), such as when there is still learning image data that has not been learned, the process returns to step S401. If the learning is ended (step S405; Yes), the whole learning process is ended.

As described above, by learning the neural net to the output layer 1501 of the identification result determining unit 15, the image identifying apparatus 100 according to the modified example 2 can identify an unknown input image. Next, an identification process for identifying an unknown input image by the image identification device 100 according to the modified example 2 will be described with reference to FIG. This process is executed when identifying an unknown image.

First, the control unit 10 acquires an unknown image to be identified by the image identifying apparatus 100 via the image input unit 31 (step S501).

Then, the cropped image generation unit 11 performs a cropped image generation process (FIG. 3) to generate a cropped image from the unknown image acquired in step S501 (step S502). Further, the non-cropped image generation unit 12 performs non-cropped image generation processing (FIG. 6) to generate a non-cropped image from the unknown image acquired in step S501 (step S503). Note that steps S502 and S503 may proceed in parallel, or, conversely to FIG. 12, the processing of step S503 may be performed before the processing of step S502.

Then, in the CNN in which the cropped image discriminator 13 and the non-cropped image discriminator 14 are connected as shown in FIG. 10, the discrimination result determination unit 15 sets the cropped image discriminator 13 to the cropped image generated by the cropped image generation unit 11. Then, the non-cropped image generated by the non-cropped image generation unit 12 is input to the non-cropped image discriminator 14. Then, the final identification result is determined by the output value output to the output layer 1501 (step S504).

Then, the control unit 10 outputs the final identification result determined by the identification result determination unit 15 to the output unit 32 (step S505), and ends the process. In step S505, the control unit 10 outputs not only the final classification result but also the output of the cropped image classifier 13 (the value of each element of the output layer 1301 in FIG. 10) and the output of the non-cropped image classifier 14 ( The value of each element of the output layer 1401 in FIG. 10 may also be output to the output unit 32.

By the identification processing described above, the image identification apparatus 100 according to the modified example 2 can improve the identification accuracy by using both the cropped image and the non-cropped image.

In the above-described embodiment and modification, the control unit 10 also functions as the cropped image discriminator 13 and the non-cropped image discriminator 14 by the control unit 10 executing a program that realizes the discriminator based on CNN. However, it is not limited to this. The image identification apparatus 100 realizes the functions of a cropped image discriminator 13 and a non-cropped image discriminator 14 (for example, a GPU (Graphics Processing Unit), a dedicated IC (Integrated Circuit), etc.) separately from the control unit 10. A device may be included.

Further, in the above-described first embodiment, a skin disease has been mainly described as an example, but the present invention is not limited to the field of dermatology, and can be widely applied to the field of general image identification. For example, it can be applied to identification of flowers, identification of micrographs of bacteria, and the like.

Further, the above-described embodiments and modified examples can be combined appropriately. For example, by combining Modification 1 and Modification 2, a region to be identified in the activation map is automatically extracted, and the cropped image discriminator 13 and the non-cropped image discriminator 14 are extracted as shown in FIG. May be connected to obtain the final identification result from the output layer 1501.

Further, although the non-cropped image generation unit 12 is described as generating an image in which the edge of the input image is emphasized in the above-described embodiment and modification, the present invention is not limited to this. It is important for the non-cropped image generation unit 12 not to crop the input image, and the input image may be directly used as the non-cropped image.

Each function of the image identifying apparatus 100 can be implemented by a computer such as a normal PC (Personal Computer). Specifically, in the above embodiment, the program of the image identification processing performed by the image identification device 100 is described as being stored in the ROM of the storage unit 20 in advance. However, the program may be stored in a flexible disk, a compact disc read only memory (CD-ROM), a digital versatile disc (DVD), a magneto-optical disc (MO), a memory card, a computer (USB) readable by a USB (universal serial bus), or the like. A computer capable of realizing the above-described functions may be configured by storing and distributing the program in a recording medium, reading the program into a computer, and installing the program.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the specific embodiments, and the present invention includes the inventions described in the claims and their equivalents. Be done. The inventions described in the initial claims of the present application will be additionally described below.

(Appendix 1)
A cropped image generation unit that generates a cropped image that has been cropped, which is a process of cutting out an image including an identification target region from an input image,
A non-cropped image generation unit that generates a non-cropped image from which the crop processing has not been performed from the input image,
A cropped image identifier for identifying the cropped image,
A non-cropped image discriminator that identifies the non-cropped image,
A discrimination result by the cropped image discriminator, a discrimination result by the non-cropped image discriminator, and a discrimination result determination unit that obtains a final discrimination result using both of them.
An image identification device including.

(Appendix 2)
The cropped image generation unit acquires the identification target area from the input image, and generates the cropped image as an image including a predetermined margin area in the identification target area,
The image identification device according to attachment 1.

(Appendix 3)
The cropped image generation unit acquires the identification target area by an identification target area determiner obtained by machine learning,
The image identification device according to attachment 2.

(Appendix 4)
The cropped image generation unit acquires the classification target area by extracting a region activated in an activation map of the non-cropped image classification device,
The image identification device according to attachment 2.

(Appendix 5)
The non-cropped image generation unit generates the non-cropped image as an image obtained by performing edge enhancement processing and sidecut processing on the input image,
The image identification device according to any one of appendices 1 to 4.

(Appendix 6)
The non-cropped image generation unit, the input image is the non-cropped image,
The image identification device according to any one of appendices 1 to 4.

(Appendix 7)
The discrimination result determining unit obtains a final discrimination result by obtaining an average of an output value of the cropped image discriminator and an output value of the non-cropped image discriminator,
The image identification device according to any one of appendices 1 to 6.

(Appendix 8)
The discrimination result determination unit connects the output layer of the cropped image discriminator and the output layer of the non-cropped image discriminator to form a fully combined layer, and an output layer through the fully combined layer finally provides Get the identification result,
The image identification device according to any one of appendices 1 to 6.

(Appendix 9)
Further, a display unit for displaying the output value of the cropped image classifier and the output value of the non-cropped image classifier,
The image identification device according to any one of appendices 1 to 8.

(Appendix 10)
The identification target area is a lesion area,
The image identification device according to any one of appendices 1 to 9.

(Appendix 11)
A cropped image generation step of generating a cropped image that has been cropped, which is a process of cutting out an image including an identification target area from the input image;
A non-cropped image generation step for generating a non-cropped image from which the crop processing has not been performed, from the input image;
A crop image learning step of learning a crop image identifier with the crop image,
A non-cropped image learning step of learning a non-cropped image identifier with the non-cropped image,
A classifier learning method including.

(Appendix 12)
A cropped image generation step of generating a cropped image that has been cropped, which is a process of cutting out an image including an identification target area from the input image;
A non-cropped image generation step for generating a non-cropped image from which the crop processing has not been performed, from the input image;
A cropped image identifying step for identifying the cropped image,
A non-cropped image identifying step for identifying the non-cropped image,
An identification result determination step of obtaining a final identification result using both the identification result by the cropped image identification step and the identification result by the non-cropped image identification step,
Image identification method including.

(Appendix 13)
In the computer of the image identification device,
A cropped image generation step of generating a cropped image that has been cropped, which is a process of cutting out an image including an identification target area from an input image,
A non-cropped image generation step of generating a non-cropped image from which the crop processing has not been performed from the input image,
A cropped image identifying step of identifying the cropped image,
A non-cropped image identifying step for identifying the non-cropped image, and
An identification result determination step of obtaining a final identification result by using both the identification result by the cropped image identification step and the identification result by the non-cropped image identification step,
A program to execute.

10... Control unit, 11... Crop image generation unit, 12... Non-cropped image generation unit, 13... Crop image discriminator, 14... Non-cropped image discriminator, 15... Discrimination result determination unit, 20... Storage unit, 31... Image Input unit, 32... Output unit, 33... Communication unit, 34... Operation input unit, 100... Image identification device, 1010, 1100... Input image, 1011, 1061... Identification target, 1012, 1062... Scale, 1020... Region, 1030 ... rectangular area, 1040, 1050, 1070... square area, 1041... margin area, 1042... margin, 1060... image, 1063... dotted line, 1101... lesion area, 1110, 1120... activation map, 1301, 1401, 1501... output layer

Claims (13)

A cropped image generation unit that generates a cropped image that has been cropped, which is a process of cutting out an image including an identification target region from an input image,
A non-cropped image generation unit that generates a non-cropped image from which the crop processing has not been performed from the input image,
A cropped image identifier for identifying the cropped image,
A non-cropped image discriminator that identifies the non-cropped image,
A discrimination result by the cropped image discriminator, a discrimination result by the non-cropped image discriminator, and a discrimination result determination unit that obtains a final discrimination result using both of them.
An image identification device including. The cropped image generation unit acquires the identification target area from the input image, and generates the cropped image as an image including a predetermined margin area in the identification target area,
The image identification device according to claim 1. The cropped image generation unit acquires the identification target area by an identification target area determiner obtained by machine learning,
The image identification device according to claim 2. The cropped image generation unit acquires the classification target area by extracting a region activated in an activation map of the non-cropped image classification device,
The image identification device according to claim 2. The non-cropped image generation unit generates the non-cropped image as an image obtained by performing edge enhancement processing and sidecut processing on the input image,
The image identification device according to claim 1. The non-cropped image generation unit, the input image is the non-cropped image,
The image identification device according to claim 1. The discrimination result determining unit obtains a final discrimination result by obtaining an average of an output value of the cropped image discriminator and an output value of the non-cropped image discriminator,
The image identification device according to claim 1. The discrimination result determination unit connects the output layer of the cropped image discriminator and the output layer of the non-cropped image discriminator to form a fully combined layer, and an output layer through the fully combined layer finally provides Get the identification result,
The image identification device according to claim 1. Further, a display unit for displaying the output value of the cropped image classifier and the output value of the non-cropped image classifier,
The image identification device according to claim 1. The identification target area is a lesion area,
The image identification device according to claim 1. A cropped image generation step of generating a cropped image that has been cropped, which is a process of cutting out an image including an identification target area from the input image;
A non-cropped image generation step for generating a non-cropped image from which the crop processing has not been performed, from the input image;
A crop image learning step of learning a crop image identifier with the crop image,
A non-cropped image learning step of learning a non-cropped image identifier with the non-cropped image,
A classifier learning method including. A cropped image generation step of generating a cropped image that has been cropped, which is a process of cutting out an image including an identification target area from the input image;
A non-cropped image generation step for generating a non-cropped image from which the crop processing has not been performed, from the input image;
A cropped image identifying step for identifying the cropped image,
A non-cropped image identifying step for identifying the non-cropped image,
An identification result determination step of obtaining a final identification result using both the identification result by the cropped image identification step and the identification result by the non-cropped image identification step,
Image identification method including. In the computer of the image identification device,
A cropped image generation step of generating a cropped image that has been cropped, which is a process of cutting out an image including an identification target area from an input image,
A non-cropped image generation step of generating a non-cropped image from which the crop processing has not been performed from the input image,
A cropped image identifying step of identifying the cropped image,
A non-cropped image identifying step for identifying the non-cropped image, and
An identification result determination step of obtaining a final identification result by using both the identification result by the cropped image identification step and the identification result by the non-cropped image identification step,
A program to execute.

Images