JP2019028650A - Image identification device, learning device, image identification method, learning method and program - Google Patents

Abstract

To provide an image identification device, a learning device, an image identification method, a learning method and a program capable of performing image recognition with high accuracy without depending upon intuition.SOLUTION: An image processing device includes acquisition means 1100 for acquiring an input image by a sensor value of an imaging device; and identification means 1300 for identifying the acquired input image by the sensor value by using a discriminator having a conversion unit. At least the conversion unit in the discriminator performs learning on the basis of a learning image by the sensor value of the imaging device and correct answer data attached to the learning image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image identification technique such as classifying an image into a predetermined class or dividing an image into regions of a plurality of classes.

  Research for classifying images into predetermined classes has been widely performed, and in recent years, tasks for classifying images into very many classes have been studied. Non-Patent Document 1 discloses an image classification technique based on deep learning.

  In addition, many researches on area division that divides an image into multiple areas have been conducted, and semantic areas such as human areas, automobile areas, road areas, building areas, and empty areas are also extracted from images. The problem of cutting out has been actively researched. Such a problem is called semantic segmentation and is considered to be applicable to image correction corresponding to the type of subject, scene interpretation, and the like.

  In performing semantic region division, many methods have been proposed in which class labels for each position of an image are identified in units of small regions (superpixels). The small area is mainly cut out from the image as a small area having similar characteristics, and various methods such as Non-Patent Document 2 have been proposed. Each small region obtained in this way is used to identify a class label using the internal feature amount or the surrounding context feature amount together. Usually, region identification is performed by learning such a local-based region classifier using various learning images.

  In recent years, research on region segmentation using deep learning has also been conducted. In Non-Patent Document 3, an intermediate layer output of CNN (Convolutional Neural Network) is used as a feature amount, and class determination results for each pixel based on a plurality of intermediate layer features are integrated. In Non-Patent Document 3, semantic region division of an image is performed in this way. In the method of Non-Patent Document 3, class determination is directly performed for each pixel without using the small region division result as described above.

  In performing such image classification such as image classification and semantic region division, what is given as input data of the discriminator is usually an image developed by the user's hand after camera internal processing or photographing. Originally, the developed image is something that the user can see and enjoy, so the image development method is determined based on the beauty of appearance. However, such a normal development method is not always suitable for an image identification task. For example, in an image that is slightly overexposed to make a white wall look beautiful, it is difficult to distinguish the cloudless sky and the wall. On the other hand, it is considered that an image in which the wall texture can be seen by photographing darkly is suitable for classifying the wall and the sky.

  In Patent Document 1, an image for which under / overexposure is suppressed by selecting a correction function from among a plurality of gamma correction functions according to an average luminance value of a RAW image obtained from an imaging device is referred to as a display image. It has been proposed to generate it separately and use it for object detection processing.

JP, 2014-11767, A

“ImageNet Classification with Deep Convolutional Neural Networks”, A.M. Krizhevsky, I.K. Suskever, G.M. E. Hinton, Proc. Neural Information Processing Systems, 2012. “SLIC Superpixels”, R.A. Achanta, A .; Shaji, K .; Smith, A.M. Lucchi, EPFL Technical Report, 2010. “Fully Convolutional Networks for Semantic Segmentation”, Long, Shelhamer, and Darrell, CVPR2015. Hypercolumns for Object Segmentation and Fine-Grained Localization, B.H. Hariharan, P.A. Arbelaez, R.A. Girstick and J.M. Malik, CVPR2015.

  However, in Patent Document 1, when the correction function prepared is a parameter that is intuitively set by a person, and whether the correction value is good or bad depends on the intuition, and high-accuracy image identification cannot be performed. was there. Therefore, an object of the present invention is to enable high-accuracy image identification.

  The present invention includes an acquisition unit that acquires an input image based on a sensor value of an imaging device, and an identification unit that identifies the input image based on the acquired sensor value using a discriminator having a conversion unit, At least the conversion unit of the discriminator is learned based on a learning image based on sensor values of the imaging device and correct data provided to the learning image.

  According to the present invention, highly accurate image identification can be performed.

1 is a schematic block diagram showing a functional configuration of an image processing apparatus in each embodiment. The flowchart which shows the learning process in 1st Embodiment, and an identification process. The figure explaining the class label in the semantic area | region division | segmentation in 1st Embodiment. The figure explaining the structure of the discriminator in 1st Embodiment. The figure explaining a gamma correction function and its approximation function in 1st Embodiment. The figure explaining the structure of the conversion part in 1st Embodiment. The figure explaining the structure of the discriminator in each embodiment. The figure explaining the structure of the adjustment part in 1st Embodiment. The figure explaining the structure of the discriminator in each embodiment.

[First Embodiment]
The details of the first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram illustrating a functional configuration of an image processing apparatus according to each embodiment, and FIG. 1A is a schematic block diagram according to the present embodiment. The image processing device functions as a learning device at the time of learning and an image identification device at the time of identification.

  Such an image processing apparatus has a hardware configuration such as a CPU, a ROM, a RAM, and an HDD. When the CPU executes a program stored in a ROM, an HD, or the like, for example, each of the functional configurations and flowcharts described below are performed. Processing is realized. The RAM has a storage area that functions as a work area where the CPU develops and executes the program. The ROM has a storage area for storing programs executed by the CPU. The HDD has a storage area for storing various types of data including various programs necessary for the CPU to execute processing, data on threshold values, and the like.

  First, an outline of a device configuration during learning in the present embodiment will be described. Here, learning is to generate a discriminator used for performing processing at the time of discrimination described later from a learning image prepared in advance. Details of the processing contents will be described later.

  In the first learning data storage unit 5100, first learning data is prepared in advance. The first learning data includes a plurality of developed learning images and class labels assigned to the learning images. The first learning data acquisition unit 2100 reads a learning image and a class label from the first learning data storage unit 5100. The first learning unit 2200 learns the first discriminator from the error between the discrimination result obtained by inputting the learning image to the discriminator and the class label. The first discriminator obtained by learning is stored in the first discriminator storage unit 5200.

  Second learning data is prepared in advance in the second learning data storage unit 5300. The second learning data includes a learning image obtained by an image sensor such as a digital camera and a sensor value before development and a class label assigned to each learning image. The second learning data acquisition unit 2300 reads the learning image and the class label from the second learning data storage unit 5300. In the conversion part addition part 2400, a 1st discriminator is read from the 1st discriminator memory | storage part 5200, and a 2nd discriminator is produced | generated by adding a conversion part to the input side. The second learning unit 2500 learns the second discriminator from the error between the discrimination result obtained by inputting the learning data in the second learning data to the second discriminator and the class label. The second discriminator obtained by learning is stored in the second discriminator storage unit 5400.

  Next, an outline of the device configuration at the time of identification will be described. Here, the term “identification” refers to image identification for an unknown input image. Details of the processing contents will be described later.

  In the input data acquisition unit 1100, the input image obtained by the sensor value before development and obtained by the imaging device and the shooting information corresponding to the input image are read. In the discriminator setting unit 1200, the second discriminator obtained by learning in advance is read from the second discriminator storage unit 5400 and set. In the identification unit 1300, the acquired input image is input to the set conversion unit of the second classifier, and the identification result is obtained. The obtained identification result is sent to the identification result output unit 1400, and the result is presented to the user or another device.

  In the present embodiment, the functional configuration at the time of learning and the functional configuration at the time of identification are described as being realized by the same device (image processing device), but may be realized by different devices. The first learning data acquisition unit 2100, the first learning unit 2200, the second learning data acquisition unit 2300, the conversion unit addition unit 2400, and the second learning unit 2500 are all realized on the same device. Although described as a thing, it is good also as an independent module. Moreover, you may implement | achieve as a program mounted on an apparatus. The first learning data storage unit 5100, the first discriminator storage unit 5200, the second learning data storage unit 5300, and the second discriminator storage unit 5400 are realized as storage inside or outside the apparatus.

  The input data acquisition unit 1100, the classifier setting unit 1200, the identification unit 1300, and the identification result output unit 1400 may all be realized on the same device, or may be independent modules. Further, the program may be implemented as a program installed on the apparatus, or may be implemented as a circuit or a program inside an imaging apparatus such as a camera. When the second discriminator storage unit 5400 is realized by different devices at the time of learning and at the time of identification, different storages may be used. In that case, the classifier obtained at the time of learning may be used by copying or moving to the storage in the identification device.

  Next, details of processing according to the present embodiment will be described. Here, as image identification, explanation will be given by taking an example of semantic area division of an image. First, the processing at the time of learning of this embodiment will be described. FIG. 2A is a flowchart showing processing at the time of learning in the present embodiment.

In the first learning data acquisition step S2100, the first learning data acquisition unit 2100 reads the learning image and the class label from the first learning data storage unit 5100 as learning data. In the first learning data storage unit 5100, a plurality of developed learning images and class labels are prepared in advance. The learning image is specifically image data taken by a digital camera or the like and developed by a developing program inside or outside the camera. Usually, it is given in a format such as JPEG, PNG, or BMP, but this embodiment is not limited to the format of the learning image. It is assumed that the number of prepared first learning images is N ₁ and the n-th first learning image is written as I _n (n = 1,..., N ₁ ). The class label in the semantic area division is an identification class assigned as a label to each area of the learning image.

FIG. 3 shows an example of class labels in semantic area division. 3A is an example of a learning image, and 540 in FIG. 3B is an example of a class label corresponding to the learning image 500. In this way, correct data to which a correct class label corresponding to an image is given is generally called GT (Ground Truth) in image identification. Class labels such as sky 541, hair 542, face 543, clothes 544, flower 545, and leaf stem 546 are given to each region. Here, a semantic class label is taken as an example, but a class label may be given according to attributes of a region such as a glossy surface, a matte surface, and a high frequency region. Also, a class in which a plurality of types of objects such as sky and tree branches are mixed may be defined. Assume that there are M kinds of area classes in total. The class label set corresponding to all pixels of the learning image I _n and GT _n.

  In the first learning step S2200, the first learning unit 2200 learns the first discriminator based on the first learning data. Here, a description will be given assuming that a CNN (Convolutional Neural Network) is used as the first discriminator. CNN is a neural network that performs identification tasks by gradually consolidating local features of an input signal by repeating a convolution layer and a pooling layer, and converting them into features that are robust against deformation and displacement. It is a network.

  FIG. 4 is a diagram for explaining the structure of the CNN used in this embodiment. The CNN is composed of 610 and 620 in FIG. 4A, and is called a convolution layer and a complete coupling layer, respectively. These are portions that perform feature extraction and pattern classification as their respective roles, and are hereinafter referred to as a feature extraction unit 610 and a classification unit 620 so as not to lose generality. Reference numeral 611 denotes an input layer, which receives a convolution calculation result at each position of the input image 630 as a signal. Reference numerals 612 and 613 denote intermediate layers, and signals are sent to the final layer 615 through a plurality of intermediate layers. In each layer, a convolution layer and a pooling layer are sequentially arranged, and a convolution operation and signal selection by pooling are repeated. The output signal in the final layer 615 of the feature extraction unit 610 is sent to the classification unit 620. In the classification unit 620, the elements in each layer are fully coupled to the previous and subsequent layers, and a signal is sent to the output layer 640 by a product-sum operation using a weight coefficient. The output layer 640 has the same number of output elements as the number of classes M, and the signal intensity of each element is compared, and the class corresponding to the element that outputs the largest signal becomes the output class label of the pixel.

The value of the output signal obtained at the output layer when entering the learning image I _n the CNN is compared with the teacher signal, the learning is performed. Here, the pixels of the learning image _{I n} (i, j) in the teacher signal t output element corresponding to the class m (n, i, j, m) is the class label of the pixel of GT _n (i, j) Is C (C = 1,..., M), the following equation 1 is defined.

Position in the input image I _n obtained as a result of the input to the discriminator output layer (i, j) output element corresponding to the class m the (i, j, m) the output signal of y (n, i, j, m), the error in the output element (i, j, m) is calculated as shown in Equation 2 below.

E _{(n, i, j, m)} = (y (n, i, j, m) −t (n, i, j, m)) ² (Formula 2)
By sequentially backpropagating the error from the output layer to the input layer by the error backpropagation method, the weight coefficient of each layer in the CNN is updated by the stochastic gradient descent method or the like.

  The weight coefficient of CNN in learning may be started from a random initial value, but a learned weight coefficient may be given as an initial value for some task. For example, the class label of the area division task needs to be given for each pixel, but the class label of the image classification task may be given one label for one image. Therefore, there is a difference of several tens to several hundreds of times that a person inputs a class label as a GT in advance. For this reason, a large number of learning images for image classification tasks are publicly available and can be easily obtained. For example, in ILSVRC (ImageNet Large-scale Visual Recognition Challenge), 1.2 million learning images for image classification are disclosed. Therefore, the initial value of the weighting coefficient of CNN may be initially learned once by such an image classification task, and the learning of the area division task may be started using the weighting coefficient based on the learning result as the initial value. In this way, the weight coefficient of CNN learned in the first learning step is stored in the first discriminator storage unit as a discriminator parameter.

  Next, in the second learning data acquisition step S2300, the second learning data acquisition unit 2300 acquires learning images, shooting information, and class labels that have not been developed from the second learning data storage unit 5200 as learning data. Read as.

  In the second learning data storage unit 5200, a plurality of undeveloped learning images and class labels are prepared in advance. Also, shooting information related to the learning image is obtained. The learning image is specifically an enumeration of sensor values in an image element such as a CCD or CMOS before being taken with a digital camera or the like and developed, and is generally called a RAW image. As for the luminance value in the RAW image, the absolute amount of the luminance of each color channel in each pixel can be obtained using the shooting information as follows.

It is assumed that the brightness Bv value and the appropriate level value opt of the sensor are obtained as shooting information. When the sensor measurement value of the R channel in the Bayer array on the RAW image corresponding to the pixel position (i, j) in a certain image is R _RAW , the absolute amount of luminance in the R channel of the pixel (i, j) The value of R _Bv can be obtained by the following Equation 3.

Similarly, G _Bv and B _Bv which are absolute luminance amounts of the G channel and the B channel are also obtained from Equations 4 and 5.

  By using such a conversion formula, it is possible to obtain a map of absolute luminance amounts related to the second learning image by the RAW image.

The number of prepared second learning images is N _2, and learning images based on the absolute luminance map obtained by conversion from the n-th second learning image are J _n (n = 1,..., N ₂ ). The class label has the same definition as the class label in the first learning image. The class label set corresponding to all pixels of the learning image J _n and GT _n. Since an image with a RAW image is more difficult than collecting images without it, the number N ₂ of _second learning images is generally less than N _{1 in} many cases. In fact, many amateur photographers do not publish RAW images, so most of the images that can be collected on the web or the like do not involve RAW images. Further, the learning image used in the second learning data can be used for the first learning data by developing.

  In conversion unit addition step S2400, conversion unit addition unit 2400 reads the first discriminator from first discriminator storage unit 5200, and adds the conversion unit to the input layer side of the read first discriminator. .

  First, the CNN weighting factor learned in the first learning step S2200 is read from the first discriminator storage unit 5200. The read weight coefficient is set to CNN. A conversion unit 651 is added to the set CNN as shown in FIG. On the input side of the conversion unit 651, a learning image 651 based on an absolute luminance amount prepared in the second learning data acquisition step S2300 is input. The image 652 converted through the conversion unit 650 is input to the input layer 611 of the CNN in the same manner as the developed image.

  The converter is added as a new layer of CNN. Normally, conversion from a RAW image to a developed image is performed by gamma correction and correction by white balance. The gamma correction function is defined as Equation 6 below.

y = x ^γ (Formula 6)
Here, x is a value of a RAW image at an arbitrary pixel, that is, a sensor value in the imaging device, and y is a luminance value after development of the pixel. The value of the control parameter γ varies depending on the camera, manufacturer, shooting mode, and the like. FIG. 5 is a diagram for explaining a gamma correction function and its approximate function in this embodiment, and FIG. 5A shows an example of the gamma function. In FIG. 5A, 701 is a curve of a gamma correction function when γ = 1, 702 is γ = 0.5, and 703 is γ = 2. The white balance is a weight for each channel of the corrected luminance values. Here, it is considered that the gamma correction function for the input signal is approximated as shown in Equation 7 below.

y = w ₂ tanh (w ₁ x + b ₁ · z ₁ ) + b ₂ · z ₂ (Formula 7)
Here, z ₁ and z ₂ are variables that vary depending on the shooting environment, and w ₁ , w ₂ , b _1, and b ₂ are weighting factors. As shown in FIG. 5B, this function can be an approximation of the γ correction function. Reference numerals 711, 712, and 713 in FIG. 5B are functions such as the following Expression 8, Expression 9, and Expression 10, respectively.

y = 1.1 tanh (x−0.5) +0.5 (Formula 8)
y = 5 tanh (x + 1) -3.8 (Formula 9)
y = 5 tanh (x-2) +4.8 (Formula 10)
FIG. 6 is a diagram showing the configuration of the conversion unit in the present embodiment, and the format in Expression 7 can be expressed by a combination of elements as shown in FIG. An arbitrary pixel of the learning image 651 based on the absolute luminance amount and a corresponding pixel in the converted image 652 are combined by an element 653 and an element 654. The absolute luminance value in the learning image 651 corresponds to x in Equation 7, is weighted by the weight w ₁ , and is input to the element 653 having a nonlinear function tanh as an input / output function. The output signal of the element 653 is weighted by the weight w ₂ and input to the element 654 having a linear function that increases monotonously as an input / output function.

From the learning image 651, the feature quantity 656 describing the scene of the image is extracted through the scene feature extractor 655. The scene description feature quantity 656 can be assumed to be HOG, FisherVector, a color histogram, or the like, but this embodiment is not limited by the type of the feature quantity. Further, the scene description feature is not limited to the image feature as described above. For example, as supplementary information in the captured image, a feature amount may be extracted from a gyro sensor value as orientation information of the imaging device with respect to the ground axis or time information by a clock. An example is shown in FIG. For example, since the ground direction can be obtained as a three-axis value from the value of the gyro sensor, it can be used as a three-dimensional scene description feature vector if it is normalized. In addition, time information by a clock can be used as a feature value with one day as one cycle if it is associated with 15 deg as a cyclic function by sin and cos. Similarly, the calendar can be used as a feature value with 15 months as one month and one cycle as one year. The scene description feature quantity 656 corresponds to z ₁ and z ₂ in Equation 7. Then, it is weighted by the product-sum operation using the weight vector b ₁ and input to the element 653, is weighted by the product-sum operation using the weight vector b ₂ , and is input to the element 654.

Here, z ₁ = z _{2 has} been described, but the scene description feature may be divided as a separate item, for example, z ₁ is HOG and z ₂ is FisherVector. The scene description feature is a feature for adjusting the bias of the tahh curve, and weighting with the weight coefficient vectors b ₁ and b ₂ corresponds to performing a kind of scene identification. For example, in an outdoor scene on a sunny day and an indoor scene surrounded by a white wall, different scene description characteristics are obtained due to the difference in the objects reflected in the image and the difference in absolute luminance. Different correction amounts are applied as conversion biases. Further, when sending the scene description feature 656 to the elements 653 and 654, a neural network having a multilayer structure may be added instead of the linear sum by the weighting factors b ₁ and b ₂ . FIG. 6 (c) shows an example, and 657 and 658 are multi-layer neural networks that input a scene description feature 656 into the input layer and output one output signal f (z) and g (z), respectively. is there. In this case, Expression 7 is expressed as Expression 11 below. In the following description, the coupling coefficient of each neural network f and g may be replaced with b ₁ and b ₂ for reading.

y = w ₂ tanh (w ₁ x + f (z ₁ )) + b ₂ · g (z ₂ ) (Formula 11)
The output signal of the element 654 is treated as the value of the corresponding pixel of the image 652 passed to the input layer of the CNN as it is. When such coupling is provided for each of the R _Bv , G _Bv , and B _Bv channels, the values of w ₂ and b ₂ represent the balance of each channel, which approximates the value of white balance. To do. In this manner, the learning image 651 before development is image-converted by conversion approximated to the development processing from the absolute luminance amount.

In the second learning step S2500, the second learning unit 2500 learns the discriminator together with the conversion unit added in the conversion unit addition step S2400. The weighting factors w ₁ , w ₂ , b ₁ , b ₂ that define the image conversion set in the conversion unit adding step S2400 are learned from the learning image and the class label acquired by the second learning data acquisition unit. When the learning image J _n is input to the conversion unit 650 in FIG. 4B and an output signal is obtained via the feature extraction 610 and the classification unit 620, the value is compared with the class label to convert the entire CNN Learn the weighting factor of the part.

  The coupling coefficient between the feature extraction unit 610 and the classification unit 620 uses the value obtained in the first learning step S2200 as an initial value. The weighting factor in the conversion unit 650 may be learned from a random initial value. Alternatively, only the conversion unit 650 may be learned independently from the CNN, and the state may be learned as an initial value together with the CNN. In order to initially learn only the conversion unit 650, the conversion unit 650 is regarded as a three-layer neural network, the learning image 651 based on the luminance absolute amount map before development is input, and the output signal of the element 653 is used as the output signal of the network. I reckon. Regression learning may be performed by back propagation by giving a luminance value of a developed image with appropriate exposure as a teacher signal. When the initial values of the weighting factors of the conversion unit 650, the feature extraction unit 610, and the classification unit 620 are determined, learning is performed through all of them. In this way, by learning through the learning image through all of the conversion unit 650, the feature extraction unit 610, and the classification unit 620, the parameters of the conversion unit 650 are also corrected to reduce the identification error with respect to the learning image. be able to. This is equivalent to correcting the image development method so that it is easy to identify, not good appearance. When the conversion unit 650, the convolution layer 610, and the complete coupling layer 620 are learned, the obtained weight coefficient is stored in the second discriminator storage unit 5400.

  The step of identifying an actual input image using the classifier learned in this way will be described in detail below. FIG. 2B is a flowchart showing processing at the time of identification according to the present embodiment.

  First, in input data acquisition step S1100, the input data acquisition unit 1100 acquires image data before development obtained from the imaging apparatus. Assume that the method of input data is the same as the pre-development image in the second learning data acquisition step S2300. That is, a RAW image based on sensor values obtained by the imaging apparatus is converted into a map of absolute luminance using Formula 3, Formula 4, and Formula 5 using imaging information.

  In the classifier setting step S1200, the classifier setting unit 1200 reads a learned classifier from the second classifier storage unit 5400. Here, the discriminator setting step S1200 is performed after the input data acquisition step S1100, but the procedure of these two steps may be reversed. When the discriminator is always secured in the memory and the input image is processed one after another, the processing after the input data acquisition step S1100 may be repeatedly performed after the discriminator setting step S1200. The discriminator set in the discriminator setting step S1200 is a discriminator composed of the conversion unit and the CNN represented in FIG.

  In the identification step S1300, the identification unit 1300 performs identification processing of the input image acquired in the input data acquisition step S1100, using the identifier set in the identifier setting step S1200. The input image acquired as the absolute luminance map is input to the converter 650 of the discriminator in FIG. 4B, and the converted image is input to the input layer 611 in the feature extraction unit 610 of the CNN. In the convolution layer 610, the signal of the input image is forwardly propagated to each layer, and the converted signal becomes an output signal of the output element 621 assigned to each identification class via the all coupling layer 620. The class label corresponding to the output element with the largest signal becomes the class identification result of the pixel.

  In the identification result output step S1400, the identification result output unit 1400 outputs the identification result obtained in the identification step S1300. The processing performed in the identification result output step S1400 depends on the application that uses the identification result, and does not limit the present embodiment. For example, in the case of an image correction application in which the image processing given for each area is changed depending on the area class, the class label of each pixel may be output to the image correction program. At this time, if processing weighting is necessary due to the ambiguity of each class, the output signal value of the output element 621 corresponding to each class label may be output as it is as the class likelihood. If only the identification results for a specific class are needed, the results for other classes may be discarded and only the identification results for the necessary classes may be output.

  In the above description, image segmentation has been described as an example of image identification processing, but the present embodiment can also be applied to an image classification task. FIG. 9 is a diagram for explaining the structure of the discriminator in each embodiment. In the case of an image classification task, as shown in FIG. 9A, output signals in all pixels of the final layer 615 of the CNN feature extraction unit 610 are obtained. , Input to the classification unit 620.

  As described above, according to the present embodiment, by learning the conversion unit that converts the input image to the classifier, an image suitable for identification can be obtained, and highly accurate image identification can be realized. Become. Further, by adding a conversion unit to the image input side of the classifier and performing additional learning using a learning image before development, it is possible to learn the classifier including development processing. As a result, it can be expected that more accurate identification can be performed without forcibly performing identification processing on an image that has undergone appearance-oriented development processing. In addition, a part of the image other than the conversion unit is pre-learned with a large amount of the developed image, and the image is converted with a learning image with relatively few pre-development images by a process of learning the discriminator including the conversion unit as an initial value. Can be learned. This is effective when it is difficult to align a large number of learning images based on the pre-development image compared to the post-development image.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the first embodiment, an example in which CNN is used as a discriminator is shown. However, when CNN is used as a discriminator, edge information and the like remain strongly from a forward propagation signal due to repetition of convolution processing. Thus, the absolute value information such as the luminance value tends to fade gradually. When color and brightness are effective features, the loss of such information causes a reduction in identification accuracy. For example, a solid wall of a pastel color or a shining object such as a sun lamp can be expected to improve the identification accuracy if the luminance of each color channel is sufficient.

  Further, depending on the imaging device, distance information may be obtained by image plane phase difference AF or the like, and combined use with distance information can be expected to improve robustness against scaling caused by the distance to the subject. . In the present embodiment, a mode will be described in which absolute value information given to each pixel of an input image is taken into a discriminator.

  In addition, the same code | symbol is attached | subjected about the structure already demonstrated by 1st Embodiment, and the description is abbreviate | omitted. Since the apparatus configuration in this embodiment is represented in FIG. 1A as in the first embodiment, detailed description thereof is omitted.

  First, the learning process of this embodiment will be described. The flowchart is the same as the flowchart of the learning process of the first embodiment shown in FIG. 2A, but the contents of the process in some steps are different from those of the first embodiment.

  Since the first learning data acquisition step S2100, the first learning step S2200, and the second learning data acquisition step S2300 are the same as those in the first embodiment, description thereof is omitted.

In the conversion unit addition step S2400, first, the conversion unit addition unit 2400 reads the CNN learned in the first learning step S2200 from the first discriminator storage unit 5200. And the conversion part addition part 2400 adds the conversion part 650 and the transmission part 660 like FIG.4 (c) with respect to this CNN. Since the conversion unit 650 is the same as that of the first embodiment, the description thereof is omitted. The transmission unit 660 is a multilayer network for transmitting absolute value information in each pixel included in an image before development. Each of the layers 661, 662, 663, ..., 665 is a layer corresponding to each convolutional layer 611, 612, 613, ..., 615 of the CNN. The number of channels in each layer corresponds to the number of information in each pixel of the image. For example, when the absolute luminance information (R _Bv , G _Bv , B _Bv ) shown in the first embodiment is used as absolute value information, each layer has three channels corresponding to R _Bv , G _Bv , B _Bv. Will have. The size of the feature surface in each channel of the layer shall be equal to the size of the feature surface of the corresponding layer of the CNN. Each layer of the transmission unit 660 is simply in a scaling relationship, and is resized by a technique such as linear interpolation, bicubic interpolation, or nearest neighbor method. If the calculation time is important, resizing may be performed by referring to a table by thinning. A coupling coefficient that is corrected by learning is not allocated to a coupling portion between layers in these transmission units. Also, since the size of the layer 661 of the transmission unit 660 corresponding to the input layer 611 of the CNN is the input image size itself, the pixel information of the input image is directly set as it is. The value of the position 669 corresponding to the pixel position of the output unit in the layer 611 on the most input side of the transmission unit 660 is input as it is to the input layer of the classification unit 620.

The value in each layer of the transmission unit 660 is input as a bias value together with the bias coefficient to the corresponding CNN convolution layer. An input signal u _lm (i, j) for the element at the position (i, j) of the channel m in the l-th convolutional layer is expressed as Equation 12 below.

Here, the first term on the right side represents coupling in the CNN, K is the number of channels in the 1-1 layer of the CNN, and H is the width of the convolution filter between the 1-1 layer and the 1st layer. h _lpqkm is the value of the position (p, q) in the filter center coordinates of the convolution filter that combines the m-th channel of the l-th layer and the k-th channel of the ( _1-1) th layer. Further, z _l−1 (i, j, k) is an output signal at the position (i, j) in the l− _1th layer, and b _lm is a bias coefficient in the mth channel of the lth layer. The second term on the right side represents the coupling from the transmission unit 660, R is the number of channels of pixel information, J _l (i, j, r) is the value in the r-th channel of the l-th layer of the pixel information transmission unit, b _lr is the bias coefficient. Among these, parameters corrected by learning are h _lpqkm , b _lm , and b _lr .

  In the second learning step S2500, the second learning unit 2500 learns the CNN internal coupling coefficient using the learning image acquired in the second learning data acquisition step S2300. The second learning unit 2500 learns the coefficient that combines the CNN with the conversion unit 650 and the transmission unit 660 added in the conversion unit addition step S2400, using the learning image together with the CNN internal coupling coefficient. Except that the above-described coupling coefficient is added as a learned parameter, the basic algorithm related to learning is the same as that in the first embodiment, and a description thereof will be omitted. The parameters corrected by learning are stored in the second discriminator storage unit 5400.

  Next, the identification process of this embodiment will be described. The flowchart is the same as the flowchart of the identification process of the first embodiment shown in FIG. 2B, but the contents of the process in some steps are different from those of the first embodiment.

  Since the processing of the input data acquisition step S1100 and the discriminator setting step S1200 is the same as that of the first embodiment, description thereof is omitted.

  In the identification step S1300, the identification unit 1300 obtains an identification result by inputting the input image based on the absolute luminance value before development into the network shown in FIG. 4C. Since the transmission of the signal in the forward propagation direction at the time of identification is the same as that at the time of learning, detailed description is omitted.

  Since the processing in the identification result output step S1400 is the same as that in the first embodiment, description thereof is omitted.

  As absolute value information, distance information obtained by image plane phase difference AF or the like of the imaging system may be given in addition to the absolute luminance value. Since the distance information gives information on the absolute size and three-dimensional shape of the target object, it is easy to distinguish between similar objects by scaling, real objects, and photographs and paintings. For example, this is effective in distinguishing between a person or a giant figure drawn on a signboard and a real person.

FIG. 4D shows a configuration when a distance information transmission unit 670 is added. Reference numeral 653 denotes a distance map having distance information of each pixel. When the distance measurement points are sparse with respect to the pixel density of the image, the distance map for each pixel can be calculated by supplementing the distance of each pixel by linear interpolation, bicubic interpolation, or nearest neighbor method. That's fine. In this case, the number of channels is 4 channels of R _Bv , G _Bv , B _Bv , and distance information. Since the learning process and the identification process in the configuration of FIG. 4D are the same as the example of the absolute luminance amount in FIG. 4C, the description thereof is omitted. Furthermore, in the case where the luminance information and the distance information are used in combination, two transmission units may be prepared in parallel as shown in FIG.

  In addition, when using distance information, it is also possible to add a gradient feature different from image information. Incorporating a gradient of distance into the feature makes it easy to distinguish a photograph from a three-dimensional object. In this case, as shown in FIG. 9B, another feature extraction unit is arranged in parallel. In this case, not only the conversion unit 650 and the transmission unit 660 but also the feature extraction unit 710 for distance information is added in the conversion unit addition step S2400 during learning. In the distance information feature extraction unit 710, the distance map is input to the input layer 711, and the output signal in the final layer 715 is given as an input signal to the classification unit 620. In the second learning step S2500, the error signal with respect to the learning image is back-propagated, the classification unit 620, the image and distance information feature extraction units 610 and 710, the conversion unit 650, the image and distance information transmission unit 660. And the weighting factor of 670 are updated by learning.

  As described above, in this embodiment, it is possible to improve the identification accuracy by inputting at least one of the luminance value information and the distance information to the intermediate layer of the neural network. In particular, information with absolute values that tend to fade toward the output layer of the CNN is inserted into the intermediate layer of feature extraction, so that the accuracy of identification can be improved for the identification of target objects whose color and brightness are important information. Improvement can be expected. Further, the distance information that can be obtained by the image plane phase difference or the like can also be used for the discriminator by a similar method, and further improvement in discrimination accuracy can be expected.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In the first embodiment, the method of performing the same image conversion on the entire input image according to the scene characteristics of the input image has been described. This means that a developing method for improving identification accuracy while approximating a normal developing method at the conversion unit is obtained by learning. Here, in order to further improve the identification accuracy, different development may be performed depending on the region. This embodiment demonstrates the form which corrects the conversion parameter of a conversion part for every area | region. In addition, the same code | symbol is attached | subjected about the structure already demonstrated in 1st, 2nd embodiment, and the description is abbreviate | omitted.

  FIG. 1B is a schematic block diagram illustrating a functional configuration of the image processing apparatus according to the present embodiment. First, a device configuration when functioning as a learning device during learning will be described. Since the first learning data acquisition unit 2100, the first learning unit 2200, the second learning data acquisition unit 2300, the conversion unit addition unit 2400, and the second learning unit 2500 are the same as those in the first implementation, Description is omitted. The first learning data storage unit 5100, the first discriminator storage unit 5200, the second learning data storage unit 5300, and the second discriminator storage unit 5400 are also the same as in the first embodiment. Therefore, explanation is omitted.

  The adjustment unit addition unit 2600 adds an adjustment unit that adjusts the conversion unit based on the primary identification result to the classifier. The third learning unit 2700 performs identification processing on the second learning data using the conversion unit and the discriminator learned by the second learning unit 2500, and based on the error generated there, the adjustment unit To learn. The learned parameters of the adjustment unit are stored in the adjustment unit storage unit 5500.

  Next, a device configuration when functioning as an image identification device at the time of identification will be described. The input data acquisition unit 1100, the discriminator setting unit 1200, and the discriminating unit 1300 are the same as those in the first embodiment, and thus description thereof is omitted. The adjustment unit setting unit 1500 reads parameters of the adjustment unit from the adjustment unit storage unit 5500 and sets an adjustment unit that adjusts the conversion unit. The re-identification unit 1600 uses the identification result in the identification unit 1300 and the adjustment unit to perform identification processing by the classifier again. The obtained re-identification result is sent to the identification result output unit 1400, and the result is presented to the user or another device.

  Next, the learning process according to the present embodiment will be described with reference to FIG. Since the first learning data acquisition step S2100 to the second learning step S2500 are the same as those in the first embodiment, description thereof is omitted.

  In the adjustment unit addition step S2600, the adjustment unit addition unit 2600 adds an adjustment unit that adjusts the development processing by the conversion unit based on the primary identification result. FIG. 7 is a diagram illustrating the configuration of the classifier according to each embodiment, and a configuration example according to this embodiment is illustrated in FIG.

  First, in this step, a class identification result is obtained by the element 621 by inputting a learning image to the discriminator learned up to the second learning step S2500. From the obtained identification result, in the second learning image, a region having a small error from the class label in GT, for example, a region having an error of 0.2 or less is excluded from the learning data. With respect to the remaining learning data, output signal vectors 691 relating to all channels in the final layer 615 of the feature extraction unit 610 are extracted. Although the vector 691 has been described as an output signal in the final layer 615 here, values in other layers may be used. For example, the values of the input layer 611 may be used, or the values of all the layers 611 to 615 may be connected and used. These values are input to the adjustment unit 690, and a signal output from the adjustment unit 690 is provided to the conversion unit 650 to adjust the image conversion process.

Here, the configuration of the adjustment unit 690 according to the present embodiment will be described with reference to FIG. The output signal of the final layer 615 in the feature extraction unit 610 and the output signal of each class in the output layer 640 are combined into a feature quantity 696. The feature quantity 696 is input to the element 693 after being weighted by the weight vector b ₃ . The feature quantity 696 is also weighted by the weight vector b ₄ and input to the element 654. The adjustment value obtained by such a structure is an approximation of the gamma correction function as in the first embodiment. Since the input signal to the element 654 is the sum of the conversion function by the conversion unit and the conversion function by the adjustment unit, it approximates development processing by a combination of two gamma correction functions. The output signal y of the element 654 corrected by both the conversion unit and the adjustment unit is described as Equation 13 below.

y = w ₂ tanh (w ₁ x + f (z ₁ )) + b ₂ · g (z ₂ ) + w ₃ tanh (b ₃ · z ₃ ) + b ₄ · z ₃ (Formula 13)
Here, z ₃ is a feature vector obtained by combining the output signal of the final layer in the convolution layer indicated by 696 and the output signal of each class in the output layer, and w ₃ , b ₃ , and b ₄ are weight coefficients. is there. The second term on the right side is an input from the adjustment unit, and is referred to as an adjustment term here.

Next, in the third learning step S <b> 2700, the third learning unit 2700 learns the weighting factors of these adjustment units 690. The third learning unit 2700 fixes the learning coefficients of all the weighting factors other than the weighting factors w ₃ , b ₃ , and b ₄ related to the adjusting unit 690 to 0, and reverses the error only for w ₃ , b ₃ , and b _4. Propagate and correct. The parameters obtained by learning are stored in the adjustment unit storage unit 5500.

  Since the learning performed here learns the output signal from the feature extraction unit 610 and the class identification signal from the classification unit as input feature amounts, it can be interpreted as learning an error pattern. That is, it is learned when an error occurs between the class identification result and the internal state of the CNN, and what image conversion is performed at that time to reduce the error. For example, if a white region is skipped at the time of primary identification and the identification result is not correct, in this embodiment, in the case of a similar error pattern, in order to reduce the error, the contrast is high in a bright portion. The conversion of the adjustment unit is corrected to be strong.

  Next, identification processing according to the present embodiment will be described with reference to FIG. First, the input data acquisition step S1100 to the identification step S1300 are performed by the same processing as in the first embodiment, and the primary identification result is obtained.

  Next, in the adjustment unit setting step S1500, the adjustment unit setting unit 1500 reads the weight coefficient of the adjustment unit from the adjustment unit storage unit 5500, and the adjustment unit 690 is set.

  In the re-identification step S1600, the re-identification unit 1600 inputs the feature vector 696 obtained by connecting the class identification signal of the primary identification result and the output signal from the feature extraction unit 610 to the adjustment unit 690, and the adjustment term is added. An image conversion function is obtained for each region. The input image is subjected to image conversion adjusted for each region through image conversion with correction by an adjustment term, and the conversion result is input to the feature extraction 610. As a result, a re-identification result is obtained in the output layer 640 via the classification unit 620. Although not shown here, the result of the re-identification step S1600 may be further used to repeatedly re-identify using the adjustment term. In that case, the calculation may be terminated at an appropriate number of repetitions or when the change in the signal of the adjustment term becomes small.

  The identification result output step S1400 is the same process as that of the first embodiment, and thus the description thereof is omitted.

  In the present embodiment, first, based on the primary identification result obtained in this way, the image conversion process is adjusted using the feature value reflecting the identification result obtained by combining the output signal and the class identification signal in the convolution layer. . Since the adjustment method is learned in a direction to reduce the error based on the learning data, it can be expected that the adjustment is performed with higher accuracy than the primary identification result.

[Other Embodiments]
In each of the above embodiments, the identification by the CNN has been described in the form of connecting the final layer 615 of the feature extraction unit 610 and the classification unit 620. However, when the fine texture of the subject is an effective feature amount, the signal from the final layer alone may not be sufficient for identification. For example, when distinguishing a white wall made of mortar and a sky with no texture caused by cloudy weather, fine texture is important information. In such a case, there is a method in which signals are extracted from all layers of the feature extraction unit 610 and passed to the classification unit 620. This is a known technique which is called a hyper column structure and is also mentioned in Non-Patent Document 4.

  Even if the hyper column structure is similarly adopted for the configuration described so far, the same processing as in each of the above embodiments can be performed. FIG. 7C shows a CNN having a hyper column structure. , 685 are output signals in the respective layers 611, 612, 613,..., 615 in the feature extraction unit 610 at the position of the pixel 621 in the output layer 640. These signal values are treated as feature vectors and input to the classification unit 620.

  A similar structure can be inserted in FIGS. 4B and 4C and FIGS. 5D and 5B. Even if the structure is a hyper column structure as described above, the learning process and the identification process can be performed with the same algorithm, and thus detailed description thereof is omitted.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 1100 Input data acquisition part 1200 Classifier setting part 1300 Identification part 1400 Identification result output part

Claims (16)

Acquisition means for acquiring an input image based on sensor values of the imaging device;
Using a discriminator having a conversion unit, and discriminating means for discriminating an input image based on the acquired sensor value,
At least the conversion unit of the classifier is learned based on a learning image based on a sensor value of the imaging device and correct data given to the learning image.   The classifier includes a classifier obtained by adding the conversion unit to a classifier learned based on first learning data including a developed learning image and correct data given to the learning image. The image identification device according to claim 1, wherein the image identification device is generated by learning based on second learning data including a learning image based on sensor values of the imaging device and correct data assigned to the learning image. .   The image discriminating apparatus according to claim 1, wherein the discriminator is a neural network.   4. The convolution neural network including a feature extraction unit that extracts image features by a multi-layer convolution operation, and a classification unit that performs pattern classification from the extracted image features. The image identification device described in 1.   The said discriminator inputs at least one of the luminance value or distance information in each pixel of the input image based on the sensor value as a weighted signal to each layer of the feature extraction unit. Image identification device. The converter is
A first output of an output signal obtained by converting an input signal composed of a weighted linear sum of weighting factors of an absolute luminance amount in each pixel of the input image and a scene feature extracted from the entire input image by a non-linear function. Layers,
A second layer that outputs an output signal obtained by converting an input signal composed of a weighted linear sum of the scene feature and the output signal of the first layer by a weighting function with a linear function;
The image identification device according to claim 3, comprising:   The image identification apparatus according to claim 6, wherein the scene feature is an image feature acquired from the entire input image.   The image identification apparatus according to claim 6, wherein the scene feature is an image feature acquired from the entire input image and imaging information when the input image is captured. Based on the identification result by the identification means, further comprising an adjustment means for adjusting the conversion unit,
The image identification apparatus according to claim 1, wherein the identification unit re-identifies the input image using a classifier having the adjusted conversion unit.   The image identification device according to claim 1, wherein an input image based on a sensor value of the imaging device is a RAW image.   The identification unit executes an identification task for classifying the input image into a plurality of classes or an identification task for dividing the input image into regions for a plurality of classes. The image identification device according to item 1. A learning device for learning a discriminator used for identifying an input image based on a sensor value of an imaging device,
First learning means for learning the first classifier based on first learning data including a learning image in which an image based on a sensor value is developed and correct data assigned to the learning image;
Adding means for adding the conversion unit to the learned first classifier to generate a second classifier;
A second discriminator used in the discriminating unit is generated by learning the second discriminator based on second learning data including a learning image based on a sensor value and correct data assigned to the learning image. 2 learning means,
A learning apparatus comprising: Acquiring an input image based on sensor values of the imaging device;
Using a discriminator having a conversion unit to identify an input image based on the acquired sensor value, and
At least the conversion unit of the classifier is learned based on a learning image based on a sensor value of the imaging device and correct data given to the learning image. A learning method for learning a discriminator used for identifying an input image based on sensor values of an imaging device,
Learning a first classifier based on first learning data including a learning image in which an image based on sensor values is developed and correct data assigned to the learning image;
Adding the converter to the learned first classifier to generate a second classifier;
A step of generating a discriminator used by the discriminating means by learning the second discriminator based on second learning data including a learning image based on sensor values and correct data assigned to the learning image. When,
A learning method characterized by comprising:   A program for causing a computer to function as the image identification device according to any one of claims 1 to 11.   A program for causing a computer to function as the learning device according to claim 12.

Images