JP2014016676A - Image region division device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve accuracy of region division when dividing regions of a principal object and the other background within an image.SOLUTION: When a specific pixel value having a high frequency value (occurrence frequency) on a histogram 207 above a rectangular frame has a high frequency value on a histogram 208 outside the rectangular frame and a lower frequency value on a histogram 209 in the rectangular frame, histogram updating means 204 updates, with respect to the specific pixel value, a frequency value in a second histogram 206 showing likelihood to be a background, which is used as an energy function for region division in region division means 201, so that this frequency value is increased relative to a frequency value on a first histogram 205 showing likelihood to be a principal object. A principal object region and a background region can be more distinctly divided by using the specific pixel value as a reference.

Description

  The present invention relates to an apparatus, a method, and a program for segmenting a main object in an image and other backgrounds.

  Sometimes you want to know the name of a flower you saw on Noyama or a roadside. Therefore, a flower image as an object is extracted from a digital flower image obtained by photographing or the like using a clustering method, and information obtained from the extracted flower image is used as a feature amount. Proposed a technique to determine the type of wild grass by calculating one or more feature quantities and analyzing the obtained feature quantities and the feature quantities of various plants registered in the database in advance using statistical methods. (For example, the technique described in Patent Document 1).

  In addition, a conventional technique is known in which an image including a main object is divided into a main object and a background using the Graph Cuts method (for example, techniques described in Non-Patent Document 1 and Patent Document 2). When region segmentation is performed, there is a possibility that the boundary is unclear due to the relationship between the main object and the background, and it is necessary to perform optimum region segmentation. Therefore, in this prior art, region division is regarded as an energy minimization problem, and a minimization method is proposed. In this prior art, the energy function is minimized by creating a graph that matches the region division and obtaining the minimum cut of the graph. This minimum cut realizes efficient area division calculation by using a maximum flow algorithm.

  In the method of dividing the main object and the background using the Graph Cuts method, the region label indicating the main object or the background to be given to each pixel in the image is updated, and the region is based on the region label and the pixel value of each pixel. A technique for performing division is known. In this case, for example, an energy function including the following cost term is defined. First, for example, a cost term that decreases as the value of a histogram for each color pixel value calculated from an image showing a main object increases is included. Further, for example, a cost term that decreases as the value of the histogram for each color pixel value calculated from the image indicating the background increases is included. Then, the main object and the background are divided into regions in the image by the energy function minimization process (for example, the method described in Non-Patent Document 1).

  Further, it may be difficult to divide the main object and the background only by the Graph Cuts method. For this reason, for example, in mounting on a so-called smartphone or the like, for example, an input device such as a touch panel is used for an approximate region where an object (for example, a flower) to be recognized exists for an image captured by the imaging device. The function to specify the rectangular frame is implemented.

Japanese Patent Laid-Open No. 2002-203242 JP 2011-35636 A

Y. Boykov and G. Funka-Lea: “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in ND Images”, Proceedings of “Interference Convence CV”. I, p. 105-112, July 2001.

  However, in the case of a flower image or the like, even when the user designates a rectangular frame as described above, the same flower is often reflected outside the rectangular frame, that is, in the background area. In such a case, when the background histogram is updated using the data of the area determined to be the background, the background histogram is actually updated using data of the same color as the flower of the main object. As a result, the area division is performed using incorrect histogram data from the next time, and the main object and the background area are easily misidentified in the main object and the background area division, and the accuracy of the area division is lowered. It had the problem that.

  An object of the present invention is to improve the accuracy of area division.

  In an example of the aspect, a device that divides a main object in an image range specified in an image and a background other than the main object into regions, and a region label indicating a main object or background to be assigned to each pixel in the image range The cost term that decreases as the value of the first histogram for each pixel value calculated from the image indicating the main object increases based on the region label and the pixel value of each pixel, and the image indicating the background Area dividing means for dividing the main object and the background in the image by the energy function minimizing process including a cost term that decreases as the value of the second histogram for each pixel value calculated from Each histogram for each pixel value on the frame, for each pixel value outside the boundary of the image range, and for each pixel value within the frame of the image range. Histogram calculating means for calculating as an on-frame histogram, an out-of-frame histogram, and an in-frame histogram; and a specific pixel value extracting means for extracting a pixel value having a power value larger than the first threshold value as a specific pixel value in the on-frame histogram; When the power value at the specific pixel value is larger than the second threshold value on the out-of-frame histogram and smaller than the third threshold value on the in-frame histogram, the power value of the second histogram at the specific pixel value is Histogram updating means for updating at least one of the two frequency values so as to increase relative to the frequency value of the first histogram;

  According to the present invention, it is possible to improve the accuracy of area division.

It is a block diagram which shows the hardware structural example of the image area division | segmentation apparatus which concerns on one Embodiment of this invention. It is a functional block diagram which shows the functional structure of the image area division | segmentation apparatus of FIG. It is a flowchart which shows the whole operation | movement of the image area division | segmentation process by this embodiment. It is explanatory drawing of a weighted directed graph. It is explanatory drawing of histogram (theta). It is a characteristic view of h _uv (X _u , X _v ). It is the figure which showed typically the relationship between the graph which has t-link and n-link, the area | region label vector X, and the graph cut. It is a flowchart which shows an area | region division process. It is a flowchart which shows a histogram update process. It is explanatory drawing of a specific pixel value. It is a figure which shows the example of a result of the area | region division process based on a histogram update process.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram illustrating a hardware configuration example of an image area dividing device 101 according to an embodiment of the present invention.

  The image area dividing device 101 is realized on a computer system that is a portable information terminal such as a so-called smartphone.

  The image area dividing device 101 includes a CPU (Central Processing Unit) 102, a ROM (Read Only Memory) 103, and a RAM (Random Access Memory) 104. The image area dividing device 101 includes an external storage device 105 such as a solid storage device, a communication interface 106, an input device 107 such as a touch panel display device, and a display device 108. Further, the image area dividing device 101 includes a portable recording medium driving device 109 capable of setting a portable recording medium 110 such as a micro SD memory card or a USB (Universal Serial Bus) memory card. The imaging device 112 is a digital camera mechanism that can capture still images and video images, and includes a lens, an autofocus drive control device, an exposure control device, an imaging sensor, and the like. The above-described devices 102 to 109 and 112 are connected to each other by a bus 111.

The ROM 103 stores a program for controlling general operations of the entire smartphone, and a control program for image area division processing shown by flowcharts of FIGS. 3, 8, and 9 described later. The CPU 102 reads this control program from the ROM 103 and executes the RAM 104 as a work memory. As a result, an image region dividing function shown in the functional block of FIG. 2 to be described later is realized. As a result, for example, the user captures a flower or the like with the imaging device 112 and An image area dividing process for dividing a main object such as from other backgrounds is executed. The image data of the main object area such as a flower obtained in this way is used as an image search server computer connected to the Internet from the communication interface 106 via the Internet (not shown) in particular so that the user searches for, for example, the type of flower. Sent to. On this computer, a flower database is searched based on the sent flower image data of the main object area. As a result, the pictorial book information of the flower hit by the search is received by the communication interface 106 together with the image data of the flower via the Internet and displayed on the display device 108.
Note that the image region dividing device 101 according to the present embodiment may be realized not on the portable information terminal but on the server computer.

FIG. 2 is a functional block diagram showing a functional configuration of the image area dividing device 101 of FIG.
The image area dividing device 101 according to the present embodiment is realized as an apparatus that divides a main object such as a flower within an image range designated by a user and a background other than the main object in an image.

  The area dividing unit 201 updates the area label indicating the main object or background to be given to each pixel in the image range designated by the user, for example, a rectangular frame, and the pixel value of the area label and each pixel, for example, the color pixel value For example, a cost term that decreases as the value of the first histogram 205 for each color pixel value calculated from the image indicating the main object increases, and a second for each color pixel value calculated from the image indicating the background, for example. The main object and the background are divided into regions in the image by the energy function minimization process including the cost term that decreases as the value of the histogram 206 increases. The area dividing unit 201 executes energy function minimization processing by, for example, the Graph Cuts method. The initial value of the first histogram at this time is calculated as, for example, a histogram for each color pixel value of an image showing a plurality of learning main objects. The initial value of the second histogram is calculated as a histogram for each color pixel value of an image showing a plurality of backgrounds for learning, for example.

  The histogram calculation unit 202, for example, for each pixel value on the frame of the image range designated by the user as a rectangular frame, for each pixel value outside the boundary of the image range (for example, outside the rectangular frame), and within the frame of the image range (for example, within the rectangular frame) ) Are calculated as an on-frame histogram 207, an out-of-frame histogram 208, and an in-frame histogram 209, respectively.

  The specific pixel value extraction unit 203 extracts, for example, a color pixel value whose frequency value is larger than the first threshold value as the specific pixel value in the on-frame histogram 207.

  The histogram updating unit 204 determines the first pixel value in the specific pixel value when the power value at the specific pixel value is larger than the second threshold value on the out-of-frame histogram 208 and smaller than the third threshold value on the in-frame histogram 209. At least one of the two power values is updated so that the power value of the second histogram 206 increases relative to the power value of the first histogram 205. In this case, the histogram update unit 204 executes, for example, the following update process. The frequency value of the second histogram 206 at the specific pixel value is updated by a predetermined number that is a multiple larger than one. Alternatively, a predetermined number greater than 0 is added to the frequency value of the second histogram 206 at the specific pixel value to update. Furthermore, conversely, the frequency value of the first histogram 205 at the specific pixel value may be updated so as to be small.

  The inventor of the present patent application has a high frequency of occurrence of a specific pixel value, for example, a color pixel value that is frequently generated on a rectangular frame specified by the user, for example, in a region such as a flower image, even outside the rectangular frame. In addition, if the occurrence frequency is not high in the rectangular frame, the specific pixel value is highly likely to be a pixel value belonging to the background. Based on this knowledge, in the above case, the first histogram 205 indicating the likelihood of the main object and the second histogram 206 indicating the likelihood of the background used in the energy function for performing the area division in the area dividing means 201. In the specific pixel value, the histogram updating means 204 causes at least one of these two power values so that the power value of the second histogram 206 increases relative to the power value of the first histogram 205. Is updated. As a result, the main object region and the background region can be more clearly divided based on the specific pixel value.

  FIG. 3 is a flowchart showing the overall operation of the image area dividing process according to the present embodiment. The processing of this flowchart is realized as processing in which the CPU 102 in FIG. 1 executes the control program stored in the ROM 103 while using the RAM 104 as a work memory together with the processing in the flowchart showing the detailed processing in FIGS. The

  First, rectangular frame determination processing is executed (step S301 in FIG. 3). In this process, the user uses, for example, the input device 107 such as a touch panel for an approximate region where an object (for example, a flower) to be recognized exists in an image captured by the imaging device 112 of FIG. Specify a rectangular frame. For example, a sliding operation with a finger on the touch panel.

  Next, histogram update processing is executed (step S302 in FIG. 3). In this process, the functions of the histogram calculation means 202, specific pixel value extraction means 203, and histogram update means 204 shown in FIG. 2 are realized. Details of this processing will be described later with reference to FIGS.

  Subsequently, an area division process (graph cut process) for dividing the main object and the background into areas is executed for each pixel in the image range (step S303 in FIG. 3). This area dividing process realizes the function of the area dividing means 201 of FIG. Details of this processing will be described later with reference to FIGS.

Once the region division processing is completed, convergence determination is performed (step S304 in FIG. 3). This convergence determination is a determination result of YES when any of the following is satisfied.
・ The number of repetitions has exceeded a certain level ・ The difference between the area that was the previous foreground and the area that was the foreground this time is less than a certain level

If the determination in step S304 does not converge and the determination is NO, the cost function g _v (X _v ), which will be described later in the rectangular frame specified by the user, is In this way, the data is corrected and updated (step S305 in FIG. 3). A histogram of the area determined as the main object by the area dividing process in step S303 and a histogram θ (c, 0), which will be described later, prepared in advance are mixed (added), for example, at a fixed ratio for each color pixel value c. The As a result, a new histogram θ (c, 0) indicating the likelihood of a main object is generated, and a new cost function g _v (X _v ) is calculated based on the histogram θ (c, 0) (see equation 7 below). Similarly, the histogram of the area determined as the background by the area dividing process in step S303 and the histogram θ (c, 1), which will be described later, prepared in advance are mixed (added) at a certain ratio for each color pixel value c, for example. ) As a result, a histogram θ (c, 1) indicating a new background is generated, and a new cost function g _v (X _v ) is calculated based on the histogram θ (c, 1) (see Equation 8 below).

  When the determination in step S304 is converged and the determination is YES, the region division process shown in the flowchart of FIG. 3 is terminated, and the main object region currently obtained is output as the final result.

Hereinafter, the area dividing process in step S303 in FIG. 3 will be described.
Now
And an element X _v is a region label vector indicating an area label for pixels v (1 ≦ v ≦ V) in the image V. This area label vector is, for example, a binary vector having an element X _v = 0 if the pixel v is in the main object area and an element X _v = 1 if it is in the background area. That is,
It is.

The area dividing process executed in the present embodiment is a process for obtaining an area label vector X of Formula 1 that minimizes an energy function E (X) defined by the following expression in the image V.
As a result of executing the energy minimization process, the main object region is obtained as a set of pixels v having the region label value X _v = 0 on the region label vector X. In the example of this embodiment, it is a flower area within a rectangular frame. Note that a set of pixels v with the region label value X _v = 1 on the region label vector X is a background region (including outside the rectangular frame).

In order to minimize the energy of Equation 3, the following equation and a weighted directed graph (hereinafter, abbreviated as “graph”) shown in FIG. 4 are defined.
Here, V is a node and E is an edge. When this graph is applied to image area division, each pixel of the image corresponds to each node V. Also, as nodes other than pixels, the following formula and shown in FIG.
A special terminal called is added. Consider the source s in association with the main object area and the sink t in the background area. The edge E represents the relationship between the nodes V. An edge E representing the relationship with surrounding pixels is n-link, and an edge E representing the relationship between each pixel and the source s (corresponding to the main object region) or sink t (corresponding to the background region) is t-link. Call.

Now, each t-link connecting the source s and a node corresponding to each pixel is regarded as a relationship indicating how much each pixel seems to be a main object region. Then, the cost value indicating the likelihood of the main object area is associated with the first term of Equation 3 and
It is defined as Here, θ (c, 0) is function data indicating a histogram (number of appearances) for each color pixel value c calculated from the area of the main object of the image. For example, as shown in FIG. Has been obtained. It is assumed that the sum total of θ (c, 0) over all color pixel values c is normalized to be 1. I (v) is a color (RGB) pixel value at each pixel v of the input image. Actually, there may be a value obtained by converting a color (RGB) pixel value into a luminance value. However, for the sake of simplicity of explanation, unless otherwise specified, “color (RGB) pixel value” or “color pixel value” will be described below. ". In Equation 6, the cost value decreases as the value of θ (I (v), 0) increases. This is because, as the number of appearances of color pixel values in the main object area obtained in advance increases, the cost value obtained by Equation 6 becomes smaller, and the pixel v seems to be a pixel in the main object area. This means that the value of the energy function E (X) in Equation 3 is pushed down.

Next, each t-link connecting the sink t and the node corresponding to each pixel is regarded as a relationship indicating how much each pixel is a background region. Then, the cost value indicating the likelihood of the background region is associated with the first term of Equation 3 and
It is defined as Here, θ (c, 1) is function data indicating a histogram (appearance frequency) for each color pixel value c calculated from the background area of the image, and is obtained in advance as shown in FIG. 5B, for example. It has been. It is assumed that the sum total of θ (c, 1) over all color pixel values c is normalized to be 1. I (v) is a color (RGB) pixel value at each pixel v of the input image, as in Equation 6. In Equation 6, the cost value decreases as the value of θ (I (v), 1) increases. This means that the more frequently appearing color pixel values of the background area obtained in advance, the smaller the cost value obtained by Equation 7 is, and the pixel v seems to be a pixel in the background area. As a result, the value of the energy function E (X) in Formula 3 is pushed down.

Next, the cost value of n-link indicating the relationship between the node corresponding to each pixel and its surrounding pixels is associated with the second term of Equation 3;
It is defined as Here, dist (u, v) indicates the Euclidean distance between the pixel v and the surrounding pixel u, and κ is a predetermined coefficient. Further, I (u) and I (v) are color (RGB) pixel values in the pixels u and v of the input image. As described above, actually, a value obtained by converting a color (RGB) pixel value into a luminance value may be used. When the region label values X _u and X _{v of the} pixel v and the surrounding pixels u are selected to be the same (X _u = X _v ), the cost value of Formula 8 is set to 0, and the energy E ( It does not affect the calculation of X). On the other hand, the cost value of Formula 8 when the region label values X _u and X _{v of} the pixel v and its surrounding pixels u are selected to be different (X _u ≠ X _v ) is, for example, the characteristic shown in FIG. It has a function characteristic having That is, the region label values X _u and X _v of the pixel v and its surrounding pixels u are different, and the difference I (u) −I (v) between the color pixel values (luminance values) of the pixel v and its surrounding pixels u. ) Is small, the cost value obtained by Equation 8 is large. In this case, the value of the energy function E (X) in Equation 3 is pushed up. In other words, when the difference between the color pixel values (luminance values) between neighboring pixels is small, the area label values of those pixels are not selected to be different from each other. In other words, in this case, the region label values are controlled to be the same between neighboring pixels and the main object region or the background region is not changed as much as possible. On the other hand, the region label values X _u and X _v of the pixel v and the surrounding pixel u are different, and the difference I (u) −I (v) between the color pixel value (luminance value) of the pixel v and the surrounding pixel u ) Is large, the cost value obtained by Equation 8 is small. In this case, the result is that the value of the energy function E (X) in Equation 3 is pushed down. In other words, if there is a large difference in color pixel values between neighboring pixels, it means that the boundary between the main object region and the background region is likely, and the region label values in the direction where the pixel v and its surrounding pixels u are different. Be controlled.

  Using the above definition, a cost value (likeness of main object area) of t-link connecting the source s and each pixel v is calculated for each pixel v of the input image by Equation (6). In addition, the cost value (likeness of background area) of t-link connecting the sink t and each pixel v is calculated by Expression 7. Further, for each pixel v of the input image, eight n-link cost values (likeness of boundary) connecting the pixel v and its surroundings, for example, each of eight pixels in eight directions, are calculated by Equation (8).

Theoretically, for each combination of region labels 0 or 1 of the region label vector X in Equation 1, the above Equation 6, Equation 7, and Equation 8 according to each region label value. The energy function E (X) of Formula 3 is calculated while the calculation result of is selected. Then, by selecting the region label vector X that minimizes the value of the energy function E (X) among all the combinations, as a set of pixels v with the region label value X _v = 0 on the region label vector X The main object area can be obtained.

  However, since the number of combinations of 0 or 1 of all region label values of the region label vector X is actually the number of pixels multiplied by 2, calculation of the energy function E (X) minimization process in a realistic time is calculated. Can not do it.

Therefore, the Graph Cuts method makes it possible to calculate the energy function E (X) minimization process in a realistic time by executing the following algorithm.
FIG. 7 schematically shows the relationship between the graph having the t-link defined by Equation 6 and Equation 7 and the n-link defined by Equation 8, the region label vector X, and the graph cut. It is the figure shown in. In FIG. 7, the pixel v is shown one-dimensionally for easy understanding.

  In the calculation of the first term of the energy function E (X) of Equation 3, among the pixels in the main object region where the region label value in the region label vector X should be 0, among Equation 6 and Equation 7, The cost value of equation (6), which is a smaller value when it seems to be a pixel in the main object area, is smaller. Accordingly, in a certain pixel, the t-link on the source s side is selected and the t-link on the sink t side is cut (case 702 in FIG. 7), and E (X) in Equation 3 using Equation 6 If the calculation result becomes small when the first term is calculated, 0 is selected as the region label value of the pixel. Then, the graph cut state is adopted. If the calculation result does not become small, the graph cut state is not adopted and another link search and graph cut are attempted.

  Conversely, for the pixels in the background region where the region label value in the region label vector X should be 1, among the equations (6) and (7), the equation The cost value is smaller. Therefore, in a certain pixel, t-link on the sink t side is selected, and t-link on the source s side is cut (case 703 in FIG. 7), and E (X) in Equation 3 using Equation 7 When the first term is calculated, if the calculation result becomes small, 1 is selected as the region label value of the pixel. Then, the graph cut state is adopted. If the calculation result does not become small, the graph cut state is not adopted and another link search and graph cut are attempted.

  On the other hand, the main object region that should be continuous when the region label value in the region label vector X is 0 or 1 by the region division (graph cut) processing related to the calculation of the first term of the energy function E (X) of Formula 3 The cost value of equation (8) is 0 between the pixels inside or inside the background area. Therefore, the calculation result of Equation 8 does not affect the calculation of the cost value of the second term of the energy function E (X). Further, the n-link between the pixels is maintained without being cut so that Equation 8 outputs a cost value of 0.

  However, when the region label value changes between 0 and 1 between neighboring pixels by the region division (graph cut) processing relating to the calculation of the first term of the energy function E (X), If the difference between the color pixel values (luminance values) is small, the cost value of Equation 8 is large. As a result, the value of the energy function E (X) in Equation 3 is pushed up. Such a case corresponds to a case where the determination of the region label value by the value of the first term happens to be reversed in the same region. Therefore, in such a case, the value of the energy function E (X) becomes large, and as a result, such inversion of the region label value is not selected. In this case, the n-link between the pixels is maintained without being cut so that the calculation result of Expression 8 maintains the above result.

  On the other hand, when the region label value changes between 0 and 1 between neighboring pixels by the region division (graph cut) processing related to the calculation of the first term of the energy function E (X), If the difference in color pixel value (luminance value) between the pixels is large, the cost value of Equation 8 is small. As a result, the value of the energy function E (X) in Equation 3 is pushed down. Such a case means that those pixel portions are likely to be the boundary between the main object region and the background region. Therefore, in such a case, the region label value is made different between these pixels, and control is performed in the direction in which the boundary between the main object region and the background region is formed. Also, in this case, in order to stabilize the boundary formation state, n-link between these pixels is cut, and the cost value of the second term of Equation 3 is set to 0 (FIG. 7). 704 case).

  The determination control process described above is repeated starting from the node of the source s while sequentially tracing the node of each pixel, whereby a graph cut as shown by 701 in FIG. 7 is executed, and the energy function E (X) The minimization process is calculated in a realistic time. As a specific method of this processing, for example, the method described in Non-Patent Document 1 can be adopted.

  If t-link on the source s side remains for each pixel, 0 is given as the area label value of that pixel, that is, a label indicating the pixel of the main object area is given. On the contrary, if t-link on the sink t side remains, 1 is given as the area label value of the pixel, that is, a label indicating the pixel in the background area is given. Finally, the main object area is obtained as a set of pixels whose area label value is 0.

  FIG. 8 is a flowchart showing the region division processing in step S303 of FIG. 3 based on the above-described operation principle.

  First, color pixel values I (V) are read one by one from the image (step S801 in FIG. 8).

  Next, it is determined whether or not the pixel read in step S801 is a pixel within a rectangular frame designated by the user (step S802 in FIG. 8).

  If the determination in step S802 is YES, the cost value indicating the main object area-likeness, the cost value indicating the background-area-likeness, and the boundary likelihood are calculated based on the above-described Expression 6, Expression 7, and Expression 8. The cost values shown are respectively calculated (steps S803, S804, and S805 in FIG. 8). Note that the initial value of θ (c, 0) is calculated from the areas of a plurality of (approximately several hundred) main objects prepared for learning. Similarly, the initial value of θ (c, 1) is calculated from a plurality (several hundreds) of background regions prepared for learning.

On the other hand, if the determination in step S802 is NO, there is no main object area outside the rectangular frame, so that the cost value g _v ( X _v ) is a constant large value K as shown in the following equation.
Here, as shown in the following equation, K is set to a value larger than the sum of the smoothing terms of an arbitrary pixel (step S806 in FIG. 8).

Further, in order to make sure that the outside of the rectangular frame is determined as the background area, the cost value g _v (X _v ) indicating the likelihood of the background area is set to 0 as shown in the following equation (step S807 in FIG. 8). .

Furthermore, since all outside the rectangular frame is the background region, the value of h _uv (X _u , X _v ) is set to 0 (step S808 in FIG. 8).

  After the above processing, it is determined whether or not there remains a pixel to be processed in the image (step S809 in FIG. 8).

  If there is a pixel to be processed and the determination in step S809 is YES, the process returns to step S801 and the above process is repeated.

  When there is no pixel to be processed and the determination in step S809 is NO, the Graph Cuts algorithm is executed while calculating the energy function E (X) of Equation 3 using the cost values obtained for all the pixels in the image. Then, the main object and the background are divided into regions (step S810).

  FIG. 9 is a flowchart showing the histogram update process in step S302 of FIG.

  First, with respect to the rectangular frame specified by the user in the rectangular frame determination process in step S301 of FIG. Are calculated as an on-frame histogram 207, an out-of-frame histogram 208, and an in-frame histogram 209 (step S901 in FIG. 9). The frequency value of the on-frame histogram 207 is θon (c, 0) for each color pixel value c, the frequency value of the out-of-frame histogram 208 is θout (c, 0) for each color pixel value c, and the frequency value of the in-frame histogram 209 is colored. Assume that θin (c, 0) for each pixel value c.

  Next, the histogram comparison update process shown in the series of processes in steps S902 to S908 in FIG. 9 is executed. Here, in step S902 in FIG. 9, the color pixel value c is set to number 1 corresponding to the first pixel value, and then the number indicating the color pixel value c is incremented by +1 in step S908 in FIG. Then, until it is determined in step S907 in FIG. 9 that the number corresponding to the color pixel value c has reached the maximum number N, the processing in steps S903 to S906 in FIG. 9 is executed for each color pixel value c. .

First, the specific pixel value _cm is acquired from the color pixel value c. Specifically, the frequency value θon (c, 0) of the on-frame histogram 207 created in step S901 corresponding to the color pixel value c is equal to or larger than the first threshold θon _m (θon (c, 0) ≧ It is determined whether or not θon _m ) (step S903). That is, it is determined whether the frequency of occurrence of the color pixel value c on the rectangular frame is high. As a result, if this determination is YES, it is determined that the color pixel value c is the specific pixel value _cm . FIG. 10 is an explanatory diagram of specific pixel values. For example, in θon (c, 0) (1 ≦ c ≦ N) indicated by 1001 in FIG. 10A, the color pixel value in the range indicated by 1002 in which the value is equal to or larger than θon _m is the specific pixel value _cm. It becomes. As shown in FIG. 10B, a cumulative histogram represented by Ron (c, 0) (1 ≦ c ≦ N) may be used instead of the normal histogram. In this case, the frequency is Instead of the predetermined threshold value θon _m for the predetermined frequency, the predetermined threshold value Ron _m for the cumulative frequency is used for the determination.

If the color pixel value c is determined to be the specific pixel value _cm and the determination in step S903 is YES, the frequency value θout (c, 0) of the out-of-frame histogram 207 created in step S901 corresponding to the color pixel value c is the first value. It is determined whether or not (θout (c, 0) ≧ θout _m ) is equal to or greater than the threshold value 2 of θout _m . That is, it is determined whether the occurrence frequency of the specific pixel value _cm is high even outside the rectangular frame (step S904 in FIG. 9).

If the occurrence frequency of the specific pixel value _cm is high even outside the rectangular frame and the determination in step S904 is YES, then the frequency value θin (c, 0) of the in-frame histogram 209 created in step S901 corresponding to the color pixel value c. Is less than the third threshold value θin _m (θin (c, 0) <θin _m ). That is, it is determined whether the occurrence frequency of the specific pixel value _cm is low within the rectangular frame (step S905 in FIG. 9).

If the occurrence frequency of the specific pixel value _cm is low in the rectangular frame and the determination in step S905 is YES, the specific pixel value _cm is a color specific to the background and is not a main color constituting the main object. I can judge.

In this case, as shown in the following Equation 12, the histogram frequency value θ (c, 1) indicating the background likelihood used in step S303 in FIG. 3 at the specific pixel value _cm (= FIG. 2). The second histogram 206) is updated by α (α> 1) times.

Alternatively, as shown in the following formula 13, the constant β is added to the histogram frequency value θ (c, 1) indicating the background likeness and updated at the specific pixel value _cm .

As described above, in certain pixel value c _m, the frequency value at the histogram indicating the background likeness used in step S303 in FIG. 3 θ (c, 1) ( = second histogram 206 of FIG. 2), the main Θ (c, 1) is updated so as to increase relative to the frequency value θ (c, 0) (= first histogram 205 in FIG. 2) of the histogram indicating the object likeness. Here, the update process may be performed so that the histogram frequency value θ (c, 0) indicating the likelihood of a main object becomes smaller at the specific pixel value _cm (step S906 in FIG. 9).

When the following determination is made for the current color pixel value c, the histogram update process is not executed, and the process proceeds to step S907. This is a case where the color pixel value c is not determined to be the specific pixel value _cm and the determination in step S903 is NO. Alternatively, the color pixel value c is determined to be the specific pixel value _cm and the determination in step S903 is YES, but the occurrence frequency of the specific pixel value _cm is not high outside the rectangular frame and the determination in step S904 is NO. It is. Alternatively, the occurrence frequency of the specific pixel value _cm is high outside the rectangular frame, and the determination in step S904 is YES. However, the occurrence frequency of the specific pixel value _cm is high even in the rectangular frame, and the determination in step S905 is NO. It is.

  The series of histogram update processes in steps S903 to S906 in FIG. 9 described above are repeatedly executed for each color pixel value c number (1 ≦ c ≦ N).

In the present embodiment, the specific pixel value _cm , which is a color pixel value having a high occurrence frequency on the rectangular frame, has a high occurrence frequency outside the rectangular frame, and if the occurrence frequency is not high in the rectangular frame, FIG. In the region dividing process executed in step S303, the histogram frequency value θ (c, 1) indicating the background likelihood with respect to the specific pixel value _cm is updated so as to be relatively increased. As a result, the probability that the specific pixel value _cm is classified into the background region is increased. As a result, in the area dividing process in the area dividing unit 201, the ratio of erroneous recognition between the background area and the main object area is reduced, and the accuracy of area division can be improved.

  FIG. 11 is a diagram illustrating an example of a result of the area division process (graph cut process) based on the above-described histogram update process.

As an input image, as shown in FIG. 11A, for example, when a single flower 1101 occupying a large amount of yellow is arranged in a background 1102 occupying a large amount of green, the main image indicating the portion of the flower 1101 The result of dividing the object area and the background area indicating the background 1102 is, for example, as shown in FIG. In FIG. 11B, the white area is determined as the main object area, and the black area is determined as the background area. In this case, in the present embodiment, as shown in FIG. 11C, the green pixel value occupies most on the frame 1103 of the rectangular frame designated by the user. Accordingly, in the on-frame histogram 207, the numerical value corresponding to the green pixel value becomes high, the determination in step S903 in FIG. 9 becomes YES, and the green pixel value is extracted as the specific pixel value _cm . Next, the green pixel value occupies a large amount even outside the rectangular frame 1103. Accordingly, in the out-of-frame histogram 208, the numerical value corresponding to the green pixel value increases, and the determination in step S904 in FIG. 9 becomes YES. Further, the green pixel value does not occupy much inside the rectangular frame 1103. Accordingly, in the in-frame histogram 209, the power value corresponding to the green pixel value is lowered, and the determination in step S905 of FIG. 9 is YES. As a result, for the green pixel value, in step S906 in FIG. 9, the histogram frequency value θ (c, 1) indicating the background likelihood is compared with the histogram frequency value θ (c, 0) indicating the main object likelihood. Updated so as to increase relatively. As a result, the region division is executed so that the main object region corresponds well to the portion of the flower 1101 in which the yellow color occupies most.

On the other hand, as shown in FIG. 11D, for example, when there are many similar pink flowers in the background 1105 as compared with the pink flower portion 1104 as shown in FIG. The result of dividing the main object area shown and the background area showing the background 1105 is, for example, as shown in FIG. In this case, in the present embodiment, as shown in FIG. 11 (f), a large number of pink pixel values occupy the upper frame 1106 of the rectangular frame designated by the user. Accordingly, in the on-frame histogram 207, the numerical value corresponding to the pink pixel value becomes high, the determination in step S903 in FIG. 9 becomes YES, and the pink pixel value is extracted as the specific pixel value _cm . Next, pink pixel values occupy a large amount outside the rectangular frame 1103. Accordingly, in the out-of-frame histogram 208, the numerical value corresponding to the pink pixel value increases, and the determination in step S904 in FIG. 9 becomes YES. However, the pink pixel values occupy most of the inside of the rectangular frame 1103. Accordingly, in the in-frame histogram 209, the power value corresponding to the pink pixel value is high, and the determination in step S905 of FIG. 9 is NO. As a result, with respect to the pink pixel value, the histogram frequency value θ (c, 1) indicating the likelihood of background increases relative to the histogram frequency value θ (c, 0) indicating the likelihood of the main object. No updates will be made. As a result, a situation in which region division is performed based on pink is avoided.

  As described above, in the present embodiment, it is possible to improve the accuracy of clipping in area division by appropriately setting the cost inside and outside the rectangular frame.

  In the above embodiment, the case where the main object is a flower has been described as an example. However, the main object is not limited to a flower, and various objects can be adopted.

Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
A device that divides a main object in an image range designated in an image and a background other than the main object into regions,
The pixel value calculated from the image representing the main object based on the region label and the pixel value of each pixel while updating the region label indicating the main object or the background to be given to each pixel in the image range Energy including a cost term that decreases as the value of the first histogram for each pixel decreases, and a cost term that decreases as the value of the second histogram for each pixel value calculated from the image indicating the background increases. Area dividing means for dividing the main object and the background into areas in the image by a function minimization process;
For each pixel value on the frame of the image range, for each pixel value outside the boundary of the image range, and for each pixel value within the frame of the image range, a histogram on the frame, an out-of-frame histogram, and an in-frame histogram, respectively. Histogram calculating means for calculating as
Specific pixel value extracting means for extracting, as the specific pixel value, a pixel value having a power value larger than the first threshold in the histogram on the frame;
When the frequency value at the specific pixel value is larger than the second threshold value on the out-of-frame histogram and smaller than the third threshold value on the intra-frame histogram, the second histogram at the specific pixel value Histogram updating means for updating at least one of the two frequency values so that the frequency value increases relative to the frequency value of the first histogram;
An image region dividing apparatus comprising:
(Appendix 2)
The histogram updating unit is configured to output the specific pixel value when a power value at the specific pixel value is larger than a second threshold value on the out-of-frame histogram and smaller than a third threshold value on the in-frame histogram. Updating the frequency value of the second histogram at a predetermined multiple that is a multiple greater than 1.
The image area dividing device according to Supplementary Note 1, wherein
(Appendix 3)
The histogram updating unit is configured to output the specific pixel value when a power value at the specific pixel value is larger than a second threshold value on the out-of-frame histogram and smaller than a third threshold value on the in-frame histogram. Updating by adding a predetermined number greater than 0 to the frequency value of the second histogram at
The image area dividing device according to Supplementary Note 1, wherein
(Appendix 4)
The initial value of the first histogram is calculated as a histogram for each pixel value of an image showing a plurality of learning main objects,
The initial value of the second histogram is calculated as a histogram for each pixel value of a plurality of images indicating the background for learning.
The image area dividing device according to any one of appendices 1 to 3, wherein
(Appendix 5)
The region dividing unit performs the energy function minimization process by a Graph Cuts method.
The image area dividing device according to any one of appendices 1 to 4, characterized in that:
(Appendix 6)
A method of dividing a main object within an image range designated in an image and a background other than the main object into regions,
The pixel value calculated from the image representing the main object based on the region label and the pixel value of each pixel while updating the region label indicating the main object or the background to be given to each pixel in the image range Energy including a cost term that decreases as the value of the first histogram for each pixel decreases, and a cost term that decreases as the value of the second histogram for each pixel value calculated from the image indicating the background increases. A region dividing step of dividing the main object and the background into regions in the image by function minimization processing;
For each pixel value on the frame of the image range, for each pixel value outside the boundary of the image range, and for each pixel value within the frame of the image range, a histogram on the frame, an out-of-frame histogram, and an in-frame histogram, respectively. A histogram calculation step to calculate as
A specific pixel value extracting step of extracting a pixel value having a power value larger than the first threshold value as the specific pixel value in the on-frame histogram;
When the frequency value at the specific pixel value is larger than the second threshold value on the out-of-frame histogram and smaller than the third threshold value on the intra-frame histogram, the second histogram at the specific pixel value A histogram update step of updating at least one of the two frequency values so that the frequency value of the first histogram increases relatively with respect to the frequency value of the first histogram;
An image region dividing method comprising:
(Appendix 7)
A computer that executes processing for dividing a main object in an image range designated in an image and a background other than the main object into a region;
The pixel value calculated from the image representing the main object based on the region label and the pixel value of each pixel while updating the region label indicating the main object or the background to be given to each pixel in the image range Energy including a cost term that decreases as the value of the first histogram for each pixel decreases, and a cost term that decreases as the value of the second histogram for each pixel value calculated from the image indicating the background increases. A region dividing step of dividing the main object and the background into regions in the image by function minimization processing;
For each pixel value on the frame of the image range, for each pixel value outside the boundary of the image range, and for each pixel value within the frame of the image range, a histogram on the frame, an out-of-frame histogram, and an in-frame histogram, respectively. A histogram calculation step to calculate as
A specific pixel value extracting step of extracting a pixel value having a power value larger than the first threshold value as the specific pixel value in the on-frame histogram;
When the frequency value at the specific pixel value is larger than the second threshold value on the out-of-frame histogram and smaller than the third threshold value on the intra-frame histogram, the second histogram at the specific pixel value A histogram update step of updating at least one of the two frequency values so that the frequency value of the first histogram increases relatively with respect to the frequency value of the first histogram;
A program for running

101 Image area dividing device 102 CPU
103 ROM
104 RAM
105 External Storage Device 106 Communication Interface 107 Input Device 108 Display Device 109 Portable Recording Medium Drive Device 110 Portable Recording Medium 111 Bus 112 Imaging Device 201 Region Dividing Unit 202 Histogram Calculation Unit 203 Specific Pixel Value Extracting Unit 204 Histogram Updating Unit 205 Histogram of 1 206 Second histogram 207 Histogram above frame 208 Histogram outside frame 209 Histogram inside frame

Claims (7)

A device that divides a main object in an image range designated in an image and a background other than the main object into regions,
The pixel value calculated from the image representing the main object based on the region label and the pixel value of each pixel while updating the region label indicating the main object or the background to be given to each pixel in the image range Energy including a cost term that decreases as the value of the first histogram for each pixel decreases, and a cost term that decreases as the value of the second histogram for each pixel value calculated from the image indicating the background increases. Area dividing means for dividing the main object and the background into areas in the image by a function minimization process;
For each pixel value on the frame of the image range, for each pixel value outside the boundary of the image range, and for each pixel value within the frame of the image range, a histogram on the frame, an out-of-frame histogram, and an in-frame histogram, respectively. Histogram calculating means for calculating as
Specific pixel value extracting means for extracting, as the specific pixel value, a pixel value having a power value larger than the first threshold in the histogram on the frame;
When the frequency value at the specific pixel value is larger than the second threshold value on the out-of-frame histogram and smaller than the third threshold value on the intra-frame histogram, the second histogram at the specific pixel value Histogram updating means for updating at least one of the two frequency values so that the frequency value increases relative to the frequency value of the first histogram;
An image region dividing apparatus comprising: The histogram updating unit is configured to output the specific pixel value when a power value at the specific pixel value is larger than a second threshold value on the out-of-frame histogram and smaller than a third threshold value on the in-frame histogram. Updating the frequency value of the second histogram at a predetermined multiple that is a multiple greater than 1.
The image area dividing device according to claim 1, wherein The histogram updating unit is configured to output the specific pixel value when a power value at the specific pixel value is larger than a second threshold value on the out-of-frame histogram and smaller than a third threshold value on the in-frame histogram. Updating by adding a predetermined number greater than 0 to the frequency value of the second histogram at
The image area dividing device according to claim 1, wherein The initial value of the first histogram is calculated as a histogram for each pixel value of an image showing a plurality of learning main objects,
The initial value of the second histogram is calculated as a histogram for each pixel value of a plurality of images indicating the background for learning.
The image area dividing device according to claim 1, wherein the image area dividing device is an image area dividing device. The region dividing unit performs the energy function minimization process by a Graph Cuts method.
The image area dividing device according to claim 1, wherein the image area dividing device is an image area dividing device according to claim 1. A method of dividing a main object within an image range designated in an image and a background other than the main object into regions,
The pixel value calculated from the image representing the main object based on the region label and the pixel value of each pixel while updating the region label indicating the main object or the background to be given to each pixel in the image range Energy including a cost term that decreases as the value of the first histogram for each pixel decreases, and a cost term that decreases as the value of the second histogram for each pixel value calculated from the image indicating the background increases. A region dividing step of dividing the main object and the background into regions in the image by function minimization processing;
For each pixel value on the frame of the image range, for each pixel value outside the boundary of the image range, and for each pixel value within the frame of the image range, a histogram on the frame, an out-of-frame histogram, and an in-frame histogram, respectively. A histogram calculation step to calculate as
A specific pixel value extracting step of extracting a pixel value having a power value larger than the first threshold value as the specific pixel value in the on-frame histogram;
When the frequency value at the specific pixel value is larger than the second threshold value on the out-of-frame histogram and smaller than the third threshold value on the intra-frame histogram, the second histogram at the specific pixel value A histogram update step of updating at least one of the two frequency values so that the frequency value of the first histogram increases relatively with respect to the frequency value of the first histogram;
An image region dividing method comprising: A computer that executes processing for dividing a main object in an image range designated in an image and a background other than the main object into a region;
The pixel value calculated from the image representing the main object based on the region label and the pixel value of each pixel while updating the region label indicating the main object or the background to be given to each pixel in the image range Energy including a cost term that decreases as the value of the first histogram for each pixel decreases, and a cost term that decreases as the value of the second histogram for each pixel value calculated from the image indicating the background increases. A region dividing step of dividing the main object and the background into regions in the image by function minimization processing;
For each pixel value on the frame of the image range, for each pixel value outside the boundary of the image range, and for each pixel value within the frame of the image range, a histogram on the frame, an out-of-frame histogram, and an in-frame histogram, respectively. A histogram calculation step to calculate as
A specific pixel value extracting step of extracting a pixel value having a power value larger than the first threshold value as the specific pixel value in the on-frame histogram;
When the frequency value at the specific pixel value is larger than the second threshold value on the out-of-frame histogram and smaller than the third threshold value on the intra-frame histogram, the second histogram at the specific pixel value A histogram update step of updating at least one of the two frequency values so that the frequency value of the first histogram increases relatively with respect to the frequency value of the first histogram;
A program for running