JP2005293367A - Image processing method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method for extracting a desired area by an easy operation. <P>SOLUTION: This image processing method for extracting an area desired by a user from an image includes a step S11 for dividing the image into several areas, a step S12 for designating one point in the image according to an instruction by the user, a correlation degree calculation step S13 for analyzing a correlation degree between an area including the designated point and another area, a step S14 for extracting a plurality of candidate areas by combining the area including the designated point and a plurality of areas having high correlation, a step S15 for presenting a plurality of candidate areas to the user, and a step S16 for making a selection in the displayed candidate areas according to the instruction by the user. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

The present invention relates to an image processing method and apparatus suitable for use in extracting a desired region including a point after a user designates an arbitrary point in an image.

With the widespread use of digital cameras and PCs (personal computers), image data taken with digital cameras can be easily edited on a PC. Furthermore, there is a demand for editing image data immediately after shooting. For example, there is a request to perform (simple) editing within a device such as a digital camera or a mobile phone, or a request to easily check a captured image.

However, since portable devices such as digital cameras and mobile phones cannot use an interface such as a mouse, for example, when you want to enlarge a certain part of an image, specify the upper left and lower right of the desired area by pressing the cross key repeatedly. For example, it was necessary to perform complicated operations.

As another technical background, in the image processing technique, several methods for dividing an image into several regions or extracting a certain region from the image have been proposed. However, for example, although there is a method for estimating an area attracted by a person as the degree of attraction, there are few methods for calculating the relationship and importance of each extracted region. Therefore, when it is desired to specify a plurality of areas from an image divided into several areas, it is necessary to perform a plurality of operations for specifying the areas corresponding to the number of divisions. Specifically, in the case of a plurality of person photographs (group photographs) or the like, an area can be extracted for each person, but enlarging to include one person on both sides requires a plurality of designation operations.

In the process of enlarging by designating a certain point in the image, the simplest method is to determine the area around the image center / predetermined point at a predetermined magnification as described in Patent Document 1. Conceivable. However, in this case, since there is no optimal magnification, for example, 1.1
The user is forced to repeatedly perform operations such as magnifying each time.

In addition, as described in Patent Document 2, there is also proposed a method of detecting an effective image area from a copy without specifying a magnification and enlarging it according to a sheet to be output. However, the effective image area is a non-blank area having no image or text, and is different from the technique for detecting the effective area in the image as the subject of the present invention.

Next, there is a method of extracting a region by calculating a certain feature from one point designated by the user.
For example, in Patent Document 3, a contour is extracted and a region is extracted from an image. However, in this method, only one area can be specified, and it is not possible to perform extraction over a plurality of areas, for example, when it is desired to select a background or the like at the same time.
JP 2000-75416 A JP 7-327129 A JP 2000-125119 A

In the above method, even when a valid result is obtained by a predetermined method,
There may be cases where the user does not have the desired area. For example, even if it is the same image, the region desired by the user may be a region including an adjacent person or a region up to the upper body. In such a case, the above-described method has a problem that it is not possible to extract a region that accurately reflects the user's wishes with a simple operation.

For example, in the method of extracting the approximate color area, there is a problem that, for example, other than the face area cannot be extracted from one point of the face area. In addition, in the case of using shapes, it is necessary to store a plurality of shapes in order to be able to search various shapes, but it can be said that it is actually difficult because of the number of types.

The present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing method and apparatus capable of extracting a desired region with a simple operation.

In order to solve the above problems, the present invention is an image processing method for extracting a region desired by a user from an image, the step of dividing the image into regions, and 1 in the image according to a user instruction.
Multiple candidate regions are extracted by combining a step that specifies points, a relevance analysis step that analyzes the degree of association between an area that includes the specified point and other areas, and an area that includes the specified point and multiple areas that have a high degree of association And a step of presenting a plurality of candidate areas to the user, and a step of selecting the candidate areas displayed according to the user's instruction.

In the present invention, when an image is divided into regions in advance and the user designates an arbitrary point in the image, the region including the point and the region related to the region are analyzed, and a plurality of points are determined depending on the degree of association. Present the area to the user. Thus, a plurality of areas are automatically presented only by the user specifying an arbitrary point in the image. If a desired area is selected from these, it is possible to easily extract the desired area.

Another aspect of the invention is characterized in that, in the relevance analysis step, a distance from a designated point is used as an index, and a region having a close distance is defined as a region having a high relevance. According to this, since the degree of association is obtained using the distance from the designated point as an index, it is possible to cope with a case where there is a desire to select and extract an area close to the area designated by the user.

Another invention is characterized in that, in the relevance analysis step, a region adjacent to a region including a designated point is set as a region having a high relevance. According to this, it is possible to cope with a case where there is a desire to select and extract an area adjacent to the area designated by the user.

Another invention is characterized in that, in the relevance analysis step, a region that includes a designated point is combined as a region having a high relevance. According to this, if there is a conjugation relationship (relationship that does not change the state of each region), it is presented as a candidate region, so select the region that is conjugated with the region specified by the user This can be done when there is a desire to extract.

Another invention is characterized in that, in the relevance analysis step, a region having a high degree of similarity is defined as a region having a high degree of relevance, using as an index the similarity of at least one of the shape and color of the region including the designated point. . According to this, it is possible to cope with a case where there is a desire to select and extract a region having a high degree of similarity in shape and color with the region designated by the user.

Another invention is characterized in that, in the relevance analysis step, the relevance between specific colors and shapes is stored in advance, and the relevance is used as an index. According to this,
The degree of relevance can be obtained by simple processing.

Another invention is characterized in that, in the relevance analysis step, a region having a relatively large area is defined as a region having a low relevance, using the area as an index. According to this, it is possible to cope with a case where there is a desire to select and extract a region designated by the user and a relatively small region.

In another aspect of the present invention, in the step of extracting the plurality of region candidates, a region having a relevance level equal to or higher than a predetermined value is set as a highly related region, and a combined region of a highly related region and a region including a designated point is selected. It is characterized by extracting as a region. According to this, the user can easily select and extract the designated area and the area having a higher degree of association.

Another invention is characterized in that, in the step of extracting the plurality of region candidates, a combined region of a plurality of related regions and a region including a designated point is extracted as a candidate region. According to this, the user can easily select and extract the designated region and a plurality of regions having a higher degree of association.

In another aspect, in the step of extracting the plurality of region candidates, the candidate region A is included in another candidate region B at a predetermined ratio, and the region C obtained by combining the candidate region A and the candidate region B is: The region C is selected as a candidate region when it falls within a predetermined rate of increase with respect to the candidate region B. According to this, since the number of candidate areas can be reduced appropriately, the user can easily select and extract a desired area.

According to another aspect of the present invention, there is provided an image processing apparatus for extracting an area desired by a user from an image, and means for dividing an image into areas, and means for designating one point in the image in accordance with a user instruction Relevance analysis means for analyzing the degree of association between the area including the designated point and another area, means for extracting a plurality of candidate areas by combining the area containing the designated point and a plurality of areas with high relevance, and a plurality of areas Means for presenting the candidate area to the user, and means for selecting the candidate area displayed according to the user's instruction.

According to still another aspect of the present invention, there is provided an image processing program for extracting a region desired by a user from an image, the step of dividing the image into regions, and designating one point in the image according to a user instruction. A step of analyzing the relevance between the region including the designated point and the other region, a step of extracting a plurality of candidate regions by combining the region including the designated point and a plurality of regions having a high degree of relevance; Presenting multiple candidate areas to the user;
And a step of selecting a candidate area displayed in accordance with a user instruction by a computer.

Embodiments of an image processing apparatus according to the present invention will be described below with reference to the drawings. The present embodiment is realized by an information device such as a mobile phone terminal, a digital camera, a personal information terminal, a personal computer, and the like, a computer constituting the computer and its peripheral devices, and a program executed by the computer. In the present embodiment, it is assumed that the information device includes at least an image display device such as a liquid crystal display device and an operator that can specify an arbitrary point in a display image such as a cross cursor. In this embodiment, when a user designates an arbitrary point in an image using such a configuration, an area related to the area including the point is analyzed,
A plurality of areas having relevance to the area including the point are presented to the user according to the degree of relevance. The user can extract a desired region in the image by selecting one of the plurality of regions. The extracted image can be stored in a predetermined storage medium, transmitted by being attached to an e-mail, or stored in a predetermined storage device and synthesized with another image.

The flow of image processing according to this embodiment will be described with reference to FIG. In this embodiment, when the user selects an image to be processed, an area division process is performed in advance (step S11).

In a simple example, the area division processing is performed when the color information of a given image is RGB (Red) Green
(Green) Blue (blue)). In contrast, first, the RGB color information is converted into HSV (Hue (Hue) Saturation (saturation) Value (lightness)) information. H
(Hue) is expressed in a ring shape, and the hue can be classified according to the angle. For example, when the color is classified into 24 colors, the color is classified by 15 degrees. Thereby, a given image can be classified into 24 approximate color regions. A divided region is formed for each successive approximate region classified into the same color.

Here, the HSV method is used, but a method having hue information such as Lab (love color; a color model for digitizing color information) may be used. Further, a classification method such as a k-means method may be used instead of a classification method in which hues such as 24 colors are fixed.

Next, the divided area is displayed on the display device of the information device constituting this embodiment, and one point in the image is designated in accordance with the user's instruction using the operation element (step S12). For example, FIG.
As shown in FIG. 5, the image 1 is divided into a plurality of areas 10 to 18 and displayed, and a designated point 20 is designated according to a user instruction.

In the following, each of the regions 11 to 18 shown in FIG. 2 is changed from region A to region H as shown in FIG.
Will be described.

Next, an area 10 including the designated point 20 (hereinafter, designated area 10) and other areas A to H (area 1).
1 to 18) are analyzed (step S13). In the relevance analysis in step S13, first, information about each region such as the color and shape of the regions A to H is calculated. For example,
As shown in FIG. 4, the distance between the designated area 10 and each of the areas A to H and the area of each of the areas A to H are obtained. In this example, it is assumed that the distance and area of each region are obtained as shown in FIG. However,
In this case, as the area of each region, the circumscribed rectangular area of the region shown by the broken line shown in FIG. 4 is used. The distance between the regions is the distance between the centroids of the regions, and the centroid is the center of the circumscribed rectangle.

Next, it is determined whether or not the degree of relevance is to be obtained. That is, it is determined whether or not it is appropriate as a target for determining the degree of association. For example, in this determination, as shown in FIG. 4, the region H is the entire image 1 (that is, the outer shape when the region is extracted matches the image 1),
Removed from the subject. In addition, the areas F and G are excluded from the target because they fall within the circumscribed rectangle 10 a of the designated area 10.

Next, in each of the areas A to E excluding the areas F to H, as shown in FIG. For example, a value obtained by normalizing the distances between the areas A to E and the designated area 10 with the maximum distance among the distances is used as an index representing the relationship related to the distance.

The index representing the relationship related to the area is represented by a numerical value obtained by normalizing the areas of the regions A to E by the maximum area among them.

Similarity (an index related to similarity) is, for example, color approximation and circumscribed rectangle shape (aspect ratio)
Can be used. In this case, for example, (index for similarity) = (dissimilarity) = 0
. A similarity index can be obtained as 5 * (color index) + 0.5 * (shape index). The index related to the similarity indicates that the larger the value is, the lower the similarity is (that is, the degree of dissimilarity).

Here, as the color index, a color distance between the average color of the designated area 10 and the average color of each area is calculated, and a value normalized by the maximum value is used. The shape index is calculated using the aspect ratio. That is, the aspect ratio of the designated area 10 is (x0: y0), and the aspect ratio of a certain area is (x1:
When y1), the aspect ratio difference P

  P = | (x0 / y0) − (x1 / y1) |

  A value obtained in each region and normalized by the maximum value of P is used as a shape index.

For example, in FIG. 3, if the color of the region D is the same color as the designated region 10 and the shape is the same, the color index is 0 and the shape index is 0. The calculation result (dissimilarity) is 0. Or, for example, the color of the area A is different from the designated area 10,
If the shape (aspect ratio) is equal, the similarity index (dissimilarity) is, for example,
The value is greater than 0 and less than or equal to 0.5.

The index related to the adjacency is obtained with 0 as the area in contact with the boundary of the designated area 10 and 1 as the other area. That is, the smaller the index related to the adjacency, the higher the relevance.

By adding all the indices shown in FIG. 6 for each area, the degree of unrelatedness of each area can be obtained. In the example shown in FIG. 6, the relevance is lower as each index is larger, and the relevance is higher as the index is smaller. That is, the sum of each index has a low relevance when the value is large, and a high relevance when the value is small. That is, the added value of each index is a numerical value that represents the degree of disassociation.

In the example shown in FIG. 6, the value of the degree of irrelevance in the area A is the smallest, and the value of the degree of irrelevance in the area C is the largest. That is, the ranking of the degree of association is from A to B in the areas A, B, D, E, and C.

Next, a candidate area to be presented to the user is selected based on the degree of association (or degree of unrelevance) obtained in step S13 of FIG. 1 (step S14). Basically, a region obtained by synthesizing the designated region 10 with a region having a low degree of unrelevance is selected as a candidate region. However, it is possible to prevent a region having an unrelated degree equal to or higher than a predetermined threshold from being a candidate. And
The selected candidate area is presented to the user (step S15), and is selected according to the user's instruction (step S16). For example, different candidate areas are sequentially selected by moving the cursor key by the user, and the user can select an area by pressing the enter key in a state where a desired area is selected.
As another method, the area candidate number may be directly selected by a numeric key or the like.

In step S15, for example, when selecting a candidate area, if four indices having values of 0 to 1 are used as indices as shown in FIG. 2 is selected as a threshold value. When the threshold value is 2, regions A and B are selected. In the example of FIG. 7, up to the candidate area 3 is the target to be presented to the user.

In FIG. 7, the number indicated by a rectangle at the upper right or upper left corner of the rectangle indicated by the broken line is the rank number of the candidate area (the smaller the number, the higher the relevance). Candidate area 1 is designated area 10 itself. The candidate area 2 is an area determined by a circumscribed rectangle of the combination area of the designated area 10 and the area A. Candidate area 3 is an area determined by a circumscribed rectangle of the combined area of designated area 10 and area B. The candidate area 4 is an area formed from the designated area 10 and the area D, and the candidate area 5 is an area formed from the designated area 10 and the area E, respectively.

It is also effective to determine an upper limit for the number of candidate areas to be presented. The maximum number of candidates displayed by the user is set in advance. For example, if the user sets five,
In order to make the designated area 10 the first candidate area, the remaining four areas A, B, D, and E are selected. The candidate area in this case is as shown in FIG. Instead of the example of determining by the threshold value, the maximum number of candidates can be determined and presented to the user in this way.

Furthermore, the number of candidate areas can be reduced or a new candidate area can be added by detecting an area having a large overlapping area with respect to the extracted candidate areas and combining them as candidate areas. Good.

In the case of FIG. 7, when the area of the candidate area 2 is S2 and the area of the candidate area 3 is S3, the overlapping area is equal to or greater than a predetermined ratio (eg, 0.8) of the area S2. When the region obtained by combining candidate region 2 and candidate region 3 falls within a predetermined ratio (eg, 1.2) of area S3, the candidate region obtained by combining candidate region 2 and candidate region 3 (candidate region 2 in FIG. 8) Is generated. Candidate area 1 is an area of designated point 20. This designated area 10 overlaps with the candidate area 2, but the designated point area 10 is not synthesized. The final candidate area in this case is as shown in FIG. In FIG. 8, candidate area 2 and candidate area 3 of FIG. 7 are combined, and the candidate numbers of candidate area 4 and candidate area 5 of FIG.

Furthermore, even in a combination of three or more regions, candidate regions can be integrated or abandoned by area ratio (see FIG. 9). In FIG. 9, the designated point area 10 and the candidate areas 1 and 2 in FIG.
The candidate 5 including 4 and 5 is added, and the candidate area 3 is changed to a candidate obtained by merging the area 1 and the area 2.

In the above configuration, the degree of association is obtained from the distance, area, dissimilarity, and adjacent indices shown in FIG. 6, but only the color and shape can be used as indices.

Further, a correlation function may be used as an index representing similarity. The reference point is the lower left, etc.
The calculation is performed with 0 having the highest degree of correlation and 1 having the lowest degree of correlation. For example, the similarity calculation method using the correlation function is performed as follows.

(1) The distance between the center of gravity (center of the circumscribed rectangle) of the designated area and each point on the area boundary line is sampled. The sampling start point is, for example, the uppermost boundary pixel point.

(2) For each divided area, the distance between the center of gravity and the point on the boundary line is similarly sampled.

(3) The number of samples in each area is matched with the number of samples in the specified area. The number of samples in the designated area is Mx, the number of samples in the comparison area is My, and among the sample values of sample number N, the sample value before change is Y1 (N), the sample value after change is Y2 (N), and the magnification s = Mx / My, d = N
Assuming that / s, n = integer part (d), and k = decimal part (d), Y2 (N) is the sample value after the change, obtained by the following equation.

For example, a sample value calculation example is as follows. Here, M1 = 200, M2 = 150
As a result, an Nth (= 10) th value of the comparison area is obtained.

(4) Find correlation coefficient with correlation function Average and variance:

Correlation coefficient:

(5) The dissimilarity is obtained by the following equation.

As described above, according to the present embodiment, as an operation for specifying a specific pixel (point) in an image, the user simply selects an arbitrary point in the image and selects a desired region. Is possible. As a result, the user can easily enlarge and print a desired area or edit an image.

The embodiment of the present invention is not limited to the above-described one, and the processing steps are integrated or distributed, and the hardware for executing the processing steps is integrated, divided, or distributed. Can do. Moreover, the program which comprises embodiment of this invention can be distributed via a computer-readable recording medium or a communication line.

In the relevance calculation process (steps S13 to S14 in FIG. 1), the following (1) to
(6) A configuration as described above can be used. (1) A region that includes an area including a designated point is regarded as a highly related area. (2) Using a similarity degree related to at least one of a shape and a color with respect to an area including a designated point as an index, an area having a high similarity degree is set as an area having a high degree of association. (3) The relationship between specific colors and shapes is stored in advance, and the relationship is used as an index. (4
) Using an area as an index, a region having a relatively large area is defined as a region having a low degree of association. (5) A region having a relevance level equal to or higher than a predetermined value is set as a highly related region, and a combined region of a highly related region and a region including a designated point is extracted as a candidate region. (6) A composite region of a plurality of related regions and a region including a designated point is extracted as a candidate region.

The flowchart which shows the flow of a process of embodiment of this invention. The figure which shows the example of the image 1 for demonstrating the flow of FIG. The figure which shows the example of the image 1 for demonstrating the flow of FIG. The figure which shows the example of a display of the image 1 for demonstrating the calculation process of FIG.1 S13. The chart for demonstrating an example of the calculation content of step S13 of FIG. The chart for demonstrating the example of a relevance (non-relevance) calculation of step S13 of FIG. The figure which shows the example of a display of the image 1 for demonstrating the process of step S14 of FIG. The figure which shows the example of a display of the image 1 for demonstrating the process of step S14 of FIG. The figure which shows the example of a display of the image 1 for demonstrating the process of step S14 of FIG.

Explanation of symbols

1 image, 10 specified area, 20 specified points

Claims (12)

An image processing method for extracting an area desired by a user from an image,
Dividing the image into regions,
Designating a point in the image according to a user instruction;
A relevance analysis step for analyzing the relevance between the region including the specified point and another region;
Extracting a plurality of candidate regions by combining a region including the designated point and a plurality of highly relevant regions;
Presenting multiple candidate areas to the user;
And a step of selecting a candidate area displayed in response to a user instruction. The image processing method according to claim 1, wherein, in the relevance analysis step, an area having a high relevance is defined as an area having a close distance with a distance from the designated point as an index. 2. The image processing method according to claim 1, wherein in the relevance analysis step, an area adjacent to the area including the designated point is set as a high relevance area. The image processing method according to claim 1, wherein in the relevance analysis step, an area that includes an area including a designated point is set as a high relevance area. 2. The relevance analysis step according to claim 1, wherein the similarity is a region having a high degree of relevance, using as an index the similarity of at least one of the shape and color of the region including the designated point. Image processing method. 6. The image processing method according to claim 5, wherein in the relevance analysis step, relevance between specific colors and shapes is stored in advance, and the relevance is used as an index. 7. The image processing method according to claim 1, wherein in the relevance analysis step, an area having a relatively large area is used as an index, and an area having a low relevance is used as an index. In the step of extracting the plurality of region candidates, a region having a relevance level equal to or higher than a predetermined value is extracted as a highly related region, and a combined region of the highly related region and the region including the designated point is extracted as a candidate region. The image processing method according to claim 1, wherein: 9. The image processing method according to claim 8, wherein, in the step of extracting the plurality of region candidates, a combined region of a plurality of related regions and a region including a designated point is extracted as a candidate region. In the step of extracting the plurality of region candidates, the candidate region A is included in another candidate region B at a predetermined ratio, and the region C obtained by combining the candidate region A and the candidate region B is compared with the candidate region B. 10. The image processing method according to claim 1, wherein the area C is a candidate area when the increase falls within a predetermined ratio. An image processing apparatus that extracts an area desired by a user from an image,
Means for segmenting the image;
Means for designating one point in the image in response to a user instruction;
Relevance analysis means for analyzing the relevance of the region including the specified point and other regions;
Means for extracting a plurality of candidate regions by combining a region including the designated point and a plurality of highly relevant regions;
Means for presenting multiple candidate areas to the user;
An image processing apparatus comprising: means for selecting a candidate area displayed in accordance with a user instruction. An image processing program for extracting an area desired by a user from an image,
Dividing the image into regions,
Designating a point in the image according to a user instruction;
A relevance analysis step for analyzing the relevance between the region including the specified point and another region;
Extracting a plurality of candidate regions by combining a region including the designated point and a plurality of highly relevant regions;
Presenting multiple candidate areas to the user;
An image processing program comprising: a step of selecting a candidate area displayed in accordance with a user instruction by a computer.

Images