JP3510040B2 - Image processing method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method for correcting poor color tone of a pupil in a subject image due to flash emission during color photography.

[0002]

2. Description of the Related Art Heretofore, there have been proposed several techniques for correcting the so-called red-eye defect in the color tone of the pupil (US Pat.
130789, JP-A-7-72537, etc.). The technique disclosed in US Pat. No. 5,130,789 encloses the target eye with a rectangular mask, designates several points in the red eye region with a pointer, and sets the red eye region based on the color information. If is within this area, it is determined to be red eye and corrected. At this time, the correction amount is changed according to the distance from the boundary of the area.

Further, in the technique disclosed in Japanese Patent Laid-Open No. 7-72537, the target eye area is designated, candidate pixels are selected by color information, and after labeling, information on each label is also added. , To evaluate whether it has red eyes. In addition, regarding the correction method, the correction amount is changed separately for the main body of the red eye and the peripheral portion.

[0004]

Problems to be Solved by the Invention US Pat. No. 5,130,78
With the technique of No. 9, in addition to surrounding the target eye with a rectangular mask, it is necessary to specify several points within the red-eye region, which imposes a burden on the operator. JP-A-7-7
Regarding the technology of No. 2537, the candidate area is set by comparing a plurality of color information with a threshold value. However, since the color information of the part surrounding the red eye greatly differs depending on the shooting scene and individual differences, the same condition is used. It is difficult to set, which causes a decrease in yield. Further, the correction method is complicated because it is necessary to separate the main body and the peripheral portion and change the correction amount.

The present invention has been made in view of the above-mentioned conventional problems, and an object of the present invention is to provide an image processing method capable of easily and accurately correcting red eye.

[0006]

Therefore, according to the invention of claim 1, as shown in a functional block diagram of FIG. 1, one or more pupils have poor color tone for a captured image in which the pupils are included in the subject. Area is set, the inside of the set area is divided into a plurality of small areas, and the pupil in which the poor color tone is present is determined based on the color information of each small area and the position information. A small area having a high possibility of being included is selected as a pupil candidate area, a pixel having a poor color tone is extracted from the selected pupil candidate area, and the poor hue quality of the extracted pixel is corrected. To do.

According to the second aspect of the present invention, a range surrounding a pair of pupils is set as a region in which the poor color tone of the pupil may exist, and the center point of the set region is calculated. And the pupil candidate area is selected based on the color information and the position information based on the center point.

In the invention according to claim 3, the range surrounding the pair of pupils is set by designating one point near the middle of the pair of pupils and setting the designated point as a center. It is characterized by setting a region having a different area, and dividing into small regions and selecting a pupil candidate region based on the color information and position information with the designated point as a reference.

According to a fourth aspect of the present invention, the region in which the poor hue of the pupil may exist specifies a range surrounding one pair of pupils in which one or more of the poor hues are present, and It is characterized in that points near the centers of a pair of pupils are designated, and division into small regions and selection of pupil candidate regions are performed based on color information and position information with the designated points as references.

Further, in the invention according to claim 5, in the division into the small areas, edge pixels are extracted from the set area on the basis of color information and divided into areas surrounded by the edge pixels. It is characterized by doing it. Further, in the invention according to claim 6, the selection of the pupil candidate region is performed by obtaining an evaluation function indicating pupil-likeness based on the color information and the position information for each combination of the two small regions as one set. The feature is that it is performed by selecting a combination with a high evaluation.

Further, the invention according to claim 7 is characterized in that the correction of the defective color tone pixel is performed based on the lightness information of a color component other than red of the defective color tone pixel. Further, claim 8
The invention according to claim 1 is characterized in that the defective color tone pixel is corrected based on information of a color selected from a plurality of color samples by an operator.

According to a ninth aspect of the present invention, in the correction of the defective color tone pixel, an operator sets an arbitrary threshold value regarding color information for extracting the defective color tone pixel, thereby correcting the defective color tone pixel correction region. It is characterized by being adjustable.

[0013]

According to the first aspect of the invention, it is possible to increase the height based on the color information and the position information from each of the small areas obtained by dividing the area where one or more pupils may have poor color tone. It is possible to select the pupil candidate area with high accuracy, and thus correct the red-eye correction by correcting the defective color tone pixel in the pupil candidate area.

Further, according to the inventions of claims 2 to 4, it is possible to select the pupil candidate region with high accuracy with a minimum burden on the operator. According to the invention of claim 5, by dividing into areas surrounded by edge pixels, it is possible to divide into small areas having substantially the same color tone base, and defective color tone pixels are extracted from within the small areas. Easier to do.

According to the sixth aspect of the present invention, by combining two small areas as one set and evaluating the combination, an evaluation function utilizing symmetry can be used, especially when red eyes occur in two pupils. The pupil candidate area can be selected with high accuracy. According to the invention of claim 7,
Based on lightness information of color components other than red, such as blue and green,
By correcting the brightness of red or red, blue, and green so as to match the tendency of the brightness, a natural feeling can be corrected.

According to the invention of claim 8, the operator can select from the color sample and correct the desired color. Further, according to the invention of claim 9, it is possible to set the correction area by enlarging or reducing it according to the degree to which the operator feels that the color tone is poor by setting the threshold value.
In addition, it is possible to absorb variations in the color information of the poor color tone region and its peripheral region that occur due to the shooting scene and individual differences.

[0017]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. FIG. 2 shows a system configuration of one embodiment. Digital image data of a color image taken by a still video camera, or a color image taken by a silver salt film camera and developed on film,
Digital image data read by a scanner is stored in the storage device 1 such as an optical disk.

The image data stored in the storage device 1 is read by the control device 2 and displayed as an image on the monitor 3. While viewing the image displayed on the monitor 3, the operator sets an image area for a subject image in which red-eye is generated by flash photography as follows, and the control device 2 sets the area. The red-eye pixels are extracted from among the pixels and corrected, and the corrected image is displayed on the monitor 3.

The red-eye correction routine according to the present invention will be described below with reference to the flowcharts shown in FIG. FIG. 3 shows a main routine for red-eye correction. In step (denoted as S in the figure. The same applies hereinafter) 1, in the subject image including the pupil displayed on the monitor 2, a rectangular area surrounding the pair of eyes is set, and at the same time, in the vicinity of the middle of the pair of eyes. One point is designated as a reference point for color information and position information described later (see FIG. 6).

As a method of setting the target area and the intermediate point, the operator may specify both the target area and the intermediate point,
You can specify only the target area and calculate the midpoint by calculation (for example, calculate the center of the specified rectangular area as the midpoint), or specify only the midpoint and set the preset area centered there. It may be the target area. The accuracy of red-eye correction is the highest because the first method manually specifies both the target area and the intermediate point.

A feature of the present invention is that, for example, in a pair of pupils,
Even if there is only one pupil with poor color tone, the area surrounding a pair of pupils is designated. Further, when both the pair of pupils have red eyes, it is possible to correct two red eyes at the same time by specifying the area once. For example, in the technique disclosed in US Pat. No. 5,130,789, the degree of intervention of the operator is large because the area is designated for each pupil.

In step 2, the color information of the point designated in step 1 above, for example, R (red), G (green), B (blue)
The total value (R + G + B) of the respective brightness values of is calculated as the feature amount. In step 3, the pixel of which the difference between the feature amount of the designated point and the feature amount is within the designated threshold is set as the skin color pixel, and the region of the skin color pixel is extracted. That is, the designated point is that the point near the middle of the pair of eyes is located at the base of the nose,
Since the color at that point is estimated to be the skin color, the region having a color close to the point can be estimated to be the skin color pixel. In addition, the hue and saturation, which are general color information, may be used as the feature amount. In this way, the pixels within the threshold value are labeled, and the labeling area including the designated point is extracted as the skin color area.

In step 4, the average value Rskin of the chromaticity [= R / (R + G + B)] of red in the skin color area is calculated. In step 5, in order to reduce the following calculation amount and the data amount necessary for the calculation, a rectangular area circumscribing the skin color area extracted in step 3 (a part between the eyes so that both eyes are circumscribing simultaneously) (Including one horizontally long area)
Set as a new target area. The setting of the rectangular area is
It is done automatically. This may be done manually, but it can also be done automatically if the operator specifies a wide area that includes an area other than the face.

In step 6, the target area is divided into small areas. This sub-region division subroutine will be described with reference to the flowchart of FIG. In step 11, edge pixels are obtained within the set area. The edge pixel is obtained by, for example, using the lightness (R + G + B) as a feature amount and obtaining the sum of the absolute values of the differences in lightness between the target pixel and four surrounding pixels. To determine. In addition, Sobel operator, P
An edge detection filter such as a rewitt operator may be used.

In step 12, the edge pixels detected in step 11 are labeled. In step 13, a pixel surrounded by edge pixels of the same label on at least three sides is changed to the same label as the edge pixel. That is, the areas surrounded by the edges have the same label. In this way, the set area is divided into small areas having the same label.

Returning to the main routine of FIG. 3, in step 7, an area having a high possibility of including a pupil having a poor color tone is selected from the divided small areas as a pupil candidate area. The pupil candidate area selection subroutine in step 7 will be described with reference to the flowchart of FIG.

In the present embodiment, the pupil-likeness is obtained by calculating the following characteristic amount for all combinations, with two of the divided small areas as one set. In step 21, the red chromaticity [= R / (R + G +
B)], the maximum value (Ri) of the chromaticity of red in the area and the point of the eye and the middle of the eye specified in step 1 are set as the origin, and the pixel of the maximum value (Ri) The position (Xi, Yi) on the X-axis and the Y-axis is set as the feature amount. Where i
Represents a value for identifying a small area.

For example, when combining the small area 1 and the small area 2, the pupil candidate area is selected as follows. In step 22, the characteristic amount C12 of the color information and the characteristic amount P12 of the position information are obtained by the following equation. Color information feature amount C12 = R1 + R2 Position information feature amount P12 = X1 + X2 + Y1 + Y2 Here, the larger the color information feature amount, the smaller the position information feature amount, the higher the pupil-likeness. For example, the characteristic amount E12 is E12 = C12−
P12 is set, and the combination of the small areas having the highest value is extracted as a pupil candidate area having a high possibility that at least one of the pupils has a poor color tone (red eye).

Here, the smaller the feature amount of the position information, the higher the likelihood of pupils is that if both eyes have red eyes,
This is because when one of X1 on the left and right is set to + and the other X2 is set to − for the point between the eyes, the sum becomes smaller. Further, only one of the pupils may have a red eye. When a combination of both pupils is selected, the value of the position information P12 becomes small even if there is no red eye in one pupil, and only one red eye exists. However, this is because the feature amount above the color is larger than that of the combination having no red eyes.

Returning to the main routine of FIG. 3, in step 8, from among the pupil candidate regions extracted in step 7,
Poor color tone (red eye) Extract pixels. The extraction method is as follows. Red chromaticity of target pixel [= R / (R + G +
B)] is the average value Rsk of the skin color area obtained in step 2.
If it is larger than in and larger than the value obtained by subtracting the set threshold value TH from the feature amount (Ri) of the color information of the target area, it is determined as a red-eye pixel. That is, the pixels that satisfy the following equation are set as red-eye pixels.

R / (R + G + B)> Rskin∩R /
(R + G + B)> Ri-TH In step 9, the red-eye pixel extracted in step 8 is corrected to an appropriate color tone. In this case, if all the pixels are corrected to have the same pixel value, it looks very unnatural, so that the brightness information other than red is utilized. For example, when the digital values are R, G, and B, R = 200, G = 50, and B = 30, the gray color is corrected according to the lowest digital value. In this case, since it is possible to correct the pupil image that reflects the lightness of R = G = B = 30 to some extent, it looks natural.

Alternatively, in order to deal with the case where the pupil color is not originally gray, for example, the digital values are R = 200, G = 50, and B = 3, respectively, similarly to the above.
When 0, G and B are maintained at the same values, and R is adjusted to the larger value of G and B, in this case, 50 of G. In addition, the value of R may be adjusted to the average value of G and B, which is 40 in this case.

Further, when the color does not match the operator's favorite pupil color, the operator selects a favorite color from the pupil color palette (sample) and corrects it based on the color information. As a method of correction, for example, the following procedure is performed. The color information of the selected color, for example YIQ, is calculated by the following equation.

[0034]

[Equation 1]

Among the red-eye pixels, the maximum value of B + G is obtained, and this is taken as Lmax. The brightness Y of the palette is multiplied by a value obtained by dividing L (= B + G) of the target pixel by Lmax (see the following equation). Y ′ = Y × L / Lmax The values of I and Q are maintained and converted back to R, G and B,
This digital value is used as the correction value. This will make it look more natural.

That is, although the color selection is performed by using the palette, the brightness is corrected by making use of the brightness information of the components other than red of the red-eye pixel, so that a natural feeling can be obtained. In the series of processes, the chromaticity of red is obtained, but other general color information such as a combination of lightness, hue, and saturation may be used. According to this embodiment, the following effects can be obtained.

The degree of operator intervention is small. Since the area is divided using edge information, compared to the method of dividing using only color information, the shooting scene,
It is less affected by individual differences and can be corrected with high yield. By performing the correction when the pupil color is not gray, it is possible to cope with cases other than Japanese.

Next, a second embodiment will be described.
In the first embodiment, there may be a case where only a part of the red-eye portion to be corrected is corrected or the non-red-eye portion is also corrected due to individual differences in the red-eye portion. In this embodiment, in order to prevent this, the operator can adjust the correction area by adjusting the threshold value for determining the presence or absence of the correction while looking at the image.

Specifically, when the operator looks at the image and determines that only part of the red-eye portion has been corrected, the threshold value TH set for red-eye pixel extraction is lowered. , Until the correction area is enlarged appropriately,
On the contrary, when the operator determines that the red eye is not corrected, the threshold value TH is increased.
Repeat until the correction area is properly reduced.

The setting of the red-eye area also depends on the subjectivity, and by doing so, it is possible to make a correction that satisfies the operator's subjectivity. In addition, it is possible to absorb variations in color information around the red eye and the red eye, which are caused by scenes and individual differences.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration and functions of the present invention.

FIG. 2 is a diagram showing a system configuration of an embodiment.

FIG. 3 is a flowchart showing a main routine of red-eye correction in the embodiment.

FIG. 4 is a flowchart showing a sub-region division subroutine of the same routine.

FIG. 5 is a flowchart showing a subroutine of pupil candidate area selection of the same routine.

FIG. 6 is a diagram showing a method of designating a target area in the embodiment.

[Explanation of symbols]

1 storage device 2 controller 3 monitors

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ identification code FI H04N 5/91 H04N 5/91 J 9/79 9/79 G (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 9/64 H04N 1/46 H04N 1/60 H04N 5/91 H04N 9/79 G03B 15/05 G06T 1/00 340

Claims (9)

(57) [Claims] 1. A captured image in which a pupil is included in a subject,
An area in which there is a possibility that one or more pupils have poor color tone is set, the set area is divided into a plurality of small areas, and the poor hue quality is obtained based on the color information of each of the small areas and the position information. Is selected as a pupil candidate region having a high possibility of including a pupil, and a pixel having a poor color tone is extracted from the selected pupil candidate region, and a poor color tone of the extracted pixel is selected. An image processing method, which comprises: 2. A region surrounding a pair of pupils is set as a region in which the poor color tone of the pupil may exist, a center point of the set region is calculated, and the center point is used as a reference. 2. The division into small areas and the selection of pupil candidate areas are performed based on the color information and the position information.
The image processing method described in. 3. The setting of the range surrounding the pair of pupils is performed by designating one point near the middle of the pair of pupils and setting a region having a preset area centered on the designated point. Done by
The image processing method according to claim 1, further comprising dividing into small regions and selecting a pupil candidate region based on color information and position information with the designated point as a reference. 4. The area in which the poor hue of the pupil may exist specifies a range surrounding a pair of pupils in which one or more pupils have poor hue, and the area near the center of the pair of pupils. 2. The image processing method according to claim 1, wherein a point is designated, and division into small areas and selection of a pupil candidate area are performed based on color information and position information with the designated point as a reference. 5. The division into the small regions is performed by extracting edge pixels from the set region based on color information and dividing the regions into regions surrounded by the edge pixels. The image processing method according to any one of claims 1 to 4. 6. The selection of the candidate pupil area is performed by obtaining an evaluation function indicating the likelihood of a pupil based on color information and position information for each combination of two small areas as one set, and selecting the combination with the highest evaluation. The image processing method according to any one of claims 1 to 5, wherein the image processing method is performed by selecting. 7. The image according to claim 1, wherein the defective color tone pixel is corrected based on the brightness information of a color component other than red of the defective color tone pixel. Processing method. 8. The image according to any one of claims 1 to 7, wherein the defective color tone pixel is corrected based on information of a color selected from a plurality of color samples by an operator. Processing method. 9. The correction of defective color tone pixels is characterized in that an operator can set a correction region of defective color tone pixels by arbitrarily setting a threshold value regarding color information for extracting defective color tone pixels. The image processing method according to any one of claims 1 to 8.