JP7510258B2 - Image processing device, image processing method, imaging device, program, and storage medium - Google Patents

Description

The present invention relates to a technology for removing fog from images captured in foggy conditions to improve visibility.

Conventional cameras used for surveillance and other purposes can sometimes produce haze when capturing images outdoors, reducing visibility. To save such images, there are known techniques for performing correction processing to remove the haze when it occurs. Correction processing includes a method that uses a tone curve to correct contrast for the entire screen, and a scattering separation method that separates Mie scattering and Rayleigh scattering in an atmospheric model and removes Mie scattering components that are not wavelength-dependent, such as haze.

The scattering separation method estimates the transmittance using the dark channel value for each region of the image and controls the gradation to remove mist and haze. Specifically, a technology has been proposed that estimates the transmittance caused by the concentration of mist and haze in an atmospheric model and performs gradation control.

In Patent Document 1, the sky region or backlit region is calculated, the dark channel value in that region is corrected, the transmittance is calculated from the dark channel value, and the mist component is removed from the input image using the transmittance.

In Patent Document 2, pixels used to derive the reference intensity of scattered light in a color image are selected, and a first reference intensity corresponding to a first color component of the scattered light and a second reference intensity corresponding to a second color component are derived. The first reference intensity and weighting value are used to correct the pixel values of the first color component of the color image, and the second reference intensity and weighting value are used to correct the pixel values of the second color component to generate a corrected image.

JP 2017-138647 A JP 2015-103167 A

However, in the conventional technology disclosed in Patent Document 1, although it is possible to correct for scattered light using dark channel values, halo noise occurs at the edges of the image, and this noise cannot be suppressed.

Similarly, the conventional technology disclosed in Patent Document 2 can also use scattered light in an atmospheric model to remove and correct components caused by the concentration of fog and haze, but it still generates halo noise at the edges of the image and cannot suppress the noise.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can effectively remove fog from an image.

The image processing device of the present invention comprises an acquisition means for acquiring image data, a first gradation correction means for acquiring a dark channel value for each region of the image data to estimate transmittance and correct the gradation, a second gradation correction means for correcting the gradation of the image data using a tone curve, and a control means for controlling which gradation correction means is to be used for the gradation correction , wherein the control means causes the first gradation correction means to perform gradation correction so as to gradually obtain a correction effect, and stops the gradation correction by the first gradation correction means when a white region is extracted at the contour of a subject in the image data .

The present invention makes it possible to provide an image processing device that can effectively remove fog and haze from an image.

1 is a diagram showing the configuration of a digital camera that is a first embodiment of an image processing apparatus of the present invention. FIG. 2 is a block diagram showing the configuration of an image processing circuit 120 according to the first embodiment. 13 is a diagram showing luminance histograms with and without the occurrence of mist and haze. 11A to 11C are diagrams showing examples of toe curve correction according to the degree of mist generation. 4 is a flowchart showing a mist and haze removal operation in the first embodiment. 10 is a flowchart showing an operation of removing mist and haze in a second embodiment. FIG. 13 is a block diagram showing the configuration of an image processing circuit 121 according to a third embodiment. 13 is a flowchart showing an operation of removing mist and haze in the third embodiment. 13 is a flowchart showing an operation of removing mist and haze in the fourth embodiment. FIG. 13 is a diagram showing the configuration of an imaging system according to a fifth embodiment. 13 is a flowchart showing an operation of removing mist and haze in the fifth embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
FIG. 1 is a diagram showing the configuration of a digital camera 100 which is a first embodiment of an image processing device of the present invention.

In FIG. 1, a lens unit 102 focuses an optical image of a subject on an imaging unit 101 having an imaging element such as a CCD or CMOS sensor. A lens driving device 103 performs zoom control, focus control, aperture control, and the like. An imaging signal processing circuit 104 performs various correction processes and data compression processes on the image signal output from the imaging unit 101. The imaging signal processing circuit 104 also has an internal image processing circuit 120 for removing mist from the image. An imaging element driving circuit 105 outputs various power sources, shooting mode instruction signals, and various timing signals to the imaging element in the imaging unit 101.

The memory unit 106 functions as a memory for temporarily storing image data, and the overall control calculation unit 107 performs various calculations and controls the entire digital camera 100. The recording medium control I/F unit 108 is an interface for recording data to the recording medium or reading data from the recording medium. The recording medium 109 is a removable semiconductor memory for recording or reading image data. Furthermore, the display unit 110 is a display device such as a liquid crystal display device that displays various information and captured images.

Figure 2 is a block diagram showing the configuration of the image processing circuit 120 in Figure 1. The image processing circuit 120 has a function to remove mist while suppressing halo noise that occurs at the edges of an image.

In FIG. 2, the image processing circuit 120 includes a first image processing unit 11, a first haze correction unit 12, a second haze correction unit 14, a second image processing unit 15, an image output unit 16, and a histogram acquisition and analysis unit 13.

Image data captured by the imaging unit 101 is input to the image processing circuit 120. The first image processing unit 11 is a first-stage image processing unit for image data. Specifically, the image data acquired from the imaging unit 101 is subjected to sensor correction processing, scratch correction processing, conversion processing for converting a Bayer-type RGB signal to a YC signal, contour emphasis processing, noise reduction processing, white balance processing, and the like. The first haze correction unit 12 is a first-stage gradation correction unit that estimates the transmittance (density map) using the dark channel value for each region of the image and performs a first gradation correction processing (gradation correction operation) that controls the gradation. The histogram acquisition and analysis unit 13 acquires a histogram of the luminance component in the corrected image after the first gradation correction processing, and detects the contrast of the image by calculating the variance value. The second haze correction unit 14 is a second-stage gradation correction unit that performs a second gradation correction processing (gradation correction operation) that controls the contrast and intermediate gradations using a tone curve.

The details of the first tone correction process are described below. First, image I in which mist and haze have occurred and image J after mist and haze have been removed are related by an atmospheric model such as equation (1).

I(x,y)=J(x,y)·t(x,y)+A(1−t(x,y) Equation (1)
x, y are two-dimensional coordinate positions in the horizontal and vertical directions in the image, t is the mist density map, and A is the ambient light. Here, the density map t(x, y) represents attenuation due to the atmosphere, and the longer the subject distance from the camera, the greater the amount of attenuation (pixel value), and the closer the subject is, the smaller the amount of attenuation (pixel value). In formula (1), by estimating the ambient light A and the density map t(x, y), it is possible to find J(x, y) after mist removal.

First, we will explain how to estimate the ambient light A using a formula. Ambient light A is the average pixel value of the RGB components of the sky region, and is calculated using formula (2).

The ave function is a function that calculates the average value in the arguments, c represents a color component, and Ωsky represents a local area within the sky region. Here, the sky region can be identified by, for example, a method of calculation based on the distribution of a histogram, a method of using a coordinate position specified in advance, or a method of using a position indicated by the user.

Next, a method for estimating the density map t(x, y) will be described. The first gradation process in this embodiment is a process for estimating the density map t(x, y) based on the dark channel value, and is based on the premise that in an outdoor image without haze, the pixel value of at least one of the RGB components is locally very small. The dark channel value I ^drk is calculated as shown in Equation (3).

c represents a color component, and Ω represents a local region including the target coordinates (x, y). As shown in equation (3), the dark channel value is the minimum value of the RGB components in the local region including the target pixel. By substituting the dark channel calculation formula (3) into the atmospheric model (1), equation (4) is obtained to calculate the dark channel value from the atmospheric model.

I ^drk = J ^drk (x, y) · t (x, y) + A (1 - t (x, y)) Equation (4)
Here, assuming that in an image without fog or haze, the pixel value of at least one of the RGB components is locally small, the dark channel value J ^drk (x, y) of the defrosted image in equation (4) is extremely close to 0. Therefore, equation (4) can be approximated as equation (5).

I ^drk (x, y) ≈ A (1-t (x, y)) Equation (5)
By modifying the approximation of equation (5), the concentration map t(x, y) can be estimated as in equation (6).

t(x,y)≈1-{ω×I ^drk (x,y)/A} Equation (6)
Here, ω is a parameter that controls the degree of mist/haze correction and is defined in the range from 0.0 to 1.0, and the larger the value, the stronger the mist/haze correction effect can be set. By substituting the airglow A and concentration map t(x, y) calculated by equations (2) to (6) described above into equation (7), it is possible to find J(x, y) after mist/haze removal.

J(x,y)={(I(x,y)-A)/t(x,y)}+A Equation (7)
As shown in formula (1), the first tone correction process based on the dark channel calculates the correction amount for each pixel, taking into account not only the ambient light A but also the density map that changes depending on the subject distance from the camera. Therefore, it is possible to effectively remove fog and haze not only from close-up subjects but also from distant subjects. However, as shown in formula (3), the calculation of the dark channel value does not take into account edge information, and as shown in formula (4), the density map is calculated by approximation, so that erroneous correction occurs around the edge portion, specifically, a phenomenon in which the edge portion appears white (hereinafter, this phenomenon is called halo noise (white area)).

Next, the second haze correction unit 14 will be described in detail. First, the histogram acquisition and analysis unit 13 generates a histogram of the input image and calculates a statistical value representing the distribution density of the generated histogram. For example, the variance ^σ2 of the histogram is calculated according to the formula (8).

σ ² = {(x1 - μ) ² · f1 + (x2 - μ) ² · f2 + ... + (xN - μ) ² } / n Equation (8)
Here, x is the input luminance level, x is the maximum value of the input luminance level, f is the frequency, n is the sum of the data (total number of pixels), and μ is the average. Note that in this embodiment, the variance of the histogram is used as a statistical value representing the distribution density, but this is not limiting.

Furthermore, the histogram acquisition and analysis unit 13 calculates the degree of occurrence of mist and haze based on the above-mentioned histogram and statistical values. Generally, an image in which mist and haze occur has a low contrast image with the histogram distribution concentrated in a certain area as shown in FIG. 3(a). Therefore, when mist and haze occur, the variance value becomes relatively small. Therefore, the mist and haze correction unit 2 calculates the degree of occurrence of mist and haze H by comparing an arbitrary threshold value Th with the variance value ^σ2 . For example, the calculation is performed as shown in Equation (9).

0.0 (if ^σ2 < 0)
H = (Th - ^σ2 ) /Th (if 0≦ ^σ2 ≦Th)
1.0 (if Th < σ ² ) Equation (9)
Here, the degree of occurrence of mist/haze expressed by the formula (9) is normalized in the range of 0.0 to 1.0, and the larger the value, the stronger the degree of occurrence of mist/haze becomes.

Then, the second haze correction unit 14 adjusts the tone curve according to the degree of mist generation calculated by the histogram acquisition and analysis unit 13. Specifically, as shown in FIG. 4, the stronger the degree of mist generation, the more a tone curve is applied that lowers the dark areas and raises the bright areas. By applying the tone curve described above, the histogram becomes smoothly distributed as shown in FIG. 3(b), and mist generation can be removed.

The second haze correction unit 14 therefore improves the contrast of the entire screen according to the histogram distribution (degree of haze occurrence), making it possible to remove haze while minimizing adverse effects. However, because it does not calculate the amount of correction taking into account the subject distance, this method is less effective for distant subjects.

In the above, the first haze correction unit 12 (one of the haze correction units) performs the first gradation correction process and then the second haze correction unit 14 (the other haze correction unit) performs the second gradation correction process, but the order may be reversed. In other words, a similar mist removal effect can be obtained by performing the second gradation correction process in the first haze correction unit 12 and then the first gradation correction process in the second haze correction unit 14.

The second image processing unit 15 performs a second stage of image processing on the image from which the mist and haze have been removed by the first dehaze correction unit 12 and the second dehaze correction unit 14. Specifically, edge enhancement processing, noise reduction processing, color correction processing, resolution conversion processing, etc. are performed. Here, edge enhancement processing, noise reduction processing, color correction processing, etc. may be performed by the first image processing unit 11 or the second image processing unit 15. The image output unit 16 outputs the image that has been subjected to image processing and dehaze processing to another image processing unit that performs image compression processing, etc., the display unit 110, etc.

In the first gradation correction process, the correction process is performed so as to gradually obtain a correction effect over time. Specifically, the value of ω in equation (6) is set gradually from a small value to a large value. Then, if halo noise is extracted from the edge, the first gradation correction process is stopped (stopped). By doing so, it is possible to suppress halo noise from the edge that may occur in the first gradation correction process.

As described above, it is also possible to consider a case where the first haze correction unit 12 performs the first tone correction process and the second haze correction unit 14 performs the second tone correction process, or a case where the order of tone correction processes is reversed. In each case, tone correction processes are performed in the following manner.

First, when the first haze correction unit 12 performs the first gradation correction process and the second haze correction unit 14 performs the second gradation correction process, the following method is used. That is, the first haze correction unit 12 gradually performs the first gradation correction process while watching the appearance of halo noise as described above, and after the gradation correction process stops, the effect of the gradation control (correction result) is judged by watching the image statistics, specifically the luminance histogram. If the effect of the gradation control is insufficient, the second haze correction unit 14 performs the second gradation correction process over time until it reaches a gradation correction amount (target gradation) that provides a sufficient effect. However, since the second gradation correction process does not need to be performed slowly, it does not necessarily need to be performed slowly because it does not need to take halo noise into consideration.

In addition, when the first haze correction unit 12 performs the second gradation correction process and the second haze correction unit 14 performs the first gradation correction process, the following method is adopted. That is, the first haze correction unit 12 performs the second gradation correction process gradually over time while observing the effect of the second gradation correction process. In the second gradation correction process, the effect of removing mist and haze is not significant in the distant view, so the process is performed while observing the degree of the effect. However, the second gradation correction process has the advantage that it is easier to see the change in effect when performed over a long period of time, but as described above, there is no need to consider halo noise, so it is not necessarily required to perform it slowly. Then, after the second gradation correction process stops, if the effect of the gradation control is insufficient, the second haze correction unit 14 performs the first gradation correction process over a long period of time until it reaches a gradation correction that provides a sufficient effect. In this case, the first gradation correction process stops when halo noise is extracted from the contour.

By using the above method, it is possible to perform good de-haze processing that suppresses the occurrence of halo noise.

Figure 3 is a flow chart showing the operation of removing mist and haze so that halo noise does not occur at the edges of an image. Note that Figure 3 shows an example in which the first haze correction unit 12 performs a first tone correction process, and the second haze correction unit 14 performs a second tone correction process.

When the flow of the mist removal process starts, in step S31, the image captured by the imaging unit 101 is input to the image processing unit 120. In step S32, the first haze correction unit 12 performs a first tone correction process. As described above, the correction process is performed within a range that does not cause halo noise in the contours of the image.

In step S33, the overall control calculation unit 107 determines the magnitude of contrast for the image resulting from the first tone correction process in step S32. If the contrast is less than a predetermined value, the process proceeds to step S34, and if the contrast is equal to or greater than the predetermined value, the tone correction process ends.

In step S34, a second gradation correction process is performed to remove any mist or haze that could not be completely removed by the first gradation correction process performed in step S32, and the gradation correction process is terminated.

Note that FIG. 3 shows an example in which the first gradation correction process is performed in step S32, and the second gradation correction process is performed in step S34. However, as already explained, it is also possible to perform the second gradation correction process in step S32, and the first gradation correction process in step S34. In either case, the first gradation correction process performs correction within a range in which halo noise does not occur.

As described above, according to the first embodiment, it is possible to perform effective mist removal processing to the extent that halo noise does not occur.

Second Embodiment
A digital camera that is a second embodiment of the image processing device of the present invention will be described below. The overall configuration of the digital camera in the second embodiment is similar to that of the first embodiment shown in Fig. 1, so a description thereof will be omitted. In this second embodiment, the first tone correction process and the second tone correction process described in the first embodiment are weighted and executed.

Figure 4 is a flowchart showing the operation of de-misting an image in the second embodiment to prevent halo noise from occurring at the edges of the image.

When the flow of the mist removal process starts, in step S41, the image captured by the imaging unit 101 is input to the image processing unit 120. In step S42, the first image processing unit 11 acquires statistics of the brightness values of the input image, specifically, a brightness histogram.

In step S43, the overall control calculation unit 107 determines the magnitude of contrast from the brightness histogram acquired in step S42. If the contrast is smaller than a predetermined value, it is determined that there is a strong mist and the process proceeds to step S44. If the contrast is equal to or greater than the predetermined value, it is determined that there is not much mist and the process proceeds to step S46.

In step S44, the weight of the first tone correction process described in the first embodiment is increased. Here, the degree of mist and haze removal is increased, even if it means allowing some halo noise to occur at the edges of the image. In addition, in step S45, the second tone correction process described in the first embodiment is performed to remove any mist and haze that could not be removed in step S44. Then, the mist and haze removal process ends. Note that the same effect can be obtained even if the order of steps S44 and S45 is reversed.

In step S46, since the contrast is not particularly low and the mist is not strong, the weight of the first gradation correction process is reduced so as to prevent halo noise from occurring at the edges of the image. In step S47, the weight of the second gradation correction process is increased in order to remove any mist that was not completely removed in step S46. The mist removal process then ends. Note that the same effect can be obtained even if the order of steps S46 and S47 is reversed.

As described above, according to the second embodiment, it is possible to perform effective mist removal processing depending on the strength of the mist.

Third Embodiment
A digital camera according to a third embodiment of the image processing device of the present invention will now be described. The overall configuration of the digital camera in the third embodiment is also similar to that of the first embodiment shown in Fig. 1. However, in this embodiment, the configuration of the image processing circuit 121 is different from that of the image processing circuit 120 in the first embodiment. In this third embodiment, the defogging method is switched depending on the region of interest.

Figure 5 is a diagram showing the configuration of the image processing circuit 121 in the third embodiment. The image processing circuit 121 specifies (sets) an area of interest, measures the subject distance of that area, switches the gradation control method according to the subject distance, and removes mist and haze.

In FIG. 5, the image processing circuit 121 includes a first image processing unit 21, a region of interest designation unit 22, a distance information acquisition unit 23, a first haze correction unit 24, a second haze correction unit 25, a second image processing unit 26, and an image output unit 27.

Image data captured by the imaging unit 101 is input to the image processing circuit 121. The first image processing unit 21 is a first-stage image processing unit for image data, and is similar to the first image processing unit 11 described in the first embodiment. The attention area designation unit 22 has a function of designating an area to be focused on within the image data, and is equipped with a GUI or the like that allows the user to designate the area to be focused on. The distance information acquisition unit 23 measures the distance to the attention area designated by the attention area designation unit 22. Here, the distance to the attention area is measured using control information of the focus lens and zoom lens arranged in the lens unit 102, etc.

The first haze correction unit 24 executes the first gradation correction process described in the first embodiment. The second haze correction unit 25 executes the second gradation correction process described in the first embodiment. The second image processing unit 26 performs a selection process or a mixing process of the image corrected by the first haze correction unit 24 and the image corrected by the second haze correction unit 25, and a second stage image processing. The first haze correction unit 24 is effective in removing haze from a region of interest that is far away, so when the distance information acquired by the distance information acquisition unit 23 is farther than the first predetermined distance, the first haze correction unit 24 is selected. The second haze correction unit 25 is effective in removing haze from a region of interest that is close, so when the distance information acquired by the distance information acquisition unit 23 is closer than the second predetermined distance, the second haze correction unit 25 is selected. If the distance information acquired by the distance information acquisition unit 23 is equal to or less than the first predetermined distance and equal to or greater than the second predetermined distance, a mixing process (combining process) is performed on the image corrected by the first haze correction unit 24 and the image corrected by the second haze correction unit 25.

Here, the image blending ratio is determined for each region using edge information of the subject. Specifically, for regions considered to be edges, the ratio of the second gradation correction process described in the first embodiment is set to a large value. On the other hand, for regions considered to be flat, the ratio of the first gradation correction process described in the first embodiment is set to a large value. Blending at such ratios makes it possible to effectively remove mist and haze while suppressing halo noise in edge areas. The specific steps are described below.

First, edges are detected from the image. To detect edges, a Sobel filter is applied to the input image I, which is foggy, to extract edge information as a grayscale image.

Next, the blending ratio Blend(x, y) is calculated based on the generated edge information (density value F(x, y)) using, for example, equation (10).

Blend(x,y)=F(x,y)/Fmax Equation (10)
Here, Fmax indicates the maximum density value, and the mixture ratio is normalized to a value between 0.0 and 1.0.

Finally, based on the mixing ratio of equation (10), the image Dehaze1(x, y) corrected by the first haze correction unit 24 and the image Dehaze2(x, y) corrected by the second haze correction unit 25 are combined as shown in equation (11).

Dehaze _Blend (x,y)
= (1.0 - Blend(x,y)) x Dehaze1(x,y) + Blend(x,y) x Dehaze2(x,y)
Equation (11)
By following the above procedure, a blended image Dehaze _Blend (x, y) in which haze removal has been effectively performed while suppressing halo noise at edges is obtained.

The second image processing unit 26 is similar to the second image processing unit 15 described in the first embodiment. Each image processing function may be performed by the first image processing unit 21 or the second image processing unit 26. The image output unit 27 outputs the image that has been subjected to image processing and fog removal processing to another image processing unit that performs image compression processing, etc., the display unit 110, etc.

Figure 6 is a flowchart showing the operation of removing mist and haze by switching the gradation control method depending on the distance of the specified area of interest.

When the flow of the mist removal process starts, in step S61, the image captured by the imaging unit 101 is input to the image processing unit 121. In step S62, the user specifies the area of interest using a GUI or the like. In step S63, the distance information acquisition unit 23 measures the distance to the area of interest specified by the user. The distance to the area of interest is measured using control information for the focus lens and zoom lens arranged in the lens unit 102.

In step S64, the overall control calculation unit 107 determines whether the distance to the subject included in the region of interest is farther than a first predetermined distance. If the distance to the subject is farther than the first predetermined distance, the process proceeds to step S65. If the distance to the subject is not farther than the first predetermined distance, the process proceeds to step S66.

In step S65, the first tone correction process described in the first embodiment is executed, which is effective in removing the haze that hangs over distant subjects.

In step S66, it is determined whether the distance to the subject included in the region of interest is closer than a second predetermined distance. If the distance to the subject is closer than the second predetermined distance, the process proceeds to step S67. If the distance to the subject is not closer than the second predetermined distance, the process proceeds to step S68.

In step S67, the second tone correction process described in the first embodiment, which is effective in removing mist from nearby objects, is executed. In step S68, based on the blending ratio calculated using equation (10), the image obtained by the first tone correction process described in the first embodiment and the image obtained by the second tone correction process are blended as shown in equation (11). This step S68 is executed when the distance to the object included in the region of interest is equal to or less than a first predetermined distance and equal to or greater than a second predetermined distance. After executing step S65, S67, or S68, the mist removal operation is terminated.

As described above, according to the third embodiment, it is possible to perform effective de-haze processing depending on the distance of the subject.

Fourth Embodiment
A digital camera that is a fourth embodiment of an image processing device of the present invention will now be described. The configuration of the image processing circuit in the fourth embodiment is similar to that of the third embodiment shown in Fig. 5. However, in this embodiment, the attention area designation unit 22 detects a subject to be focused on and automatically designates the area in which the subject exists.

Figure 7 is a flowchart showing the operation of detecting a subject of interest and removing mist by switching the gradation control method based on the distance to the subject of interest.

When the flow of the mist removal process is started, in step S71, the image captured by the imaging unit 101 is input to the image processing unit 121. In step S72, a subject to be focused on is detected, and the area in which the subject exists is automatically specified. To detect a subject to be focused on, for example, the user specifies an object or person to be focused on. Then, the object or person to be focused on is detected using object detection or AI technology, and the area including them is specified. Alternatively, a moving object or person may be detected using motion detection, and the area including them may be specified.

In step S73, the distance information acquisition unit 23 measures the distance to the area including the specified subject of interest. The distance to the area of interest is measured using control information of the focus lens and zoom lens arranged in the lens unit 102.

In step S74, the overall control calculation unit 107 determines whether the distance to the subject of interest is farther than a first predetermined distance. If the distance to the subject is farther than the first predetermined distance, the process proceeds to step S75. If the distance to the subject is not farther than the first predetermined distance, the process proceeds to step S76.

In step S75, the first tone correction process described in the first embodiment is executed, which is effective in removing the haze that hangs over distant subjects.

In step S76, it is determined whether the distance to the subject of interest is closer than a second predetermined distance. If the distance to the subject is closer than the second predetermined distance, the process proceeds to step S77. If the distance to the subject is not closer than the second predetermined distance, the process proceeds to step S78.

In step S77, the second tone correction process described in the first embodiment, which is effective in removing mist from nearby objects, is executed. In step S78, a blending process is performed on the image obtained by the first tone correction process described in the first embodiment and the image obtained by the second tone correction process. This step S78 is executed when the distance to the target object is equal to or less than a first predetermined distance and equal to or greater than a second predetermined distance. After executing step S75, S77, or S78, the mist removal operation is terminated.

As described above, according to the fourth embodiment, it is possible to perform effective de-haze processing depending on the distance of the detected subject of interest.

Fifth Embodiment
A network camera as a fifth embodiment of the image processing device of the present invention will be described below. The configuration of an image processing circuit in the fifth embodiment is made up of a network camera, a network, and a client device, as shown in FIG.

In this fifth embodiment, the defrosting method is switched depending on the recognition/analysis application executed on the client device. In this embodiment, an example will be described using a face recognition application that identifies human faces, and a people counting application that counts the number of subjects present on the screen.

As for the characteristics of each app, face recognition apps perform recognition based on minute facial features, so it is necessary to suppress the occurrence of artifacts such as the aforementioned halo noise as much as possible. On the other hand, people counting apps do not perform detection based on minute structures, but rather detect rougher structures and shapes compared to face recognition apps and count people. Therefore, when removing fog when running a people counting app, the detection accuracy of the subject is improved by prioritizing the removal of fog rather than suppressing halo noise. Note that the recognition and analysis apps are not limited to these, and may be, for example, an analysis app that creates a heat map of people's movements, an app that tracks specific people, an app that analyzes the attributes of the subject, etc.

Figure 11 is a flowchart showing the operation of de-misting in this embodiment.

When the flow of the mist removal process starts, in step S111, the image captured by the imaging unit 101 is input to the image processing unit 121. In step S112, information on the recognition/analysis app running on the client device is acquired.

In step S113, the overall control calculation unit 107 determines whether the type of the acquired recognition/analysis app is a face identification app. If it is a face identification app, the process proceeds to step S114. If it is not a face identification app, the process proceeds to step S115.

In step S114, the second tone correction process described in the first embodiment is performed, which is effective in removing mist while minimizing adverse effects.

In step S115, it is determined whether the type of the acquired recognition/analysis app is a people counting app. If it is a people counting app, the process proceeds to step S116. If it is neither a face recognition app nor a people counting app, the process proceeds to step S117.

In step S116, the first tone correction process described in the first embodiment is executed, which is effective in removing the haze that hangs over distant subjects.

In step S117, similar to step S68 described in the third embodiment, a blending process is performed on the image obtained in the first gradation correction process and the image obtained in the second gradation correction process based on the edge information. After step S114, step S116, or step S117 is performed, the mist removal operation is terminated.

As described above, according to the fifth embodiment, it is possible to perform effective fog and haze removal processing depending on the recognition and analysis application executed on the client device.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: Digital camera, 101: Imaging unit, 102: Lens unit, 104: Imaging signal processing circuit, 107: Overall control and calculation unit, 120, 121: Image processing circuit

Claims (12)

An acquisition means for acquiring image data;
a first gradation correction means for acquiring a dark channel value for each region of the image data, estimating transmittance, and correcting gradation;
a second gradation correction means for correcting the gradation of the image data using a tone curve;
A control means for controlling which gradation correction means is to be used for gradation correction;
Equipped with
The image processing device is characterized in that the control means causes the first gradation correction means to perform gradation correction so as to gradually obtain a correction effect, and when a white area is extracted at the contour of a subject in the image data, the control means stops the gradation correction by the first gradation correction means . The image processing device according to claim 1, characterized in that the control means causes one of the first gradation correction means and the second gradation correction means to perform a first stage of gradation correction on the image data, and causes the other of the first gradation correction means and the second gradation correction means to perform a second stage of gradation correction on the image data after the first stage of gradation correction, based on a correction result of the image data after the first stage of gradation correction. The image processing device according to claim 2, characterized in that the control means causes the other of the first gradation correction means and the second gradation correction means to perform the second gradation correction on the image data after the first gradation correction when the contrast of the image data after the first gradation correction is lower than a predetermined value. The image processing device according to claim 3, characterized in that the contrast is calculated based on a luminance histogram of the image data after the first stage of gradation correction is performed. The image processing device according to claim 3 or 4, characterized in that the control means causes the other of the first gradation correction means and the second gradation correction means to perform the second stage of gradation correction until the contrast reaches a target value. The image processing device according to any one of claims 2 to 5, characterized in that the control means causes the first gradation correction means to perform the first stage of gradation correction. 6. The image processing apparatus according to claim 2, wherein the control means causes the second tone correction means to perform the first-stage tone correction.
The image processing device according to claim 1, further comprising an analysis means for performing an analysis using the image data, and controlling the gradation correction operation of the first gradation correction means and the gradation correction operation of the second gradation correction means based on the result of the analysis by the analysis means. An imaging means for imaging a subject;
An image processing device according to any one of claims 1 to 8 ,
An imaging device comprising: An acquisition step of acquiring image data;
a first gradation correction step of acquiring a dark channel value for each region of the image data, estimating transmittance, and correcting gradation;
a second gradation correction step of correcting the gradation of the image data using a tone curve;
a control step of controlling which gradation correction step is to be used for gradation correction;
having
The image processing method is characterized in that, in the control step, gradation correction is performed in the first gradation correction step so as to gradually obtain a correction effect, and when a white area is extracted at the contour of the subject in the image data, the gradation correction in the first gradation correction step is stopped . A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 8 . 9. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.

Images