JP2010086139A - Image processing method, image processing apparatus, and image pickup apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To quickly correct a blur component and a distortion component of an image by image restoration processing. <P>SOLUTION: An image processing method includes: a step S1 of acquiring an image; and a step S7 of performing the image restoration processing on the image using an image restoration filter created such that its filter values have a two-dimensional distribution based on aberration information of an optical system 101. The image restoration filter is a filter in which a position of a cell having a maximum absolute value among the filter values is shifted with respect to a position of a center cell of the image restoration filter by a shift amount according to an amount of distortion aberration of the optical system, and which reduces the blur component and the distortion component of the image. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing technique for reducing a blur component and a distortion component included in an image.

  An image obtained by imaging a subject with an imaging device such as a digital camera has image degradation caused by spherical aberration, coma aberration, field curvature, astigmatism, etc. of an imaging optical system (hereinafter simply referred to as an optical system). A blur component is included as a component. Such a blur component is generated when a light beam emitted from one point of a subject to be collected again at one point on the imaging surface when an aberration is not caused and there is no influence of diffraction is formed by forming an image with a certain spread.

  The blur component here is optically represented by a point spread function (PSF) and is different from the blur due to the focus shift. In addition, color bleeding in a color image can also be said to be a difference in blurring for each wavelength of light when it is caused by axial chromatic aberration, chromatic spherical aberration, and chromatic coma in the optical system.

  As a method for correcting a blur component of an image, a method for correcting the blur component using information of an optical transfer function (OTF) of an optical system is known. This method is called image restoration or image restoration. Hereinafter, processing for correcting (reducing) a blur component of an image using information on the optical transfer function (OTF) of the optical system is called image restoration processing.

  The outline of the image restoration process is as follows. A degraded image (input image) including a blur component is denoted by g (x, y), and an original image that is not degraded is denoted by f (x, y). Further, a point spread function (PSF) that is a Fourier pair of the optical transfer function is assumed to be h (x, y). At this time, the following equation holds. Here, * indicates convolution and (x, y) indicates coordinates on the image.

g (x, y) = h (x, y) * f (x, y)
Further, when the above equation is converted into a display format on a two-dimensional frequency plane by Fourier transform, a product format for each frequency is obtained as in the following equation. H is a Fourier transform of the point spread function (PSF) and is an optical transfer function (OTF). (U, v) indicates the coordinates on the two-dimensional frequency plane, that is, the frequency.

G (u, v) = H (u, v) · F (u, v)
In order to obtain the original image from the deteriorated image, both sides may be divided by H as follows.

G (u, v) / H (u, v) = F (u, v)
A restored image corresponding to the original image f (x, y) is obtained by performing inverse Fourier transform on the F (u, v) and returning to the actual surface.

Here, ^assuming that the result of inverse Fourier transform of H ⁻¹ is R, the original image can be similarly obtained by performing convolution processing on the actual image as in the following equation.

g (x, y) * R (x, y) = f (x, y)
This R (x, y) is called an image restoration filter. Since an actual image has a noise component, using an image restoration filter created by taking the perfect reciprocal of the optical transfer function (OTF) as described above, the noise component is amplified together with the deteriorated image. A good image cannot be obtained. With respect to this point, for example, a method of suppressing the recovery rate on the high frequency side of the image according to the intensity ratio of the image signal and the noise signal, such as a Wiener filter, is known. Deterioration of the color blur component of the image is corrected, for example, if the blur amount for each color component of the image becomes uniform by correcting the blur component.

  Here, since the optical transfer function (OTF) varies depending on the optical system state such as the focal length (zoom state) and the aperture diameter of the optical system, the image restoration filter used for the image restoration process is also changed accordingly. There is a need to.

  The image may include a distortion component. Distortion is generally a geometric distortion in which an image expands or contracts greatly toward the periphery, and is caused by distortion of the optical system.

In Patent Document 1, as a method for correcting geometric distortion of an image due to distortion, information on an imaging state such as a zoom state and an imaging distance at the time of imaging is acquired, and distortion corresponding to an imaging state prepared in advance is acquired. A technique for correcting geometric distortion with reference to aberration data is disclosed. The distortion data is a function that depends on the image height.
JP 2006-270918 A

  In order to satisfactorily correct an image deteriorated due to various aberrations of the optical system and obtain a high-quality image, it is necessary to perform the process of reducing the blur component and the distortion component described above. However, in the software for performing these processes and the imaging apparatus in which the software is mounted, the processing speed must be increased so as not to cause the user to feel stress.

  Further, with the correction technique disclosed in Patent Document 1, the distortion component of the image can be reduced using the distortion aberration data corresponding to the imaging state, but the blur component of the image cannot be reduced. It is possible to reduce not only the distortion component but also the blur component by sequentially performing the distortion component correction method disclosed in Patent Document 1 and the image restoration processing using the conventional image restoration filter. However, the number of processing steps increases and the processing speed decreases.

  The present invention provides an image processing method, an image processing apparatus, and an imaging apparatus that can correct blur components and distortion components of an image at high speed.

  An image processing method according to an aspect of the present invention includes an image acquisition step and an image for the image using an image restoration filter created so that a filter value has a two-dimensional distribution based on aberration information of the optical system. And an image restoration step for performing a restoration process. In the image restoration filter, the position of the cell where the absolute value of the filter value is maximum has a deviation amount corresponding to the distortion amount of the optical system with respect to the position of the center cell of the image restoration filter. It is a filter that reduces distortion components as well as blur components.

  An image processing apparatus according to another aspect of the present invention includes a storage unit that stores an image restoration filter created so that a filter value has a two-dimensional distribution based on aberration information of an optical system, and an image restoration filter. And an image restoration means for performing an image restoration process on the image. In the image restoration filter, the position of the cell having the maximum absolute value of the filter value has a deviation amount corresponding to the distortion amount of the optical system with respect to the central cell of the image restoration filter, and the image blurring is caused. It is a filter that reduces distortion components as well as components.

  Note that an imaging apparatus that includes an imaging system that acquires an image by photoelectrically converting a subject image formed by an optical system and the image processing apparatus that processes the image also constitutes another aspect of the present invention.

  According to the present invention, since the image restoration filter is created so as to reduce the distortion component together with the blur component, the blur component and the distortion component can be simultaneously reduced by the image restoration processing. Therefore, the blur component and the distortion component can be reduced by high-speed processing.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration of an imaging apparatus such as a digital camera or a video camera (using an image processing method) provided with the image processing apparatus according to the first embodiment of the present invention.

  A light beam from a subject (not shown) forms an image on an image sensor 102 constituted by a CCD sensor, a CMOS sensor, or the like by an imaging optical system 101.

  The imaging optical system 101 includes a variable power lens (not shown), a diaphragm 101a, and a focus lens 101b. By moving the zoom lens in the optical axis direction, zooming that changes the focal length of the imaging optical system 101 is possible. The diaphragm 101a adjusts the amount of light reaching the image sensor 102 by increasing or decreasing the aperture diameter of the diaphragm. The focus lens 101b is controlled in position in the optical axis direction by an unillustrated autofocus (AF) mechanism or manual focus mechanism in order to perform focus adjustment according to the subject distance.

  The subject image formed on the image sensor 102 is converted into an electric signal by the image sensor 102. An analog output signal from the image sensor 102 is converted into a digital image signal by the A / D converter 103 and input to the image processing unit 104.

  The image processing unit 104 includes an image generation unit 104a that generates a color input image by performing various processes on the input digital imaging signal. The image processing unit 104 includes an image recovery processing unit (image recovery unit) 104b that performs an image recovery process on the input image. The imaging element 102 to the image generation unit 104a correspond to the imaging system.

  The image restoration processing unit 104b obtains information on the state of the imaging optical system 101 (hereinafter referred to as an imaging state) from the state detection unit 107. The imaging state includes, for example, the focal length (zoom position), aperture diameter (aperture value, F number), and focus lens position (subject distance) of the imaging optical system 101. Note that the state detection unit 107 may obtain information on the imaging state from the system controller 110 or may be obtained from the imaging optical system control unit 106 that controls the imaging optical system 101.

  The image restoration processing unit 104b selects an image restoration filter corresponding to the imaging state from the storage unit (storage unit) 108, and performs image restoration processing on the input image. The state detection unit 107, the image restoration processing unit 104b, and the storage unit 108 constitute an image processing device in the imaging device.

  FIG. 2 shows a flowchart of processing (image processing method) relating to image recovery performed by the image recovery processing unit 104b (hereinafter referred to as the image processing unit 104). The image processing unit 104 is configured by an image processing computer and executes the processing according to a computer program.

  In step S <b> 1, the image processing unit 104 acquires an image generated based on an output signal from the image sensor 102 (hereinafter referred to as an input image) as an object for image restoration processing. Next, in step S <b> 2, the image processing unit 104 acquires imaging state information from the state detection unit 107. Here, there are three imaging states, that is, a zoom position, an aperture diameter, and a subject distance.

  Next, in step S <b> 3, the image processing unit 104 selects an image restoration filter corresponding to the imaging state acquired in step S <b> 2 from the image restoration filters stored in the storage unit 108.

  Here, the storage unit 108 stores (stores) only image restoration filters for discretely selected imaging states in order to reduce the number of image restoration filters (data number). For this reason, when the image restoration filter corresponding to the imaging state acquired in step S2 or corresponding to the imaging state very close to the imaging state is not stored in the storage unit 108, an image restoration filter as close as possible to the imaging state is selected. select. Then, the image restoration filter to be actually used is created by correcting the image restoration filter in the following steps S4 to S6 so as to be optimized to the imaging state acquired in step S2.

  FIG. 3 schematically illustrates the image restoration filter stored in the storage unit 108 for the discretely selected imaging state. As described above, the image restoration filter stored in the storage unit 108 is an imaging state space with the three imaging states of the zoom position (state A), the aperture diameter (state B), and the subject distance (state C) as axes. It is arranged discretely inside. The coordinates of each point (black circle) in the imaging state space indicate the image restoration filter stored in the storage unit 108.

  In FIG. 3, the image restoration filter is arranged at the grid point on the line orthogonal to each imaging state, but the image restoration filter may be arranged away from the grid point. Further, the types of imaging states are not limited to the zoom position, the aperture diameter, and the subject distance, and the number thereof may not be three, and a four-dimensional or more imaging state space is formed by four or more imaging states. The image restoration filter may be discretely arranged therein.

  A method of creating an actually used image restoration filter from the discretely arranged image restoration filters will be described later.

  An example of the image restoration filter is shown in FIG. In the image restoration filter, the number of cells (tap) is determined according to the amount of aberration of the imaging optical system 101. The image restoration filter shown in FIG. 4 is a two-dimensional filter having 21 × 21 cells. Each cell corresponds to one pixel of the image.

  By making the image restoration filter into a two-dimensional filter divided into 100 or more cells, for the aberration that spreads widely from the imaging position such as spherical aberration, coma aberration, axial chromatic aberration, off-axis color flare, etc. by the imaging optical system 101 However, a good image restoration result can be obtained. In addition, by convolving the input image with such an image restoration filter in real space, the image can be restored without performing Fourier transform.

  Each cell is set to have a value as shown in FIG. 5 according to aberration information such as spherical aberration, coma aberration, axial chromatic aberration, and off-axis color flare of the imaging optical system 101. FIG. 5 shows cell values (filter values) in one cross section of the image restoration filter. The image restoration filter is created so that the filter value has a two-dimensional distribution. In the example of FIG. 5, the filter recovers aberration having asymmetry such as coma aberration at a position off the center of the image.

  The image restoration filter is created by calculating or measuring an optical transfer function (OTF) of the imaging optical system 101 or the like, and inverse Fourier transforming the inverse function. In general, since it is necessary to consider the influence of noise, it is possible to select and use a method of creating a Wiener filter or a related recovery filter.

  The optical transfer function here includes not only the imaging optical system 101 but also factors that degrade the optical transfer function until the input image is generated by the image processing unit 104 from the output signal of the imaging element 102. Is desirable. That is, the image restoration filter performs an inverse Fourier transform on a function generated based on an inverse function of an optical transfer function from when light enters the imaging optical system 101 until an input image is acquired by imaging by the imaging element 102. It is good to make it by.

  Factors that degrade the optical transfer function other than the imaging optical system 101 include the following. For example, a low-pass filter (not shown) disposed in front of the image sensor 102 suppresses high frequency components with respect to the frequency characteristics of the optical transfer function. Similarly, the infrared cut filter disposed in front of the image sensor 102 affects each PSF of the RGB channel, particularly the PSF of the R channel, which is an integral value of the point spread function (PSF) of the spectral wavelength. Furthermore, the shape and aperture ratio of the pixel aperture of the image sensor 102 also affect the frequency characteristics. In addition, spectral characteristics of a light source that illuminates a subject and spectral characteristics of various wavelength filters can be cited as factors that degrade the optical transfer function. Therefore, it is desirable to create an image restoration filter based on an optical transfer function in a broad sense considering these.

  The imaging optical system 101 may be provided as a part of the imaging device, or may be an exchangeable type that can be attached to and detached from the imaging device.

  If the input image is an RGB color image, three image restoration filters corresponding to the R, G, and B color components may be created. This is because the imaging optical system 101 has chromatic aberration, and the blur is different for each color component. Therefore, in order to obtain an optimum image restoration filter for each color component, the characteristics should be made different based on the chromatic aberration. That is, it is preferable to create three image restoration filters in which the two-dimensional distribution of cell values shown in FIG. 5 is different for each color component.

  It should be noted that the number of vertical and horizontal cells (cell arrangement) of the image restoration filter need not be a square arrangement as shown in FIG. 4, but can be arbitrarily changed if the cell arrangement is taken into consideration during the convolution process. Can do.

  In the example of FIG. 5, the position of the cell having the largest absolute value of the cell value (filter value) in the image restoration filter (hereinafter referred to as the maximum value cell) is shifted from the position of the center cell of the image restoration filter. . In FIG. 4, the black-filled cell is the maximum value cell, and the cell marked with “x” is the center cell.

  When convolution processing is performed on an input image using such an image restoration filter, the blur component is reduced (corrected) by the distribution of cell values in the filter. In addition, since the maximum value cell is shifted from the center cell, the image after correcting the blur component is shifted according to the shift amount of the maximum value cell with respect to the center cell. By setting the shift amount of the maximum value cell with respect to the center cell based on the distortion aberration amount of the imaging optical system 101, the distortion component included in the input image can also be reduced (corrected). In other words, the image restoration filter used in the present embodiment is a filter that can reduce the distortion component as well as the blur component included in the input image.

  Therefore, by performing image restoration processing described later using this image restoration filter, it is possible to simultaneously correct the blur component and the distortion component included in the input image.

  A specific selection and creation (correction) method of the image restoration filter will be described. In FIG. 3, it is assumed that the imaging state indicated by a large white circle is the actual imaging state acquired in step S2. When there is an image restoration filter (hereinafter referred to as a storage filter) stored (stored) in the storage unit 108 in or near the actual imaging state, the storage filter is selected and used for image restoration processing. When the storage filter does not exist in or near the actual imaging state, the image restoration filter is selected or created (corrected) by the following method.

  First, in step S3 described above, the image processing unit 104 calculates the distance in the imaging state space between the actual imaging state and the imaging states corresponding to the plurality of storage filters. Then, the storage filter corresponding to the imaging state at the shortest distance among the calculated distances is selected.

  By selecting such a storage filter, the difference amount between the actual imaging state and the imaging state corresponding to the storage filter (hereinafter referred to as state difference amount) is minimized. Therefore, the correction amount for the storage filter can be reduced, and an image restoration filter closer to the image restoration filter corresponding to the actual imaging state can be created.

  In FIG. 3, it is assumed that the storage filter corresponding to the imaging state indicated by a small white circle is selected. In addition, although the imaging state space is conceptually illustrated in FIG. 3, coordinate value information is necessary for the actual data of each storage filter. For this reason, the storage filter itself may be provided with coordinate value information, or the data of each storage filter may be arranged in a multidimensional array space in which addresses (coordinates) are determined in advance.

  Next, in step S4, the image processing unit 104 calculates state difference amounts ΔA, ΔB, and ΔC between the imaging state corresponding to the storage filter selected in step S3 and the actual imaging state. In step S5, a state correction coefficient is calculated based on these state difference amounts ΔA, ΔB, and ΔC. In step S6, the image processing unit 104 corrects the storage filter selected in step S3 using the state correction coefficient. Thereby, an image restoration filter corresponding to an actual imaging state can be created.

  As described above, the image restoration filter in this embodiment has a blur correction function for reducing a blur component and an image shift function for reducing a distortion component. For this reason, the position of the maximum value cell is previously shifted with respect to the center cell in each storage filter.

  However, a storage filter having only cell value distribution data for correcting the blur component is prepared, and the maximum cell shift amount (deviation amount) data for correcting the distortion component according to the imaging state. May be prepared separately. In this case, in the storage filter prepared in the imaging state space, the maximum value cell does not have a shift amount (the maximum value cell matches the center cell). Therefore, as shown in FIG. Can be small.

  In this case, the maximum value cell shift amount data may be stored in the storage unit 108 in advance. If the data of the shift amount of the maximum value cell is stored in the storage unit 108 as a function corresponding to the image height, the capacity required for the storage unit 108 can be reduced.

  Then, before the actual image restoration process is performed, an image restoration filter as shown in FIG. 5 may be created using the storage filter of FIG. 6 and the shift amount data of the maximum value cell.

  Next, in step S7, the image processing unit 104 performs image restoration processing on the input image acquired in step S1 using the selected or created image restoration filter. That is, by performing convolution processing on the input image using the image restoration filter, it is possible to reduce (or remove) the blur component and the distortion component of the image caused by the aberration of the imaging optical system 101.

  At this time, as described above, by performing the image restoration process using the image restoration filter for each of the R, G, and B color components in which the shift amount of the maximum value cell is changed according to the chromatic aberration of magnification, The lateral chromatic aberration component (color shift component) can also be reduced (corrected). Also, image restoration processing using an image restoration filter for each of R, G, and B color components in which the cell value distribution in the filter relative to the maximum value cell of the image restoration filter is varied according to chromatic aberration (color flare). By performing the above, it is possible to reduce (correct) the color blur component of the input image.

  Note that the selection and correction processing of the image restoration filter in step S2 to step S6 may be performed by a device (such as a personal computer) different from the imaging device. In this case, a process of storing (installing) the image restoration filter obtained by the other apparatus in the storage unit 108 of the imaging apparatus may be performed.

  The effect of the image restoration process using the image restoration filter of this embodiment will be described with reference to FIG. Here, the effect of one color component among R, G, and B is shown. However, when there is a color blur component in the input image, the image restoration process using the image restoration filter for each color component as described above. Can be done.

  FIG. 7A shows a subject image that should have a rectangular profile if there is no influence of the aberration of the imaging optical system 101, and is deteriorated as a blurred image by the aberration of the imaging optical system 101, and further the distortion of the imaging optical system 101. An input image deviated from the original position P due to aberration is shown.

  (B) shows a restored image when an image restoration process is performed on the input image shown in (a) using a conventional image restoration filter having no image shift function. In this recovered image, the subject image is corrected so as to have an original rectangular profile, but remains unchanged from the original position P because the distortion component is not corrected.

  In (c), the recovered image shown in (b) is further subjected to image shift processing as distortion correction to correct the distortion component so that the subject image is positioned at the original position P. An image is shown.

  On the other hand, when an image restoration process is performed on the input image shown in (a) using the image restoration filter of this embodiment having both the blur correction function and the image shift function, the subject image is displayed as shown in (b1). It is corrected to have an original rectangular profile. At the same time, the position of the subject image is shifted to the original position P.

  As described above, according to the present embodiment, one blur processing correction process (image restoration process) and one distortion component correction process (image shift process), which conventionally had to be performed as separate processes, are performed once. ) Image restoration processing.

  In FIG. 1, an output image that has undergone image restoration processing by the image processing unit 104 is stored in a predetermined format in an image recording medium 109 such as a semiconductor memory or an optical disk. This output image is a high-quality image that has been sharpened by image restoration processing and has reduced distortion components.

  The output image is displayed on the display unit 105 or output to the outside of the imaging apparatus (printer or database).

  The system controller 110 controls a series of operations such as photoelectric conversion in the image sensor 102, image processing in the image processing unit 104, recording on the image recording medium 109, and image display on the display unit 105. The imaging optical system control unit 106 controls zoom driving and focus driving of the imaging optical system 101 according to instructions from the system controller 110.

  Note that the imaging optical system 101 may be designed so that the lateral magnification of the center image height of the imaging optical system 101 is 1.25 times or more larger than the lateral magnification of the maximum image height. Thereby, the diameter of the front lens located closest to the subject in the imaging optical system 101 can be reduced, or the field curvature can be corrected.

  Further, since the shift amount of the maximum value cell with respect to the center cell is determined according to the distortion aberration amount of the imaging optical system 101, the size (number of cells) of the image restoration filter may be changed according to the distortion aberration amount. .

  In FIG. 8, an input image distorted into a barrel shape by a distortion component is schematically shown by a solid line. Table 1 shows the distortion amount and the image shift amount (the number of moving pixels) at the diagonal end in the input image. When the distortion component is corrected, it is necessary to shift and enlarge the subject image in the peripheral portion of the image, so that the insufficient pixels are subjected to interpolation processing.

  Further, since the above-described optical transfer function (OTF) changes according to the angle of view (image height) even in the same imaging state, the above-described image restoration processing is a region divided according to the image height in the image. It is desirable to change the image restoration filter every time. Convolution processing may be performed while scanning the image restoration filter on the image, and the image restoration filter may be sequentially changed for each divided region.

  Further, by correcting the image restoration filter in accordance with the pixel signal value in the input image to be subjected to the image restoration processing, it is possible to suppress an increase in noise that becomes noticeable in a dark region of the input image. In the dark area of the input image, the noise component is relatively large with respect to the image signal as compared with the bright area, and thus there is a possibility that the noise may be enhanced by the image restoration process. For this reason, an increase in noise can be suppressed for dark regions by correcting the value of each cell of the image restoration filter to reduce the image restoration effect. The image restoration process may not be performed for pixels or regions whose signal values are below a certain threshold.

  If there is a correction error in the recovered image after the image recovery process, another image process for correcting the correction error may be performed. Thereby, it is possible to further reduce the residual aberration component in the recovered image caused by the error between the actual imaging state and the imaging state corresponding to the image restoration filter used for the image restoration process.

  In addition, when a saturated luminance region exists in the input image, an aberration component appears remarkably in the peripheral portion. However, when the image restoration process is performed, the original luminance value is not known for a pixel whose luminance value is saturated, and there is a possibility that good correction cannot be performed. In this case, further improvement in image quality can be achieved by detecting the remaining aberration component from the recovered image after the image recovery process and adaptively performing the correction process. This adaptive correction processing corrects the aberration finally remaining in the recovered image in which the blur component and the distortion component are reduced by the image recovery processing in the present embodiment, and thus is performed at a later stage than the image recovery processing. It is desirable.

  In the present embodiment, the imaging apparatus using the image processing method of the present invention (mounted with the image processing apparatus) has been described. However, the image processing method of the present invention is also implemented by an image processing program installed in a personal computer. be able to. In this case, the personal computer corresponds to the image processing apparatus of the present invention. The personal computer captures (acquires) an image (input image) before image recovery processing generated by the imaging device, performs image recovery processing by an image processing program, and outputs the resulting image.

  Next, a second embodiment of the present invention will be described. The image processing method of the present embodiment is also implemented by an imaging apparatus similar to the imaging apparatus shown in FIG. For this reason, in the present embodiment, the same reference numerals as those in the first embodiment are assigned to components having the same or similar functions as those in the first embodiment. Note that the processing of this embodiment may be performed by a personal computer (image processing apparatus).

  In the present embodiment, as described in the first embodiment, a case where a color shift component is corrected in addition to a blur component and a distortion component of an image by an image restoration process will be described. The color misregistration component of the image is generated due to the chromatic aberration of magnification of the imaging optical system 101, that is, the imaging magnification is different for each color component. The color blur component described above is a difference in the blur method for each color component and can be included in the blur component in a broad sense. Therefore, the color blur component is considered separately from the color shift component to be corrected in the present embodiment.

  An example of the image restoration filter for each color component is shown in FIG. In FIG. 9, image recovery filters indicated by (R), (G), and (B) are image recovery filters for R, G, and B, respectively. As described in the first embodiment, these image restoration filters are filters that simultaneously correct the blur component and the distortion component, but do not have a function of correcting a color shift component corresponding to the chromatic aberration of magnification.

  The image restoration filters indicated by (R1), (G1), and (B1) are also image restoration filters for R, G, and B, respectively. However, these image restoration filters have a function of correcting the color misregistration component in addition to the simultaneous correction function of the blur component and the distortion component.

  In each image restoration filter, the black cell is the maximum value cell, and the cell with x is the center cell.

  The image restoration filters indicated by (R), (G), and (B) have different filter characteristics because the optical transfer function (OTF) differs for each color component, but the position of the maximum value cell (shift amount with respect to the center cell). Are the same as each other.

  On the other hand, the image restoration filters indicated by (R1), (G1), and (B1) have different filter characteristics and also have different maximum value cell positions (shift amounts with respect to the center cell). The relative displacement amount of the position of these maximum value cells is an amount corresponding to the amount of lateral chromatic aberration. Furthermore, the shift amount of the maximum value cell position with respect to the position of the central cell in the image restoration filters (R1), (G1), and (B1) for the respective color components depends on the image height in the imaging optical system 101. It is set to a value corresponding to the distortion aberration amount and the magnification chromatic aberration amount.

  Although FIG. 9 shows a 21 × 21 cell image restoration filter, when the amount of aberration is large, the number of cells is increased according to the amount of aberration, such as 50 × 50 cells or 100 × 100 cells. May be.

  In the image restoration filters indicated by (R1), (G1), and (B1), the blur component (color blur component) for each color can be corrected by changing the distribution of cell values in the filter for each color. Further, the distortion component and the color misregistration component can be corrected by shifting the position of this distribution (shifting the position of the maximum value cell from the position of the center cell and making the shift amount different).

  Next, the effect of the image restoration filter indicated by (R1), (G1), and (B1) will be described with reference to FIG. 10A shows a subject image that should have a rectangular profile if there is no influence of the aberration of the imaging optical system 101, and is deteriorated as a blurred image due to the aberration of the imaging optical system 101, and further the distortion of the imaging optical system 101. An input image deviated from the original position P due to aberration is shown.

  A solid line, a broken line, and an alternate long and short dash line represent a G component, an R component, and a B component, respectively. In (a), a color shift occurs between RGB due to lateral chromatic aberration.

  (B) shows a restored image when an image restoration process is performed on the input image shown in (a) using a conventional image restoration filter having no image shift function. In this recovered image, the R, G, and B subject images are corrected to have the original rectangular profile, but the distortion component and the color shift component are not corrected, and thus remain shifted from the original position P. .

  In (c), by performing image shift processing as distortion correction on the recovered image shown in (b) with different shift amounts for each color component, the distortion component and the color shift component are corrected, and R, G , B shows an image corrected so that the subject image is positioned at the original position P.

  On the other hand, when image restoration processing is performed on the input image shown in (a) using the image restoration filter of this embodiment having both the blur correction function and the image shift function for each color component, as shown in (b1). In addition, the R, G, and B subject images are corrected to have an original rectangular profile. At the same time, the positions of the R, G, and B subject images are also shifted to the original position P.

  As described above, according to the present embodiment, the blur component correction processing (image restoration processing), the distortion component correction processing (image shift processing), and the color misregistration component, which conventionally had to be performed as separate processes. This correction processing can be performed in one (once) image restoration processing.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment when the present invention is implemented.

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. 3 is a flowchart illustrating processing performed by the imaging apparatus according to the first embodiment. FIG. 3 is a schematic diagram of an imaging state space in which an image restoration filter stored in a storage unit is disposed in the first embodiment. FIG. 3 is an explanatory diagram of an image restoration filter used in the imaging apparatus according to the first embodiment. The figure which shows distribution of the cell value of the said image restoration filter. FIG. 6 is a diagram illustrating another example of the image restoration filter stored in the storage unit according to the first embodiment. Explanatory drawing of the effect of the image restoration process in Example 1. FIG. Explanatory drawing of a distortion aberration. Explanatory drawing of the image restoration filter used in Example 2 of this invention. Explanatory drawing of the effect of the image restoration process in Example 2. FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Imaging optical system 101a Aperture 101b Focus lens 102 Imaging element 104 Image processing part 105 Display part 106 Imaging optical system control part 107 State detection part 108 Storage part 110 System controller

Claims (7)

An image processing method for processing an image obtained by imaging using an optical system,
Obtaining the image;
An image restoration step for performing an image restoration process on the image using an image restoration filter created so that a filter value has a two-dimensional distribution based on aberration information of the optical system,
In the image restoration filter, the position of the cell having the maximum absolute value of the filter value has a shift amount corresponding to the distortion amount of the optical system with respect to the position of the center cell of the image restoration filter, An image processing method comprising: a filter that reduces a distortion component together with a blur component of the image.   The image restoration filter is created by performing an inverse Fourier transform on a function generated based on an inverse function of an optical transfer function from when light enters the optical system until the input image is acquired by the imaging. The image processing method according to claim 1, wherein:   3. The image recovery step according to claim 1, wherein in the image recovery step, information indicating an imaging state at the time of the imaging is acquired, and the image recovery filter corresponding to the imaging state information is selected or created and used. Image processing method.   4. The image processing method according to claim 1, wherein in the image restoration step, a convolution process using the image restoration filter is performed on the image. 5. The image restoration filter has a plurality of image restoration filters having a two-dimensional distribution of different filter values for each color,
The shift amount of the cell position where the absolute value of the filter value in each image restoration filter is the maximum with respect to the position of the center cell is a shift amount corresponding to the distortion aberration amount and the magnification chromatic aberration amount depending on the image height. The image processing method according to claim 1, wherein the image processing method is provided. An image processing apparatus that processes an image obtained by imaging using an optical system,
Storage means for storing an image restoration filter created so that the filter value has a two-dimensional distribution based on the aberration information of the optical system;
Image recovery means for performing image recovery processing on the image using the image recovery filter,
In the image restoration filter, the position of the cell having the maximum absolute value of the filter value has a deviation amount corresponding to the distortion amount of the optical system with respect to the central cell of the image restoration filter, and the image An image processing apparatus that is a filter that reduces a distortion component as well as a blur component. An imaging system for photoelectrically converting a subject image formed by an optical system to obtain an image;
An imaging apparatus comprising: the image processing apparatus according to claim 6 that processes the image.