JP4596987B2 - Imaging device - Google Patents

Description

  The present invention relates to an image pickup apparatus, and more particularly to an image pickup apparatus suitable for use in simultaneously acquiring images having different angles of view.

  Conventionally, in an image input device such as a video camera or a digital camera, the focal length of the lens is changed according to the distance to the subject to be photographed and the size of the subject with respect to the angle of view, and can be freely enlarged or reduced. The zoom function is widely used. As a method of these zoom functions, a new pixel is interpolated between pixels using an optical zoom that is usually realized by mechanically moving an internal lens and a part of an image output from the imager. Thus, it is roughly divided into electronic zoom that electronically enlarges the subject. Here, the electronic zoom has an advantage that the device can be configured in a small size and at a low cost without a driving part as compared with the optical zoom, but there is a problem that the image quality is inferior.

To solve such a problem, as shown in FIG. 29, a fixed focal length image input optical system having a function of compressing a peripheral portion of an input image and a light receiving element mainly receiving a uniform pixel density. An electronic zoom image input method characterized by realizing a zoom image with an equivalent resolution in the operation region by providing a function for correcting and converting a received light image of a light receiving element including distortion due to compression. (For example, refer to Patent Document 1). Therefore, if such a method is employed, it is expected that an image with little image degradation in the central portion can be obtained for both a wide-angle image and a telephoto image, although the image quality degradation in the peripheral portion cannot be avoided.
JP-A-10-233950

  As described above, since the original image having the above optical distortion has a characteristic that the peripheral portion is compressed more than the central portion, from the same original image, a wide-angle image, a telephoto image in which the center is enlarged, etc. A plurality of different angles of view can be obtained simultaneously. However, when functions that take advantage of these characteristics are provided to users by installing them in consumer products such as digital cameras, it is a challenge for users to be able to use these functions easily and easily. .

  The present invention has been made in view of the above problems, and an object thereof is to provide an image pickup apparatus that is user-friendly and easy to use for the above functions.

In order to solve the above problems, the present invention proposes the following means.
According to the first aspect of the present invention, there is provided an optical system having a distortion characteristic that enlarges the central part and compresses the peripheral part, and an image that converts a subject image formed by the optical system into image data and outputs an original image. An angle of view setting means for setting a first angle of view and a second angle of view narrower than the first angle of view for the same original image data with the sensor, and the set A distortion aberration correction circuit that corrects distortion aberration of original image data of an angle of view and outputs corrected image data, and an image that displays an image based on the corrected image data of the first angle of view in which the distortion aberration is corrected A display device, an image recording device that records the corrected image data of the first and second angles of view in which the distortion is corrected, the first angle of view, distortion characteristics, and the size of the image to be displayed. Display fill that performs different filtering for each area Means, first recording filter means for performing different filtering for each region in accordance with the first angle of view, distortion characteristics, and the size of the image to be recorded, and the second angle of view, distortion characteristics And a second recording filter unit that performs different filtering for each region in accordance with the size of the image to be recorded .

  According to the present invention, an object image formed by an optical system having a distortion characteristic that enlarges the central portion and compresses the peripheral portion is converted into image data, and the same original image output from the image sensor as original image data. A first angle of view and a second angle of view different from the first angle of view are set for the image data by the angle of view setting means, and the original image data of the first and second angles of view are The distortion aberration is corrected by the distortion aberration correction circuit, and is stored in the storage device as corrected image data of the first and second angles of view. Therefore, the user can obtain image data having a first angle of view related to the same original image and a second angle of view different from the first angle of view.

  According to a second aspect of the present invention, in the image pickup apparatus according to the first aspect, the angle-of-view setting unit can individually set the size of the first angle of view and the size of the second angle of view. An image pickup apparatus characterized by the above is proposed.

  According to this invention, the size of the first field angle and the size of the second field angle are individually set by the field angle setting means. Therefore, the user can individually set the sizes of the first angle of view and the second angle of view.

  According to a third aspect of the present invention, in the image pickup device according to the second aspect, the angle-of-view setting unit sets a first angle of view setting unit that sets the size of the first angle of view, and the second And a second field angle setting unit for setting the size of the field angle.

  According to the present invention, the first field angle setting unit sets the first field angle size, and the second field angle setting unit sets the second field angle size. Therefore, the user can individually set the sizes of the first angle of view and the second angle of view.

  According to a fourth aspect of the present invention, in the image pickup device according to the second aspect, the angle-of-view setting unit switches the magnitude of the angle of view between the first angle of view and the second angle of view. There has been proposed an image capturing apparatus including an angle switching unit and an angle-of-view switching display control unit that controls display related to a switching result by the imaging angle-of-view switching unit.

  According to this invention, the angle of view for setting the size of the angle of view is switched between the first angle of view and the second angle of view by the imaging angle-of-view switching unit, and the display relating to the switching result is It is controlled by the corner switching display control unit. Therefore, the operations related to the setting of the size of the first angle of view and the setting of the size of the second angle of view can be made common, and the mounting area related to this operation can be suppressed. Furthermore, the user can visually confirm the switching result.

  According to a fifth aspect of the present invention, in the image pickup device according to the fourth aspect, the angle-of-view switching display control unit superimposes the display related to the switching result on the display of the original image data that is the target of the angle-of-view setting. An image pickup device characterized in that the image is displayed is proposed.

  According to the present invention, the display related to the switching result is displayed by the view angle switching display control unit so as to be superimposed on the display of the original image data to be set for the view angle. Therefore, the user can visually check the viewable angle of view on the original image.

  According to a sixth aspect of the present invention, in the image pickup device according to the fourth aspect, the angle-of-view switching display control unit displays the display related to the switching result at a position different from the original image data to be set as the angle of view. An image pickup apparatus characterized by being displayed on the screen has been proposed.

  According to the present invention, the display related to the switching result is displayed at a position different from the original image data that is the target of the angle of view setting by the angle of view switching display control unit. Therefore, the user can confirm the operable angle of view by displaying the position different from the original image.

According to a seventh aspect of the present invention, there is provided a first imaging mode in which only the first angle of view is imaged, and a plurality of angles of view of the first and second angles of view . second imaging mode for simultaneously imaging, and includes a plurality imaging modes including a third imaging mode, for capturing an image of the first field angle in the second field angle to display to the image display device , an imaging mode switching means consists of a button or a lever for switching the imaging mode, an imaging mode setting means for setting the imaging mode according to the state the imaging mode switching means, an output of the image pickup mode setting means further comprising a mode switching display control means for controlling the display in accordance with, the control of the output and the imaging mode switching display control means of the first field angle setting means, the display filter means and said first Recording Change the setting data unit, according to the control of the output and the imaging mode switching display control unit of the second field angle setting means, and changing the setting of the second recording filter means An image pickup device is proposed.

  According to the present invention, the first imaging mode in which only an image with one angle of view is captured by the imaging mode switching means, the second imaging mode in which a plurality of images with the first and second angles of view are simultaneously imaged, In addition, the imaging mode is switched between a plurality of imaging modes including a third imaging mode that is monitored at the first angle of view and is imaged at the second angle of view. It is controlled by the display control means. Therefore, the imaging mode can be switched and the switching result can be confirmed visually.

  According to an eighth aspect of the present invention, in the image pickup device according to the seventh aspect, the mode switching display control unit superimposes the display according to the switching result on the display of the original image data that is a target of the angle of view setting. An image pickup apparatus characterized by displaying the image is proposed.

  According to the present invention, the display related to the switching result is displayed by the mode switching display control unit so as to be superimposed on the display of the original image data to be set as the field angle. Therefore, the user can visually confirm the set imaging mode on the original image.

  According to a ninth aspect of the present invention, in the image pickup device according to the seventh aspect, the mode switching display control unit displays the display according to the switching result at a position different from the original image data that is the target of the angle of view setting. An image pickup apparatus characterized by displaying is proposed.

  According to this invention, the display related to the switching result is displayed by the mode switching display control unit at a position different from the original image data that is the target of the angle of view setting. Therefore, the user can confirm the set mode by displaying the position different from the original image.

  According to a tenth aspect of the present invention, in the image capturing device according to the seventh aspect, in the first imaging mode, the mode switching display control unit sets the display of the target original image as a color display, and In the second imaging mode, the display of the target original image is displayed in color and the frame of the second field angle is displayed. In the third imaging mode, the second field angle is displayed. There has been proposed an image pickup apparatus characterized by displaying a frame, displaying in a color inside the frame, and displaying different from the color display outside the frame.

  According to this invention, the mode switching display control unit displays the target original image in color in the first imaging mode, and displays the target original image in the second imaging mode. Is displayed in color and a frame of the second angle of view is displayed. In the third imaging mode, a frame of the second angle of view is displayed, the inside of the frame is displayed in color, and the outside of the frame is The display is different from the color display. Therefore, it is possible to make it easier for the user to visually confirm the setting of the imaging mode.

  According to an eleventh aspect of the present invention, in the image pickup device according to the tenth aspect, the mode switching display control unit includes a monochrome display including a monochrome display, a display by changing luminance, a tone enhancement, or a faint color outside the frame. An image pickup apparatus characterized in that the display is made by doing is proposed.

  According to the present invention, the mode switching display control unit sets the outside of the frame as a single color display including black and white display, a display by changing the luminance, a display by emphasizing or fainting the color tone. Therefore, for the user, it is possible to make it easier to visually confirm the setting of the imaging mode, what the imaging range and the non-imaging range are, and the like.

According to a twelfth aspect of the present invention, in the image pickup device according to the first aspect, the corrected image data of the first angle of view and the second image recorded in the image recording device and related to the same original image data. There has been proposed an image pickup apparatus further comprising a display control circuit for sequentially switching and displaying the corrected image data of the corners at regular intervals.

According to the present invention, the corrected image data of the first angle of view and the corrected image data of the second angle of view related to the same original image data recorded in the image recording device are displayed at regular intervals by the display control circuit. , Displayed sequentially. Therefore, the user can automatically check all the corrected image data recorded in the same original image with different angles of view.

According to a thirteenth aspect of the present invention, in the image pickup device according to the fourth aspect, the corrected image data of the first angle of view relating to the same original image data recorded in the image recording device, or the second The same original image data that is different from the displayed corrected image data when one of the corrected image data of the angle of view is displayed for a certain period of time and switching by the imaging angle of view switching unit is detected within the certain period of time. An image pickup apparatus is further provided that further includes a display control circuit that displays the other corrected image data according to the above.

According to the present invention, either the corrected image data of the first angle of view or the corrected image data of the second angle of view related to the same original image data recorded in the image recording device is displayed for a certain period of time. When the switching by the imaging angle-of-view switching unit is detected within a certain time, the other corrected image data related to the same original image data, which is different from the displayed corrected image data, is displayed by the display control circuit. The Accordingly, the user can sequentially display and check a plurality of corrected image data recorded in the same original image with different angles of view by operating the imaging angle of view switching means.

According to a fourteenth aspect of the present invention, in the image pickup device according to the first aspect, the first image angle correction includes an index indicating that the image recording apparatus is created from the same original image. An image pickup apparatus is proposed that records image data and correction data for the second angle of view.

According to the present invention, the corrected image data of the first angle of view and the corrected data of the second angle of view, which are both indexed indicating that they are created from the same original image, are recorded in the image recording apparatus. . Therefore, it is possible to easily confirm the relationship between the corrected image data of the first field angle and the correction data of the second field angle.
According to a fifteenth aspect of the present invention, in the image pickup device according to the seventh aspect, the display filter means, the first recording filter means, and the second recording filter means share the same circuit. It has a configuration and operates with display settings when displaying a through image. After the shutter is released, it is set and executed sequentially for first recording and second recording according to the imaging mode. An image pickup device is proposed.

  According to the present invention, there is an effect that it is possible for a user to provide an image pickup apparatus that can be used easily and easily.

(First embodiment)
FIG. 1 is a block diagram showing a digital camera which is an image pickup apparatus according to the first embodiment of the present invention.
As shown in FIG. 1, the digital camera according to the present embodiment includes an optical system 101 having distortion, an image sensor 102, a distortion correction image processing device 103 serving as a distortion correction circuit, an image display device 104, Image storage device 105 as storage device, field angle setting means, first field angle setting unit, second field angle setting unit, imaging field angle switching unit, field angle switching display control unit, shooting mode switching unit, mode switching The control unit 106 is a display control means, a display control circuit, and an operation unit 107.

  The optical system 101 having distortion aberration is an optical system having distortion aberration characteristics that enlarges the central portion and compresses the peripheral portion. The image sensor 102 is a photoelectric conversion element for converting an optical image picked up through the optical system 101 having distortion aberration into an electric signal, and is, for example, a CCD or a CMOS sensor. The distortion correction image processing apparatus 103 converts the image signal output from the image sensor 102 into a format necessary for display and compression. Further, optical distortion is corrected, image processing for improving image quality is performed, and data compression processing for recording is performed. Details of the configuration will be described later.

  The image display device 104 is composed of a liquid crystal or the like and displays a captured image. The image storage device 105 is a semiconductor memory, a magnetic memory, or the like for recording captured image data. The control device 106 controls the optical system 101 having distortion aberration, the image sensor 102, the distortion correction image processing device 103, the image display device 104 and the like according to a control program stored in a ROM (Read Only Memory) or the like (not shown). . The operation unit 107 includes buttons and switches corresponding to various functions used by the user to operate the camera, an information display unit, and the like.

  2 (a) and 2 (b) show optical images when an optical image taken with a normal optical system without distortion as shown in FIG. 2 (c) is taken with an optical system 101 having distortion. Illustrated. Among these, FIG. 2A is an example of an optical image taken by an optical system in which distortion is generated such that the optical image is compressed toward the peripheral part independently in the vertical and horizontal directions, and FIG. Is an example of an optical image photographed by an optical system in which a barrel-shaped distortion aberration is generated such that the optical image is compressed as the distance from the central portion increases concentrically by a coaxial optical system. 2A to 2C are examples of optical images obtained by photographing the same subject.

  As described above, the optical system 101 having distortion according to the present invention is configured such that the central image is enlarged and the peripheral image is reduced. It can also be realized as an optical distortion that combines the optical system of (b). In general, it is possible to perform correction with higher image quality by generating distortion as shown in FIG. Hereinafter, the present embodiment will be described by taking the distortion aberration of FIG. 2A as an example.

  FIG. 3 is a schematic diagram showing image data taken by the optical system 101 having longitudinal and transverse independent distortion aberrations as in FIG. 2A, and FIG. 3A is taken. FIG. 3B shows an original image, that is, an original image of the subject, that is, an image taken with an optical system having no optical aberration.

  In FIG. 3A, the center of the subject is enlarged both vertically and horizontally, and the periphery of the subject is compressed both vertically and horizontally, as in FIG. 3 (a) and 3 (b), (1) to (13) indicate coordinate positions having equidistant distances in the horizontal direction of the subject, and A to O are equidistant distances in the vertical direction. The coordinate position which has is shown. Therefore, in FIG. 3A and FIG. 3B, the coordinate positions (for example, ((7), H) in FIG. 3A and ((7)) in FIG. H)) corresponds to the same image. Note that an optical system having such distortion can be realized by combining vertical cylindrical and horizontal cylindrical lenses 401 and 402 as shown in FIG.

  When a landscape as shown in FIG. 3B is photographed by the digital camera in the present embodiment, the optical image is enlarged through the optical system 101 having distortion aberration, the central part is enlarged, and the peripheral part is compressed. An image is formed on the image sensor 102 as an image as shown in FIG. Then, the optical image having this distortion is converted into an electric signal by the image sensor 102.

  Note that a Bayer color filter is attached to the image sensor 102, and pixel data including Bayer color information is sequentially output. The pixel data is converted into a digital signal through an analog front-end circuit (CDS / PGA / A / D) (not shown) and input to the distortion correction image processing apparatus 103.

  As shown in FIG. 5, the distortion correction image processing apparatus 103 includes an image correction unit 501, an RGB synchronization unit 502, a first filter 503, an aberration correction unit 504, an enlargement / reduction unit 505, 2 filter 507, YC conversion unit 508, image compression unit 509, display conversion unit 510, and frame memory 511.

  The image correction circuit 501 performs pixel defect correction, white balance correction, and shading correction (light quantity correction) on the captured image input from the image sensor 102. As shown in FIGS. 6A and 6B, this shading correction is due to the effect of brightening due to the peripheral portion being compressed by distortion of the optical system and the effect of ambient light being cast by a lens frame or the like. In consideration of the dimming effect, correction is performed by a correction circuit as shown in FIG. 7 (a) or (b).

  In the case of FIG. 7A, specific correction processing is performed using a reference table memory 701 that outputs a correction coefficient as data when a physical coordinate position in an image input from the image sensor 102 is designated as an address. Is called. Further, in the case of FIG. 7B, the influence of the brightening due to the distortion aberration is expressed as a function F (x, y) of the physical coordinate position (x, y), and the ambient light is blocked by a lens frame or the like. An arithmetic unit 702 is configured as a product of the effects of dimming due to the generated effect as a function G (x, y) of the physical coordinate position (x, y). The arithmetic unit 702 includes a physical coordinate position ( By inputting x, y) as an address, a correction coefficient is output as a calculation result.

  Although not shown, the capacity of the reference table memory 701 is limited to a plurality of points, and correction coefficients for coordinate positions that are not in the table are calculated from the data of neighboring coordinates stored in the reference table memory 701 by linear interpolation or the like. May be calculated.

  In addition, such shading correction is caused by the effect of brightening due to the peripheral portion being compressed by the distortion of the optical system as shown in FIG. This can also be realized by adding an optical attenuating filter in which the light transmittance is changed based on a characteristic that takes into account the influence to an optical system having distortion. In addition, the influence of brightening due to the peripheral portion being compressed by distortion of the optical system due to the size and transmittance of the on-chip lens of the imager as shown in FIG. Correction can also be performed by changing the aperture ratio based on characteristics that take into account the effect of dimming due to the effect of being removed.

  The frame memory 511 stores the image signal that has been subjected to image correction by the image correction circuit 501. As described above, the reason why the image signal is once recorded in the frame memory as described above is that, as shown in FIG. 8, the data arrangement in the progressive or interlaced raster scan output from the imager is performed by the subsequent spatial image processing. This is for rearranging to block line scan, which is the order of data that can be processed continuously without going through a frame memory in a continuous processing system, and the other is to convert wide-angle images and telephoto images as will be described later. This is to enable simultaneous shooting.

  The RGB synchronization circuit 502 reads image data corresponding to the angle of view designated by the photographer from the frame memory 511, and generates all RGB color data at all coordinate positions from this Bayer image data by interpolation.

  Note that such synchronization processing is usually generated by interpolating from surrounding pixel data of the same color filter. This will be described with reference to FIG. Now, focusing on the red data, for example, in FIG. 9A, the data at the position 902 is generated by interpolating from the data 901 and the data 903, and the data at the position 904 is the data 901 and It is generated by interpolation from the data 907. At this time, if there is no distortion in the optical system, interpolation data may be generated as an average value of two pixels. However, the imaged Bayer image data is distorted by optical distortion as shown in FIG. The image data is obtained by sampling evenly by the image sensor. Accordingly, the actual landscape (corrected image) is as shown in FIG. 9B, and the coordinate positions of the obtained pixels are not uniform. ((1) to (13) in FIGS. 9A and 9B indicate coordinate positions that are equally spaced in the horizontal direction with respect to the subject, and A to O are coordinates that are equally spaced in the vertical direction. Therefore, for example, the same image corresponds to ((2), B) in FIG. 9A and ((2), B) in FIG. )

For this reason, in order to interpolate the captured image with high image quality, it is necessary to perform correction according to the distance in the actual landscape (corrected image). That is, when the pixel data 902 is interpolated from the pixel data 901 and the pixel data 903 in FIG. 9A, the correction between the pixel data 901 and the pixel data 902 after correction (FIG. 9B) Is XA, and the distance between the corrected pixel data 903 and the pixel data 902 is XB, the R color data at the position 902 is
902 R data = (901 R data × (XB / (XA + XB))
+ (903 R data x (XA / (XA + XB))
Need to be interpolated.

Similarly,
R data at position 904 = (R data at 901 × (YB / (YA + YB))
+ (907 R data x (YA / (YA + YB))
Need to be interpolated. Note that R data, B data, and G data at other coordinate positions can be generated in the same manner.

Further, the above shows an example in which interpolation is performed from two neighboring points. However, correction of G data or the like from four neighboring points can be performed with higher image quality. In this case, the correction may be performed in the same manner by performing weighting inversely proportional to the inter-pixel distance of the subject after distortion correction, and can be generated by the following arithmetic expression.
905 G data = (904 G data × (XB / (XA + XB))
+ (G data of 906 × (XA / (XA + XB)) / 2)
+ (G data of 902 × (YB / (YA + YB))
+ (908 G data × (YA / (YA + YB)) / 2)
Also, R data and B data can be generated in the same manner by combining with generation from the two neighboring points, or by performing interpolation from neighboring pixels in the oblique direction.

  The first filter 503 performs spatial filter processing on the RGB color signals independently of the RGB data by the RGB synchronization unit 502 prior to distortion correction processing. This is processing performed to prevent the image quality from deteriorating due to the high-frequency component of the image being folded back when the central portion of the image is reduced when performing distortion correction. Note that the characteristics of the low-pass filter must be changed according to the angle of view (zoom position) to be photographed, and the output image of the image sensor 102 is reduced as shown in FIGS. 10 (a) to 10 (b). When it is necessary to perform the processing, it is necessary to perform the processing only for the central region indicated by 1001 in FIG. 10A, and it is necessary not to perform the processing on the region 1002 that must be enlarged on the opposite side of the peripheral portion of the image. is there.

  FIG. 11 shows an example in which the first filter 503 is configured as a 3 × 3 spatial filter, and can function as a low-pass filter by setting the values of the coefficients (C11 to C33). Note that the values of the coefficients (C11 to C33) stored in the register need to be changed according to the coordinate position of the pixel, and in the area of the peripheral portion 1002 in FIG. And the others are set to “0” to control the filter not to operate. Further, it is desirable that the filter coefficient of the filter processing region 1001 in the central part is changed according to the reduction ratio determined by the angle of view and the degree of distortion depending on the coordinate position in order to perform image processing with high image quality.

  In the configuration example of FIG. 11, all the coefficients are configured so that they can be set independently in consideration of the influence of distortion. However, the image quality is sacrificed to some extent, and the influence of distortion is ignored, and the target is vertically and horizontally. It is possible to reduce the number of required multipliers and reduce the circuit scale by adding the pixel data at the position and then multiplying by the common coefficient, and at the expense of image quality. It can also be a constant value. However, in this case, it is necessary to blur only the area portions 1001 and 1002. Further, when the distortion is large and the reduction ratio at the center is large, the circuit scale becomes large, but it is necessary to configure a larger filter such as 4 × 4 or 5 × 5. Further, the configuration of the filter is not limited to the matrix filter as shown in the present embodiment.

  The aberration correction unit 504 inputs the image signal subjected to the low-pass filter processing to each color of RGB independently for the central portion of the image reduced by the distortion correction in the first filter 503. The specific configuration of the aberration correction unit 504 is as shown in FIG. 12, and the corresponding coordinate calculation unit that calculates the coordinate position on the image having the corresponding distortion aberration from the coordinates of the image data after distortion aberration correction. 1201, an IF image memory 1202 for storing image data processed by the first filter 503 and the like in the previous stage, and recorded data necessary for obtaining pixel data of target coordinates calculated by the corresponding coordinate calculation unit 1201 The pixel of the target coordinate is calculated from the neighboring pixel data reading unit 1203 that calculates the coordinates of the neighboring pixel data and reads out the data of the coordinates from the IF image memory 1202, and the neighboring pixel data read by the neighboring pixel data reading unit 1203. And a correction pixel value calculation unit 1204 for calculating data.

Next, a distortion correction method in the aberration correction unit 504 will be described with reference to FIG.
FIG. 13A shows an image after distortion aberration correction, and the purpose of this aberration correction unit is to calculate pixel data at the intersection of the lattices in this figure. Here, a case where the pixel 1301 having the coordinate value (Xn, Yn) is obtained will be described as an example.

  First, when coordinate values (Xn, Yn) are input to the corresponding coordinate calculation unit 1201, the corresponding coordinate calculation unit 1201 uses the corresponding coordinate values (Xs, Ys) in the original image before correcting optical distortion and lateral chromatic aberration. calculate. In addition, when there is one-dimensional distortion aberration in the vertical and horizontal directions as in the present embodiment, each coordinate value can be obtained by independently calculating. Although this calculation depends on the lens design, since it is one-dimensional, it can be realized with a simple arithmetic circuit, or a table memory in which the input coordinate is an address and the output coordinate value is recorded in the corresponding data, or The number of table memories is limited to several discrete points, and can be obtained by a polygonal line approximation method that realizes the middle by linear interpolation.

  The coordinate position calculated in this way is set to 1302 in FIG. The grating shown in FIG. 13B is described so as to correspond to the grating shown in FIG. 13A, and has an optical distortion aberration in which the central part is enlarged and the peripheral part is compressed. Show. Accordingly, it can be understood that 1302 is a coordinate where the corresponding image is shown, although the physical coordinate position is deviated from the target coordinate position.

  However, as shown in FIG. 13C, the positions of the pixels actually imaged by the image sensor 102 are equal grid positions on the image having the distortion, and are not the pixel position data to be obtained. . Therefore, the coordinate position data 1304 to 1307 recorded in the vicinity of the target pixel 1303 (Xs, Ys) are read from the IF image memory 1202 by the neighboring pixel data reading unit 1203, and linear interpolation or the like is performed by the correction pixel value calculation unit 1204. The value of the target pixel 1303 (Xs, Ys) is calculated by the above means. In the present embodiment, the method of calculating by linear interpolation from the four neighboring pixels is shown. However, the interpolation calculation is not limited to this method, and the nearest pixel can be substituted or calculated from the sixteen neighboring pixels by cubic interpolation. Or other interpolation methods may be used.

  In the present embodiment, a configuration is employed in which aberration correction is performed independently for each of the RGB color signals. Therefore, even when the optical system has lateral chromatic aberration, it is possible to correct aberration simultaneously with distortion by adjusting the correction amount for each color. On the other hand, if the lateral chromatic aberration is small enough not to be corrected, the correction pixel value calculation unit 1204 can be made common to each color, so that the circuit scale can be reduced.

  In this embodiment, a method of performing optical distortion aberration independently in the vertical and horizontal directions has been described. However, when a coaxial lens is used, concentric distortion aberration is generated according to the distance from the center. . The coordinate value in such a case can generally be obtained by Y = polynomial. Also, use a table memory in which the input coordinates are the addresses and the output coordinate values are recorded in the corresponding data, or the number of table memories is limited to a few discrete points, and a polygonal line approximation method that achieves the middle by linear interpolation, etc. You can also.

  The enlargement / reduction unit 505 receives the image data whose distortion has been corrected by the aberration correction unit 504, and calculates an interpolated pixel value by linear interpolation, cubic interpolation, or the like for each RGB signal. As shown in FIG. 14, the enlargement / reduction unit 505 includes a corresponding coordinate calculation unit 1401, an IF image memory 1402, a neighboring pixel data reading unit 1403, and a corrected pixel value calculation unit 1404.

  When the coordinate value (Xn, Yn) is input, the corresponding coordinate calculation unit 1401 calculates the corresponding coordinate value (Xs, Ys) in the original image before the image is enlarged or reduced. The IF image memory 1402 stores image data subjected to distortion aberration correction. The pixel data reading unit 1403 reads stored pixel data in the vicinity in order to calculate the pixel value at the target coordinate position after enlargement / reduction. The corrected pixel value calculation unit 1404 calculates a pixel value at a target coordinate position from neighboring pixel data.

FIG. 15 is a diagram for explaining the enlargement / reduction process.
In the drawing, a solid line grid portion 1518 indicates a position where a pixel of input image data stored in the IF image memory 1402 is recorded, and a broken line grid portion 1519 indicates a pixel to be obtained for performing reduction processing. The coordinate position is shown. Here, when obtaining the pixel value of 1501 on the dashed grid, the coordinate position of the pixel in the grid section 1519 of the broken line is input to the corresponding coordinate calculation section 1401 and converted to the coordinate position of the solid grid section 1518 of the input image. Is done. The coordinate position data is input to the neighboring pixel data reading unit 1403, and the neighboring pixel data reading unit 1403 calculates the coordinate positions on the solid grid of 16 points near the pixel value calculation target 1501; These pixel data are read from the IF image memory 1402. These image data are output to the corrected pixel value calculating unit 1404, and the corrected pixel value calculating unit 1404 calculates the pixel value of 1501 to be obtained by cubic interpolation. This interpolation calculation is not limited to this method, and the nearest pixel may be substituted, or may be calculated by linear interpolation from the four neighboring pixels, or other interpolation methods may be used.

  Thus, the image data resized to a desired size is input to the second filter 507 shown in FIG. 5 and subjected to spatial filter processing. This is because, as a result of distortion correction, the edge enhancement filter processing performed to recover the image quality that has deteriorated due to a decrease in the resolution of the image as the peripheral portion of the image is enlarged, the first filter 503, This is a noise reduction filter process for reducing a noise level difference from the central portion where noise is reduced by reduction by distortion correction.

  This processing is desirably performed only for the peripheral area as indicated by 1002 in FIG. 10A, in which the image sensor output image that changes according to the angle of view (zoom position) to be photographed must be enlarged. This is because the resolution of the reduced area at the center of the image is not lowered, and it is prevented that only the peripheral part of the image is blurred as it is. Further, since the noise reduction effect can be obtained by reducing the central portion of the image, only the peripheral portion of the image is prevented from becoming an image with noticeable noise. Therefore, it is not necessary to perform edge enhancement processing and noise reduction processing at least as much as the image peripheral portion for the central portion of the image.

  This filter can be realized with the same configuration as the low-pass filter of FIG. 11, and can function as an edge enhancement filter by setting the values of the coefficients (C11 to C33). Note that the values of the coefficients (C11 to C33) stored in the register need to be changed according to the coordinate position of the pixel. In the region such as the central portion 1001 in FIG. By setting “1” and setting others to “0”, it is possible to control so that the filter does not act.

  In the configuration example of FIG. 11, all the coefficients can be set independently with the influence of distortion, but the influence of distortion is ignored at the expense of some image quality, and the target positions are vertically and horizontally. It is also possible to reduce the circuit scale by reducing the number of necessary multipliers by multiplying the pixel data first and then multiplying by the common coefficient. In addition, when the distortion is large and the enlargement ratio of the peripheral portion is large, the circuit scale becomes large. However, the image quality can be improved by configuring as a larger filter such as 4 × 4 or 5 × 5. However, it can also be a constant value at the expense of image quality. However, in this case, it is necessary to blur only the area portions 1001 and 1002. The filter is not limited to the matrix filter as shown in the present embodiment.

  After restoring the image quality of the peripheral portion as described above, the image data is input to the display conversion unit 510 shown in FIG. 5 and converted into a format necessary for input of an image display device such as a TFT liquid crystal panel (NTSC format). The image is resized, interlaced, etc. in accordance with the above, and then output to the image display device 104 shown in FIG. Further, this image signal is simultaneously converted from the RGB color space to the YCbCr color space by the YC conversion unit 508 and subjected to compression processing by the image compression unit 509 such as JPEG, and then the image storage device shown in FIG. 105 is output.

  In the present embodiment, as shown in FIG. 5, the aberration correction and the enlargement / reduction are performed separately. However, both of these functions generate data of a new coordinate position from the surrounding pixel data. This is a function, and when calculating the coordinate position to be generated, the part due to aberration and the part due to enlargement / reduction are calculated together, so that this part is shared and the circuit scale is reduced as shown in FIG. It is possible. In FIG. 16, components having the same functions and operations as the components shown in FIG. 5 are denoted by the same reference numerals.

  Further, as shown in FIG. 17, with the Bayer data as it is, the first filter 1703 applies a low-pass filter process to the central portion of the image, and this image data is input and aberration correction / enlargement / reduction is performed. It is also possible to change the order of processing so that the processing unit 1706 performs aberration correction and enlargement / reduction processing, and then the RGB synchronization unit 1702 performs RGB synchronization processing. However, in this case, it is possible to correct aberrations independently for RGB as in the configurations of FIGS. 5 and 16, but it is difficult to correct high-quality magnification chromatic aberration because only the peripheral same color pixels are corrected. Become. Further, in order to simplify the circuit, the configuration is such that it cannot cope with chromatic aberration of magnification correction. In FIG. 17, components having the same functions and operations as those of the components shown in FIG.

  Further, as shown in FIG. 18, after the RGB synchronization unit 502 performs RGB synchronization, the YC conversion unit 1808 performs conversion to a luminance / color difference signal (YCbCr signal), and then the first filter 1803 performs conversion. The image center portion is subjected to low-pass filter processing for each luminance / color difference signal, and the aberration correction / enlargement / reduction processing unit 1806 performs aberration correction and enlargement / reduction processing. It is also possible to change the order of processing so that the filter 1807 performs edge enhancement filter processing and noise reduction filter processing in the peripheral portion. In FIG. 18, components having the same functions and operations as those of the components shown in FIG.

  In this case as well, it is difficult to correct the lateral chromatic aberration, which is possible by correcting the aberration independently for RGB in the configurations of FIGS. 5 and 16, but on the other hand, the luminance signal that greatly affects the resolution. Only high-precision processing can achieve both high image quality and low cost.

  Further, as shown in FIG. 19, with the Bayer data as it is, the first filter 1703 performs low-pass filter processing on the center of the image, and in this state, the aberration correction / enlargement / reduction processing unit 1706 corrects and corrects aberrations. After that, the RGB synchronization unit 1702 performs RGB synchronization, and then the YC conversion unit 1908 converts the luminance / color difference signal (YCbCr signal), and then the second filter 1907. , The processing order can be changed so as to perform edge enhancement filter processing and noise reduction filter processing in the peripheral portion. In FIG. 19, components having the same functions and operations as the components shown in FIGS. 5, 17, and 18 are denoted by the same reference numerals.

  In this case as well, it is difficult to correct the lateral chromatic aberration that has been made possible by correcting the aberration independently for RGB in the configurations of FIGS. 5 and 16, but it is highly accurate only for the luminance signal that greatly affects the resolution. It is possible to achieve both high image quality and low cost by performing appropriate processing.

Next, the user interface of the digital camera and the operation method thereof according to this embodiment will be described.
A high-quality telephoto image and a wide-angle image can be obtained simultaneously from image data picked up through an optical system having distortion in the present invention. Therefore, in addition to the normal imaging mode, it is possible to set three types of imaging modes: a wide-angle / telephoto simultaneous imaging mode and a wide-angle monitor / telephoto imaging mode. This wide-angle / telephoto simultaneous shooting mode is effective when you want to shoot both a wide-angle image and a telephoto image with a magnified center at the same time, such as a sports day or a school performance for children. This is a shooting mode that is most suitable for applications such as shooting an enlarged image of a subject while easily tracking a subject that moves rapidly.

  In the present embodiment, these three types of imaging modes are sequentially switched and used as shown in FIG. 20 by a changeover switch. At this time, the photographer is shooting in any mode. It is necessary to indicate clearly. Therefore, in the present invention, during normal wide-angle shooting and telephoto shooting, as shown in FIG. 21 (c), a general color display is used, and when telephoto shooting is performed while monitoring at a wide angle, FIG. ), The part that is actually captured in the image is displayed in color, the part that is not recorded is displayed in monochrome, such as black and white, the brightness is changed, the color tone is emphasized, and the display is made lighter. To do.

  In addition, when performing wide-angle telephoto simultaneous photographing, a wide-angle telephoto simultaneous photographing is performed on the photographer by displaying a frame indicating the photographing range on the telephoto side in addition to the normal photographing display as shown in FIG. Notify that there is. In this way, the user can intuitively understand the shooting mode when viewing a monitor such as a TFT liquid crystal.

  FIG. 22 shows a configuration example of the operation unit for controlling such a shooting mode, and the operation of the digital camera will be described based on FIG. The photographer operates the mode button 2202 to select a desired mode from the three shooting modes. At this time, when the wide-angle / telephoto simultaneous photographing mode or the wide-angle monitor / telephoto photographing mode is selected, the photographer needs to be able to set the zoom magnifications on the telephoto side and the wide-angle side independently. In the example of FIG. 22A of the present embodiment, the Tele1 button 2205 / Wide1 button 2206 on the wide-angle side and the Tele2 button 2207 / Wide2 button 2208 on the telephoto side are provided independently, so both angles of view are the subject. Can be set according to the situation. After the angle of view is determined in this way, shooting can be performed by pressing the shutter button 2201.

  The zoom buttons do not necessarily have to be provided independently for the two systems of wide-angle and telephoto. As shown in FIG. 22B, one tele button 2209 / wide button 2210 is replaced with a T / W switching button 2203. It can also be used by switching. In this case, it is desirable to place the changeover switch near the center of the zoom key.Although not shown, which side is set is displayed on the monitor with characters and symbols, or with the color of the frame. , To the photographer. Further, as shown in FIG. 22C, a switching lever 2204 in which a zoom button and a changeover switch are integrated may be provided. Note that the photographing mode can be changed even during photographing.

  FIG. 22 shows an example in which two wide-angle and telephoto images can be simultaneously photographed. However, the original image photographed by the optical system having the optical distortion according to the present invention is not limited to two, but any number of three or more. Can be taken simultaneously. In such a case, in the configuration of FIG. 22 (a), it is necessary to further increase the Tele button and the Wide button, but in the configurations of FIG. 22 (b) and FIG. 22 (c), operation can be performed by providing similar operation buttons. It is obvious that this is possible.

Next, the relationship between the button operation of the operation unit and the screen display in the wide-angle / telephoto simultaneous shooting mode and the wide-angle monitor / telephoto shooting mode will be described.
When the mode button is pressed in the normal shooting mode (wide angle) state of FIG. 21 (d), the camera shooting mode is the wide angle / telephoto simultaneous shooting mode as shown in FIG. 20, and this is displayed to the user. Therefore, the image on the image display device 104 changes as shown in FIG. Further, when the mode button is pressed, the shooting mode of the camera becomes the wide-angle monitor / telephoto mode as shown in FIG. 20, and the image on the image display device 104 is displayed in FIG. ). In this state, when the mode button is pressed, the shooting mode of the camera is set to the normal shooting mode as shown in FIG. 20, and the image on the image display device 104 is displayed in FIG. ) State. When the Tele button is operated in the normal shooting state, the state shown in FIG. 21C (telephoto) is obtained, and when the Wide button is operated, the state returns to the state shown in FIG. 21D (wide angle).

The operation when the T / W switch button and the zoom are operated in the wide-angle / telephoto simultaneous shooting mode or the wide-angle monitor / telephoto shooting mode with the camera having the operation unit shown in FIG. To do.
FIG. 23B shows a state in which the shooting mode is the wide-angle / telephoto simultaneous shooting mode and the zoom operation is effective for the Wide image. In this figure, the frame line on the side where the zoom operation is enabled is shown boldly and shown to the user. It should be noted that the current zoom operation status can be shown to the user by changing the thickness of the frame line, changing the color or shape, or changing the brightness.

  When the switch lever 2204 is tilted to the right from this state, the Tele button is operated, and the entire screen is zoomed up and enlarged as shown in FIG. At this time, the displayed frame line is also enlarged. On the contrary, when the switching lever 2204 is tilted to the left, the imaged range is widened and reduced as shown in FIG. At this time, the displayed frame line is also reduced in conjunction. Therefore, when the zooming operation is performed by tilting the switching lever 2204 left and right, the display changes as shown in FIGS. 23A to 23C, and the photographing range can be displayed to the user.

  Next, when the switch lever 2204 is pushed in again from the state shown in FIG. 23B, this operation becomes an operation of the T / W switch button, and the angle of view at which the zoom operation is effective changes from the wide angle side to the telephoto side. Accordingly, as shown in FIG. 23E, the thickness of the frame line is switched between the wide angle side and the telephoto side. When the switching lever 2204 is tilted to the right from this state, the Tele button is operated, and the display on the entire screen does not change as shown in FIG. 23D, and the frame line indicating the telephoto side shooting range is reduced. This displays that the shooting range is narrowed, that is, that the inside of the frame is enlarged.

  On the other hand, when the switching lever 2204 is tilted to the left, as shown in FIG. 23 (f), the display on the entire screen does not change and the frame indicating the photographing range on the telephoto side becomes larger. As a result, it is displayed that the photographing range is widened, that is, the inside of the frame is enlarged. Therefore, when the zooming operation is performed by tilting the switching lever 2204 left and right, the display changes as shown in FIGS. 23D to 23E and the photographing range can be displayed to the user. Further, when the switching lever 2204 is pushed in again from the state shown in FIG. 23 (e), the T / W switching button is operated, and the field angle at which the zoom operation is effective is shifted from the telephoto side to the wide angle side. As shown in FIG. 23B, the thickness of the frame is switched between the wide-angle side and the telephoto side.

  In the above description, the operation of the T / W switching button has been described between (b) and (e) in the center of the figure, but the T / W switching button can be used at any angle of view for both wide angle and telephoto. It is possible to change the angle of view in which the zoom operation is effective by the operation of. Further, in the above description, the switching display has been shown as a method of superimposing the captured image on the image display device 104 of FIG. 1, but the method of display to the user is not limited to this, for example, FIG. As shown in d), an LED may be arranged on the operation unit, or a liquid crystal display panel or the like may be arranged to display on such a part, or a state such as a toggle switch is fixed. A button may be adopted for display. Moreover, you may display to a user combining these several methods suitably.

  When performing wide-angle telephoto simultaneous shooting, the output signal of the image sensor 102 is held in a temporary storage medium such as a DRAM (not shown) in the distortion correction image processing apparatus 103, and first, according to the wide-angle setting value set by the user. Various parameters (brightness correction, filter coefficient, enlargement / reduction ratio, etc.) in the distortion correction image processing apparatus 103 are set. Next, the above-described image processing is executed to obtain a wide-angle captured image in which distortion is corrected, and this is compressed and recorded in the image storage device 105.

  Further, from the image data held in a temporary storage medium such as DRAM (not shown) in the distortion correction image processing apparatus 103, this time, the distortion aberration image processing apparatus 103 is set according to the telephoto set value set by the user. Various parameters (luminance correction, filter coefficient, enlargement / reduction ratio, etc.) are set. Then, the image processing as described above is executed to obtain a telephoto image in which distortion is corrected, and this is compressed and recorded in the image storage device 105.

  In the wide-angle telephoto simultaneous shooting mode, both the wide-angle shot image and the telephoto shot image can be obtained by one shooting in this way. Then, the file name recorded at this time is given the same prefix so that it can be easily discriminated that the images were simultaneously shot later. In addition, the recorded image is displayed as a preview after shooting, but the wide-angle and telephoto images can be displayed sequentially for a certain period of time, or either the wide-angle or telephoto images can be displayed simultaneously. It can also be configured so that the photographed image is clearly displayed to the user and the display can be switched by operating the wide-angle / telephoto switching button. Note that the order of image processing at the time of simultaneous photographing for generating a wide-angle / telephoto image from an original image having distortion within the camera may be either the wide-angle image or the telephoto image.

  During normal shooting mode, wide-angle monitor / telephoto shooting mode, according to the setting value of the shooting angle of view set by the user from image data held in a temporary storage medium such as DRAM (not shown) in the distortion image processing apparatus 103 Thus, various parameters (brightness correction, filter coefficient, enlargement / reduction ratio, etc.) in the distortion aberration image processing apparatus 103 are set. Then, the above-described image processing is executed to obtain a captured image in which distortion is corrected, and this is compressed and recorded only once in the image storage device 105. The recorded image is displayed as a preview after shooting. In this case, the shot image is only displayed for a certain period of time, and the operation of the wide-angle / telephoto switching button or the mode button is invalidated.

  Naturally, the digital camera according to the present embodiment can also display captured images. When displaying images captured in the wide-angle and telephoto simultaneous shooting mode, the wide-angle and telephoto images are automatically displayed for a certain period of time. It is possible to configure the display so that it is repeatedly displayed one after another, and the normal frame advance button clearly indicates that either the wide-angle side or the telephoto side image is a simultaneously shot image. In this case, the display can be switched by operating the wide-angle / telephoto switching button in addition to the normal frame advance button operation. Further, the normal frame advance button may be advanced to display only one angle of view.

  In the digital camera according to the present embodiment, it is necessary to correct distortion of the optical system even when displaying a through image on the display unit as a viewfinder. For such applications, imagers often have a finder mode in which vertical lines are thinned out and read at a high frame rate as shown in FIG. However, when an image having distortion and a peripheral portion compressed in this manner is thinned out evenly and corrected as described above, and then the necessary number of lines is obtained by interpolation, the image quality is greatly deteriorated. Therefore, as shown in FIG. 24B, a high-quality through image display is realized by thinning the peripheral part and thinning the central part so that the lines obtained after distortion correction are uniform.

  The lens distortion information and lens frame information are stored in an information storage means such as a ROM provided in the lens together with the focal length, F value, etc., and this information is read from the camera body for filtering and shading correction. By configuring so as to perform coordinate transformation for distortion correction, even when applied to an interchangeable lens camera, it is possible to perform appropriate distortion correction according to the interchangeable lens.

Next, an overview of the zoom function of the digital camera will be described.
An image captured through the optical system 101 having distortion aberration has distortion aberration such that the central portion is enlarged and the peripheral portion is compressed as shown in FIG. When performing wide-angle shooting using the image pickup apparatus of the present invention, distortion correction is performed using all the pixels indicated by reference numeral 2503 in FIG. 25A, the central portion is reduced, and the peripheral portion is enlarged. As shown in FIG. 25B, image data of the original size in which distortion is corrected is obtained.

  When performing telephoto shooting, a pixel in the range indicated by 2501 in FIG. 25A is cut out to perform distortion correction, while the central portion is left as it is and the peripheral portion is enlarged as shown in FIG. 25C. Then, the original size image data in which distortion is corrected is obtained. When shooting at an intermediate angle of view between the wide angle and the telephoto, for example, a pixel in the range indicated by 2502 in FIG. 25A is cut out to correct distortion, and the central portion is reduced and the peripheral portion is enlarged. Thus, as shown in FIG. 25D, image data of the original size with the distortion corrected is obtained. In FIGS. 25A to 25D, reference numerals 2501 to 2503 denote the same field angle range. By doing so, it is possible to realize a zoom without a driving unit. Compared to an electronic zoom photographed with an optical system having no distortion, the range to be cut out is large at the time of telephoto photography, so that the image quality is high, and in particular, there is no deterioration because there is no need to enlarge the central portion.

Next, a flow of processing when a subject is captured using the digital camera according to the first embodiment of the present invention having the configuration of FIG. 5 and simultaneous wide-angle and telephoto shooting is performed will be described.
First, when the user uses the camera in the wide-angle / telephoto simultaneous shooting mode and observes the subject through the liquid crystal monitor, the digital camera according to the present embodiment drives the image sensor 102 in the finder mode and sets the vertical direction to 200 lines. Continuously thinned images are obtained. Next, the image correction unit 501 performs pixel defect correction, white balance correction, and shading correction (light amount correction) on the image. At this time, as described above, correction is performed in consideration of the influence of distortion. The image-corrected data is continuously stored in the frame memory 511.

  When the user is performing an operation for determining the angle of view on the wide angle side, image data corresponding to the angle of view on the wide angle side that the user is observing on the liquid crystal monitor from this frame memory 511 in the subsequent stage. Data is read out by rearranging in the order of easy processing. The RGB synchronization unit 502 performs RGB synchronization processing on the read Bayer data. At this time, as described above, correction is performed in consideration of the influence of distortion.

  The data subjected to the RGB simultaneous processing is filtered in the first filter 503. At this time, the central portion is strong considering the influence of distortion, and the image signal is filtered with a coefficient suitable for reducing the image signal to a size to be displayed on the liquid crystal monitor. Since the peripheral portion is compressed due to distortion, if the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the low-pass filter is set so that the coefficient of the matrix filter is only 1 for the portion that is not reduced. Do not apply.

  Next, the data filtered by the first filter 503 is input to the aberration correction unit 504, coordinate conversion processing for correcting distortion is performed, and then the enlargement / reduction unit 505 performs a liquid crystal monitor. Is reduced to the size to be displayed. At this time, if there is an enlarged portion in the image data, edge enhancement and noise reduction processing is performed on the portion by the second filter 507.

  The data output from the second filter 507 is input to the display conversion unit 510, converted into a format (size, scan format) necessary for the liquid crystal monitor display, and stored in the display frame memory in the display conversion unit 510. Written. Then, the display conversion unit 510 reads the display frame memory in the display conversion unit 510 and continues to display an image on the liquid crystal monitor.

  When the user switches the angle of view with the Tele button 2209 or the Wide button 2210 for controlling the zoom shown in FIG. 22B in a state in which the angle of view on the wide angle side can be operated, the image that the user is observing on the liquid crystal monitor Corresponding to the corner, the angle of view of the image data read from the frame memory 511 is changed and read. Along with this, since the degree of distortion and the angle of view (number of image data) output from the frame memory 511 change, the processing of the RGB synchronization unit 502 and the first filter are interlocked with this. The parameters to 503, the distortion correction unit 504, the enlargement / reduction unit 505, and the second filter 507 are also changed.

  When the user operates the T / W switching button 2203 shown in FIG. 22B, the angle of view on the side that has been operated so far (for example, the wide angle side) is determined, and the angle of view on the other side (for example, the telephoto side). It becomes the mode to operate. At this time, even if the user switches the angle of view with the Tele button 2209 or the Wide button 2210 for controlling the zoom shown in FIG. 22B, the angle of view that the user is observing on the liquid crystal monitor is not changed, and the liquid crystal monitor Continues to display the angle of view corresponding to the wide-angle image. Accordingly, the angle of view of the image data read from the frame memory 511 is read according to the angle of view on the wide angle side. The parameters corresponding to the angle of view on the wide angle side are also used for the processing of the RGB synchronization unit 502 and the parameters for the first filter 503, the distortion correction unit 504, the enlargement / reduction unit 505, and the second filter 507.

  The frame shown in FIG. 22 (b) changes in accordance with the operation of the Tele button 2209 or Wide button 2210 for controlling the zoom, and shows a telephoto-side shooting area displayed on the liquid crystal monitor. When the button 2209 is pressed and the telephoto lens is tilted to the telephoto side, the size is small.

  When the user also determines the angle of view on the telephoto side and operates the shutter button 2201 shown in FIG. 22B, after the processing such as AE / AF is executed, the image sensor outputs all pixels in a progressive or interlaced manner. Driven in normal shooting mode. Then, the image correction unit 501 performs pixel defect correction, white balance correction, and shading correction (light amount correction) on this image. At this time, as described above, correction is performed in consideration of the influence of distortion.

  The image-corrected data is stored in the frame memory 511 for one frame. First, in order to generate wide-angle image data, data corresponding to the image data set by the user at the wide-angle side is rearranged in the order of arrangement that can be easily processed in the subsequent stage from the frame memory 511.

  The read data is RGB-synchronized from the Bayer data in the RGB synchronization unit 502. At this time, as described above, correction is performed in consideration of the influence of distortion. The RGB synchronized data is low-pass filtered by the first filter 503. At this time, the central part is strong in consideration of the influence of distortion, and filtering is performed with a coefficient suitable for the degree of reduction for the part to be reduced to fit the image size to be recorded and set by the user. To do. Since the peripheral portion is compressed by distortion, when the peripheral portion is not reduced from the image size to be recorded and the peripheral image size, the coefficient of the spatial filter shown in FIG. , C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied.

  The data subjected to the low-pass filter processing is subjected to coordinate conversion processing for correcting distortion aberration in the aberration correction unit 504. The data subjected to the coordinate conversion processing is enlarged / reduced by the enlargement / reduction unit 505 to the image size to be recorded. Next, the enlarged / reduced data is subjected to edge enhancement and noise reduction processing for the enlarged part in the second filter 507.

  The data output from the second filter 507 is converted into a format (size, scan format) necessary for the liquid crystal monitor display in the display conversion unit 510 and written in the display frame memory in the display conversion unit 510. . The display conversion unit 510 reads the display frame memory in the display conversion unit 510 and continues to display an image on the liquid crystal monitor.

  On the other hand, the data output from the second filter 507 is also input to the YC conversion unit 508, and the YC conversion unit 508 converts the RGB signal into the color space and converts it into a YCbCr signal. Also, a process of thinning out the CbCr signal to 422 or the like is performed in accordance with the JPEG compression mode. The signal converted into the color space in the YC conversion unit 508 is subjected to JPEG compression in the image compression unit 509 and recorded as a wide-angle captured image in an image storage device such as a memory card.

  Next, in order to generate image data on the telephoto side, data is read from the frame memory 511 by rearranging the image data set by the user at the angle of view on the telephoto side in an order that can be easily processed later. The read data is RGB-synchronized from the Bayer data in the RGB synchronization unit 502. At this time, as described above, correction is performed in consideration of the influence of distortion.

  The RGB synchronized data is input to the first filter 503 and subjected to low-pass filter processing. At this time, the central part is strong in consideration of the influence of distortion, and filtering is performed with a coefficient suitable for the degree of reduction for the part to be reduced to fit the image size to be recorded and set by the user. To do. Since the peripheral portion is compressed by distortion, when the peripheral portion is not reduced from the image size to be recorded and the peripheral image size, the coefficient of the spatial filter shown in FIG. , C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied. When the central portion is not reduced, the filter coefficient is set to 1 only for the center C22 for the entire area of the image so that the low-pass filter is not applied.

  The low-pass filtered data is subjected to coordinate transformation processing for correcting distortion in the aberration correction unit 504, and then enlarged / reduced to an image size to be recorded in the enlargement / reduction unit 505. In the second filter 507, edge enhancement and noise reduction processing are performed on the enlarged portion.

  The data output from the second filter 507 is converted into a format (size, scan format) necessary for the liquid crystal monitor display in the display conversion unit 510 and written in the display frame memory in the display conversion unit 510. . The display conversion unit 510 reads the display frame memory in the display conversion unit 510 and continues to display an image on the liquid crystal monitor. At this point, the liquid crystal monitor is switched from displaying the wide-angle image so far to displaying the telephoto image.

  On the other hand, the data output from the second filter 507 is converted from an RGB signal to a color space by a YC conversion unit 508 and converted to a YCbCr signal. Also, a process of thinning out the CbCr signal to 422 or the like is performed in accordance with the JPEG compression mode. The signal converted into the color space in the YC conversion unit 508 is subjected to JPEG compression processing in the image compression unit 509 and recorded as a telephoto image on an image recording apparatus such as a memory card.

  Next, the flow of processing when a subject is captured using the digital camera shown in FIG. 16 and simultaneous wide-angle and telephoto shooting is performed will be described. The process in this case is almost the same as the process flow of the digital camera having the configuration shown in FIG. 5 described above, and a coordinate conversion process for correcting distortion and a process for enlarging / reducing the image size to be displayed or recorded. Together with the calculation to interpolate the target coordinate data from the surrounding pixels and calculate the position of the corrected target coordinate position in the image before correction. However, the only difference is that the aberration correction and the enlargement / reduction process are shared and the simultaneous process is realized.

  Next, the flow of processing when a subject is captured using the digital camera shown in FIG. 17 and simultaneous wide-angle and telephoto shooting is performed will be described. Note that the image correction unit 501 performs pixel defect correction and the like, and continuously saves them in the frame memory 511, and processes image data corresponding to the angle of view on the wide angle side from the frame memory 511 in the subsequent stage. The processing up to the process of rearranging the data in an easy order and reading the data is the same as the processing in the configuration of FIG.

  The Bayer image data read from the frame memory 511 is low-pass filtered in the first filter 1703. At this time, the central portion is strong in consideration of the influence of distortion, and the image signal is filtered with a coefficient suitable for reducing the image signal to a size to be displayed on the liquid crystal monitor. Since the peripheral portion is compressed due to distortion, when the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the coefficient of the spatial filter shown in FIG. C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied. In addition, since it is a filtering process for the Bayer image, the circuit may be a single system, but the process is performed by switching the coefficient separately for each color only for the pixels to which the surrounding same color filter is attached. .

  The Bayer image data that has undergone the low-pass filter processing is data of coordinates targeted by both the coordinate conversion processing for correcting distortion aberration and the processing for reducing the image size displayed on the liquid crystal monitor in the aberration correction enlargement / reduction unit 1706. Is processed by interpolating from the surrounding pixels, and by calculating the position of the target coordinate position after correction in the image before correction, it is processed at the same time . Since this process is also Bayer data, only the pixels of the same color filter are processed for each color.

  The Bayer image data subjected to the aberration correction and enlargement / reduction processing is RGB-synchronized from the Bayer data in the RGB synchronizer 1702. When there is an enlarged portion of the RGB synchronized data, the second filter 507 performs edge enhancement and noise reduction processing.

  The data output from the second filter 507 is converted into a format (size, scan format) necessary for the liquid crystal monitor display in the display conversion unit 510 and written in the display frame memory in the display conversion unit 510. . The display conversion unit 510 reads the display frame memory in the display conversion unit 510 and continues to display an image on the liquid crystal monitor.

  When the user switches the angle of view by tilting the switching lever 2204 that controls the zoom shown in FIG. 22C in a state where the angle of view on the wide angle side can be operated, the image that the user is observing on the liquid crystal monitor Corresponding to the corner, the angle of view of the image data read from the frame memory 511 is changed and read. Along with this, since the degree of distortion and the angle of view (number of image data) output from the frame memory 511 change, the processing of the RGB synchronizer 1702 and the first filter 1703 are interlocked with this. The parameters for the aberration correction enlargement / reduction unit 1706 and the second filter 507 are also changed.

  When the user operates to push in the switching lever 2204 shown in FIG. 22C, the angle of view on the side that has been operated (for example, the wide angle side) is determined, and the angle of view on the other side (for example, the telephoto side) is operated. Mode. Even if the user switches the angle of view by tilting the switching lever 2204 for controlling the zoom shown in FIG. 22 (c) to the left or right, the angle of view that the user is observing on the liquid crystal monitor remains unchanged, and the liquid crystal monitor has a wide angle. Continue to display the angle of view corresponding to the image. Accordingly, the angle of view of the image data read from the frame memory 511 is read according to the angle of view on the wide angle side. Also, parameters corresponding to the angle of view on the wide angle side are used for the processing of the RGB synchronization unit 1702 and the parameters for the first filter 1703, the aberration correction enlargement / reduction unit 1706, and the second filter 507.

  The frame that shows the telephoto-side shooting area displayed on the liquid crystal monitor changes according to the tilting of the switching lever 2204 that controls the zoom shown in FIG. 22C to the left and right. This frame is the switching lever 2204. If the switch lever 2204 is turned to the telephoto position to the telephoto side, the distance is small, and if the switch lever 2204 is tilted to the left to the wide angle side, the distance is increased.

  When the user also determines the angle of view on the telephoto side and operates the shutter button 2201 shown in FIG. 22 (c), the image sensor outputs all pixels in a progressive or interlaced manner after processing such as AE / AF is executed. Driven in normal shooting mode. At this time, the image correction unit 501 executes pixel defect correction, white balance correction, and shading correction (light amount correction) for the image. At this time, correction is performed in consideration of the influence of distortion as described above.

  The image-corrected data is stored in the frame memory 511 for one frame. First, in order to generate image data on the wide angle side, data corresponding to the image data set by the user at the angle of view on the wide angle side is rearranged in an order that can be easily processed later, from the frame memory 511. .

  The read data is low-pass filtered on the Bayer image in the first filter 1703. At this time, the central portion is strong in consideration of the influence of distortion, and filtering is performed with a coefficient suitable for reducing the image signal to a recording size. Since the peripheral portion is compressed by distortion, if the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the coefficient of the spatial filter shown in FIG. C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied. In addition, since it is a filtering process with respect to a Bayer image, it processes separately for every color only for the pixel to which the surrounding same color filter was affixed.

  The Bayer image subjected to the low-pass filter processing is performed by the aberration correction enlargement / reduction unit 1706 around the coordinate data for which the coordinate conversion processing for correcting distortion aberration and the processing for enlargement / reduction to the recorded image size are performed. By using the processing that is generated by interpolation from the pixels, the calculation is performed simultaneously by calculating the position where the target coordinate position after correction is in the image before correction. Since this process is also Bayer data, only the pixels of the same color filter are processed for each color.

  The Bayer image data subjected to the aberration correction and enlargement / reduction processing is RGB-synchronized from the Bayer data in the RGB synchronizer 1702. The RGB synchronized data is subjected to edge enhancement and noise reduction processing for the enlarged portion in the second filter 507.

  The data output from the second filter 507 is converted into a format (size, scan format) necessary for the liquid crystal monitor display in the display conversion unit 510 and written in the display frame memory in the display conversion unit 510. . The display conversion unit 510 reads the display frame memory in the display conversion unit 510 and displays an image on the liquid crystal monitor.

  The data output from the second filter 507 is converted from an RGB signal to a color space by a YC conversion unit 508 and converted to a YCbCr signal. Also, a process of thinning out the CbCr signal to 422 or the like is performed in accordance with the JPEG compression mode. The signal converted into the color space in the YC conversion unit 508 is subjected to JPEG compression processing in the image compression unit 509 and is recorded as a wide-angle captured image on an image recording apparatus such as a memory card.

  Next, in order to generate image data on the telephoto side, data is read from the frame memory 511 by rearranging the image data set by the user at the angle of view on the telephoto side in an order that can be easily processed later. The read data is low-pass filtered on the Bayer image in the first filter 1703. At this time, the central portion is strong in consideration of the influence of distortion, and filtering is performed with a coefficient suitable for reducing the image signal to a recording size. Since the peripheral portion is compressed by distortion, if the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the coefficient of the spatial filter shown in FIG. C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied. In addition, since it is a filtering process with respect to a Bayer image, it processes separately for every color only for the pixel to which the surrounding same color filter was affixed. If the central portion is not reduced, the filter coefficient is set to 1 only at the center C22 for the entire area of the image so that the low-pass filter is not applied.

  The Bayer data subjected to the low-pass filter processing is processed by the aberration correction enlargement / reduction unit 1706 around the data of the coordinates targeted by both coordinate conversion processing for correcting distortion aberration and processing for enlargement / reduction to the image size to be recorded. By using the processing that is generated by interpolation from the pixels, the calculation is performed simultaneously by calculating the position where the target coordinate position after correction is in the image before correction. Since this processing is also for Bayer data, only the pixels of the same color filter are targeted.

  The Bayer image data subjected to the aberration correction and enlargement / reduction processing is RGB-synchronized from the Bayer data in the RGB synchronizer 1702. At this time, as described above, correction is performed in consideration of the influence of distortion. The RGB synchronized data is subjected to edge enhancement and noise reduction processing for the enlarged portion in the second filter 507.

  The data output from the second filter 507 is converted into a format (size, scan format) necessary for the liquid crystal monitor display in the display conversion unit 510 and written in the display frame memory in the display conversion unit 510. . The display conversion unit 510 reads the display frame memory in the display conversion unit 510 and continues to display an image on the liquid crystal monitor. At this point, the liquid crystal monitor is switched from displaying the wide-angle image so far to displaying the telephoto image.

  The data output from the second filter 507 is converted from an RGB signal to a color space by a YC conversion unit 508 and converted to a YCbCr signal. Also, a process of thinning out the CbCr signal to 422 or the like is performed in accordance with the JPEG compression mode. The signal converted into the color space in the YC conversion unit 508 is subjected to JPEG compression processing in the image compression unit 509 and recorded as a telephoto image on an image recording apparatus such as a memory card.

  Next, a flow of processing when a subject is captured using the digital camera shown in FIG. 18 and simultaneous wide-angle and telephoto shooting is performed will be described. Note that the image correction unit 501 performs pixel defect correction and the like, and continuously saves them in the frame memory 511, and processes image data corresponding to the angle of view on the wide angle side from the frame memory 511 in the subsequent stage. The processes up to rearrangement in an easy order to read data and synchronize the data with RGB are the same as the process in the configuration of FIG.

  The RGB synchronized data is converted from an RGB signal to a color space by a YC converter 1808 and converted to a YCbCr signal. The converted YCbCr signal is low-pass filtered in the first filter 1803. At this time, the central portion is strong in consideration of the influence of distortion, and the image signal is filtered with a coefficient suitable for reducing the image signal to a size to be displayed on the liquid crystal monitor. Since the peripheral portion is compressed by distortion, if the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the coefficient of the spatial filter shown in FIG. C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied.

  The YCbCr signal that has undergone the low-pass filter processing is used as a target coordinate data in the aberration correction / enlargement / reduction unit 1806 for both coordinate conversion processing for correcting distortion aberration and processing for reducing the image size displayed on the liquid crystal monitor. Is processed by interpolating from the surrounding pixels, and by calculating the position of the target coordinate position after correction in the image before correction, it is processed at the same time . The YCbCr signal subjected to aberration correction and enlargement / reduction processing is subjected to edge enhancement and noise reduction processing in the second filter 1807 if there is a portion subjected to enlargement processing.

  The YCbCr signal output from the second filter 1807 is converted into a format (data format, size, scan format) required for the liquid crystal monitor display in the display conversion unit 1810, and the display frame in the display conversion unit 1810 is converted. Written to memory. The display conversion unit 1810 continues to display the image on the liquid crystal monitor by reading the display frame memory in the display conversion unit.

  When the user switches the angle of view on the wide-angle side with the Tele button 2205 or the Wide button 2206 for controlling the zoom on the wide-angle side shown in FIG. 22A, in correspondence with the angle of view that the user observes on the liquid crystal monitor, The angle of view of the image data read from the frame memory 511 is also changed. Along with this, since the degree of distortion and the angle of view (number of image data) output from the frame memory 511 change, the processing of the RGB synchronizer 502 and the YC converter 1808 are interlocked with this. The parameters for the first filter 1803, the aberration correction / enlargement / reduction unit 1806, and the second filter 1807 are also changed.

  Further, when the user switches the telephoto side angle of view by using the Tele button 2207 or the Wide button 2208 for controlling the zoom on the telephoto side shown in FIG. 22A, the angle of view of the image data read from the frame memory 511 is changed. First, the angle of view on the wide-angle side is continuously read. Accordingly, since the degree of distortion and the angle of view (number of image data) output from the frame memory 511 do not change, processing of the RGB synchronizer 502, YC converter 1808, first filter 1803, aberration correction / enlargement / Parameters for the reduction unit 1806 and the second filter 1807 are not changed.

  What is changed in accordance with the operation of the Tele button 2207 and the Wide button 2208 for controlling the zoom on the telephoto side shown in FIG. 22A is a frame indicating the telephoto side photographing area displayed on the liquid crystal monitor. However, when the Tele button 2207 is pressed and the telephoto is tilted to the telephoto side, the change is small. In the present embodiment, the user can also perform an operation of simultaneously switching the wide-angle side view angle and the telephoto-side view angle as described above.

  When the user determines the angle of view on the wide-angle side and the telephoto side and operates the shutter button 2201 shown in FIG. 22A, after the processing such as AE / AF is executed, the image sensor progressively or It is driven in the normal shooting mode that outputs in interlace. At this time, the image correction unit 501 executes pixel defect correction, white balance correction, and shading correction (light amount correction) for the image. At this time, correction is performed in consideration of the influence of distortion as described above.

  The image-corrected data is stored in the frame memory 511 for one frame. First, in order to generate image data on the wide angle side, data corresponding to the image data set by the user at the angle of view on the wide angle side is rearranged in an order that can be easily processed later, from the frame memory 511. .

  The read data is RGB-synchronized from the Bayer data in the RGB synchronization unit 502. At this time, as described above, correction is performed in consideration of the influence of distortion. The RGB synchronized data is converted from an RGB signal to a color space by a YC converter 1808 and converted to a YCbCr signal.

  The converted YCbCr signal is subjected to low-pass filter processing for each YCbCr image in the first filter 1803. At this time, the central portion is strong in consideration of the influence of distortion, and filtering is performed with a coefficient suitable for reducing the image signal to a recording size. Since the peripheral portion is compressed by distortion, if the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the coefficient of the spatial filter shown in FIG. C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied.

  The YCbCr signal that has undergone the low-pass filter processing is processed by the aberration correction / enlargement / reduction unit 1806 as data of coordinates targeted by both coordinate conversion processing for correcting distortion aberration and processing for enlargement / reduction to the image size to be recorded. Using the process that is generated by interpolating from surrounding pixels, the target coordinate position after correction is the pre-correction image, and it is processed at the same time by calculating where it was. .

  The YCbCr signal subjected to aberration correction and enlargement / reduction processing is subjected to edge enhancement and noise reduction processing in the second filter 1807 with respect to the enlarged portion. The YCbCr signal output from the second filter 1807 is converted into a format (data format, size, scan format) required for the liquid crystal monitor display in the display conversion unit 1810, and the display frame in the display conversion unit 1810 is converted. Written to memory. The display conversion unit 1810 reads the display frame memory in the display conversion unit 1810 and continues to display an image on the liquid crystal monitor. Also, the YCbCr signal output from the second filter 1807 is subjected to JPEG compression by the image compression unit 509 by thinning out the CbCr signal to 422 or the like in accordance with the JPEG compression mode, so that an image such as a memory card is obtained. It is recorded in the storage device as a wide-angle shot image.

  Next, in order to generate image data on the telephoto side, data is read from the frame memory 511 by rearranging the image data set by the user at the angle of view on the telephoto side in an order that can be easily processed later. The read data is RGB-synchronized from the Bayer data in the RGB synchronization unit 502. At this time, as described above, correction is performed in consideration of the influence of distortion.

  The RGB synchronized data is converted from an RGB signal to a color space by a YC converter 1808 and converted to a YCbCr signal. The YCbCr signal is low-pass filtered by the first filter 1803 for each YCbCr image. At this time, the central portion is strong in consideration of the influence of distortion, and filtering is performed with a coefficient suitable for reducing the image signal to a recording size. Since the peripheral portion is compressed by distortion, if the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the coefficient of the spatial filter shown in FIG. C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied.

  The YCbCr signal that has undergone the low-pass filter processing is processed by the aberration correction / enlargement / reduction unit 1806 with coordinate data targeted for both the coordinate change process for correcting distortion aberration and the process for enlarging / reducing the recorded image size. By using the process that is generated by interpolating from surrounding pixels, the calculation is performed simultaneously by calculating the position where the target coordinate position after correction is in the image before correction.

  The YCbCr signal subjected to aberration correction and enlargement / reduction processing is subjected to edge enhancement and noise reduction processing in the second filter 1807 with respect to the enlarged portion. The YCbCr signal output from the second filter 1807 is converted into a format (data format, size, scan format) required for the liquid crystal monitor display in the display conversion unit 1810, and the display frame in the display conversion unit 1810 is converted. Written to memory. The display conversion unit 1810 reads the display frame memory in the display conversion unit 1810 and displays an image on the liquid crystal monitor. Also, the YCbCr signal output from the second filter 1807 is subjected to JPEG compression by the image compression unit 509 by thinning out the CbCr signal to 422 or the like in accordance with the JPEG compression mode, so that an image such as a memory card is obtained. It is recorded in the storage device as a wide-angle shot image.

  Next, the flow of processing when capturing an object using the image capturing apparatus (digital camera) having the distortion aberration correcting image processing apparatus shown in FIG. 19 and performing simultaneous wide-angle and telephoto shooting will be described. The image correction unit 501 performs pixel defect correction and the like, and continuously saves them in the frame memory 511, and processes image data corresponding to the wide-angle side view angle from the frame memory 511 in the subsequent stage. The process is the same as the process in the configuration of FIG. 17 until data is read out in order, easily subjected to low-pass filter processing, aberration correction and enlargement / reduction processing, and RGB synchronization. The details are omitted.

  The RGB synchronized data is converted from an RGB signal to a color space by a YC converter 1908 and converted into a YCbCr signal. The YCbCr signal is subjected to edge enhancement and noise reduction processing in the second filter 1907 if there is an enlarged portion of the signal.

  The signal output from the second filter 1907 is converted into a format (data format, size, scan format) necessary for liquid crystal monitor display in the display conversion unit 1910, and the display frame memory in the display conversion unit 1910 Is written to. The display conversion unit 1910 reads the display frame memory in the display conversion unit 1910 and continues to display an image on the liquid crystal monitor.

  When the user switches the angle of view by tilting the switching lever 2204 for controlling the zoom shown in FIG. 22C in the state in which the angle of view on the telephoto side can be operated, the image that the user observes on the liquid crystal monitor The angle of view of the image data read from the frame memory 511 is changed corresponding to the angle. Since the degree of distortion and the angle of view (number of image data) output from the frame memory 511 are determined by the angle of view on the wide angle side, the processing of the RGB synchronizer 1702, the first filter 1703, aberration correction / enlargement / Parameters for the reduction unit 1706 and the second filter 1907 are also determined by the angle of view on the wide angle side. Therefore, it does not change with the telescopic angle of view operation, but changes according to tilting the switching lever 2204 for controlling the zoom shown in FIG. This frame changes so that it is small when the switch lever 2204 is tilted to the right to telephoto, and it is large when the switch lever 2204 is tilted to the left and moved to the wide angle side.

  Next, when the user performs an operation of pushing in the switching lever 2204 shown in FIG. 22C, the angle of view on the side that has been operated (for example, the telephoto side) is determined, and the image on the other side (for example, the wide-angle side) is determined. It becomes the mode to operate the corner. When the angle of view is switched by tilting the switching lever 2204 for controlling the zoom shown in FIG. 22C in the state in which the angle of view on the wide angle side can be operated, it corresponds to the angle of view observed by the user on the LCD monitor. Then, the angle of view of the image data read from the frame memory 511 is changed and read. Along with this, since the degree of distortion and the angle of view (number of image data) output from the frame memory 511 change, the processing of the RGB synchronizer 1702 and the first filter 1703 are interlocked with this. The parameters for the aberration correction / enlargement / reduction unit 1706 and the second filter 1907 are also changed.

  When the user also determines the angle of view on the telephoto side and operates the shutter button 2201 shown in FIG. 22 (c), the image sensor outputs all pixels in a progressive or interlaced manner after processing such as AE / AF is executed. Driven in normal shooting mode. At this time, the image correction unit 501 executes pixel defect correction, white balance correction, and shading correction (light amount correction) for the image. At this time, correction is performed in consideration of the influence of distortion as described above.

  The image-corrected data is stored in the frame memory 511 for one frame. First, in order to generate image data on the wide angle side, data corresponding to the image data set by the user at the angle of view on the wide angle side is rearranged in an order that can be easily processed later, from the frame memory 511. .

  The read Bayer image data is low-pass filtered in the first filter 1703. At this time, the central portion is strong in consideration of the influence of distortion, and filtering is performed with a coefficient suitable for reducing the image signal to a recording size. Since the peripheral portion is compressed by distortion, if the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the coefficient of the spatial filter shown in FIG. C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied. Further, since the filtering process is performed on the Bayer image, the process is performed separately for each color only for the pixels to which the surrounding same color filter is attached. If the central portion is not reduced, the filter coefficient is set to 1 only for the center C22 for the entire area of the image so that the low-pass filter is not applied.

  The Bayer image data that has undergone the low-pass filter processing is data of coordinates targeted by the aberration correction / enlargement / reduction unit 1706 for both coordinate change processing for correcting distortion aberration and processing for enlargement / reduction to the image size to be recorded. Is processed by interpolating from the surrounding pixels, and by calculating the position of the target coordinate position after correction in the image before correction, it is processed at the same time .

  The Bayer image data that has been subjected to aberration correction and enlargement / reduction processing is RGB-synchronized by an RGB synchronizer 1702. The RGB synchronized data is converted from an RGB signal to a color space by a YC converter 1908 and converted into a YCbCr signal. Also, a process of thinning out the CbCr signal to 422 or the like is performed in accordance with the JPEG compression mode. The YCbCr signal is subjected to edge enhancement and noise reduction processing in the second filter 1907 for the enlarged portion.

  The YCbCr signal output from the second filter 1907 is converted into a format (data format, size, scan format) necessary for liquid crystal monitor display in the display conversion unit 1910, and the display frame memory in the display conversion unit 1910 Is written to. Further, the display conversion unit 1910 reads the display frame memory in the display conversion unit 1910 and continues to display an image on the liquid crystal monitor. Further, the YCbCr signal output from the second filter 1907 is also input to the image compression unit 509, where JPEG compression is performed in the image compression unit 509, and recorded as a wide-angle captured image in an image storage device such as a memory card. The

  Next, in order to generate image data on the telephoto side, data is read from the frame memory 511 by rearranging the image data set by the user at the angle of view on the telephoto side in an order that can be easily processed later. The read Bayer image data is subjected to low-pass filter processing in the first filter 1703. At this time, the central portion is strong in consideration of the influence of distortion, and filtering is performed with a coefficient suitable for reducing the image signal to a recording size. Since the peripheral portion is compressed by distortion, if the peripheral portion is not reduced from the image size to be obtained and the image size of the peripheral portion, the coefficient of the spatial filter shown in FIG. C12, C13, C21, C23, C31, C32, C33) are set to 0 so that the low-pass filter is not applied. In addition, since it is a filtering process with respect to a Bayer image, it processes separately for every color only for the pixel to which the surrounding same color filter was affixed.

  The Bayer image data that has undergone the low-pass filter processing is coordinate data for which the aberration correction / enlargement / reduction unit 1706 performs both coordinate conversion processing for correcting distortion and processing for enlargement / reduction to the image size to be recorded. Is processed by interpolating from the surrounding pixels, and by calculating the position of the target coordinate position after correction in the image before correction, it is processed at the same time .

  The Bayer image data subjected to the aberration correction and enlargement / reduction processing is RGB-synchronized from the Bayer data in the RGB synchronizer 1702. The RGB synchronized data is converted from an RGB signal to a color space by a YC converter 1908 and converted into a YCbCr signal. Also, a process of thinning out the CbCr signal to 422 or the like is performed in accordance with the JPEG compression mode. The YCbCr signal is subjected to edge enhancement and noise reduction processing in the second filter 1907 for the enlarged portion.

  The YCbCr signal output from the second filter 1907 is converted into a format (data format, size, scan format) necessary for liquid crystal monitor display in the display conversion unit 1910, and the display frame memory in the display conversion unit 1910 Is written to. Further, the display conversion unit 1910 reads the display frame memory in the display conversion unit 1910 and continues to display an image on the liquid crystal monitor. At this point, the liquid crystal monitor is switched from displaying the wide-angle image so far to displaying the telephoto image. Further, the YCbCr signal output from the second filter 1907 is also input to the image compression unit 509, where JPEG compression is performed in the image compression unit 509, and recorded as a telephoto image in the image storage device 105 such as a memory card. Is done. As described above, when the wide-angle and telephoto shot images created from the same image data are stored in the image storage device 105, in order to indicate that they are created from the same image data, For example, an index may be assigned so that the same prefix is added to the file name. Thereby, it becomes possible to easily confirm the relationship between captured images.

(Second Embodiment)
FIG. 26 shows a configuration of a surveillance camera which is an image capturing apparatus according to the second embodiment of the present invention. As shown in FIG. 26, the surveillance camera according to the second embodiment of the present invention includes a camera head 2606 and a reception recording system 2613, and the camera head 2606 further includes an optical system 2601 having distortion aberration. An image sensor 2602, a processing compression device 2603, an image transmission device 2604, and a camera control device 2605. The reception system 2613 includes an image reception device 2607, a distortion aberration correction device 2608, and an image storage. A device 2609, an image display device 2610, a reception display control device 2611, and an operation unit 2612 are included.

  An optical system 2601 having distortion aberration is an optical system having distortion aberration characteristics in which a central portion is enlarged and a peripheral portion is compressed. The image sensor 2602 is a photoelectric conversion element for converting an optical image taken through the optical system 101 having distortion aberration into an electrical signal, and is, for example, a CCD or a CMOS sensor. The processing compression device 2603 corrects pixel defects in the image signal output from the image sensor 2602 or performs shading correction and then performs reversible or irreversible compression processing. The image transmission device 2604 converts data in accordance with a communication protocol and adds an error correction code in order to communicate by wire or wireless. The camera control device 2605 controls the entire camera head 2606 according to a predetermined control program.

  On the other hand, the image receiving device 2607 receives an image signal transmitted from the camera head 2606. The distortion correction device 2608 decompresses the data read from the image storage device 2609 and then converts the image data into a format that can be displayed on the image display device 2610. The image storage device 2609 records the original image data that has been compressed with the distortion received by the image receiving device 2607 as it is. The image display device 2610 is a display device such as a liquid crystal that displays image data output from the distortion correction device 2608. The reception display control device 2611 receives inputs from the operation unit 2612 for an operator of the monitoring camera system to specify a display state and a camera head state, and controls each unit in the reception recording system 2613. At the same time, a control signal is transmitted to the camera head 2606 via the image receiving device 2607. Since this embodiment relates to a surveillance camera system, the camera head 2606 is configured separately from the reception recording system 2613, and image data from a plurality of camera heads 2606 is received by one reception recording system 2613. It can also be configured to receive and simultaneously hold records.

The operation of the surveillance camera according to the second embodiment of the present invention will be described below.
An optical image picked up through the optical system 2601 having distortion is converted into an electrical signal such as a Bayer array by the image sensor 2602 and input to the processing compression device 2603. The configuration of the processing compression device 2603 is as shown in FIG. 27, and the input image signal is subjected to simple white balance correction by roughly correcting the pixel defect and RGB signals in the image correction unit 2701. The RGB synchronizer 2702 generates an RGB signal for each pixel from the Bayer array by interpolation. Then, the YC conversion unit 2703 converts the RGB signals into luminance / color difference signals, and the image compression unit 2704 executes compression processing such as JPEG.

  Information on optical characteristics for correcting distortion of the optical system, information on peripheral brightening caused by this distortion and peripheral darkening caused by the lens frame, and white processing including simple white balance processing Information for accurately correcting the balance is recorded in the distortion aberration information storage unit 2705 in the form of a table created by pairing the characteristic value or parameter itself or an index for specifying the characteristic value or parameter. Has been. The processing compression apparatus 2603 records the characteristic value, parameter, or index together in the header portion of the JPEG file. When the index is recorded in the header of the JPEG file, the table is transmitted to the reception recording system 2613 as another file in advance, or is transmitted as an associated file together with the JPEG file. The transmission of information necessary for correction by this index has an advantage that it is not necessary to increase the capacity of the JPEG file.

  Further, in the present embodiment, a configuration is described in which accurate white balance correction and shading correction are performed on the reception recording system side. However, accurate white balance correction and shading correction are performed in the processing compression device 2603 on the camera head side. The image correction unit 2701 may be configured in advance. In addition, processing such as encryption may be added before and after compression.

  In this way, the image signal that has been executed up to the compression process in the processing compression apparatus 2603 is sent to the image transmission apparatus 2604, and in order to perform wired or wireless communication, the data is converted in accordance with the communication protocol, It is transmitted after adding an error correction code. The image transmission device 2604 also receives a camera control signal from the reception recording system and has a role of transmitting this information to the camera control device 2605. Upon receiving this signal, the camera control device 2605 performs control of the shutter and diaphragm of the optical system, control of the drive mode of the image sensor 2602, parameter setting of the processing compression device 2603, and the like. Further, the image signal compressed by the camera head is a signal that is not subjected to correction of distortion and is compressed while having distortion.

  As described above, the image signal transmitted from the camera head 2606 is received by the reception recording system 2613 and sent to the image receiving device 2607. The image receiving device 2607 performs processing such as demodulation processing and error correction, and then sends the processed image to the distortion correction device 2608 and the image storage device 2609.

  The image storage device 2609 is a large-capacity storage device such as a semiconductor memory, a magnetic disk device, or a magneto-optical disk device, and is specified such as when a moving image sent from the camera head 2606 or every certain time or when a screen changes. A still image is recorded every time the above condition is satisfied. This recorded image is compressed with the distortion of the optical system, and as described above, the distortion of the lens is corrected, the correct white balance correction, and the shading are corrected. Information is also added in the form of characteristic values, parameters, or indexes, and stored together. In the case of an index, a table for associating with characteristic values and parameters is also stored.

  FIG. 28 is a block diagram of the distortion correction apparatus 2608 shown in FIG. As described above, the image signal sent from the image receiving device 2607 is decompressed by JPEG compression by the decompression unit 2801, and accurate white balance correction and shading correction are performed by the image correction unit 2806. The image-corrected data is subjected to low-pass filter processing by the first filter 2802 on the reduced portion in the center of the image. The low-pass filtered data is resized to a size suitable for correction and display of distortion by the aberration correction / enlargement / reduction unit 2803, and edge enhancement and noise of the peripheral part are performed by the second filter 2804. The reduction processing is performed, and the display conversion unit 2805 performs interlace processing, encoding to NTSC, and the like.

  At this time, the data for accurate white balance correction and shading correction stored together with the image data in the image storage device 2609 is held in the distortion aberration information holding unit 2807, and based on this data, The image correction unit 2806 performs white balance correction and shading correction. Further, the shading correction processing may be performed using a reference table memory as shown in FIGS. 7A and 7B, or may be performed by an arithmetic circuit. After performing this correction, the corrected image data is sent to the first filter 2802, and a marker indicating that the correction has been completed is recorded in the header portion of the file and recompression processing is performed. Thereafter, it is re-recorded in the image storage device 2609. The re-recorded image data is appropriately read out and controlled so as to be directly input to the first filter 2802 via the expansion unit 2801. This is because the processing by the image correction unit 2806 has already been performed, and the previous control is performed based on the content of the marker previously recorded in the header portion.

  Similarly to the first embodiment, the first filter 2802 is a low-pass filter for the center of the image indicated by 1001 in FIG. 10, and FIG. 11 shows a configuration example thereof. The aberration correction enlargement / reduction unit 2803 has a structure in which FIG. 12 and FIG. 14 are integrated. In the corresponding coordinate calculation unit 1401 shown in FIG. 14, the coordinate position in the broken grid 1519 in FIG. Are converted to coordinate positions in the solid line grid 1518, and the corresponding coordinate values in the original image before correction of optical distortion and lateral chromatic aberration are calculated together.

  The correction pixel value may be calculated by 16-point cubic interpolation or 4-point linear interpolation. The second filter 2804 is an edge enhancement filter and noise reduction filter for the peripheral portion of the image indicated by reference numeral 1002 in FIG. 10, and FIG. 11 shows a configuration example thereof.

  The reception recording system 2613 shown in FIG. 26 also includes an operation unit 2612, and sends a signal for operating the camera to the camera head 2606 through the reception display control device 2611 and the image reception device 2607 by the operation of the surveillance camera operator. By controlling each part in the reception recording system 2613 through the reception display control device 2611, the display angle of view can be changed and a past recorded image reproduction operation can be performed.

  Further, as described above, the image storage device 2609 stores the original image before correcting the distortion aberration and the optical system information necessary for the correction together. Even if the angle of view is changed by displaying the image, the image quality is not deteriorated unlike the normal electronic zoom.

  Also, information on lens distortion and lens frame is stored in an information storage means such as a ROM provided in the lens frame together with focal length and F value, and this information is read from the camera body to store distortion aberration information. By being configured to be held in the unit 2705 and configured to perform coordinate conversion for filtering, shading correction, and distortion correction, even when applied to an interchangeable lens camera, appropriate distortion according to the interchangeable lens Aberration correction can be performed.

  As described above, the embodiments of the present invention have been described in detail with reference to the drawings. However, the specific configuration is not limited to these embodiments, and includes design changes and the like without departing from the gist of the present invention. .

1 is a configuration diagram of a digital camera image capturing device according to a first embodiment. FIG. It is the figure which illustrated the optical image by the presence or absence of the distortion aberration in an optical system. It is a figure which illustrates an original image and the image imaged with the optical system which has a distortion aberration. It is a figure which illustrates the structure of the optical system which has a distortion aberration. It is a figure which shows the structural example of a distortion aberration correction image processing apparatus. It is the figure which showed the influence of the brightening of the peripheral part by a distortion aberration, and the dimming by a lens frame. It is a figure which illustrates the structure of an image correction circuit. It is the figure which showed the mode of the rearrangement of the data by a frame memory. It is a figure explaining the process in the case of performing RGB synchronization with respect to the image imaged with the optical system which has a distortion aberration. It is a figure which shows the area | region which performs a filter process by the 1st filter circuit and the 2nd filter circuit. It is a figure which shows the structural example of a filter circuit. It is a figure which shows the structural example of an aberration correction part. It is a figure for demonstrating the distortion aberration correction process in an aberration correction part. It is a figure which shows the structural example of an expansion / reduction part. It is a figure for demonstrating the expansion / reduction process in an expansion / reduction part. It is a figure which shows the structural example of a distortion aberration correction image processing apparatus. It is a figure which shows the structural example of a distortion aberration correction image processing apparatus. It is a figure which shows the structural example of a distortion aberration correction image processing apparatus. It is a figure which shows the structural example of a distortion aberration correction image processing apparatus. It is a figure which shows switching of imaging | photography mode. It is a figure which shows the mode transition of each imaging | photography mode. It is a figure which shows the structural example of an operation part. It is a figure which shows the operation | movement transition at the time of operating T / W switch button etc. It is a figure explaining the process in the case of reading the image imaged with the optical system which has a distortion aberration with a high frame rate. It is a figure for demonstrating a zoom function. It is a block diagram of the surveillance camera which concerns on 2nd Embodiment. It is a figure which shows the structural example of the process compression apparatus which concerns on 2nd Embodiment. It is a figure which shows the structural example of the distortion correction apparatus which concerns on 2nd Embodiment. It is a figure which shows a prior art example.

Explanation of symbols

101, 2601 Optical system with distortion 102, 2602 Image sensor 103 Distortion correction image processing device 104, 2610 Image display device 105, 2609 Image storage device 106 Control device 107, 2612 Operation unit 401, 402 Cylindrical lens 501, 2701, 2806 Image correction unit 502, 1702, 2702 RGB synchronization unit 503, 1703, 1803, 2802 First filter 504 Aberration correction unit 505 Enlargement / reduction unit 506, 1706, 1806, 2803 Aberration correction / enlargement / reduction unit 507 , 1807, 1907, 2804 Second filter 508, 1808, 1908, 2703 YC conversion unit 509 Image compression unit 510, 1810, 2805 Display conversion unit 511 Frame memory 1201, 1401 Corresponding coordinate calculation unit 1202, 1402 IF image memory 1203, 1403 Neighborhood image data reading unit 1204, 1404 Correction image value calculation unit 2201 Shutter button 2202 Mode button 2203 T / W switching button 2204 Switching lever 2205 Tele1 button 2206 Wide1 button 2207 Tele2 button 2208 Wide2 button 2209 Tele button 2210 Wide button 2211 LED
2603 Processing compression device 2604 Image transmission device 2605 Camera control device 2606 Camera head 2607 Image reception device 2608 Distortion aberration correction device 2611 Reception display control device 2613 Reception system 2704 Compression unit 2705 Distortion aberration information storage unit 2801 Expansion unit 2807 Distortion aberration information holding unit

Claims (15)

An optical system with distortion characteristics that expand the central part and compress the peripheral part,
An image sensor that converts a subject image formed by the optical system into image data and outputs an original image;
An angle-of-view setting means for setting a first angle of view and a second angle of view narrower than the first angle of view for the same original image data;
A distortion correction circuit that corrects distortion of the original image data of the set angle of view and outputs corrected image data;
An image display device that displays an image based on the corrected image data of the first angle of view in which the distortion is corrected;
And an image recording apparatus for recording the corrected image data of the distortion the aberration is corrected first and second field angle,
Display filter means for performing different filtering for each region in accordance with the first angle of view, distortion characteristics, and the size of an image to be displayed;
First recording filter means for performing different filtering for each region in accordance with the first angle of view, distortion characteristics, and the size of an image to be recorded;
Second recording filter means for performing different filtering for each region in accordance with the second angle of view, distortion characteristics, and the size of the image to be recorded;
An image pickup apparatus.   The image capturing apparatus according to claim 1, wherein the angle of view setting unit can individually set the size of the first angle of view and the size of the second angle of view.   The field angle setting unit includes a first field angle setting unit that sets the size of the first field angle, and a second field angle setting unit that sets the size of the second field angle. The image capturing apparatus according to claim 2, wherein   The angle-of-view setting unit controls an imaging angle-of-view switching unit that switches the size of the angle of view between the first angle of view and the second angle of view, and a display related to a switching result by the imaging angle-of-view switching unit. The image capturing apparatus according to claim 2, further comprising an angle-of-view switching display control unit.   The image capturing apparatus according to claim 4, wherein the angle-of-view switching display control unit displays the display related to the switching result so as to be superimposed on the display of the original image data to be set as the angle of view.   The image capturing apparatus according to claim 4, wherein the angle-of-view switching display control unit displays a display related to the switching result at a position different from original image data that is a target of angle-of-view setting. A first imaging mode for imaging only the first angle of view , a second imaging mode for simultaneously imaging a plurality of angles of view of the first and second angles of view , and an image of the first angle of view the comprises a plurality imaging modes including a third imaging mode, for imaging with the second field angle to display to the image display apparatus, an imaging mode switching and the like buttons or a lever for switching the imaging mode Means, an imaging mode setting means for setting an imaging mode according to the state of the imaging mode switching means, and a mode switching display control means for controlling display according to the output of the imaging mode setting means ,
In accordance with the output of the first view angle setting means and the control of the imaging mode switching display control means, the settings of the display filter means and the first recording filter means are changed, and the second view angle is changed. The image capturing apparatus according to claim 1 , wherein the setting of the second recording filter unit is changed according to the output of the setting unit and the control of the imaging mode switching display control unit .   The image capturing apparatus according to claim 7, wherein the mode switching display control unit displays a display related to the switching result so as to be superimposed on a display of original image data that is a target for setting an angle of view.   The image capturing apparatus according to claim 7, wherein the mode switching display control unit displays the display related to the switching result at a position different from the original image data that is a target for setting an angle of view.   The mode switching display control unit displays the target original image in color in the first imaging mode, and displays the target original image in color in the second imaging mode. The frame of the second angle of view is displayed, and in the third imaging mode, the frame of the second angle of view is displayed, the inside of the frame is displayed in color, and the outside of the frame is displayed in color. The image pickup apparatus according to claim 7, wherein the display is different from the display.   The image capturing according to claim 10, wherein the mode switching display control unit is configured to display the outside of the frame by a single color including black and white display, a display by changing luminance, a display by enhancing or fainting a color tone. apparatus. Display control for sequentially switching and displaying the corrected image data of the first angle of view and the corrected image data of the second angle of view related to the same original image data recorded in the image recording apparatus at regular intervals. The image capturing apparatus according to claim 1, further comprising a circuit. Either one of the corrected image data of the first angle of view or the corrected image data of the second angle of view related to the same original image data recorded in the image recording device is displayed for a certain period of time. And further comprising a display control circuit for displaying the other corrected image data related to the same original image data, which is different from the displayed corrected image data, when switching by the imaging angle-of-view switching unit is detected. The image capturing apparatus according to claim 4. The image recording apparatus, and characterized in that to record the same original index to indicate that it has been created from the image has been added together, the correction data of the second field angle and the correction image data of the first field angle The image pickup device according to claim 1. The display filter means, the first recording filter means, and the second recording filter means share the same circuit, operate at a display setting when displaying a through image, and release a shutter. The image pickup apparatus according to claim 7, wherein the image pickup apparatus is sequentially set and executed for the first recording and the second recording according to the image pickup mode.

Images