JP5948167B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus that uses a plurality of photoelectric conversion layers having a plurality of light receiving portions and processes image signals read from the respective light receiving portions of the photoelectric conversion layers.

  When image signal processing is performed by converting a layered image signal into a Bayer image signal, it is necessary to interpolate color information from surrounding pixels for each pixel during the image signal processing. There is a problem that false colors are generated when If a low-pass filter or the like is used to suppress the false color generated in the interpolation calculation process, it becomes possible to suppress the false color, but a new problem that the resolution is lowered occurs.

  Therefore, Patent Document 1 and Patent Document 2 disclose inventions related to image signal processing for leaving a sense of resolution.

  Patent Document 1 generates a luminance signal from an electrical signal detected by an upper imaging device in a composite imaging device composed of two upper and lower imaging devices, and generates a color signal from the electrical signal detected by a lower imaging device. . An invention for obtaining an image with a sense of resolution by synthesizing these high-accuracy luminance signals and color signals is disclosed.

  Patent Document 2 sets a predetermined number of calculated directions and a plurality of direction identification numbers for a target pixel being processed, and corresponds to each of four pixel level values including the target pixel corresponding to each set direction identification number. Multiply by the coefficient to be. Among them, the representative value closest to the level value of the target pixel is selected, and this representative value is replaced with the level value of the target pixel. An invention is disclosed in which the above process is performed on all pixels to obtain an image from which noise has been removed while suppressing a decrease in resolution.

JP 2011-238773 A

Japanese Patent No. 4051196

  In the invention described in Patent Document 1, an image sensor having a narrow pixel pitch on the upper side and an image sensor having a wide pixel pitch on the lower side are overlapped in a direction perpendicular to the light receiving surface. The light receiving part and the light receiving part are arranged so as to overlap each other. In addition, a color filter is disposed between the upper image sensor and the lower image sensor, and the structure is very complicated. Therefore, it cannot be applied to the multilayer solid-state imaging device. Moreover, the imaging device described in Patent Document 1 has a problem that the yield in a factory is poor. That is, there is a problem that the cost increases.

  In addition, when the invention described in Patent Document 2 is applied to a stacked solid-state image sensor, the invention described in Patent Document 2 is based on a Bayer-type image signal. An image signal to be detected (hereinafter referred to as a laminated image signal) is converted into an image signal (hereinafter referred to as a Bayer image signal) to be detected by a Bayer solid-state imaging device, and then applied. When the layered image signal is converted to the Bayer image signal, false colors and noise are generated when the interpolation calculation process is performed in the subsequent image processing. In addition, when the image signal is converted into a Bayer image signal, the green (G) luminance signal having a large influence on the resolution of the image signal is reduced. Therefore, there is a problem that the image to be acquired becomes an image with a reduced resolution.

  Furthermore, as a problem peculiar to the multilayer solid-state imaging device, when the number of pixels of the multilayer solid-state imaging device and the Bayer solid-state imaging device is the same, the multilayer solid-state imaging device can detect three colors with one pixel. Therefore, the data amount of the stacked image signal is larger than that of the Bayer image signal. Therefore, there is a problem that it takes time for image signal processing.

  In addition, since the arrangement of the light receiving units is different between the stacked solid-state imaging device in which three light receiving units are arranged in one pixel and the Bayer type solid-state imaging device in which one light receiving unit is arranged in one pixel, the number of pixels is the same. Even in the case of an image sensor, the output order of the image signals sent from the image sensor differs between the stacked image signal detected by the image sensor and the Bayer image signal. Therefore, the Bayer type image signal processing circuit that is widely distributed in the world cannot be diverted to the stacked type image signal processing circuit. Therefore, since it is necessary to manufacture an image signal processing circuit for laminated image signal processing, there is a problem that costs increase.

  The present invention has been made in view of such a situation, and suppresses false color and noise generated when converting a layered image signal into a Bayer image signal, while reducing the amount of data. An object of the present invention is to provide an imaging device that provides an image with a sense of image.

A first invention for solving the above-described problem is an image pickup device, which is a stacked type detected using a stacked solid-state image pickup device in which at least three photoelectric conversion layers are stacked on the top of a substrate per pixel. In an imaging apparatus for processing the image signal, a luminance signal generating means for generating a luminance signal from the laminated image signal, a low-pass filter means for smoothing the laminated image signal, and a contour signal for generating a contour signal from the luminance signal An image signal conversion means for converting an image signal into a Bayer image signal by performing a thinning process on the laminated image signal to which the generation means and the smoothing operation are applied; a second luminance signal and a color signal from the Bayer image signal; YCbCr conversion means for generating the luminance signal synthesis means for synthesizing the contour signal and the second luminance signal, and YCbCr for synthesizing the color signal and the luminance signal synthesized by the luminance signal synthesis means. Characterized in that it comprises a forming unit.

  In addition, a second invention for solving the above-described problem is an image sensor, which is detected using a stacked solid-state image sensor in which at least three photoelectric conversion layers are stacked on the top of the substrate per pixel. In an imaging apparatus for processing a laminated image signal, a luminance signal generating means for generating a luminance signal from the laminated image signal, a contour signal generating means for generating a contour signal from the luminance signal, and a binning process for the laminated image signal Then, the image signal conversion means for rearranging the binned image signal to become a Bayer type image signal, the YCbCr conversion means for generating the second luminance signal and the color signal from the Bayer type image signal, the contour signal, and the second luminance A luminance signal synthesizing unit that synthesizes the signal; and a YCbCr synthesizing unit that synthesizes the color signal and the luminance signal synthesized by the luminance signal synthesizing unit.

  The present invention relates to a conventional stacked solid-state imaging device and an image signal processing circuit for processing a Bayer image signal to convert a stacked image signal into a Bayer image signal and perform image signal processing. Therefore, there is no increase in costs such as a new cost due to a decrease in yield.

  In addition, the present invention applies a low-pass filter in the state of the stacked image signal before conversion to the Bayer image signal when converting the stacked image signal to the Bayer image signal. It is possible to effectively suppress false colors and noise in the acquired image. This is because the stacked image signal has a color signal also in adjacent pixels, and therefore, even if a low-pass filter is applied, it is unlikely that the image signal is far from the original color information. The binning process is the same. Focusing on the amount of data, the color signal is converted from a layered color signal to a Bayer type color signal in both thinning and binning methods using a low-pass filter. Can be reduced. As for the luminance signal, a new contour signal is generated from a layered luminance signal with high resolution before high-frequency components are removed by a low-pass filter or binning processing, and the generated contour signal and Bayer-type luminance signal are generated. By combining, a new luminance signal is generated. In the generated luminance signal, the contour portion is emphasized while reducing the data amount. Therefore, the acquired image is an image that leaves a sense of resolution while reducing the amount of data.

  In addition, the luminance signal of the present invention can prevent luminance noise generated in the luminance signal by not performing interpolation processing. An image generated using the above color signal and luminance signal is an image in which a decrease in resolution is suppressed. In addition, by reducing the data amount of the luminance signal, it is possible to reduce the time spent on image signal processing.

  In addition, the present invention makes it possible to use an image signal processing circuit for processing a Bayer image signal by converting a stacked image signal into a Bayer image signal. Therefore, since it is not necessary to manufacture an image signal processing circuit for a laminated image signal, manufacturing costs can be reduced.

It is the block diagram which showed the structure of the imaging device to which the image processing method which concerns on this invention is applied. 3 is a flowchart of an image processing method according to the first embodiment of the present invention. It is a flowchart of the image processing method which concerns on 2nd Example of this invention. It is a conceptual diagram explaining the concept of the binning process of step # 23 of 2nd Example of this invention.

  The best mode for carrying out the present invention will be described below with reference to the accompanying drawings. In addition, this invention is not limited by this embodiment.

  FIG. 1 is a configuration diagram showing a configuration of an imaging apparatus to which an image processing method according to the present invention is applied.

  Reference numeral 100 denotes an imaging apparatus. The imaging apparatus 100 according to the present embodiment is a digital camera, captures a subject, and stores the captured image data. The imaging apparatus 100 has a configuration in which the optical system 200 can be replaced.

  Reference numeral 101 denotes a stacked solid-state imaging device. The stacked solid-state imaging device 101 is a CMOS or CCD that converts light detected through the optical system 200 into an image signal.

  Here, the image signal will be described.

  An image signal detected by the multilayer solid-state imaging device is defined as a multilayer image signal. An image signal detected by the Bayer type solid-state imaging device is a Bayer type image signal. The solid-state image sensor of the present invention is a stacked image sensor. Therefore, the image signal detected by the multilayer solid-state imaging device 101 is a multilayer image signal. The laminated image signal is converted into a Bayer image signal from the laminated image signal by performing signal processing in the image signal conversion unit 1023.

  Further, among image signals, a signal having data relating to color is referred to as a color signal, and a signal having data relating to luminance is referred to as a luminance signal.

  In this embodiment, the stacked solid-state imaging device 101 uses a stacked solid-state imaging device in which three photoelectric conversion layers are stacked in one pixel. However, the photoelectric conversion layer stacked on one pixel of the stacked solid-state imaging device 101 is not limited to three layers, and may be four layers, five layers, or more.

  Reference numeral 102 denotes an FPGA. The FPGA 102 constitutes a luminance signal generation unit 1021, a digital low-pass filter unit 1022, and an image signal conversion unit 1023 by internal circuits. An ASIC or the like can be used as an intermediate device instead of the FPGA 102.

  Reference numeral 103 denotes a CPU. The CPU 103 includes a signal processing unit 1031, a YCbCr conversion unit 1032, and a color signal noise removal unit 1033 inside. Specifically, the multilayer solid-state imaging element 101 detects a multilayer image signal, and image signal processing is applied to the detected multilayer image signal by the FPGA 102. The signal processing unit 1031 performs various image processing such as gamma correction and noise removal on the image signal converted into the Bayer type image signal, and the YCbCr conversion unit 1032 performs the Bayer type image signal after the various image processing. YCbCr conversion is performed. The color signal noise removing unit 1033 removes noise from the CbCr signal. Further, various devices such as an AE mechanism and an AF mechanism (not shown) are also controlled.

  Reference numeral 104 denotes a DSP. The DSP 104 has a CPU 103 inside, and performs signal processing on the luminance signal and the color signal. The DSP 104 includes a contour signal generation unit 1041, a luminance signal synthesis unit 1042, and a YCbCr synthesis unit 1043 inside.

  Reference numeral 105 denotes a first SDRAM. The first SDRAM 105 is connected to the DSP 104.

  Reference numeral 106 denotes an external storage device. The external storage device 106 is a medium for storing captured images. Examples of the external storage device 106 are represented by, for example, an SD memory card (registered trademark), a multimedia card (registered trademark), an xD picture card (registered trademark), and a smart media (registered trademark) that are detachable from the imaging device. Various recording media such as a semiconductor memory card, a portable small hard disk, a magnetic disk, and a magneto-optical disk can be used.

  Reference numeral 107 denotes an external display device. The external display device 107 can display an image immediately after shooting, an image read from the external storage device 106, and the like. The external display device 107 also displays various menu screens for manual setting such as camera operation mode, white balance, image pixel count, and sensitivity, and allows user settings that can be manually set according to user operations. Display the screen for the face. As the external display device 107, for example, liquid crystal or organic EL can be used.

  Reference numeral 200 denotes an exchangeable optical system. The optical system 200 includes a lens CPU and a diaphragm (not shown) in addition to the lens optical system.

  Next, a characteristic configuration of the present invention among the above configurations will be described.

  Reference numeral 1021 denotes a luminance signal generation unit. The luminance signal generation unit 1021 generates a luminance signal from the image signal sent from the stacked solid-state imaging device 101. Note that the luminance signal generation unit 1021 corresponds to a luminance signal generation unit.

  Reference numeral 1022 denotes a digital low-pass filter unit. The digital low-pass filter unit 1022 applies a digital low-pass filter to the image signal transmitted from the multilayer solid-state imaging device 101 via the luminance signal generation unit 1021. This is for removing high-frequency components of the image signal that appear as false colors when interpolation processing is performed in a later process. The digital low-pass filter unit 1022 corresponds to low-pass filter means.

  The low-pass filter used in the digital low-pass filter unit 1022 removes high-frequency components of the image signal by applying a Gaussian filter to the image signal. As the digital low-pass filter of the present invention, in addition to a Gaussian filter, it is also possible to use a generally known filter such as a smoothing filter, a median filter whose edges are easily preserved and whose outline is difficult to blur, and an epsilon filter. .

  Reference numeral 1023 denotes an image signal converter. The image signal converter 1023 converts the stacked image signal into a Bayer image signal. Note that the image signal conversion unit 1023 corresponds to an image signal conversion unit.

  Reference numeral 1031 denotes a signal processing unit. The signal processing unit 1031 performs various image signal processes such as white balance correction, demosaicing, OB subtraction, noise reduction, gamma correction, color reproduction, and edge enhancement.

  Reference numeral 1032 denotes a YCbCr converter. The YCbCr converter 1032 converts the image signal sent from the signal processor 1031 into a YCbCr luminance signal and color signal. The YCbCr conversion unit 1032 corresponds to YCbCr conversion means.

  Reference numeral 1033 denotes a color signal noise removing unit. The color signal noise removal unit 1033 removes noise from the color signal of the image signal converted from RGB to YCbCr by the YCbCr conversion unit 1032.

  Reference numeral 1041 denotes a contour signal generator. The contour signal generation unit 1041 generates a contour signal from the luminance signal generated by the luminance signal generation unit 1021. A method for generating the contour signal will be described later. The contour signal generation unit 1041 corresponds to a contour signal generation unit.

  Reference numeral 1042 denotes a luminance signal synthesis unit. The luminance signal synthesis unit 1042 synthesizes the contour signal processed by the contour signal generation unit 1041 and the luminance signal of the image signal converted from RGB to YCbCr by the YCbCr conversion unit 1032. The synthesis method will be described later. Note that the luminance signal synthesis unit 1042 corresponds to a luminance signal synthesis unit.

  Reference numeral 1043 denotes a YCbCr combining unit. The YCbCr combining unit 1043 combines the luminance signal generated by the luminance signal combining unit 1042 and the color signal from which noise has been removed by the color signal noise removing unit 1033. Note that. The YCbCr synthesis unit 1043 corresponds to YCbCr synthesis means.

  A character string used for a luminance signal in the present invention will be described. Let the luminance signal generated by the luminance signal generation unit 1021 be a luminance signal (Y). The luminance signal of the image signal converted into YCbCr by the YCbCr conversion unit 1032 is defined as a luminance signal (Y ′). Finally, a luminance signal newly generated by synthesizing the luminance signal (Y ′) and the contour signal in the luminance signal synthesis unit 1042 is defined as a luminance signal (Y ″).

  Next, image signal processing performed by the imaging apparatus according to the present invention will be described with reference to the flowchart of FIG.

In step # 1, the luminance signal generation unit 1021 generates a luminance signal (Y). When the photographing operation is performed, the photographed image signal is output from the stacked solid-state imaging device 101 and sent to the luminance signal generation unit 1021. The luminance signal generation unit 1021 generates a luminance signal (Y) from the sent image signal. The sent image signal is sent to the digital low-pass filter unit 1022. Since the image pickup device in the present invention is the stacked solid-state image pickup device 101, the image signal output is a color signal and a luminance signal for each color of red (R), green (G), and blue (B) for each pixel. have. The luminance signal generation unit 1021 generates a luminance signal from the signals of each color, and generates a new luminance signal (Y) that is combined into one luminance signal for each pixel using Expression 1 below. The newly generated luminance signal (Y) is sent to the contour signal generation unit 1041.
(Formula 1) Y = k1 * R + k2 * G + k3 * B
(Y: generated luminance signal, k1, k2, k3: coefficient)

  In step # 2, the contour signal generation unit 1041 generates a contour signal from the luminance signal (Y) sent from the luminance signal generation unit 1021. The contour signal generation method removes noise from the luminance signal (Y) sent from the luminance signal generation unit 1021. Next, a contour signal is generated by detecting a contour from the luminance signal (Y) from which noise has been removed. As a method of detecting a contour, there is a method of generating a contour signal by applying a contour detection filter such as a Laplacian filter or a Sobel filter. Or you may produce | generate an outline signal from the difference of the signal after application of a low-pass filter, and the luminance signal (Y) before application like an unsharp mask. The generated contour signal is sent to the luminance signal synthesis unit 1042.

  In step # 3, the digital low-pass filter unit 1022 applies a low-pass filter to the image signal transmitted from the luminance signal generation unit 1021. The image signal to which the low-pass filter is applied is sent to the image signal conversion unit 1023.

  In step # 4, the image signal conversion unit 1023 converts the image signal detected by the stacked solid-state imaging device 101 from a stacked image signal to a Bayer image signal. The conversion method performs thinning readout. A specific method of conversion will be described. In the stacked image signal, there are image signals of red (R), green (G), and blue (B) in each pixel. Therefore, the red (R) signal is read from the pixels in the odd rows and the odd columns, the green (G) signal is read from the pixels in the odd rows and the even columns, and the green (G) signal is read from the even rows and the odd columns. The even number column reads the blue (B) signal. By performing the reading method described above for all the pixels, the stacked image signal detected by the stacked solid-state imaging device 101 can be converted into a Bayer image signal. Note that. Although a general Bayer pattern is shown as an example of a color pattern read from each pixel, it is not limited to this color pattern.

  In step # 5, the signal processing unit 1031 performs various image signal processing such as gamma correction and noise removal on the image signal converted into the Bayer image signal by the image signal conversion unit 1023.

  In step # 6, the YCbCr conversion unit 1032 converts the image signal signal-processed by the signal processing unit 1031 into YCbCr.

  In step # 7, the luminance signal synthesizer 1042 performs the luminance signal (Y ′) among the image signals converted into YCbCr in step # 6 and the contour processed by the contour signal generator 1041 in step # 2. A luminance signal (Y ″) is newly generated from the signal.

  Next, a method for generating the luminance signal (Y ″) will be described.

  The luminance signal (Y ″) synthesized by the luminance signal synthesis unit adds the contour signal generated by the contour signal generation unit 1041 to the luminance signal (Y ′) of the image signal converted by the YCbCr conversion unit 1032. To be generated.

  Since the newly generated luminance signal (Y ″) is obtained by adding the contour signal to the luminance signal (Y ′) of the image signal converted by the YCbCr conversion unit 1032, the portion where the luminance difference such as the contour is generated is emphasized. It becomes a luminance signal (Y "). Therefore, an image using this luminance signal (Y ″) is an image with a sense of resolution.

  In step # 8, the color signal noise removing unit 1033 performs signal processing on the color signal of the image signal converted into YCbCr in step # 6 to remove noise.

  In step # 9, the YCbCr combining unit 1043 generates a new image signal by combining the contour signal (Y ") generated in step # 7 and the color signal subjected to signal processing in step # 8. To do.

  In step # 10, the SDRAM 105 temporarily stores the image signal generated in step # 9. The external display device 107 reads out from the SDRAM 105 and displays it as a preview image. The external storage device 106 reads an image signal from the SDRAM 105 and stores it as a captured image.

  By performing the above steps, a layered image signal is converted to generate a Bayer image signal, and an image with a sense of resolution is recorded while reducing pattern noise generated during image signal processing. It becomes possible.

  The reason why it is possible to record an image with a sense of resolution while reducing pattern noise by performing the first embodiment will be described.

  First, pattern noise will be described.

  It has already been mentioned that false color is likely to occur during the subsequent interpolation calculation process when a layered image signal captured using a layered solid-state image sensor is simply thinned and converted to a Bayer type image signal. It was. In addition, the image signal value of the adjacent pixel greatly changes in the contour portion. Therefore, when the interpolation calculation process is performed, mosaic noise is noticeably generated in the contour portion. This is pattern noise.

  Therefore, in order to reduce this pattern noise, the present invention applies a low-pass filter to the laminated image signal in the digital low-pass filter unit 1022 after generating a luminance signal. This is because a high-frequency component of the image signal value is removed by applying a low-pass filter, the change of the image signal value is smoothed, and false color and pattern noise generated during the interpolation calculation process are suppressed. However, since the change in the image signal value is smoothed by applying a low-pass filter, the resolution is lowered. As a result of applying the low-pass filter to the image signal, the outline portion eliminates mosaic noise, that is, pattern noise, but causes a reduction in resolution.

  Therefore, the present invention generates a luminance signal (Y ″) in which the contour portion is emphasized by adding the contour signal to the luminance signal (Y ′), and the image in which the contour portion is blurred due to a decrease in resolution. By replacing the luminance signal (Y ′) of the signal with the luminance signal (Y ″), the blurred outline is emphasized. As a result, an image with a sense of resolution can be recorded.

  Next, the reason why the feeling of resolution can be left will be described.

  In the image signal processing, conversion from RGB to YCbCr is performed to obtain a luminance signal. The luminance signal includes only black and white brightness information. However, the delicateness of the luminance signal image, which is an image of only the luminance signal, affects the impression of resolution of the entire image. Therefore, in order to obtain an image with a sense of resolution, a delicate luminance signal image is required.

  On the other hand, it is said that the human eye has a visual sensitivity to green (G) more than other colors. Therefore, it can be said that the green (G) luminance signal (Y) has a high ratio in the impression of resolution of the image.

  The image sensor used in the present invention is a stacked solid-state image sensor, and an image signal to be detected is a stacked image signal. Since the stacked image signal includes a green (G) light receiving portion in each pixel, it is easy to obtain a green (G) luminance signal from each pixel, unlike the Bayer image signal. .

  Therefore, a new luminance signal generated using more green (G) luminance signals can be felt to have higher resolution to the human eye than a luminance signal generated from a Bayer image signal. Needless to say.

  Therefore, the contour signal generated using this luminance signal also has a high resolution. Therefore, the luminance signal obtained by adding the contour signal to the luminance signal generated from the Bayer-type image signal is a luminance signal that has a sense of resolution more than the luminance signal generated from the Bayer-type image signal. Needless to say. Therefore, it goes without saying that an image generated using a luminance signal that leaves a sense of resolution becomes an image that leaves a sense of resolution.

  From the above, in the present invention, it is possible to acquire an image with a sense of resolution while reducing pattern noise.

  Next, the second embodiment will be described with reference to the flowchart of FIG.

  Unlike the first embodiment, the second embodiment performs binning processing.

  In step # 21, the luminance signal generation unit 1021 generates a luminance signal (Y). Since the method of generating the luminance signal (Y) is the same as that in the first embodiment, the description is omitted. The image signal output from the stacked solid-state imaging device 101 is sent to the image signal conversion unit 1023.

  In step # 22, the contour signal generation unit 1041 performs signal processing on the luminance signal (Y) generated by the luminance signal generation unit 1021 in step # 21. Specifically, the contour signal is generated by removing low frequency components from the luminance signal. The generated contour signal is sent to the luminance signal synthesis unit 1042. (Same as step # 2 in the first embodiment.)

  In step # 23, the image signal conversion unit 1023 detects the layered solid-state imaging device 101 and converts the layered image signal sent from the luminance signal generation unit 1021 into a Bayer type image signal. The conversion method is performed using a binning process. Specifically, a 2 × 2 binning process is performed to generate one large pixel from four adjacent pixels (FIG. 4A → FIG. 4B). When one large pixel is generated from the four pixels, each large pixel has four pixels of red (R), green (G), and blue (B). Therefore, a simple average of four pixels of the same color is used as a color signal for each color after binning processing. This is performed for each color of red (R), green (G), and blue (B). Next, one pixel is returned to four pixels (FIG. 4 (b) → FIG. 4 (c)). The one large pixel has a red color (R), a green color (G), and a blue color (B) in each pixel, and thus becomes a stacked image signal. Therefore, in order to convert the stacked image signal into a Bayer image signal, the stacked color image signals are rearranged into a Bayer image signal array. However, since one red pixel (R), one green color (G), and one blue color (B) are stacked on each pixel, green (G) is insufficient. Therefore, green (G) compensates for the deficiency by duplicating the already existing green (G).

  In step # 24, the signal processing unit 1031 performs signal processing on the image signal converted into the Bayer-type image signal by the image signal conversion unit 1023.

  In step # 25, the YCbCr conversion unit 1032 converts the image signal signal-processed by the signal processing unit 1031 into YCbCr.

  In step # 26, the luminance signal synthesis unit 1042 performs the luminance signal (Y ′) of the image signal converted into YCbCr in step # 25 and the contour signal subjected to signal processing in the contour signal generation unit 1041 in step # 22. A new luminance signal (Y ″) is generated from the above. Since the generation method of the luminance signal (Y ″) is the same as that in the first embodiment, a description thereof will be omitted.

  In step # 27, the color signal noise removing unit 1033 performs signal processing on the color signal of the image signal converted into YCbCr in step # 25 to remove noise.

  In step # 28, the YCbCr combining unit 1032 generates a new image signal by combining the luminance signal (Y ") generated in step # 26 and the color signal subjected to signal processing in step # 27. To do.

  In step # 29, the SDRAM 105 temporarily stores the image signal generated in step # 28. The external display device 107 reads an image signal from the SDRAM 105 and displays it as a preview image. The external storage device 106 reads an image signal from the SDRAM 105 and stores it as a captured image.

  By performing the above steps, it is possible to record an image with a sense of resolution while reducing pattern noise.

  In both the first and second embodiments of the present invention, the contour signal is generated from the luminance signal (Y) by the contour signal generation unit 1041 and is generated by the generated contour signal and the YCbCr conversion unit. The luminance signal synthesis unit 1042 synthesizes the luminance signal (Y ′) to generate a new luminance signal (Y ″).

  However, the luminance signal generated by the YCbCr conversion unit 1032 and the luminance signal obtained by performing signal processing on the luminance signal (Y) generated by the luminance signal generation unit 1021 without generating the contour signal by the contour signal generation unit 1041. Are combined to generate a new luminance signal.

  The image obtained by performing signal processing for synthesizing the luminance signal newly generated by the above method and the color signal subjected to noise removal by the color signal noise removing unit 1033 by the YCbCr synthesizing unit 1043 also leaves a sense of resolution. Images can be acquired. This is because the acquired luminance signal of the image is based on the luminance signal (Y) generated from the stacked image signal. Therefore, since the luminance of green (G), which has a large influence on the resolution, is higher than that of the Bayer image signal, it is possible to acquire an image with a sense of resolution.

  However, there is a problem in obtaining this image. When the signal processing is performed on the luminance signal and the color signal to which the signal processing is applied, the luminance brightness may be different between the luminance signal and the color signal. Therefore, it is necessary to match the color signal by adjusting the luminance signal. Therefore, adjustment processing such as brightness is required.

  Therefore, in the present invention, the contour signal is generated and used instead of synthesizing the luminance signal. Since the contour signal holds only data regarding the contour, it is not necessary to adjust the brightness. Therefore, since the same signal processing is applied to the luminance signal (Y ′) to which the contour signal is added and the color signal to be synthesized, it is not necessary to adjust the brightness, and therefore the luminance signal (Y ′) has the contour. The same applies when signals are added. That is, the necessary brightness adjustment processing can be reduced.

  In the present invention, the process performed by the image signal converter 1023 is different between the first embodiment and the second embodiment.

  In the first embodiment, a luminance signal is generated from a laminated image signal detected by the laminated solid-state imaging device 101, and then a thinning process is performed to convert the laminated image signal into a Bayer image signal. ing.

  On the other hand, in the second embodiment, binning processing is performed using 2 × 2 pixels in the vertical and horizontal directions in the state of a stacked image signal, and the signals are combined into one pixel. Since this one pixel is a laminated image signal, it has red (R), green (G), and blue (B) information. The red (R), green (G), and blue (B) of each pixel is arranged in a Bayer type as shown in FIG. At this time, green (G) is deficient by one pixel, but supplementation is performed using the green (G) on the other side. By performing the above procedure, the layered image signal is converted into the Bayer image signal.

  The reason why the binning process is performed in the second embodiment will be described. When binarization processing is not performed on a stacked image signal and the stacked image signal is converted into a Bayer image signal by arranging the stacked image signal in a Bayer format, the stacked image signal requires only one pixel ( FIG. 4 (b)) is arranged in a Bayer pattern, so that there are four pixels (FIG. 4 (c)). Therefore, when all the stacked image signals are converted into Bayer image signals by this method, the number of pixels is doubled both vertically and horizontally, and the amount of image signal data increases. Therefore, the time required for image signal processing increases. Therefore, in order to suppress an increase in the data amount of the image signal, a 2 × 2 binning process (FIG. 4 (a) → FIG. 4 (b)) using two vertical and horizontal pixels is performed in advance. The increase in the number of pixels when arranged in a Bayer type is suppressed (FIG. 4 (a) → FIG. 4 (c)). That is, an increase in the data amount of the image signal is suppressed, and the time required for image signal processing is also suppressed.

  Specifically, the data amount of the stacked image signal in a state before image signal processing is 2 (pixels) × 2 (pixels) × 3 (red (R) + green (G) + blue ( B)) (color). By performing binning processing on this stacked image signal, 2 (pixels) × 2 (pixels) × 1 (any one of red (R), green (G), and blue (B)) and the contour signal 2 (pixel) × 2 (pixel) × 1 (layer). As a result, the data amount of the image signal in the second embodiment is 2/3 of the stacked image signal.

  The same applies to the first embodiment. More specifically, the data amount of the stacked image signal, which is the state before image signal processing, is the same as that in the first embodiment. Since the layered image signal is converted into a Bayer image signal by performing a thinning process, 2 (pixel) × 2 (pixel) × 1 (red (R), green (G), and blue (B) Any one color) and 2 (pixels) × 2 (pixels) × 1 (layer) of the contour signal. As a result, as in the second embodiment, the data amount of the image signal in the first embodiment is 2/3 of the stacked image signal.

  Next, the difference in necessity of the low-pass filter that arises from the difference in the processes performed in the image signal conversion unit between the first embodiment and the second embodiment will be described.

  In the first embodiment, thinning processing is performed when converting a laminated image signal into a Bayer image signal. For example, attention is paid to a pixel in which red (R) is left. The green (G) and blue (B) of this pixel have no effect on the converted Bayer image signal. The same applies to red (R) and blue (B) for pixels that have left green (G), and red (R) and green (G) for pixels that have left blue (B). Accordingly, since the color signal of the converted Bayer image signal is only thinned out from the color signal of the original pixel, a false color is likely to occur when performing the interpolation calculation process. Therefore, in order to suppress this false color, a low-pass filter was applied to perform a high frequency component removal process.

  On the other hand, in the second embodiment, binning processing is performed when converting a laminated image signal into a Bayer image signal. In this binning process, for each color, four adjacent pixels of the same color are combined to generate one pixel. At this time, since one pixel is generated by averaging the four pixels, each pixel after being converted into a Bayer-type image signal is affected by the color information possessed by the surrounding pixels. . Even if one pixel including a high-frequency component exists among these four pixels, the high-frequency component is removed by the influence of the other three pixels when the binning process is performed. Therefore, the false color generated when performing the interpolation calculation process is smaller in the second embodiment than in the first embodiment and the second embodiment. Therefore, it is possible to omit the process of applying the low-pass filter.

  Although there is a method of obtaining the same effect by applying an optical low-pass filter used in the Bayer type solid-state imaging device, in the first embodiment using the low-pass filter, the digital low-pass filter unit 1022 is provided in the present invention. Thus, the high-frequency component of the image signal causing the false color is removed. This applies an optical low-pass filter to remove the high-frequency component of the light detected by the stacked solid-state imaging device 101, resulting in an image signal that has lost resolution. Accordingly, both the luminance signal and the contour signal generated from the image signal that has lost the sense of resolution, as compared with the case where the digital low-pass filter is used, as shown in the first embodiment of the present invention, The image will be lost.

  In the above embodiments, an interchangeable lens type digital camera is used, but it goes without saying that there is no problem even if the present invention is applied to a digital camera that is not interchangeable lens.

DESCRIPTION OF SYMBOLS 100 Imaging device 101 Multilayer solid-state image sensor 102 FPGA
1021 Luminance signal generation unit 1022 Digital low-pass filter unit 1023 Image signal conversion unit 103 CPU
1031 Signal processor 1032 YCbCr converter 1033 Color signal noise remover 104 DSP
1041 Contour signal generation unit 1042 Luminance signal synthesis unit 1043 YCbCr synthesis unit 106 External storage device 107 External display device 200 Optical system

Claims (2)

In an imaging apparatus that processes a laminated image signal detected using a laminated solid-state imaging device in which at least three photoelectric conversion layers are laminated on a substrate on one pixel,
A luminance signal generating means for generating a luminance signal from the laminated image signal;
Low-pass filter means for smoothing the laminated image signal;
Contour signal generating means for generating a contour signal from the luminance signal;
Image signal conversion means for converting an image signal into a Bayer image signal by performing a thinning process on the laminated image signal to which the smoothing operation is applied ;
YCbCr conversion means for generating a second luminance signal and a color signal from the Bayer-type image signal;
Luminance signal synthesis means for synthesizing the contour signal and the second luminance signal;
An imaging apparatus comprising: a YCbCr synthesizing unit that synthesizes the color signal and the luminance signal synthesized by the luminance signal synthesizing unit. In an imaging apparatus that processes a laminated image signal detected using a laminated solid-state imaging device in which at least three photoelectric conversion layers are laminated on a substrate on one pixel,
A luminance signal generating means for generating a luminance signal from the laminated image signal;
Contour signal generating means for generating a contour signal from the luminance signal;
Image signal conversion means for binning the laminated image signal and rearranging the binned image signal to become a Bayer image signal;
YCbCr conversion means for generating a second luminance signal and a color signal from the Bayer-type image signal;
Luminance signal synthesis means for synthesizing the contour signal and the second luminance signal;
An imaging apparatus comprising: a YCbCr synthesizing unit that synthesizes the color signal and the luminance signal synthesized by the luminance signal synthesizing unit.

Images