JP6910680B1 - Imaging system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging system capable of outputting an image signal having a high frame rate and having distance information. An imaging system has a pixel unit including 2 m (m is a natural number) pixel units each containing a plurality of pixels of the same color, and each of the 2 m pixel units has a plurality of pixels on the first side. And a binning processing unit that divides into a pixel and a second side pixel and generates a first binning signal obtained by binning the first side pixel and a second binning signal obtained by binning the second side pixel. ,including. [Selection diagram] Fig. 1

Description

The present invention relates to an imaging system.

Many electronic devices such as mobile phones, smartphones, notebook personal computers, tablet personal computers, digital cameras, digital video cameras, or in-vehicle cameras are equipped with a solid-state image sensor (image sensor) together with a display device. In these electronic devices, the subject is acquired as an image converted into an electric signal by an image sensor.

In recent years, not only display devices but also image sensors have been improved in resolution. For example, a Quad Bayer array in which each of four pixels arranged in a Bayer array is further divided into four. Image sensors are known. That is, in a quad Bayer array, each color has 2 x 2 pixels. In the quad Bayer arrangement, since each of the four divided pixels has a photoelectric conversion element, high-resolution imaging is possible.

Further, it is also possible to measure the distance between the subject and the image sensor by using the difference in the incident light to two adjacent pixels among the four divided pixels (see, for example, Patent Document 1). .).

On the other hand, in increasing the resolution of the image sensor, problems such as a decrease in the frame rate or an increase in power consumption due to an increase in the number of pixels occur. Therefore, in order to improve the frame rate, a binning process that outputs a plurality of pixels included in the image sensor as one pixel is known (see, for example, Patent Document 2).

International Publication No. 2017/210781 JP-A-2019-5634

However, when the binning process is performed on the pixels of the quad Bayer array, if the four divided pixels of the same color are binned, the distance signal component included in the pixel value is lost, so that the subject and the image sensor The distance to and cannot be measured. In other words, when the binning process is executed on the pixels of the quad Bayer array, the distance information is lost.

The present invention has been made in view of the above problems, and one of the problems with the present invention is to provide an imaging system capable of outputting an image signal having a high frame rate and distance information.

In the imaging system according to the embodiment of the present invention, a plurality of pixel ^{units including 2 m} (m is a natural number) pixel units each ^{containing a plurality of pixels of the same color and a plurality of 2 m} pixel units in each of the pixel units. The pixel is divided into a first side pixel and a second side pixel, and a first binning signal obtained by binning the first side pixel and a second binning signal obtained by binning the second side pixel are obtained. Includes a binning processing unit to be generated.

The 2 ^m pixel unit includes a first pixel unit and a second pixel unit, and the second pixel unit may be adjacent to each other at diagonal positions of the first pixel unit.

The first side and the second side may be on the left and right, respectively.

The first side and the second side may be the upper side and the lower side, respectively.

In the second k-1 frame (k is an integer of 1 or more), the first side and the second side are the left side and the right side, respectively, and in the second k frame, the first side and the second side. May be the upper side and the lower side, respectively.

Further, a distance measuring processing unit that generates distance information including the distance to the subject based on the pixel value of the first binning signal and the pixel value of the second binning signal may be included.

The pixel unit may include an optical element layer in which one convex lens is provided for each of ^{the 2 m pixel units.} Further, the pixel portion is provided with one convex lens for each of the plurality of pixels and a groove along the direction perpendicular to the direction from the first side pixel to the second side pixel on the convex lens. It may include an optical element layer including a diffraction grating.

Further, the imaging system according to the embodiment of the present invention includes a first pixel unit, a second pixel unit, a third pixel unit, and a fourth pixel unit, each containing a plurality of pixels of the same color. A unit and a plurality of pixels of each of the first pixel unit and the second pixel unit are divided into a first side pixel and a second side pixel, and a third pixel unit and a fourth pixel unit are divided. Each of the plurality of pixels of the above is divided into a third side pixel and a fourth side pixel, the first binning signal obtained by binning the first side pixel, and the second side pixel binned. The first side includes a second binning signal, a third binning signal obtained by binning the third side pixel, and a binning processing unit that generates a fourth binning signal obtained by binning the fourth side pixel. The pixel and the second side pixel are divided along the column direction, and the third side pixel and the fourth side pixel are divided along the row direction.

The third pixel unit may be adjacent at a diagonal position of the second pixel unit.

Further, based on the pixel value of the first binning signal, the pixel value of the second binning signal, the pixel value of the third binning signal, and the pixel value of the fourth binning signal, distance information including the distance to the subject is generated. It may include a distance measuring processing unit.

According to the imaging system according to the embodiment of the present invention, the frame rate can be improved by the binning process, and the pixel value of the binning signal after the binning process has a distance signal component, so that the subject and the image are captured. The distance to the system 10 can be measured. Therefore, by using the imaging system, for example, it is possible to focus on the subject with high accuracy.

It is a block diagram which shows the structure of the image pickup system which concerns on one Embodiment (1st Embodiment) of this invention. It is a schematic diagram which shows the plane structure of the pixel part of the image pickup system which concerns on one Embodiment (1st Embodiment) of this invention. It is a schematic diagram which shows the cross-sectional structure of the pixel part of the image pickup system which concerns on one Embodiment (1st Embodiment) of this invention. It is a schematic diagram explaining the binning process executed in the binning process part of the imaging system which concerns on one Embodiment (1st Embodiment) of this invention. It is a schematic diagram explaining the binning process which generates the 1st binning signal (G1') and the 2nd binning signal (G2') in the imaging system which concerns on one Embodiment (1st Embodiment) of this invention. It is a schematic diagram explaining the detection of the distance signal component of a subject in the image pickup system which concerns on one Embodiment (1st Embodiment) of this invention. It is a schematic diagram explaining the distance measurement processing executed by the distance measurement processing unit of the imaging system which concerns on one Embodiment (1st Embodiment) of this invention. It is a schematic diagram which shows the pixel arrangement and the cross-sectional structure of the pixel part of the image pickup system which concerns on one Embodiment (1st Embodiment) of this invention. It is a schematic diagram explaining the binning process executed in the binning process part of the imaging system which concerns on one Embodiment (second embodiment) of this invention. In the imaging system according to one embodiment (second embodiment) of the present invention, the schematic diagram illustrating the binning process for generating the first binning signal (G1 ″) and the second binning signal (G2 ″). be. It is a schematic diagram which shows the pixel arrangement and the cross-sectional structure of the pixel part of the image pickup system which concerns on one Embodiment (second Embodiment) of this invention. It is a schematic diagram explaining the binning process executed in the binning process part of the imaging system which concerns on one Embodiment (third Embodiment) of this invention. It is a schematic diagram explaining the binning process executed in the binning process part of the imaging system which concerns on one Embodiment (fourth Embodiment) of this invention. It is a schematic diagram explaining the pixel array which is binning processed in the binning processing part of the imaging system which concerns on one Embodiment (fifth Embodiment) of this invention. It is a schematic diagram explaining the binning process executed in the binning process part of the imaging system which concerns on one Embodiment (sixth Embodiment) of this invention. It is a schematic diagram explaining the binning process which generates the 1st binning signal (G1 ″'') in the image pickup system which concerns on one Embodiment (sixth Embodiment) of this invention. It is a schematic diagram explaining the binning process which generates the 2nd binning signal (G2 ″'') in the image pickup system which concerns on one Embodiment (sixth Embodiment) of this invention.

Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The disclosure of each embodiment is merely an example. That is, a configuration that can be easily conceived by a person skilled in the art by making appropriate changes while maintaining the gist of the invention is naturally included in the scope of the present invention. In order to clarify the description, the drawings may be schematically represented by the width, thickness, shape, etc. of each part as compared with the actual embodiment. However, these are merely examples and do not limit the interpretation of the present invention. Further, in the present specification and the present drawing, detailed description may be omitted as appropriate for the same configuration as described above with respect to the above-mentioned drawings.

In the present specification, the expression "α includes A, B, or C" does not exclude the case where α includes a plurality of combinations of A to C unless otherwise specified. Further, the expression "α includes A, B, or C" or "α includes A, B, and C" does not exclude the case where α includes other configurations unless otherwise specified.

In the present specification and drawings, the same reference numerals are used when referring to a plurality of identical or similar configurations as a whole. Further, when a plurality of parts in one structure are described separately, the same code may be used, and a hyphen and a natural number may be used.

In the present specification and drawings, the "row direction" may indicate the "X direction" and the "column direction" may indicate the "Y direction". For example, the X direction of the subject may correspond to the row direction of the pixel portion. Similarly, the Y direction of the subject may correspond to the column direction of the pixel portion.

In the present specification, the edge in the X direction means an edge detected by scanning the pixel portion in the X direction, and the edge in the Y direction means an edge detected by scanning the pixel portion in the Y direction.

In the present specification, "left side" and "right side" represent any direction in the row direction, and the opposite of "left side" is "right side". Similarly, "upper" and "lower" represent any direction in the column direction, with the opposite of "upper" being "lower".

In the present specification, "left side", "right side", "upper side", and "lower side" are referred to as "first side", "second side", "third side", and "fourth side". It may be described as either "side".

The following embodiments can be combined with each other as long as there is no technical contradiction.

<First Embodiment>
The configuration of the imaging system 10 according to the embodiment of the present invention will be described with reference to FIGS. 1 to 7.

[1. Configuration of imaging system 10]
FIG. 1 is a block diagram showing a configuration of an imaging system 10 according to an embodiment of the present invention. As shown in FIG. 1, the imaging system 10 includes a pixel unit 100, a binning processing unit 200, a distance measuring processing unit 300, a sensor unit 400, an image signal processing unit 500, a storage unit 600, and a display unit 700.

The pixel unit 100 can convert the light emitted by the subject into an electric signal. That is, the pixel unit 100 can acquire the information of the subject as an electric signal. Specifically, each of the plurality of pixels included in the pixel unit 100 converts light into an electric signal (hereinafter, the electric signal generated by the pixel of the pixel unit 100 is referred to as a "pixel signal"). In other words, the pixel unit 100 can generate and output a pixel signal. The details of the configuration of the pixel unit 100 will be described later.

The binning processing unit 200 can receive the pixel signal and execute the binning processing on the pixel signal. That is, the binning processing unit 200 can output a binning signal including pixel values obtained by adding or averaging the pixel values of several pixels of the same color included in the pixel signal. In other words, the binning processing unit 200 can execute the binning process on the pixel signal, generate the binning signal from the pixel signal, and output the binning signal. Here, the pixel value is a value that reflects the image information in the pixel, and is not only a value corresponding to the intensity of light acquired by the pixel unit 100 but also a distance corresponding to the distance between the subject and the pixel unit. Also includes signal components.

The binning process by the binning process unit 200 may be a process for an analog signal or a process for a digital signal. A typical pixel signal is an analog signal, and the binning processing unit 200 can execute binning processing on the pixel signal which is an analog signal. Further, the binning processing unit 200 can also execute the binning processing on the pixel signal converted into the digital signal by the A / D conversion circuit. In the binning process for a pixel signal which is an analog signal, the binning signal is also an analog signal, but the binning signal may be converted into a digital signal by an A / D conversion circuit. For example, the binning signal input to the image signal processing unit 500 is an A / D converted digital signal. The details of the binning process executed by the binning process unit 200 will be described later.

The distance measuring processing unit 300 receives the binning signal, calculates the difference (difference value) between the pixel values of the two binning signals in the X direction or the Y direction, and calculates the distance signal from the position and the difference value in the X direction or the Y direction. Can be generated. Further, the distance measuring processing unit 300 can calculate the distance from the imaging system 10 to the subject based on the generated distance signal, and generate distance information (distance data) converted into the actual distance to the subject. can. That is, the distance measurement processing unit 300 can execute the distance measurement processing using the binning signal and output the distance data. The binning signal processed by the distance measuring processing unit 300 may be an analog signal or a digital signal, but the distance data is input to the image signal processing unit 500 and thus converted into a digital signal. Is preferable. The distance measuring processing unit 300 can be executed by, for example, an information processing device (computer). The information processing device includes, for example, a calculation unit such as a central processing unit (CPU) and a storage unit such as a register or a memory. The distance measuring process of the distance measuring processing unit 300 can be executed by reading a program stored in advance in the storage unit by the calculation unit. The details of the distance measuring process executed by the distance measuring processing unit 300 will be described later.

The sensor unit 400 can detect the movement or orientation of the pixel unit 100 (for example, the amount of blurring of the pixel unit 100) and output the sensor detection signal to the image signal processing unit 500. The image signal processing unit 500 can correct the binning signal (for example, blur correction) based on the sensor detection signal. The number of sensors included in the sensor unit 400 may be one or a plurality. As the sensor included in the sensor unit 400, for example, an acceleration sensor or an angular velocity sensor can be used.

The image signal processing unit 500 can receive the binning signal and execute various signal processing such as gamma correction, aberration correction, blur correction, and noise reduction correction on the binning signal. Further, the image signal processing unit 500 can receive the distance data, generate an image signal including the distance information of the subject based on the distance data, and output the image signal to the display unit 700. That is, the image signal processing unit 500 can generate an image signal including distance information of the subject based on the binning signal and the distance data. As the image signal processing unit 500, for example, an image signal processor or the like can be used.

The storage unit 600 can store data related to various signal processing executed by the image signal processing unit 500. For example, the storage unit 600 can store the binning signal output by the binning processing unit 200 in each frame. The image signal processing unit 500 can read the binning signals in a plurality of frames from the storage unit 600. As the storage unit 600, for example, a memory or the like can be used.

The display unit 700 can receive the image signal and display an image based on the image signal. As the display unit 700, for example, a liquid crystal display device or an OLED display device can be used.

The binning processing unit 200 or the distance measuring processing unit 300 may be included in the image sensor together with the pixel unit 100, or may be included in the image signal processor. Further, the binning processing unit 200 or the distance measuring processing unit 300 may be provided separately from the image sensor and the image signal processor.

[2. Configuration of pixel unit 100]
The configuration of the pixel unit 100 of the imaging system 10 according to the embodiment of the present invention will be described with reference to FIGS. 2 (A), 2 (B) and 3.

2 (A) and 2 (B) are schematic views showing a planar configuration of a pixel portion 100 of an imaging system 10 according to an embodiment of the present invention. Specifically, FIG. 2A shows the pixel array 101 in the pixel unit 100, and in the pixel unit 100, as shown in FIG. 2B, the pixel array 101 is used as one unit in the row direction and the column direction. It is repeatedly arranged in.

The pixel array 101 of the pixel unit 100 includes a first pixel unit 110, a second pixel unit 120, a third pixel unit 130, and a fourth pixel unit 140. The second pixel unit 120 is adjacent to the first pixel unit 110 at a diagonal position (diagonal direction) of the first pixel unit 110. The third pixel unit 130 is adjacent to the first pixel unit 110 in the row direction. The fourth pixel unit 140 is adjacent to the first pixel unit 110 in the column direction. Further, when the position of the first pixel unit 110 and the position of the second pixel unit 120 in the pixel array 101 are relatively represented, the first pixel unit 110 is located on the upper side or the left side of the pixel array 101. The second pixel unit 120 is located below or to the right of the pixel array 101.

The first pixel unit 110 includes the first pixels 111-1 to 111-4. The second pixel unit 120 includes the second pixels 121-1 to 121-4. The third pixel unit 130 includes third pixels 131-1 to 131-4. The fourth pixel unit 140 includes fourth pixels 1411-1 to 141-4. The first pixels 111-1 and 111-2 are located on the upper side in the first pixel unit 110, and the first pixels 111-3 and 111-4 are located on the lower side in the first pixel unit 110. positioned. Further, the first pixels 111-1 and 111-3 are located on the left side in the first pixel unit 110, and the first pixels 111-2 and 111-4 are located on the right side in the first pixel unit 110. Is located in. The positions of the second pixels 121-1 to 121-4 in the second pixel unit 120, the positions of the third pixels 131-1 to 131-4 of the third pixel unit 130, and the fourth pixel unit 140. The positions of the fourth pixels 1411-1 to 141-4 are the same as the positions of the first pixels 111-1 to 111-4 in the first pixel unit 110.

FIG. 3 is a schematic view showing a cross-sectional configuration of the pixel portion 100 of the imaging system 10 according to the embodiment of the present invention. Specifically, FIG. 3 is a schematic cross-sectional view cut along the lines A1-A2 shown in FIG. 2 (A).

As shown in FIG. 3, the pixel unit 100 includes a circuit layer 1000, a photoelectric conversion element layer 1100, a color filter layer 1200, and an optical element layer 1300.

The circuit layer 1000 includes a plurality of switch units 1010. Further, the circuit layer 1000 includes a transistor, a capacitive element, a horizontal signal line, a vertical signal line, a power supply line for supplying power to the transistor, and the like.

The photoelectric conversion element layer 1100 is provided on the circuit layer 1000 and includes a photoelectric conversion element 1110 and an element separation unit 1120. The photoelectric conversion element 1110 can convert light into electricity and generate an exposure signal. The photoelectric conversion element 1110 is electrically connected to the circuit layer 1000, and the exposure signal is transmitted to the circuit layer 1000. The element separation unit 1120 can insulate a plurality of photoelectric conversion elements 1110 from each other. The element separation unit 1120 is composed of, for example, a light-shielding film or a reflective film made of an inorganic material or an organic material.

The color filter layer 1200 is provided on the photoelectric conversion element layer 1100, and includes a green color filter 1210, a red color filter 1230, a blue color filter (not shown in FIG. 3), and a light-shielding portion 1240. The light-shielding unit 1240 can, for example, block stray light at the boundary of pixels or prevent color mixing of light between adjacent pixels. The light-shielding portion 1240 is composed of a film containing, for example, aluminum, tungsten, copper, or an alloy containing them.

The optical element layer 1300 is provided on the color filter layer 1200 and includes a microlens array 1310. The microlens array 1310 includes a plurality of convex lenses, and one convex lens is arranged in each of the pixel units (the first pixel unit 110 and the third pixel unit 130 in FIG. 3). The diameter of the convex lens of the microlens array 1310 is formed so as to substantially match the length of one side of the pixel unit.

The pixel unit 100 has an optical element layer 1300 provided on the color filter layer 1200, and has a structure called a so-called on-chip lens (OCL).

A green color filter 1210 is arranged in the first pixels 111-1 and 111-2, and a red color filter 1230 is arranged in the third pixels 131-1 and 131-2. That is, the first pixels 111-1 to 111-4 include the green color filter 1210, and the third pixels 131-1 to 131-4 include the red color filter 1230. Although not shown, the second pixels 121-1 to 121-4 include a green color filter, and the fourth pixels 1411-141-4 include a blue color filter. Therefore, the first pixel 111-1 to 111-4 and the second pixel 121-1 to 121-4 are green pixels having a green color filter, and the third pixel 131-1 to 131-4 is a red color. It is a red pixel having a filter, and the fourth pixel 1411-14-14-1-4 is a blue pixel having a blue color filter.

In the pixel array 101 of the pixel unit 100, four pixel units adjacent to each other correspond to two greens, one red, and one blue, and each of the pixel units includes four pixels of the same color. That is, the pixel array 101 is a so-called quad Bayer array. In the quad-bayer arrangement, more green pixels are arranged than red pixels and blue pixels so that the brightness of green, which has high luminosity factor for the human eye, is high.

In the present specification and drawings, for convenience, the first pixels 111-1 to 111-4 are designated as the first green pixel (G1), and the second pixels 121-1 to 121-4 are designated as the second green pixel (G1). It may be described as G2).

The pixel array 101 of the pixel unit 100 is not limited to the quad Bayer array. That is, in the pixel array 101 of the pixel unit 100, the first green pixel (G1) and the second green pixel (G2) may be adjacent to each other in the row direction or may be adjacent to each other in the column direction. ..

The circuit layer 1000 can support a photoelectric conversion element layer 1100, a color filter layer 1200, and an optical element layer 1300 or a microlens array 1301. The circuit layer 1000 is, for example, a silicon substrate, and a silicon oxide film, a silicon nitride film, or the like may be formed.

The photoelectric conversion element layer 1100 includes a photoelectric conversion element 1110 provided in each of the pixels and generating an electric charge according to the intensity of the received light. The photoelectric conversion element 1110 may be a photoelectric conversion element in which an electromotive force is generated by light, or may be a photoelectric conversion element whose electrical conductivity is changed by light. Although details are not shown, a control circuit for controlling the photoelectric conversion element 1110 may be formed in the photoelectric conversion element layer 1100.

Therefore, the pixel unit 100 can generate and output a pixel signal by controlling the photoelectric conversion element 1110 in each pixel.

[3. Binning process by binning process unit 200]
FIG. 4 is a schematic diagram illustrating a binning process executed by the binning process unit 200 of the imaging system 10 according to the embodiment of the present invention. In the following, for convenience, the binning signal and the symbols in parentheses of the pixels may represent the pixel values.

The binning processing unit 200 selectively binners a plurality of pixels included in the pixel array 101, and the first binning signal 210 (G1'), the second binning signal 220 (G2'), and the third binning signal 230. (R') and a fourth binning signal 240 (B') are generated. In the following, for convenience, the first binning signal 210 (G1'), the second binning signal 220 (G2'), the third binning signal 230 (R'), and the fourth binning signal 240 (B') , It will be described as being converted into the binning signal array 201 corresponding to the pixel array 101. Therefore, the second binning signal 220 (G2') is adjacent to the first binning signal 210 (G1') at the diagonal position (diagonal direction) of the first binning signal 210 (G1'). There is. The third binning signal 230 (R') is adjacent to the first binning signal 210 (G1') in the row direction. The fourth binning signal 240 (B') is adjacent to the first binning signal 210 (G1') in the row direction. Therefore, when the position of the first binning signal 210 (G1') and the position of the second binning signal 220 (G2') in the binning signal array 201 are relatively represented, the first binning signal 210 (G1') is represented. ) Is located above or to the left of the binning signal array 201, and the second binning signal 220 (G2') is located below or to the right of the binning signal array 201.

The third binning signal 230 (R') and the fourth binning signal 240 (B') are the third pixels 131-1 to 131-4 and the fourth pixel contained in the third pixel unit 130, respectively. It is generated by binning the fourth pixels 1411-1 to 141-4 included in the unit 140. On the other hand, the first binning signal 210 (G1') is the first pixel 111-1 to 111-4 in the first pixel unit 110 and the second pixel 121-1 to the second pixel unit 120. It is generated by dividing each of 121-4 into a first side and a second side along the column direction or the row direction, and the pixels on the first side are binned. Similarly, the second binning signal 220 (G2') is generated by binning the second side pixel. Hereinafter, the binning process for generating the first binning signal 210 (G1') and the second binning signal 220 (G2') will be described with reference to FIGS. 5 (A) and 5 (B).

5 (A) and 5 (B) show the first binning signal 210 (G1') and the second binning signal 220 (G2') in the imaging system 10 according to the embodiment of the present invention, respectively. It is a schematic diagram explaining the generated binning process.

As shown in FIG. 5 (A), the binning processing unit 200 includes the left first pixels 111-1 (G1 _UL ) and 111-3 (G1 _DL ) and the second pixel in the first pixel unit 110. _{The second pixels 121-1 (G2 UL} ) and 121-3 (G2 _DL ) on the left side of the unit 120 are binned to generate a first binning signal 210 (G1').

Further, as shown in FIG. 5B, the binning processing unit 200 has the first pixels 111-2 (G1 _UR ) and 111-4 (G1 _DR ) and the second pixel 111-2 (G1 UR) and the second pixel 111-2 (G1 UR) on the right side in the first pixel unit 110. _{The second pixels 121-2 (G2 UR} ) and 121-4 (G2 _DR ) on the right side in the pixel unit 120 of the above are binned to generate a second binning signal 220 (G2').

Therefore, each pixel value of the first binning signal 210 to the fourth binning signal 240 is calculated by the formula shown in Table 1.

As can be seen from Table 1, the information on the left side of the first pixel unit 110 and the information on the left side of the second pixel unit 120 are aggregated in the first binning signal 210 (G1'). On the other hand, the information on the right side of the first pixel unit 110 and the information on the right side of the second pixel unit 120 are aggregated in the second binning signal 220 (G2'). Further, since each of the first binning signal 210 to the fourth binning signal 240 has four pixels binned, it is possible to improve the frame rate in the imaging system 10.

[4. Distance measurement processing by the distance measurement processing unit 300]
First, a method of detecting a distance signal component of a subject will be described with reference to FIG.

FIG. 6 is a schematic diagram illustrating the detection of a distance signal component of a subject in the imaging system according to the embodiment of the present invention. In FIGS. 6A to 6C, the first pixel L and the second pixel R are alternately arranged in the X direction. Here, for example, the first pixel L is the left first pixels 111-1 and 111-3 in the first pixel unit 110, and the second pixel R is in the first pixel unit 110. The first pixels 111-2 and 111-4 on the right side of. An optical element layer that changes the signal distribution depending on the incident angle of light is provided on the first pixel L and the second pixel R (not shown). Further, FIGS. 6 (A) to 6 (C) show changes in the signal intensities of the first pixel L and the second pixel R and changes in the difference signal (LR) around the edge. There is. The difference signal (LR) is a distance signal component, and is, for example, a difference value between the intensity of the pixel signal of the first pixel L and the intensity of the pixel signal of the second pixel R.

FIG. 6A shows a case where the edge of the subject is in the X direction and the distance between the subject and the imaging system 10 is shorter than the focusing distance. Due to the difference in the incident angle characteristics, the pixel signal of the first pixel L and the pixel signal of the second pixel R are shifted to the right and left at the edge position of the subject, respectively. As a result, the difference signal becomes a downwardly convex curve.

FIG. 6B shows a case where the edge of the subject is in the X direction and the distance between the subject and the imaging system 10 coincides with the focusing distance. In this case, the pixel signal of the first pixel L and the pixel signal of the second pixel R substantially match at the edge position of the subject. As a result, the difference signal becomes approximately 0.

FIG. 6C shows a case where the edge of the subject is in the X direction and the distance between the subject and the imaging system 10 is longer than the focusing distance. Due to the difference in the incident angle characteristics, the pixel signal of the first pixel L and the pixel signal of the second pixel R are shifted to the left and right at the edge position of the subject, respectively. As a result, the difference signal becomes an upwardly convex curve.

However, when the adjacent first pixel L and the second pixel R are binned (for example, when the first pixels 111-1 to 111-4 in the first pixel unit 110 are binned), The distance signal component is lost, and as a result, the difference signal cannot be detected.

FIG. 7 is a schematic diagram illustrating distance measurement processing executed by the distance measurement processing unit 300 of the imaging system 10 according to the embodiment of the present invention. Specifically, FIG. 7A is a diagram showing signal intensities of the first binning signal 210 (G1') and the second binning signal 220 (G2'), and FIG. 7B is a diagram showing signal intensities. , The difference signal (G1'-G2') of the first binning signal 210 (G1') and the second binning signal (G2') is shown.

The distance measuring processing unit 300 differs the difference value between the pixel value G1'of the first binning signal 210 (G1') and the pixel value G2' of the second binning signal 220 (G2') in each pixel array 101. Generated as a signal (G1'-G2'). As described above, the information on the left side of the first pixel unit 110 and the information on the left side of the second pixel unit are aggregated in the first binning signal 210 (G1'), and the second binning signal 220 (G2') is collected. ) Gathers information on the right side of the first pixel unit 110 and information on the right side of the second pixel unit 120. In other words, the first binning signal 210 (G1') and the second binning signal (G2') contain information on the left side and information on the right side of the pixel array 101, respectively. That is, in the imaging system 10, even after the binning process, each of the first binning signal 210 (G1') and the second binning signal 220 (G2') contains a distance signal component, and the difference is based on the distance signal component. A signal can be generated. Therefore, when the edge of the subject is in the X direction and the distance between the subject and the imaging system 10 is longer than the focusing distance, the difference signal (G1'-G2') is the subject as shown in FIG. 7B. Corresponds to the edge position of, and becomes an upwardly convex curve.

The distance signal contrast _diff is shown in, for example, the pixel value G1'of the first binning signal 210 (G1'), the pixel value G2' of the second binning signal 220 (G2'), and FIG. 7 (A). It can be expressed by the equation (1) using the edge contrast C.

When the distance signal _{diff diff} is positive, the subject is farther than the focusing distance of the imaging system 10, and when the distance signal _{diff diff} is negative, the subject is closer than the focusing distance of the imaging system 10. Therefore, the imaging system 10 can focus on the subject based on the sign of the _{distance signal diff diff.}

Further, the distance measuring processing unit 300 can _{calculate the distance from the imaging system 10 to the subject based on the distance signal diff diff} , and generate distance information (distance data) converted into the distance to the actual subject. ..

As described above, in the imaging system 10 according to the present embodiment, not only the frame rate is improved by the binning process, but also the binning signal after the binning process has a distance signal component, so that the subject and the imaging system 10 The distance to and can be calculated, and for example, the subject can be focused with high accuracy.

<Modification example 1>
A modified example of the imaging system 10 according to the first embodiment will be described with reference to FIG. In the following, description of the same configuration as the above-described imaging system 10 configuration may be omitted.

FIG. 8 is a schematic view showing the pixel arrangement 101a and the cross-sectional configuration of the pixel portion 100a of the imaging system 10 according to the embodiment of the present invention. In FIG. 8A, the configuration of the optical element layer 1300a is shown by a broken line for convenience.

The optical element layer 1300a is provided on the color filter layer 1200 and includes a microlens array 1310a and a transmission diffraction grating 1320a. The microlens array 1310a includes a plurality of convex lenses, one for each of the pixels (in FIG. 8B, the first pixels 111-1 and 111-2, and the third pixels 131-1 and 131-2). A number of convex lenses are arranged. The diameter of the convex lens of the microlens array 1310a is formed so as to substantially match the length of one side of the pixel.

The transmission type diffraction grating 1320a is provided on the microlens array 1310a, and grooves are provided along the row direction. In other words, in the transmission type diffraction grating 1320a, irregularities are periodically and repeatedly arranged along the row direction. The groove of the transmission type diffraction grating 1320a is provided between the first pixel unit 110 and the third pixel unit 130 and between the fourth pixel unit 140 and the second pixel unit 120. Not limited to.

The pixel portion 100a according to the first modification is provided with a convex lens of the microlens array 1310a in each pixel. Therefore, in the first pixels 111-1 to 111-4 in the first pixel unit 110, there is no optical difference due to the convex lens alone. However, when the transmission type diffraction grating 1320a is provided, the first pixels 111-1 and 111-3 on the left side and the first pixels 111-2 and 111-4 on the right side in the first pixel unit 110 are optical. Difference appears. The same applies to the second pixel unit 120. Therefore, also in the pixel array 101a, by executing the same binning process as described above, the distance signal component is included in each of the first binning signal 210 (G1') and the second binning signal 220 (G2'). Therefore, a difference signal can be generated based on the distance signal component.

Therefore, even in the configuration of the pixel portion 100a according to the first modification, the imaging system 10 not only improves the frame rate by the binning process, but also has the distance signal component in the binning signal after the binning process. , The distance between the subject and the imaging system 10 can be calculated, and the subject can be focused with high accuracy, for example.

<Second Embodiment>
The configuration of the imaging system 10 according to the embodiment of the present invention will be described with reference to FIGS. 9 and 10. In this embodiment, a binning process different from the binning process described in the first embodiment will be described. In the following, the description of the same configuration as that described in the first embodiment may be omitted.

FIG. 9 is a schematic diagram illustrating a binning process executed by the binning process unit 200 of the imaging system 10 according to the embodiment of the present invention.

The binning processing unit 200 selectively binners a plurality of pixels included in the pixel array 101, and the first binning signal 210A (G1''), the second binning signal 220A (G2''), and the third binning. A signal 230 (R') and a fourth binning signal 240 (B') are generated. In the following, for convenience, the first binning signal 210A (G1''), the second binning signal 220A (G2''), the third binning signal 230 (R'), and the fourth binning signal 240 (B') ) Is converted into the binning signal array 201A corresponding to the pixel array 101. Therefore, the second binning signal 220A (G2 ″) and the first binning signal 210A (G1 ″) at the diagonal position (diagonal direction) of the first binning signal 210A (G1 ″). Adjacent. The third binning signal 230 (R ″) is adjacent to the first binning signal 210A (G1 ″) in the row direction. The fourth binning signal 240 (B ″) is adjacent to the first binning signal 210A (G1 ″) in the row direction. Therefore, when the position of the first binning signal 210A (G1 ″) and the position of the second binning signal 220A (G2 ″) in the binning signal array 201A are relatively represented, the first binning signal 210A ( The G1'') is located above or to the left of the binning signal sequence 201A, and the second binning signal 220A (G2'') is located below or to the right of the binning signal sequence 201A.

10 (A) and 10 (B) show the first binning signal 210A (G1 ″) and the second binning signal 220A (G2 ″) in the imaging system 10 according to the embodiment of the present invention. It is a schematic diagram explaining the generated binning process.

As shown in FIG. 10A, the binning processing unit 200 includes the upper first pixels 111-1 (G1 _UL ) and 111-2 (G1 _UR ) and the second pixel in the first pixel unit 110. The upper second pixels 121-1 (G2 _UL ) and 121-2 (G2 _UR ) in the unit 120 are binned to generate a first binning signal 210A (G1 ″).

Further, as shown in FIG. 10B, the binning processing unit 200 includes the lower first pixels 111-3 (G1 _DL ) and 111-4 (G1 _DR ) in the first pixel unit 110, as well as the first pixel unit 110. The lower second pixels 121-3 (G2 _DL ) and 121-4 (G2 _DR ) in the second pixel unit 120 are binned to generate a second binning signal 220A (G2'').

Therefore, the pixel values of the first binning signal 210A and the second binning signal 220A are calculated by the formulas shown in Table 2.

As can be seen from Table 2, the information on the upper side of the first pixel unit 110 and the information on the upper side of the second pixel unit 120 are aggregated in the first binning signal 210A (G1 ″). On the other hand, in the second binning signal 220A (G2 ″), the information on the lower side of the first pixel unit 110 and the information on the lower side of the second pixel unit 120 are aggregated. Further, since each of the first binning signal 210A and the second binning signal 220A has four pixels binned, it is possible to improve the frame rate in the imaging system 10.

The distance measuring process by the distance measuring processing unit 300 using the first binning signal 210A (G1 ″) and the second binning signal 220A is the same as that of the first embodiment. That is, in each pixel array 101, the distance measuring processing unit 300 has a pixel value G1'' of the first binning signal 210A (G1'') and a pixel value G2'' of the second binning signal 220A (G2''). The difference value with and is generated as a difference signal (G1 ″ -G2 ″). Further, the distance measuring processing unit 300 can calculate the distance from the imaging system 10 to the subject based on the distance signal, and generate the distance information (distance data) converted into the distance to the actual subject.

As described above, also in the imaging system 10 according to the present embodiment, not only the frame rate is improved by the binning process, but also the binning signal after the binning process has a distance signal component, so that the subject and the imaging system 10 The distance to and can be calculated, and for example, the subject can be focused with high accuracy.

<Modification 2>
A modified example of the imaging system 10 according to the second embodiment will be described with reference to FIG. In the following, description of the same configuration as the above-described imaging system 10 configuration may be omitted.

FIG. 11 is a schematic view showing the pixel arrangement 101b and the cross-sectional configuration of the pixel portion 100b of the imaging system 10 according to the embodiment of the present invention. In FIG. 11A, the configuration of the optical element layer 1300b is shown by a broken line for convenience.

The optical element layer 1300b is provided on the color filter layer 1200 and includes a microlens array 1310b and a transmission diffraction grating 1320b. The microlens array 1310b includes a plurality of convex lenses, one for each of the pixels (in FIG. 11B, the first pixels 111-1 and 111-3, and the fourth pixels 141-1 and 141-3). A number of convex lenses are arranged. The diameter of the convex lens of the microlens array 1310b is formed so as to substantially match the length of one side of the pixel.

The transmission type diffraction grating 1320b is provided on the microlens array 1310b, and a groove is provided along the row direction. In other words, in the transmission type diffraction grating 1320b, irregularities are periodically and repeatedly arranged along the column direction. Grooves of the transmission type diffraction grating 1320b are provided between the first pixel unit 110 and the fourth pixel unit 140 and between the third pixel unit 130 and the second pixel unit 120. Not limited to.

The pixel portion 100b according to the second modification is provided with a convex lens of the microlens array 1310b in each pixel. Therefore, in the first pixels 111-1 to 111-4 in the first pixel unit 110, there is no optical difference due to the convex lens alone. However, when the transmission type diffraction grating 1320b is provided, in the upper first pixels 111-1 and 111-2 and the lower first pixels 111-3 and 111-4 in the first pixel unit 110, An optical difference appears. The same applies to the second pixel unit 120. Therefore, also in the pixel array 101b, by executing the same binning process as described above, a distance signal component is added to each of the first binning signal 210 (G1 ″) and the second binning signal 220 (G2 ″). Since it can be included, a difference signal can be generated based on the distance signal component.

Therefore, even in the configuration of the pixel portion 100b according to the second modification, the imaging system 10 not only improves the frame rate by the binning process, but also has the distance signal component in the binning signal after the binning process. , The distance between the subject and the imaging system 10 can be calculated, and the subject can be focused with high accuracy, for example.

<Third Embodiment>
The configuration of the imaging system 10 according to the embodiment of the present invention will be described with reference to FIG. In this embodiment, a binning process different from the binning process described in the first embodiment and the second embodiment will be described. In the following, the description may be omitted for the same configurations as those described in the first embodiment and the second embodiment.

FIG. 12 is a schematic diagram illustrating a binning process executed by the binning process unit 200 of the imaging system 10 according to the embodiment of the present invention.

FIG. 12 shows four adjacent pixel arrays 101 among the plurality of pixel arrays 101 repeatedly arranged in the pixel unit 100. The second pixel array 101-2 is adjacent to the first pixel array 101-1 at a diagonal position (diagonal direction) of the first pixel array 101-1. The third pixel array 101-3 is adjacent to the first pixel array 101-1 in the row direction. The fourth pixel array 101-4 is adjacent to the first pixel array 101-1 in the column direction.

The binning processing unit 200 executes the binning processing described in the first embodiment in each of the first pixel array 101-1 and the second pixel array 101-2. That is, each of the first pixel array 101-1 and the second pixel array 101-2 is converted into a binning signal array 201 including the first binning signal 210 and the second binning signal 220 by the binning process. .. On the other hand, the binning processing unit 200 executes the binning processing described in the second embodiment in each of the third pixel array 101-3 and the fourth pixel array 101-4. That is, each of the third pixel array 101-3 and the fourth pixel array 101-4 is converted into the binning signal array 201A including the first binning signal 210A and the second binning signal 220A by the binning process. .. In other words, the binning processing unit 200 executes the binning process so that the binning signal array 201 and the binning signal array 201A are alternately arranged in the row direction and the column direction.

The distance measuring process by the distance measuring processing unit 300 after the binning process is the same as that of the first embodiment and the second embodiment, and thus the description thereof will be omitted. Since the binning signal has a distance signal component, the distance between the subject and the imaging system 10 can be calculated, and for example, the subject can be focused with high accuracy. In particular, since the distance measuring processing unit 300 can detect the edges of the subject in the X and Y directions, it is possible to focus on the subject with higher accuracy.

<Fourth Embodiment>
The configuration of the imaging system 10 according to the embodiment of the present invention will be described with reference to FIG. In this embodiment, a binning process different from the binning process described in the first to third embodiments will be described. In the following, the description may be omitted for the same configurations as those described in the first to third embodiments.

FIG. 13 is a schematic diagram illustrating a binning process executed by the binning process unit 200 of the imaging system 10 according to the embodiment of the present invention.

FIG. 13 shows the pixel array 101. In the second k-1 frame (k is a natural number), the binning processing unit 200 executes the binning processing described in the first embodiment. That is, in the second k-1 frame, the pixel array 101 is converted into the binning signal array 201 including the first binning signal 210 and the second binning signal 220 by the binning process. On the other hand, in the second k frame, the binning processing unit 200 executes the binning processing described in the second embodiment. That is, in the second k frame, the pixel array 101 is converted into the binning signal array 201A including the first binning signal 210A and the second binning signal 220A by the binning process.

The distance measuring process by the distance measuring processing unit 300 after the binning process is the same as that of the first embodiment and the second embodiment, and thus the description thereof will be omitted. Since the binning signal has a distance signal component, the distance between the subject and the imaging system 10 can be calculated, and for example, the subject can be focused with high accuracy. In particular, since the distance measuring processing unit 300 can detect the edges of the subject in the X and Y directions for each frame, it is possible to focus on the subject with higher accuracy.

In the present embodiment, it has been described that different binning processes are executed in consecutive frames as the second k-1 frame and the second k frame, but the frame period of the binning process is not limited to this. In the present embodiment, different binning processes may be included in any of the plurality of frames.

<Fifth Embodiment>
In the first to fourth embodiments, each of the two pixel units is divided into a pixel on the first side and a pixel on the second side, and the pixel on the first side is binned to form the first pixel. A binning signal is generated, and the second side pixel is binned to generate a second binning signal, but the number of pixel units used in these binning processes is not limited to two. Therefore, in the present embodiment, an example in which the number of pixel units is other than two will be described. In the following, the description may be omitted for the same configurations as those described in the first to fourth embodiments.

FIG. 14 is a schematic diagram illustrating pixel arrays 101c to 101e that are binned by the binning processing unit 200 of the imaging system 10 according to the embodiment of the present invention.

The pixel array 101c shown in FIG. 14 (A) includes two first pixel units 110c-1 and 110c-2 and two second pixel units 120c-1 and 120c-2. The binning processing unit 200 includes a pixel on the first side of each of the first pixel units 110c-1 and 110c-2 and a pixel on the first side of each of the second pixel units 120c-1 and 120c-2. Is binned to generate a first binning signal. Further, the binning processing unit 200 binning the pixels on the second side of the first pixel units 110c-1 and 110c-2 and the pixels on the second side of the second pixel units 120c-1 and 120c-2. And generate a second binning signal. Therefore, the information on the first side of each of the first pixel unit 110c and the second pixel unit 120c is aggregated in the first binning signal, and the first pixel unit 110c and the first pinning signal are collected in the second binning signal. Information on the second side of each of the two pixel units 120c is aggregated. Therefore, even in the binning process for the pixel array 101c, since the binning signal after the binning process has a distance signal component, the distance between the subject and the imaging system 10 is calculated, and the subject is focused with high accuracy, for example. be able to.

The pixel array 101d shown in FIG. 14B includes four first pixel units 110d-1 to 110d-4 and four second pixel units 120d-1 to 120d-4. The binning processing unit 200 includes the pixels on the first side of each of the first pixel units 110d-1 to 110d-4 and the pixels on the first side of each of the second pixel units 120d-1 to 120d-4. Is binned to generate a first binning signal. Further, the binning processing unit 200 is located on the second side of each of the first pixel units 110d-1 to 110d-4 and the second side of each of the second pixel units 120d-1 to 120d-4. The pixels are binned to generate a second binning signal. Therefore, the information on the first side of each of the first pixel unit 110d and the second pixel unit 120d is aggregated in the first binning signal, and the first pixel unit 110d and the first pinning signal are collected in the second binning signal. Information on the second side of each of the two pixel units 120d is aggregated. Therefore, even in the binning process for the pixel array 101d, since the binning signal after the binning process has a distance signal component, the distance between the subject and the imaging system 10 is calculated, and the subject is focused with high accuracy, for example. be able to.

The pixel array 101e shown in FIG. 14C includes eight first pixel units 110e-1 to 110e-8 and eight second pixel units 120e-1 to 120e-8. The binning processing unit 200 includes the pixels on the first side of each of the first pixel units 110e-1 to 110e-8 and the pixels on the first side of each of the second pixel units 120e-1 to 120e-8. Is binned to generate a first binning signal. Further, the binning processing unit 200 has a pixel on the second side of each of the first pixel units 110e-1 to 110e-8 and a second side of each of the second pixel units 120e-1 to 120e-8. The pixels are binned to generate a second binning signal. Therefore, the information on the first side of each of the first pixel unit 110e and the second pixel unit 120e is aggregated in the first binning signal, and the first pixel unit 110e and the first pinning signal are collected in the second binning signal. Information on the second side of each of the two pixel units 120e is aggregated. Therefore, even in the binning process for the pixel array 101e, since the binning signal after the binning process has a distance signal component, the distance between the subject and the imaging system 10 is calculated, and the subject is focused with high accuracy, for example. be able to.

Here, focusing on the pixel unit to be binned, in the pixel arrangement 101 shown in FIG. 2, two pixel units (one first pixel unit 110 and one second pixel unit 120), FIG. 14 In the pixel array 101c shown in (A), four pixel units (two first pixel units 110c and two second pixel units 120c), and in the pixel array 101d shown in FIG. 14 (B), eight. In the pixel unit (4 first pixel units 110d and 4 second pixel units 120d) and the pixel arrangement 101e shown in FIG. 14C, 16 pixel units (8 first pixel units) The binning process is executed on the 110e and the eight second pixel units 120e). That is, in the imaging system 10, binning processing can be executed on 2, 4, 8, 16, ..., 2 ^m (m is a natural number) pixel units.

As described above, in the imaging system 10 according to the present embodiment, ^{even if the binning process is executed not only for the two pixel units but also for the 2 m} pixel units, the binning signal after the binning process is a distance. Since it has a signal component, it is possible to calculate the distance between the subject and the imaging system 10 and focus on the subject with high accuracy, for example.

<Sixth Embodiment>
In the first to fifth embodiments, the binning process in the case where one pixel unit of the pixel unit 100 includes 2 × 2 pixels has been described. However, the 2 × 2 pixels included in the pixel unit is an example, and the number of pixels included in the pixel unit may be 2n × 2n (n is a natural number). Since the number of 2n × 2n pixels can be equally divided into two into 2n × n pixels or n × 2n pixels, for a pixel unit containing 2n × 2n pixels, , The binning process described in the first to fourth embodiments can be applied.

Further, even when one pixel unit includes 2n × 1 pixel, it can be divided into two into n × 1 pixel. Similarly, even when one pixel unit includes 1 × 2 n pixels, it can be divided into two into 1 × n pixels. Therefore, the binning process described in the first embodiment or the second embodiment can be applied to the pixel unit including 2n × 1 or 1 × 2n pixels.

On the other hand, since the number of pixels cannot be equally divided into two for a pixel unit containing 2n + 1 × 2n + 1 pixels, the binning process described in the first to fourth embodiments cannot be directly applied. Can not. Therefore, in the present embodiment, with reference to FIGS. 15 to 17, a binning process applied to a pixel unit including 3 × 3 pixels will be described as an example. In the following, the description may be omitted for the same configurations as those described in the first to fourth embodiments.

FIG. 15 is a schematic diagram illustrating a binning process executed by the binning process unit 200 of the imaging system 10B according to the embodiment of the present invention.

FIG. 15 shows the pixel array 101B in the pixel portion 100B of the imaging system 10B. As shown in FIG. 15, the pixel array 101B of the pixel unit 100B includes a first pixel unit 110B, a second pixel unit 120B, a third pixel unit 130B, and a fourth pixel unit 140B. The second pixel unit 120B is adjacent to the first pixel unit 110B at a diagonal position (diagonal direction) of the first pixel unit 110B. The third pixel unit 130B is adjacent to the first pixel unit 110B in the row direction. The fourth pixel unit 140B is adjacent to the first pixel unit 110 in the column direction. Further, when the position of the first pixel unit 110B and the position of the second pixel unit 120B are relatively represented in the pixel array 101B, the first pixel unit 110B is located on the upper side or the left side of the pixel array 101B. The second pixel unit 120B is located on the lower side or the right side of the pixel array 101B.

The first pixel unit 110B includes the first pixels 111B-1 to 111B-9. The second pixel unit 120B includes the second pixels 121B-1 to 121B-9. The third pixel unit 130B includes the third pixels 131B-1 to 131B-9. The fourth pixel unit 140B includes fourth pixels 141B-1 to 141B-9. Here, the first pixels 111B-1 to 111B-9 and the second pixels 121B-1 to 121B-9 are the first green pixel (G1) and the second green pixel (G2), respectively. Further, the third pixel 131B-1 to 131B-9 and the fourth pixel 141B-1 to 141B-9 are a red pixel (R) and a blue pixel (B), respectively.

The first pixel 111B-1 (G1 _UL ), 111B-2 (G1 _UM ), and 111B-3 (G1 _UR ) are located on the upper side in the first pixel unit 110B, and the first pixel 111B-7. (G1 _DL ), 111B-8 (G1 _DM ), and 111B-9 (G1 _DR ) are located lower in the first pixel unit 110B. Further, the first pixels 111B-4 (G1 _ML ), 111B-5 (G1 _MM ), and 111B-6 (G1 _MR ) are located between the upper side and the lower side in the first pixel unit 110B. There is. In other words, the first pixels 111B-1 (G1 _UL ), 111B-4 (G1 _ML ), and 111B-7 (G1 _DL ) are located on the left side of the first pixel unit 110B and are the first pixels. 111B-3 (G1 _UR ), 111B-6 (G1 _MR ), and 111B-9 (G1 _DR ) are located on the right side in the first pixel unit 110B. Further, the first pixels 111B-2 (G1 _UM ), 111B-5 (G1 _MM ), and 111B-8 (G1 _DM ) are located between the left side and the right side in the first pixel unit 110B. .. The positions of the second pixels 121B-1 (G2 _UL ) to 121B-9 (G2 _DR ) in the second pixel unit 120B, and the third pixels 131B-1 (R _UL ) to 131B of the third pixel unit 130B. The same applies to the positions of −9 (R _DR ) and the positions of the fourth pixels 141B-1 (B _UL ) to 141B-9 (B _DR ) in the fourth pixel unit 140B.

The binning processing unit 200 selectively binners the plurality of pixels described above, and the first binning signal 210B (G1''''), the second binning signal 220B (G2''''), and the third binning signal 230B. (R'''), and a fourth binning signal 240B (B''') are generated. That is, by the binning process, the pixel array 101B has the first binning signal 210B (G1''''), the second binning signal 220B (G2''''), and the third binning signal 230B (R''''). And converted to a binning signal array 201 containing a fourth binning signal 240B (B'''). The second binning signal 220B (G2''') is the first binning signal 210B (G1'''') at the diagonal position (diagonal direction) of the first binning signal 210B (G1''''). Is adjacent to. The third binning signal 230B (R ″ ″) is adjacent to the first binning signal 210B (G1 ″ ″) in the row direction. The fourth binning signal 240B (B ″ ″) is adjacent to the first binning signal 210B (G1 ″ ″) in the column direction. Therefore, when the position of the first binning signal 210B (G1'''') and the position of the second binning signal 220B (G2'''') in the binning signal array 201B are relatively represented, the first binning signal is represented. The 210B (G1'''') is located above or to the left of the binning signal array 201B, and the second binning signal 220B (G2'''') is located below or to the right of the binning signal array 201B.

The third binning signal 230B (R''') and the fourth binning signal 240B (B''') are the third pixels 131B-1 to 131B-9 and the third pixels 131B-1 to 131B-9 included in the third pixel unit 130B, respectively. It is generated by binning the fourth pixels 141B-1 to 141B-9 included in the fourth pixel unit 140B. On the other hand, the first binning signal 210B (G1''') is the first pixel 111B-1 to 111B-9 in the first pixel unit 110B and the second pixel 121B- in the second pixel unit 120B. It is generated by dividing 1 to 121B-9 into two and binning one of the two divided pixels. Similarly, the second binning signal 220B (G2 ″'') is generated by binning the other pixel divided into two. Hereinafter, the binning process for generating the first binning signal 210B (G1 ″ ″) and the second binning signal 220B (G2 ″ ″) will be described with reference to FIGS. 16 and 17.

FIG. 16 is a schematic diagram illustrating a binning process for generating a first binning signal 210B (G1 ′ ″) in the imaging system 10B according to the embodiment of the present invention. The binning processing unit 200 includes the first pixel 111B-1 (G1 _UL ), 111B-4 (G1 _ML ), 111B-7 (G1 _DL ), and the second pixel unit on the left side of the first pixel unit 110B. In addition to the second pixel 121B-1 (G2 _UL ), 121B-4 (G2 _ML ), and 121B-7 (G2 _DL ) on the left side in 120B, the second pixel in the first pixel unit 110B located in the middle. Pixel 111B-2 (G1 _UM ), 111B-5 (G1 _MM ), and 111B-8 (G1 _DM ) of 1 and the second pixel 121B-2 (G2 _UM ), 121B- in the second pixel unit 120B. 5 (G2 _MM ) and 121B-8 (G2 _DM ) are binned to generate a first binning signal 210B (G1''').

FIG. 17 is a schematic diagram illustrating a binning process for generating a second binning signal 220B (G2 ′ ″) in the imaging system 10B according to the embodiment of the present invention. The binning processing unit 200 includes the first pixel 111B-3 (G1 _UR ), 111B-6 (G1 _MR ), 111B-9 (G1 _DR ), and the second pixel unit on the right side of the first pixel unit 110B. In addition to the second pixel 121B-3 (G2 _UR ), 121B-6 (G2 _MR ), and 121B-9 (G2 _DR ) on the right side in 120B, the second pixel in the first pixel unit 110B located in the middle. Pixel 111B-2 (G1 _UM ), 111B-5 (G1 _MM ), and 111B-8 (G1 _DM ) of 1 and the second pixel 121B-2 (G2 _UM ), 121B- in the second pixel unit 120B. 5 (G2 _MM ) and 121B-8 (G2 _DM ) are binned to generate a second binning signal 220B (G2''').

_{The first pixel 111B-2 (G1 UM} ), 111B-5 (G1 _MM ), and 111B-8 (G1 _DM ) in the first pixel unit 110B located in the middle, and the second pixel unit 120B in the second pixel unit 120B. The two pixels 121B-2 (G2 _UM ), 121B-5 (G2 _MM ), and 121B-8 (G2 _DM ) not only generate the first binning signal 210B (G1'''), but also the second It is also used for the binning signal 220B (G2'''). Therefore, the pixel values of the first binning signal (G1''') and the second binning signal 220B (G2''') are the first pixel 111B-2 (G1 _UM ) in the first pixel unit 110B. , 111B-5 (G1 _MM ), and 111B-8 (G1 _DM ) and the second pixel 121B-2 (G2 _UM ), 121B-5 (G2 _MM ), and 121B-8 in the second pixel unit 120B. It is calculated by setting each pixel value of (G2 _{DM) to 1/2.}

Therefore, each pixel value of the first binning signal 210B to the fourth binning signal 240B is calculated by the formula shown in Table 3.

As can be seen from Table 3, the information on the left side of the first pixel unit 110B and the information on the left side of the second pixel unit 120B are aggregated in the first binning signal 210B (G1 ″''). On the other hand, in the second binning signal 220B (G2''', the information on the right side of the first pixel unit 110B and the information on the right side of the second pixel unit 120B are aggregated. Since the binning signal after the binning process contains a distance signal component, high frame rate imaging is possible.

Since the distance measuring process by the distance measuring processing unit 300 after the binning process is the same as that of the first embodiment, the description thereof will be omitted. However, even in the binning process described in the present embodiment, the binning signal after the binning process is a distance signal. Since it contains a component, the distance between the edge portion of the subject in the X direction and the imaging system 10B can be measured, and the subject can be focused with high accuracy, for example.

In addition to the first embodiment, the binning process described in the second to fifth embodiments can be applied to the present embodiment.

Although each embodiment of the present invention has been described above with reference to the drawings, the present invention is not limited to the above-described embodiment and can be appropriately modified without departing from the spirit of the present invention. For example, a system in which a person skilled in the art appropriately adds, deletes, or changes the design based on the imaging system of each embodiment is also included in the scope of the present invention as long as it has the gist of the present invention. Further, each of the above-described embodiments can be appropriately combined as long as there is no contradiction with each other, and technical matters common to each embodiment are included in each embodiment even if there is no explicit description.

In addition, even if other effects different from the effects brought about by the above-described embodiments of the above-described embodiments, those that are clear from the description of the present specification or those that can be easily predicted by those skilled in the art are of course. Is understood to be brought about by the present invention.

10, 10B: Imaging system,
100, 100B: Pixel part,
101, 101c, 101d, 101e, 101B: Pixel array,
110, 110c, 110d, 110e, 110B: first pixel unit,
111, 111B: 1st pixel,
120, 120c, 120d, 120e, 120B: second pixel unit,
121, 121B: Second pixel,
130, 130B: Third pixel unit,
131, 131B: 3rd pixel,
140, 140B: 4th pixel unit,
141, 141B: Fourth pixel,
200: Binning processing unit,
201, 201A, 201B: Binning signal arrangement,
210, 210A, 210B: First binning signal,
220, 220A, 220B: Second binning signal,
230: Third binning signal,
240: Fourth binning signal,
300: Distance measurement processing unit, 400: Sensor unit, 500: Image signal processing unit, 600: Storage unit, 700: Display unit,
1000: Circuit layer, 1010: Switch part, 1100: Photoelectric conversion element layer, 1110: Photoelectric conversion element, 1120: Element separation part, 1200: Color filter layer, 1210: Green color filter, 1230: Red color filter, 1240: Shading Unit, 1300, 1300a: Optical element layer, 1310, 1310a, 1310b: Microlens array, 1320a, 1320b: Transmission type diffraction grating

Claims (11)

^{A pixel unit containing 2 m} (m is a natural number) pixel units, each containing a plurality of pixels of the same color,
The plurality of pixels included in each of the ^{2 m} pixel units are divided into a first side pixel and a second side pixel, and the first one included in each of the 2 ^{m pixel units.} A binning processing unit that generates a first binning signal obtained by binning the pixels on the side of the second side and a second binning signal obtained by binning the pixels on the second side included in each of the 2 ^{m pixel units.} , Imaging system. The 2 ^m pixel unit includes a first pixel unit and a second pixel unit.
The imaging system according to claim 1, wherein the second pixel unit is adjacent at a diagonal position of the first pixel unit. The imaging system according to claim 1 or 2, wherein the first-side pixel and the second-side pixel are divided along a column direction. The imaging system according to claim 1 or 2, wherein the first-side pixel and the second-side pixel are divided along a row direction. In the second k-1 frame (k is an integer of 1 or more), the first side and the second side are on the left side and the right side, respectively.
The imaging system according to claim 1 or 2, wherein in the second k frame, the first side and the second side are upper and lower sides, respectively. Moreover,
Any one of claims 1 to 5, which includes a distance measuring processing unit that generates distance information including a distance to a subject based on the pixel value of the first binning signal and the pixel value of the second binning signal. The imaging system according to claim 1. The imaging system according to any one of claims 1 to 6, wherein the pixel portion includes an optical element layer in which one convex lens is provided for each of the ^{2 m pixel units.} The pixel portion has one convex lens for each of the plurality of pixels, and a groove on the convex lens along a direction perpendicular to the direction from the first side pixel to the second side pixel. The imaging system according to any one of claims 1 to 6, further comprising an optical element layer including a provided diffraction grating. A first pixel unit, a second pixel unit, a third pixel unit, and a pixel unit including a fourth pixel unit, each containing a plurality of pixels of the same color.
The plurality of pixels of each of the first pixel unit and the second pixel unit are divided into a first side pixel and a second side pixel, and the third pixel unit and the fourth pixel unit are divided. A first binning signal obtained by dividing each of the plurality of pixels of the pixel unit into a third side pixel and a fourth side pixel, and binning the first side pixel, the second side pixel. A binning processing unit that generates a second binning signal obtained by binning a pixel, a third binning signal obtained by binning a pixel on the third side, and a fourth binning signal obtained by binning a pixel on the fourth side. Including
The first-side pixel and the second-side pixel are divided along the column direction.
An imaging system in which the third side pixel and the fourth side pixel are divided along a row direction. The imaging system according to claim 9, wherein the third pixel unit is adjacent at a diagonal position of the second pixel unit. Moreover,
Distance information including the distance to the subject based on the pixel value of the first binning signal, the pixel value of the second binning signal, the pixel value of the third binning signal, and the pixel value of the fourth binning signal. The imaging system according to claim 9 or 10, further comprising a distance measuring processing unit for generating the image.

Images