KR20210137886A - Image sensor - Google Patents

Abstract

An imaging device may encode light passing through an imaging optical lens disposed in a multi-lens array, transmit it to a sensing element, and restore an image based on information sensed by the sensing element. The imaging device includes: a mask array comprising a plurality of mask elements blocking light in a partial direction among light passing through the imaging optical lenses, and passing the light in a direction different from the partial direction; and the plurality of sensing elements for sensing the light passing through the imaging optical lenses and the mask array.

Description

image sensor {IMAGE SENSOR}

Hereinafter, a technique for sensing an image is provided.

Due to the development of optical technology and image processing technology, a photographing device is being used in a wide range of fields such as multimedia content, security, and recognition. For example, the photographing device may be mounted on a mobile device, a camera, a vehicle, a computer, or the like to capture an image, recognize an object, or acquire data for controlling the device. The volume of the photographing apparatus may be determined by the size of the lens, the focal length of the lens, the size of the sensor, and the like. When the size of the lens is reduced, the focal length of the lens may be reduced, and in order to reduce the volume of the photographing apparatus, a multi-lens composed of small lenses may be used.

According to an exemplary embodiment, the image sensor includes: a mask array including a plurality of mask elements blocking light in a partial direction among light passing through the imaging optical lenses, and passing light in a direction different from the partial direction; and a sensing array including a plurality of sensing elements sensing the light passing through the imaging optical lenses and the mask array.

The image sensor may be disposed on the sensing array and further include a color filter configured to filter a partial wavelength band from light incident on each sensing element, and the mask array may be disposed between the color filter and the sensing array. .

The image sensor may further include a converging lens array disposed on the sensing array to transmit light to each sensing element, and the mask array may be disposed between the converging lens array and the sensing array.

The mask array and the sensing element may be spaced apart within 1 μm.

The mask array and the sensing element may contact each other.

The region corresponding to the individual sensing element in the mask array may include an aperture region occupying an area corresponding to an aperture ratio to a total area of the corresponding region; and a masked region occupying the remaining area in which the plurality of mask elements are disposed in the corresponding region.

The aperture ratio may be 40% or more and 60% or less.

An area occupied by an aperture part in each partial area of the mask array may be greater than or equal to an area corresponding to a set aperture ratio.

The mask array is segmented into a plurality of group regions corresponding to a plurality of sensing device groups, and each of the plurality of group regions in the mask array includes a plurality of sensing groups grouped to represent one pixel. A sensing element group including elements may be covered.

In the mask array, the masking pattern of the group region may be repeated.

Each of the plurality of group regions of the mask array may have the same masking pattern.

The region corresponding to the individual masking pattern in the mask array may include more than the number of imaging optical lenses of the imaging optical system.

The plurality of mask elements may have one of two or more transmission levels.

Each of the plurality of mask elements may be divided into a plurality of regions, and transmittance of each mask element may be determined according to a ratio of an open region and a closed region among the plurality of regions.

The image sensor may further include a processor for reconstructing an image based on sensing information sensed by the plurality of sensing elements.

The processor converts the sensed information into a frequency domain to generate frequency information, divides the frequency information by a frequency conversion result of a blur kernel corresponding to the masking pattern of the mask array, and deblurs. A high-resolution image may be reconstructed by generating (deblurred) frequency information and inversely transforming the deblurred frequency information into a time domain.

The mask array may include a plurality of masking patterns, and each masking pattern may cover a sensing element group including two or more sensing elements in the sensing array.

A camera device according to an embodiment includes: an imaging lens array including optical imaging lenses that transmit light received from the outside; a sensing array including a plurality of sensing elements for sensing the light passing through the imaging lens array; and a mask array including a plurality of mask elements and disposed between the imaging lens array and the sensing array.

The mask array may be disposed on one of a position in contact with the sensing array and a position inside the sensing array on the plurality of sensing elements.

In the mask array, the region corresponding to the individual sensing element may include: an opening region occupying an area corresponding to an opening ratio to a total area of the corresponding region; and a masking region occupying the remaining area in which the plurality of mask elements are disposed in the corresponding region.

An area occupied by an opening part in each of the partial regions in the mask array may be equal to or greater than an area corresponding to a set opening ratio.

In the mask array, the masking pattern of the group region may be repeated.

The camera device converts the sensing information sensed by the plurality of sensing elements into a frequency domain to generate frequency information, and divides the frequency information by a frequency conversion result of a blur kernel corresponding to the masking pattern of the mask array. ) may further include a processor for generating deblurred frequency information and reconstructing a high-resolution image by inversely transforming the deblurred frequency information into a time domain.

1A and 1B show the structure of an imaging device according to an embodiment.
2 is a diagram illustrating a state in which a sensing element according to an embodiment receives a light beam through a lens element.
3 is a diagram illustrating a relationship between the number of sensing elements and the number of lens elements according to an exemplary embodiment.
4 illustrates a reduction in a focal length according to a multi-lens array structure in an imaging apparatus according to an exemplary embodiment.
5 illustrates a blur kernel according to a multi-lens array structure in an imaging apparatus according to an embodiment.
6 illustrates a blur kernel of an imaging device including a mask array according to an exemplary embodiment.
7 is a cross-sectional view of an imaging device in which a mask array is disposed according to an embodiment.
8 illustrates a design of a masking pattern for a mask array according to an exemplary embodiment.
9 and 10 show exemplary shapes of a mask array according to an embodiment.
11 is a view for explaining arrangement of a masking pattern for each sensing element group of an image sensor according to an exemplary embodiment.
12A to 12B illustrate an arrangement of a mask array according to an exemplary embodiment.
13 is a block diagram illustrating a configuration of an imaging apparatus according to an exemplary embodiment.
14 is a block diagram illustrating a configuration of an electronic terminal according to an embodiment.
15 and 16 are diagrams illustrating examples of a device in which an image sensor is implemented according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these examples. It should be understood that the embodiments described below are not intended to limit the embodiments, and include all modifications, equivalents or substitutes thereto.

Terms used in the examples are used only to describe specific examples, and are not intended to limit the embodiments. The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

1A and 1B show the structure of an imaging device according to an embodiment. 1A is a perspective view of an imaging device, and FIG. 1B is a cross-sectional view of the imaging device.

The imaging apparatus 100 includes a lens array 110 and an image sensor 120 . The lens array 110 includes lens elements, and the image sensor 120 includes optical sensing elements. The lens elements may be disposed along the plane of the lens array 110 , and the optical sensing elements may be disposed along the plane of the sensing array 121 in the image sensor 120 . A plane of the lens array 110 may be disposed parallel to a plane of the sensing array 121 . The lens array 110 is a multi-lens array (MLA) for imaging, and may also be referred to as an imaging lens array.

In the present specification, an optical sensing element (hereinafter, 'sensing element') is an element that senses optical information based on light incident on the element, and may output a value indicating the intensity of the incident light. The optical sensing device may include, for example, a complementary metal oxide semiconductor (CMOS), a charge-coupled device (CCD), and a photo diode.

In the present specification, a picture element (hereinafter, a pixel) is basic unit information constituting an image, and light reflected from a physical location on a subject corresponding to the pixel location provides optical information sensed by the sensing element. can indicate A pixel location is the location of a pixel in an image, which may follow a pixel coordinate system, and a physical location may follow a world coordinate system.

For reference, a pixel constituting a color image has a plurality of color values (eg, a red value, a green value, and a blue value in the case of an RGB color system) for one pixel position. In the display field, a unit pixel constituting a display is sub-pixels related to a plurality of colors (eg, in the case of an RGB color system, a red sub-pixel, a green sub-pixel, and a blue sub-pixel) in order to express color values of one pixel position. ) may be included. Contrary to this, in the image sensor field, it is common to refer to a sensing element (eg, a photodiode having a color filter disposed at the front end) that senses one color value instead of dividing the pixel into sub-pixels for each color. Also, in the image sensor field, a pixel is used interchangeably to refer to a sensing element and a value sensed by the sensing element. However, in the present specification, for clarity, a pixel is basic unit information constituting an image, and a sensing element is a hardware element that outputs a pixel value of a corresponding pixel in response to light received from a subject.

In this specification, an example in which each pixel is expressed as a single sensing element is mainly described, but the present disclosure is not limited thereto, and one pixel may be expressed as a plurality of sensing elements. A plurality of sensing elements grouped to represent one pixel may be referred to as a sensing element group. There is a limit to the amount of light that can be sensed by one sensing element, and sensitivity may be improved by expressing one pixel using values sensed by a plurality of sensing elements. 11, which will be described later, describes an example in which one pixel value is sensed by a sensing element group including four sensing elements.

The image sensor 120 may include a sensing array 121 , an optical filter 122 , and a condensing lens array 123 . However, the present invention is not limited thereto, and instead of the optical filter 122 , an individual condensing micro lens 123a of the condensing lens array 123 passes a predetermined wavelength band and blocks the remaining wavelength band. It may be configured to have

The condensing lens array 123 may include a plurality of condensing microlenses for concentrating the light passing through the lens array 110 to the sensing array 121 . For example, the condensing lens array 123 may include the same number of condensing microlenses as the number of sensing elements included in the sensing array 121 . The plurality of condensing microlenses may be disposed between the imaging optical lens and the sensing array 121 to condense and transmit light passing through the imaging optical lens to the sensing element 121a corresponding to each condensing microlens 123a. For example, as shown in FIG. 1B , a condensing microlens 123a is disposed on each sensing element 121a of the sensing array 121 to condense light to the sensing element 121a disposed below. can In addition, as shown in FIG. 1B , a color filter 122a may be disposed between each condensing microlens 123a and the sensing element 121a.

The optical filter 122 may be a filter having an optical characteristic of passing a predetermined wavelength band and blocking the remaining wavelength band. For example, the optical filter 122 may be implemented as a color filter array (CFA) including a plurality of color filters disposed along a filter plane. Each color filter 122a may be a filter that passes light of a wavelength band corresponding to an arbitrary color and blocks light of the remaining band. For example, as the color filter 122a, there may be a red pass filter, a green pass filter, and a blue pass filter. The red pass filter may pass light of a wavelength band corresponding to red and block light of the remaining band. The green pass filter may pass light of a wavelength band corresponding to green and block light of the remaining band. The blue pass filter may pass light of a wavelength band corresponding to blue and block light of the remaining band. The color filters that individually pass color lights in the color filter array may be arranged in a Bayer pattern or other pattern along the filter plane. The optical filter 122 may also be an infrared cut filter that passes the visible light band and blocks the infrared band.

The quality of the image captured and restored by the image sensor 120 may be determined by the number of sensing elements included in the sensing array 121 and the amount of light incident on the sensing element 121a. For example, the resolution of the image may be determined by the number of sensing elements included in the sensing array 121 , the sensitivity of the image may be determined by the amount of light incident on the sensing element 121a, and the amount of incident light may be determined by the sensing element 121a. ) may be determined based on the size of As the size of the sensing element 121a increases, the amount of incident light may increase, and the dynamic range of the sensing array 121 may increase. Accordingly, as the number of sensing elements included in the sensing array 121 increases, the image sensor 120 may capture a high-resolution image, and as the size of the sensing element 121a increases, the image sensor 120 becomes highly sensitive in low light. It can work advantageously for image taking.

The individual lens elements 111 of the lens array 110 may cover a certain sensing area 129 of the sensing array 121 corresponding to its own lens size. The sensing area 129 covered by the lens element 111 in the sensing array 121 may be determined according to the lens size of the corresponding lens element 111 . The sensing region 129 may indicate a region on the sensing array 121 in which light rays of a certain viewing angle range reach after passing through the corresponding lens element 111 . The size of the sensing region 129 may be expressed as a distance or a diagonal length from the center of the sensing region 129 to an outermost point. In other words, the light passing through the respective lens element 111 may be incident to the sensing elements included in the sensing region 129 .

Each of the sensing elements of the sensing array 121 may generate sensing information based on light rays passing through the lenses of the lens array 110 . For example, the sensing element 121a may sense an intensity value of light received through the lens element 111 as sensing information. The imaging device 100 determines intensity information corresponding to an original signal with respect to points included in the field of view of the imaging device 100 based on the sensing information output by the sensing array 121, The captured image may be restored based on the determined intensity information.

Also, the sensing element 121a may generate a color intensity value corresponding to a desired color as sensing information by sensing the light passing through the color filter 122a. Each of the plurality of sensing elements constituting the sensing array 121 may be arranged to sense a color different from that of a spatially adjacent adjacent sensing element.

When the diversity of the sensing information is sufficiently secured and a full rank relationship is formed between the sensing information and the original signal information corresponding to points included in the field of view of the imaging apparatus 100, the sensing array 121 A photographed image corresponding to the maximum resolution may be derived. The diversity of sensing information may be secured based on parameters of the imaging apparatus 100 such as the number of lenses included in the lens array 110 and the number of sensing elements included in the sensing array 121 .

In the multi-lens array structure for imaging, the imaging optical lens and the sensing array 121 may be arranged in a fractional alignment structure. For example, the fractional alignment structure may indicate a structure in which the sensing region 129 covered by the individual lens element 111 includes a non-integer number of sensing elements.

When the lens elements included in the lens array 110 have the same lens size, the number of lens elements included in the lens array 110 and the number of sensing elements included in the sensing array 121 are relatively prime. could be a relationship. A ratio P/L between the number L of lens elements corresponding to one axis of the lens array 110 and the number P of sensing elements corresponding to one axis of the sensing array 121 may be determined as a real number. Each of the lens elements may cover a number of sensing elements equal to an offset corresponding to P/L. For reference, the sensing region 129 illustrated in FIG. 1A may include 7/3 = 2.3 sensing elements along the vertical axis and 11/3 = 3.67 sensing elements along the horizontal axis. Furthermore, the lens element 111 may cover a plurality of non-integer condensing microlenses. Accordingly, the number of converging microlenses in the image sensor 120 is the same as the number of sensing elements of the sensing array 121 , and the number of lens elements (eg, imaging optical lenses) of the lens array 110 is It may be smaller than the number of microlenses.

Through the fractional alignment structure as described above, the imaging device 100, the optical center axis (OCA) of each lens element 111 is slightly different from each other with respect to the sensing array 121 (slightly) arrangement can have In other words, the lens elements 111 may be disposed to be eccentric with respect to the sensing element 121a. Accordingly, each lens element 111 of the lens array 110 receives different light field (LF) information. The light field information received by the fractional alignment structure will be described with reference to FIG. 2 below.

2 is a diagram illustrating a state in which a sensing element according to an embodiment receives a light beam through a lens element.

The light field may be emitted from any target point and may represent a field representing the direction and intensity of light rays reflected at any point on the subject. The light field information may indicate information in which a plurality of light fields are combined. Since the direction of a chief ray of each lens element is also changed, each sensing region receives different light field information, so that the imaging apparatus may optically acquire more sensing information.

As shown in FIG. 2 , the sensing array 220 may receive and detect light rays corresponding to individual points 230 (X1 to X10). A plurality of rays emitted from each of the individual points 230 may form a light field. Light rays emitted from the first point X1 may form a first light field, and may be incident on the first sensing element S1 , the fourth sensing element S4 , and the seventh sensing element S7 . . Light rays emitted from each of the remaining points X2 to X10 may also form a corresponding light field. The individual points 230 may be points on an arbitrary object (eg, a subject). The rays emitted from the individual points 230 may be rays reflected from an object, such as sunlight. 2 is an exemplary cross-sectional view of an imaging device. For convenience of description, the lens array 210 includes three lens elements along one axis, and the sensing array 220 includes ten sensing elements S1 to S10. Although shown to be, it is not limited thereto.

The sensing elements S1 to S10 may overlap and sense light rays passing through the plurality of lens elements. The sensing element S1 may generate sensing information (eg, intensity value) in which light rays emitted from the points X1 to X3 are superimposed, and the other sensing elements S2 to S10 also overlap the sensing information. can create The image sensor may restore such overlapped sensing information.

The sensing information generated by the sensing elements S1 to S10 shown in FIG. 2 is, according to Equation 1 below, original signal information (eg, intensity) corresponding to light rays incident from each of the points 230 . value) can be modeled.

[Equation 1]

In Equation 1, S may represent a matrix indicating sensing information (eg, a detected intensity value) sensed by an individual sensing element. X may represent a matrix indicating a signal value (eg, a color intensity value of the incident light beam) corresponding to a light beam incident from an individual point to the sensing elements S1 to S10 . T is a transformation matrix and may represent a relationship between sensing information detected by the sensing elements S1 to S10 and signal information corresponding to incident light. Light rays, lens elements, and sensing elements S1 to S10 corresponding to the individual points X1 to X10 in the structure shown in FIG. 2 may be modeled as in Equation 2 below. In Equation 2 below, the individual points X1 to X10 may be modeled as being disposed at an infinite focal point from the image sensor. The distance between the individual points X1 to X10 and the image sensor may be greater than a threshold distance.

[Equation 2]

For convenience of explanation in Equation 2, signal information (eg, light intensity values) of light rays corresponding to each of the individual points X1 to X10 is expressed as X1 to X10. In addition, sensing information (eg, a sensing intensity value) detected by the sensing elements S1 to S10 is indicated by S1 to S10. The relationship between the sensing information corresponding to the sensing elements S1 to S10 constituting the sensing array 220 and the original signal corresponding to the rays X1 to X10 incident from individual points (eg, the transformation matrix described above) ) may be determined based on the arrangement between the above-described lens element and the sensing element, the number of lens elements constituting the lens array 210 , the number of sensing elements S1 to S10 constituting the sensing array 220 , and the like.

Equation 2 described above describes the case where the individual points X1 to X10 are infinity-focused from the image sensor, and the individual points X1 to X10 are located at finite focal points from the image sensor. In this case, the original signal received by each sensing element may vary depending on the distance between the subject and the image sensor and the geometry of the image sensor.

The imaging apparatus may acquire a plurality of low-resolution input images through the various sensing information acquired as described above, and may reconstruct a higher-resolution output image from the low-resolution input images. 3 below describes a method of generating a single image by rearranging a plurality of low-resolution input images.

3 is a diagram illustrating a relationship between the number of sensing elements and the number of lens elements according to an exemplary embodiment.

As described above, the imaging optical lens and the sensing array may be arranged in a fractional arrangement, and FIG. 3 shows an example in which the ratio P/L between the number L of lens elements and the number P of sensing elements is 10/3.

According to the above-described geometric structures of the lens array and the sensing array, sensing elements covered by each lens element may receive light field information that is not the same as light field information sensed by a sensing element covered by another lens element. . For example, in the structure shown in FIG. 2 , the first sensing element S1 includes a first light field at a first point X1, a second light field at a second point X2, and a third point X3. It is possible to receive light field information that is a combination of the third light field of . On the other hand, the second sensing element S2 adjacent thereto may receive light field information in a combination of the fourth light field, the fifth light field, and the sixth light field in the structure shown in FIG. 2 . In this way, each sensing element may receive light field information different from light field information sensed by other sensing elements.

The imaging device and/or the image sensor may determine, in order to restore a high-resolution image, a pixel position in an image of pixels representing the same or adjacent point on a subject in a plurality of captured low-resolution images based on a correlation between light field information can be rearranged.

For reference, if the image is a color image, it may have color values according to the color system as pixel values, but the image sensor cannot sense three colors at the same time at one point due to a physical limitation. Since a color filter that allows only one color to pass through is generally disposed at the front end of the sensing element, a color that can be sensed at a location of each sensing element may be different from that of an adjacent sensing element. The imaging device and/or the image sensor may display a color value (eg, a red color value) that is not sensed by a sensing element at a specific location (eg, a sensing element having a blue pass filter disposed at the front end) located in the vicinity thereof. Interpolation may be performed using a color value sensed by a sensing element (eg, a sensing element having a red pass filter disposed at the front end). The imaging device and/or the image sensor may acquire three color channel images by performing interpolation for each color channel. However, the color value interpolation described above is purely an example, and other methods may be performed according to design. The imaging device and/or the image sensor may perform pixel rearrangement, which will be described later, for each color channel. For example, in the RGB color system, the imaging device and/or the image sensor may reconstruct a high-resolution red channel image by rearranging pixels of the low-resolution red channel images. Similarly, the imaging device and/or image sensor may reconstruct a high-resolution blue channel image and a green channel image. Accordingly, the imaging device and/or the image sensor may acquire a high-resolution color image. However, the present invention is not limited thereto, and the imaging device and/or image sensor acquires low-resolution color images by merging three channel images obtained through interpolation as described above, and rearranges pixels of the low-resolution color images. It is also possible to restore high-resolution color images.

The imaging device and/or the image sensor may configure pixel information of a high-resolution image by rearranging positions of pixels corresponding to sensing elements that receive similar light field information to be adjacent to each other. As described above, each sensing element may receive information in which a plurality of light fields are overlapped. The more the information sensed by the two sensing elements includes the same light field, the higher the correlation between the two pieces of information. The rearrangement of pixel positions may be performed in consideration of the depth at which the corresponding pixel is photographed. For example, the depth at which the pixel is photographed may be set to an arbitrary depth value, estimated by stereo image matching, or measured by a depth sensor. As another example, the rearrangement of the pixel position may be performed by a neural network designed to rearrange the pixel position in consideration of the depth at which the object is captured without measuring and/or estimating the depth at which the pixel is captured. This rearrangement of pixel positions may be referred to as pixel shuffle. Illustratively, a neural network designed to output a single high-resolution output image from several low-resolution input images may be used to rearrange pixel positions. The neural network may be trained based on a training data set obtained by photographing subjects of various depths.

The image sensor may determine light field information to be sensed by each sensing element, assuming that points that reflect light rays are disposed at an infinity focus position that is farther than a threshold distance from the image sensor. The image sensor may rearrange the pixel positions so that the pixel positions of pixels having output values output by the sensing element receiving the light field emitted from points spatially adjacent to each other on the subject are adjacent to each other.

For reference, the individual points X1 to X10 in FIG. 2 may be illustrated in order of being spatially adjacent to each other at an infinite focal length. The first point X1 may be adjacent to the second point X2 , and the second point X2 may be adjacent to the first point X1 and the third point X3 .

Both the light field information sensed by the first sensing element S1 and the light field information sensed by the eighth sensing element S8 among the sensing elements 311 before being rearranged are the second point X2 and the second point X2. A light field corresponding to three points X3 may be included. Accordingly, the first sensing element S1 and the eighth sensing element S8 may receive light field information similar to each other. When Equation 2 is rearranged so that pixels corresponding to light field information similar to each other are adjacent to each other, Equation 3 below can be expressed.

[Equation 3]

The sensing elements 312 rearranged according to Equation 3 above may be illustrated as shown in FIG. 3 . The first sensing element S1 may be covered by the first lens, the eighth sensing element S8 may be covered by the third lens, and the fifth sensing element S5 may be covered by the second lens. Since sensing information sensed by each sensing element corresponds to a pixel constituting an image, the image sensor and/or imaging device may rearrange the pixels so that sensing information corresponding to rays passing through different lenses are adjacent to each other. . In the rearranged image 325, the pixel positions of pixels having values sensed by sensing elements that have received similar light field information in the low-resolution images 321, 322, 323, and 324 photographed by individual lenses are adjacent to each other. It may be a placed image.

4 illustrates a reduction in a focal length according to a multi-lens array structure in an imaging apparatus according to an exemplary embodiment.

The volume of the imaging device may be determined by the focal length of the lens element. This is because the image sensor must be disposed to be spaced apart from the lens element by a distance corresponding to the focal length of the lens element in order to collect the light refracted by the lens element. The focal length of the lens element is determined by the viewing angle of the imaging apparatus 100 and the size of the lens element. When the viewing angle is fixed, the focal length is increased in proportion to the size of the lens element, and in order to take an image in a constant viewing angle range, the size of the lens element must be increased as the size of the sensing array increases.

As described above, in order to increase the sensitivity of the image while maintaining the viewing angle and the resolution of the image, the volume of the image sensor is increased. In order to increase the sensitivity of the image while maintaining the resolution of the image, the size of each sensing element must be increased while maintaining the number of sensing elements included in the sensing array, so that the size of the sensing array is increased. In order to maintain the viewing angle, as the size of the sensing array increases, the size of the lens element increases and the focal length of the lens element increases, thereby increasing the volume of the image sensor.

As the size of each of the lens elements included in the lens array is reduced, that is, as the number of lenses included in the same area on the lens array is increased, the focal length of the lens element may become smaller, so that the thickness of the imaging device is reduced. A thin camera may be implemented. As shown in FIG. 4 , compared to the focal length f in the case of the single lens 410 , the focal length f′ in the case of the multi-lens 420 may be reduced. For example, when 2Х2 multi-lenses 420 are used instead of the single lens 410, the focal length f'=f/2.

However, in the multi-lens 420 , the area S' on which light is incident on the individual lenses may be smaller than the area S' on which the light is incident on the single lens 410 . For example, the incident area of the 2Х2 multi-lens 420 may be S'=S/4. Also, an incident solid angle Ω corresponding to an angular range of light rays incident on an individual sensing element (eg, S2 of FIG. 4 ) may increase due to a decrease in focal length. For example, the incident solid angle Ω′=4 Ω corresponding to the individual sensing elements of the multi-lens 420 may be. In FIG. 5 , a blur kernel according to an increase in the incident solid angle will be described.

5 illustrates a blur kernel according to a multi-lens array structure in an imaging apparatus according to an embodiment.

When the field of view (FoV) of each lens in the structure of the multi-lens array 510 is designed to be the same, as described above in FIG. 4 , as the number of lenses increases relative to the same area, the focal length decreases and the incident solid angle This can increase. As the incident solid angle increases, signals overlapped by the sensing element may increase, and as the signals overlap, information sensed by the sensing element may be blurred. For example, in a single-lens imaging device, if FoV is 100 degrees and the number of sensing elements of the image sensor 520 is 100, FoV of each sensing element is 1 degree. If the number of lenses is two in the same area, the number of sensing elements covered by each lens in the corresponding image sensor 520 becomes 50. Accordingly, the FoV of each sensing element becomes 2 degrees. If the number of lenses is 50 in the same area, the number of sensing elements covered by each lens in the image sensor 520 becomes two, and the FoV of each sensing element becomes 50 degrees. As the FoV that can be sensed by an individual sensing element increases, more light field information overlaps, and thus the blur level of the sensing information also increases.

내지

의 라이트 필드 정보가 렌즈 구경 S 만큼 집광되고, 집광된 라이트 필드 정보가 제i 센싱 소자에서 센싱될 수 있다.

는 센싱 소자의 최외곽 지점에서 집광되는 광선 다발을 나타내고,

는 해당 센싱 소자의 반대편 최외곽 지점에서 집광되는 광선 다발을 나타낼 수 있다. 여기서 i는 1 이상 n이하의 정수이고, n은 이미지 센서(520)에 포함된 센싱 소자들의 총 개수일 수 있다. 집광된 광선 다발의 신호 세기는 하기 수학식 4 및 수학식 5와 같이 나타낼 수 있다.The blur kernel is a kernel in which blur generated due to overlap of light field information in individual sensing devices is modeled, and may also be referred to as a blur model or a blur filter. Through the multi-lens array 510

inside

The light field information of may be condensed by the lens aperture S, and the condensed light field information may be sensed by the ith sensing element.

denotes a bundle of rays condensed at the outermost point of the sensing element,

may represent a bundle of light rays condensed at the outermost point opposite to the corresponding sensing element. Here, i is an integer greater than or equal to 1 and less than or equal to n, and n may be the total number of sensing elements included in the image sensor 520 . The signal intensity of the focused ray bundle can be expressed as Equations 4 and 5 below.

[Equation 4]

[Equation 5]

In Equation 4, x _a may represent the concentration of light rays oriented in the a direction, and in Equation 5, x _b may represent the concentration of light rays oriented in the b direction.

The ith sensing element may sense an intensity value accumulating all light field information in FoV(Ω). When the intensity value of the accumulated light field information within the viewing angle of the i-th sensing element is discretely approximated, it can be expressed as Equation 6 below.

[Equation 6]

In Equation 6, s[i] may represent an intensity value sensed by the i-th sensing element. In an imaging device configured with an imaging optical system including a multi-lens array 510, the blur kernel of the image may be modeled as overlapping light field information in FoV of each sensing element with the same size (eg, 1). . The sensing information s[i] sensed by the i-th sensing element can be modeled as Equation 7 below as a convolution relationship between the _{original light field information x Ω [i] and the Uniform Blur kernel h[i].} have.

[Equation 7]

In Equation 7 described above, h[i] may represent a blur kernel, and t[i] may represent a noise component. When the above-described Equation 7 is Fourier transformed, the convolution operation can be replaced with multiplication as shown in Equation 8 below.

[Equation 8]

In Equation 8, X _Ω (f) may represent frequency information of an original signal, H(f) may represent a frequency response characteristic of a blur kernel, and T(f) may represent frequency information of noise. From Equation 8 described above, frequency information X _Ω (f) of the original signal to be restored can be calculated as Equation 9 below.

[Equation 9]

When an inverse Fourier transform is applied to Equation 9 above, deblurred x _Ω [i] may be obtained.

However, since T(f), which is a noise component in Equation 9, is an unknown component even if it is modeled with a statistical probability distribution, an error may occur due to T(f))/H(f). As shown in FIG. 5 , a noise component may be amplified due to the uniform blur kernel h[i] in the spatial domain. The frequency response characteristic H(f) in which the uniform blur kernel h[i] is frequency-converted may include zero crossing points 590, and in the reciprocal of the frequency response characteristic H(f), these zero crossing points 590 As a result, impulse components may be generated. Since the impulse components are multiplied by the noise component, noise may be greatly amplified during the deblur process.

6 illustrates a blur kernel of an imaging device including a mask array according to an exemplary embodiment.

The blur kernel h'[i] of the imaging device may be designed such that a noise component is suppressed. For example, the blur kernel h'[i] may be designed such that the zero crossing points 690 are minimized in the frequency response characteristic H'(f) of the frequency domain.

The mask array may include a mask element disposed between the imaging lens array and the sensing element to block light in some directions. For example, the mask array blocks light in some directions among the light passing through the multi-lens array and selectively passes light in other directions to form a blur kernel h'[i] as shown in FIG. can

7 is a cross-sectional view of an imaging device in which a mask array is disposed according to an embodiment.

The imaging device may include an imaging lens array 710 and an image sensor 720 . The imaging lens array 710 may include optical imaging lenses that transmit light received from the outside, and may be disposed in the form described above in FIG. 1 . The imaging optical lens may form an image forming optical system.

The imaging optical system is an optical system that performs imaging to the sensing array 721 , and the optical characteristics are the imaging lens array 710 and the sensing array 721 together with the focal length, size, shape and structure of the imaging lens array 710 . ) may be determined by a geometrical relationship between them. For reference, the imaging optical system may further include a blocking unit 711 that prevents light passing through individual imaging optical lenses from reaching other sensing regions covered by other imaging optical lenses. In addition, the imaging optical system may further include an aperture (not shown) for transmitting light to the imaging optical lens.

The sensing array 721 may include a plurality of sensing elements for sensing light received from the outside. Each of the sensing elements may receive light in a plurality of directions, and a light bundle incident on the imaging device along one direction may be condensed into light rays in the corresponding direction by an imaging optical lens. A sensing element belonging to an arbitrary sensing region may receive a light beam corresponding to each of a plurality of directions focused by an imaging optical lens covering the sensing region. In FIG. 7 , a light beam 791 in a first direction and a light beam 792 in a second direction passing through the imaging optical lens and directed toward the sensing element will be described as examples.

As described above, since the imaging optical system includes the imaging lens array 710 having a multi-lens structure, individual sensing elements of the sensing array 721 may overlap and sense light received from various directions. In other words, the overlap of light imaging on the sensing array 721 may be modeled as the blur kernel shown in FIG. 6 .

The mask array 724 includes a plurality of mask elements and may be disposed on the sensing array 721 . The mask element may be disposed on a position where the sensing element of the sensing array 721 is disposed, and may absorb and block some or all of light directed to the corresponding position. The mask array 724 may correct the imaging optical system corresponding to the blur kernel shown in FIG. 5 to the optical system corresponding to the blur kernel h'[i] shown in FIG. 6 . The mask array 724 may be interpreted as coding light passing through the imaging optical lens and transmitting it to the sensing array 721 , and may also be referred to as a coded mask array 724 . The mask array 724 may be disposed in a position in contact with the sensing array or within the sensing array. It is preferable that there is no gap between the mask array 724 and the sensing element (for example, the photodiodes 721a to 721e) regardless of where they are disposed.

The plurality of mask elements may be arranged in a pattern in which zero crossing of the frequency response characteristic of the blur kernel is minimized. For example, the first ray 791 passing through the imaging optical lens may be incident on the sensing element through between the mask elements. The second ray 792 may be incident on the mask element and absorbed. The second ray 792 may be a ray that induces zero crossing in the frequency response characteristic of the blur kernel described with reference to FIG. 6 . Accordingly, the mask array 724 may block light rays in the direction causing the zero crossing.

The imaging device and/or the image sensor 720 may overlap and sense light field information in a direction selectively filtered through the mask array 724 . Accordingly, the imaging device including the mask array 724 and/or the image sensor 720 may reconstruct an image with reduced noise. While noise due to loss of light quantity by the mask array 724 slightly increases, the noise suppression effect by the blur kernel corrected by the mask array 724 may significantly increase. As a result, image restoration quality may be improved by the masking pattern of the mask array 724 .

The plurality of mask elements may be formed and/or disposed along the masking pattern. The masking pattern may be repeated in units of sensing elements or groups of sensing elements. A first masking pattern disposed on the first sensing device 721a, a second masking pattern disposed on the second sensing device 721b, a third masking pattern disposed on the third sensing device 721c, and a fourth The fourth masking pattern disposed on the sensing device 721d and the fifth masking pattern disposed on the fifth sensing device 721e may be the same.

The design of the masking pattern for the mask array 724 will be described with reference to FIG. 8 below.

8 illustrates a method of designing a masking pattern of a mask array according to an exemplary embodiment.

The plurality of mask elements may be formed in a pattern in which a cost function determined based on a frequency response characteristic of the filter is minimized. The masking pattern arranged for each individual sensing element unit or individual sensing element group unit in the mask array is determined as a pattern in which the Euclidean norm of 1/H(f), which is the reciprocal of the frequency response characteristic of the blur kernel, is minimized. can

First, in step 810 , a target aperture ratio of the mask array may be set. The aperture ratio is a ratio of light incident to a pattern region corresponding to the masking pattern to light transmitted therethrough, and may represent, for example, an area ratio of an open region to an area of the pattern region. The pattern region may indicate a region corresponding to a sensing element or a sensing element group in the mask array. The aperture ratio may be set, for example, at a ratio of 10% to 90%, or 30% to 70%. Preferably, the aperture ratio may be set at a ratio of 40% to 60%.

A masking pattern according to the aperture ratio set in operation 820 may be generated. The mask array may be divided into pattern regions corresponding to individual sensing elements, and the pattern region may be divided into a plurality of spaces. Each cell may define a unit area in which a mask element may be formed. A cell in which the mask element is formed may be referred to as a closed space, and a cell in which the mask element is not formed may be referred to as an open space. In other words, in the mask array, a closed cell can absorb all or part of the amount of light, and an open cell can transmit light. Masking patterns in which open and closed cells are combined may be generated according to the target aperture ratio set in the above-described step 810 . When the pattern area includes N×N cells and the target aperture ratio is 50%, N ² /2 masking patterns may be generated. As will be described later with reference to FIG. 10 , the closed cells may be classified according to transmittance.

In step 820, an example of generating the entire combination has been described for convenience of description, but is not limited thereto. For another example, predefined patterns may be combined, or a masking pattern to be searched for may be generated by Swap or Toggle between patterns within the given combinations. For example, if the condition that the pattern is symmetric is added, the pattern may be searched only for a repeated region, and the generated pattern may be used symmetrically.

Subsequently, a cost function value for each masking pattern generated in step 830 may be calculated. A cost function based on the reciprocal of the frequency response characteristic of the blur kernel described above with reference to FIGS. 5 and 6 may be used. Due to the mask elements formed along the masking pattern, the blur kernel of the imaging optical system may be modified, and a frequency response characteristic H'(f) of the modified blur kernel may be calculated. The cost function based on the reciprocal 1/H'(f) of the frequency response characteristic of the modified blur kernel is the Euclidean norm of the reciprocal of the function indicating the frequency response characteristic, the variance of the inverse, and/or It can be determined based on the reciprocal of the Euclidean norm of the function. For example, a cost function such as Equation 10 below may be used.

[Equation 10]

은 주파수 응답 특성 H'(f)의 역수의 유클리디언 놈을 나타낼 수 있다.

인자에 의해 제로 크로싱 포인트가 최소화될 수 있다.

은 주파수 응답 특성 H'(f)의 유클리디언 놈의 역수를 나타낼 수 있다.

에 의해 H'(f)의 크기가 0이 되지 않도록 설계될 수 있다.

은 주파수 응답 특성 H'(f)의 역수의 분산을 나타낼 수 있다. 1/H'(f) 이 주파수 공간에서 분산(Variation)이 최소화되도록 설계될 수 있다. 수학식 10의 비용 E는 상술한 개별 비용 인자들의 가중 평균 값으로서, α, β, 및 γ는 가중치를 나타낼 수 있다.In Equation 10 above,

may represent the Euclidean norm of the reciprocal of the frequency response characteristic H'(f).

The zero crossing point can be minimized by the factor.

may represent the reciprocal of the Euclidean norm of the frequency response characteristic H'(f).

It can be designed so that the size of H'(f) does not become 0 by .

may represent the variance of the reciprocal of the frequency response characteristic H'(f). 1/H'(f) may be designed to minimize variance in this frequency space. The cost E in Equation 10 is a weighted average value of the above-described individual cost factors, and α, β, and γ may represent weights.

In step 840, a masking pattern that minimizes this cost function value may be determined. A cost function value for each masking pattern for combinations of closed and open cells within a given condition may be calculated, and a masking pattern in which noise amplification is minimized may be determined by the cost function. The image sensor may include a mask array in which the searched masking pattern is repeatedly disposed.

9 and 10 show exemplary shapes of a mask array according to an embodiment.

9 illustrates an example in which a light transmission state is divided into a binary state for each individual cell of the mask array 924 . In other words, in FIG. 9 , the open cell 991 transmits all light incident on the corresponding cell, and the closed cell 992 absorbs all light incident on the corresponding cell.

The mask array 924 may be segmented into a plurality of group regions. The group region 924a is a region in the mask array 924 , and may indicate a region that covers one or a plurality of sensing elements of the adjacent sensing array 921 . 9 illustrates an example in which the group region 924a of the mask array 924 covers one sensing element, but as will be described later in FIG. 11 , the group region 924a of the mask array 924 represents one pixel. In order to do this, a sensing element group including a plurality of grouped sensing elements may be covered. The masking pattern 990 of the group region 924a may be repeated. In other words, each of the plurality of group regions of the mask array 924 may have the same masking pattern 990 .

A pattern region of the mask array 924 corresponding to an individual sensing element may include an aperture region and a masked region. The opening area may occupy an area corresponding to an aperture ratio to the total area of the corresponding area, and the masking area may occupy the remaining area. For example, the masking pattern 990 illustrated in FIG. 9 may be a pattern designed with a target aperture ratio of 50%. The masking pattern 990 is classified into a total of 7Х7=49 cells, and may include 24 closed cells 992 and 25 open cells 991 .

In addition, the masking pattern 990 may ensure an opening ratio of the partial region 995 as well as an opening ratio relative to the entire area. An area occupied by an aperture part in each of the partial regions 995 may be equal to or greater than an area corresponding to a set aperture ratio. In FIG. 9 , the partial region 995 of the masking pattern 990 includes a total of 4Х4=16 cells, and includes 8 closed cells 992 and 8 open cells 991 , so that the aperture ratio is 50%. can Also, even if the partial region composed of 4Х4=16 cells is moved to another position of the masking pattern 990, the aperture ratio may be 50%. In other words, the masking pattern 990 may be designed such that regions having a target aperture ratio are evenly distributed.

In addition, the region corresponding to the individual masking pattern 990 may include more than the number of imaging optical lenses of the imaging optical system. As described above with reference to FIG. 5 , information sensed by the sensing element may be blurred in proportion to the number of lenses of the multi-lens array. In order to restore an image with high resolution by offsetting a blur level proportional to the number of lenses, the pattern area may include cells corresponding to the number of lenses to provide a deblur function. When the lens array 910 of FIG. 9 includes 7Х7=49 imaging optical lenses, the masking pattern 990 may include at least 7Х7=49 cells. Information passing through the lens array 910 toward the sensing array 921 may be blurred by 1/49, and the masking pattern 990 may provide a corresponding 49 times deblurring capacity. .

10 illustrates an example in which an individual mask element of a mask array has one of two or more levels of transmission. The mask element may block and/or absorb some of the arriving light, and the transmission level may indicate a level and/or a rate at which the incident light is transmitted. The mask element may be formed and/or disposed along the plane of the mask array based on the above-described masking pattern.

The individual mask element may be divided into a plurality of regions, and transmittance of the individual mask element may be determined according to a ratio of an open region and a closed region among the plurality of regions. 10 illustrates a masking pattern 1000 designed with five transmission levels. Each mask element may be composed of four regions divided by the same size. The compartment 1010 in which the mask element is not formed may have a first transmission level (eg, 100% transmission). The first mask element 1021 may have a second transmission level (eg, 75% transmission), and only one of the regions of the first mask element 1021 may be a closed region. The second mask element 1022 may have a third transmission level (eg, 50% transmission) and may be only half (eg, two regions) closed regions. The third mask element 1023 may have a fourth level of transmission (eg, 25% transmission), and the three regions may be closed regions. The fourth mask element 1024 may have a fifth transmission level (eg, 0% transmission), and all regions may be closed regions. As such, the aperture ratio determined according to the number or area of the open and closed areas may be interpreted as the amount of transmitted light compared to the amount of light incident on the area.

11 is a view for explaining arrangement of a masking pattern for each sensing element group of an image sensor according to an exemplary embodiment.

9 illustrates an example in which each group region of the mask array covers a single sensing element, and FIG. 11 shows a sensing element in which each group region 1141 of the mask array 1140 includes a plurality of sensing elements 1121a. An example of covering a group 1121 is shown. In the example illustrated in FIG. 11 , a pixel value of an image pixel may be determined based on values sensed by the sensing element group 1121 (eg, 2Х2 sensing elements 1121a of FIG. 11 ). The mask array 1140 includes a plurality of masking patterns. In FIG. 9 , one masking pattern is disposed per sensing device, but in FIG. 11 , one masking pattern 1190 per sensing device group 1121 may be disposed. . All of the masking patterns of the mask array 1140 may have the same shape.

12A to 12B illustrate an arrangement of a mask array according to an exemplary embodiment.

The mask array may be disposed in various positions capable of blocking and/or superimposing light in a specific direction in the sensor for the camera. The mask array 1240a illustrated in FIG. 12A may be disposed on the color filter 1220 between the condensing lens array 1230 and the sensing array 1210 . The mask array 1240b illustrated in FIG. 12B may be disposed between the color filter 1220 and the sensing array 1210 . As described above, the mask array 1240b and the sensing array 1210 may be spaced apart from each other within 1 μm.

13 is a block diagram illustrating a configuration of an imaging apparatus according to an exemplary embodiment.

The imaging device 1300 may include a lens array 1310 and an image sensor.

The lens array 1310 may include imaging optical lenses that transmit light received from the outside.

The image sensor may indicate a sensor that senses light that has passed through the lens array 1310 . The image sensor may include a mask array 1324 , a sensing array 1321 , and a processor 1330 . Since the mask array 1324 and the sensing array 1321 have been described above with reference to FIGS. 1 to 12B , descriptions thereof will be omitted.

The processor 1330 may reconstruct an image based on sensing information sensed by the sensing elements. The processor 1330 of the image sensor may also be referred to as an image signal processor (ISP). The processor 1330 converts the sensed information into a frequency domain to generate frequency information, and divides the frequency information by a frequency conversion result of a blur kernel corresponding to the masking pattern of the mask array 1324. Blurred frequency information may be generated. The processor 1330 may reconstruct a high-resolution image by inversely transforming the deblurred frequency information into a time domain. The sensing information may be utilized for not only image restoration, but also depth estimation of a subject, refocusing, dynamic range imaging, and high-sensitivity imaging in a low-light environment.

14 is a block diagram illustrating a configuration of an electronic terminal according to an embodiment.

The electronic terminal 1400 may include an imaging module 1410 and a processor 1420 .

The imaging module 1410 may include a lens array 1411 and an image sensor, and the image sensor may include a mask array 1412 and a sensing array 1413 . In FIG. 13 , the processor 1330 is described as being included in the image sensor, but in the embodiment of FIG. 14 , it is described as being independently positioned. Since the lens array 1411 , the image sensor, and the processor 1420 have been described above, a detailed description thereof will be omitted. The processor 1420 illustrated in FIG. 14 may be an application processor 1420 (AP).

15 and 16 are diagrams illustrating examples of a device in which an image sensor is implemented according to an embodiment.

The image sensor and/or imaging device may be applied to various technical fields. Since a lens array composed of a plurality of lenses and a sensor composed of a plurality of sensing elements can be designed to be spaced apart by a relatively short focal length, the imaging device has a large size and thickness, which enables high-quality imaging. can be implemented as an ultra-thin camera.

The image sensor and/or imaging device may be mounted on the mobile terminal. A mobile terminal is a movable terminal that is not fixed to an arbitrary position, and is, for example, a portable device such as a smartphone, a tablet, a foldable smartphone, an artificial intelligence speaker, and a vehicle ( vehicle) and the like.

As shown in FIG. 15 , the imaging module 1510 may be applied to a front camera or a rear camera of a smartphone. The imaging module 1510 has a structure in which a large full-frame sensor and a multi-lens array are combined, and may be applied to a mobile phone camera.

Also, the imaging module 1510 may be implemented in a vehicle with a thin structure or a curved structure. As shown in FIG. 16 , the imaging device 1610 may be implemented as a front camera or a rear camera having a curve in the vehicle 1600 . In addition to this, the imaging device 1610 includes a DSLR camera, a drone, a CCTV camera, a camera for a webcam, a 360 degree camera, a camera for film and broadcasting, a VR/AR camera, a flexible or extendable camera It can also be applied to fields such as (Flexible/Stretchable Camera), insect eye camera, and contact lens type camera. Furthermore, the imaging apparatus may be applied to a multi-frame super resolution image restoration in which resolution is increased by using information about a plurality of captured frames.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the devices and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU). ), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using a general purpose computer or special purpose computer. The processing device may execute an operating system (OS) and software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination thereof, which configures the processing device to operate as desired or independently or collectively configures the processing device to operate as desired. can command The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components or equivalents Appropriate results can be achieved even if substituted or substituted by

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (23)

In the image sensor,
a mask array comprising a plurality of mask elements blocking light in a partial direction among the light passing through the imaging optical lenses, and passing light in a direction different from the partial direction; and
A sensing array including a plurality of sensing elements for sensing the light passing through the imaging optical lenses and the mask array
An image sensor comprising a. According to claim 1,
A color filter disposed on the sensing array to filter some wavelength bands from light incident on each sensing element
further comprising,
The mask array is
disposed between the color filter and the sensing array,
image sensor. According to claim 1,
A condensing lens array disposed on the sensing array to transmit light to each sensing element
further comprising,
The mask array is
disposed between the condensing lens array and the sensing array,
image sensor. According to claim 1,
The mask array and the sensing element,
spaced within 1 μm,
image sensor. According to claim 1,
The mask array and the sensing element,
touching each other,
image sensor. According to claim 1,
Regions corresponding to individual sensing elements in the mask array,
an aperture region occupying an area corresponding to an aperture ratio to a total area of the corresponding area; and
A masked region occupying the remaining area in which the plurality of mask elements are disposed in the corresponding region
An image sensor comprising a. 7. The method of claim 6,
The aperture ratio is
A ratio of 40% or more and 60% or less,
image sensor. According to claim 1,
An area occupied by an aperture part in each partial area of the mask array is equal to or greater than an area corresponding to a set aperture ratio;
image sensor. According to claim 1,
The mask array is
is segmented into a plurality of group regions corresponding to a plurality of sensing element groups,
Each of the plurality of group regions in the mask array,
Covering a sensing element group including a plurality of sensing elements grouped to represent one pixel,
image sensor. 10. The method of claim 9,
In the mask array, the masking pattern of the group area is repeated,
image sensor. 10. The method of claim 9,
each of the plurality of group regions of the mask array all have the same masking pattern,
image sensor. According to claim 1,
Regions corresponding to individual masking patterns in the mask array,
Containing more than the number of the imaging optical lenses of the imaging optical system,
image sensor. According to claim 1,
A plurality of mask elements,
having one of two or more transmission levels;
image sensor. According to claim 1,
Each of the plurality of mask elements,
divided into a plurality of areas,
The transmittance of each mask element is
determined according to a ratio of an open area and a closed area among the plurality of areas,
image sensor. According to claim 1,
A processor for reconstructing an image based on sensing information sensed by the plurality of sensing elements
An image sensor further comprising a. 16. The method of claim 15,
The processor is
Frequency information is generated by converting the sensed information into a frequency domain, and the frequency information is divided by a frequency conversion result of a blur kernel corresponding to a masking pattern of the mask array to a deblurred frequency generating information and reconstructing a high-resolution image by inversely transforming the deblurred frequency information into a time domain,
image sensor. According to claim 1,
The mask array includes a plurality of masking patterns,
Each masking pattern is
Covering a sensing element group including two or more sensing elements in the sensing array,
image sensor. In the camera device,
an imaging lens array including imaging optical lenses that transmit light received from the outside;
a sensing array including a plurality of sensing elements for sensing the light passing through the imaging lens array; and
A mask array including a plurality of mask elements and disposed between the imaging lens array and the sensing array
A camera device comprising a. 19. The method of claim 18,
The mask array is
disposed on one of a position in contact with the sensing array and a position inside the sensing array on the plurality of sensing elements,
camera device. 19. The method of claim 18,
Regions corresponding to individual sensing elements in the mask array,
an opening region occupying an area corresponding to an opening ratio with respect to the total area of the corresponding region; and
A masking area occupying the remaining area in which the plurality of mask elements are disposed in the corresponding area
A camera device comprising a. 19. The method of claim 18,
An area occupied by an opening part in each of the partial regions in the mask array is equal to or greater than an area corresponding to a set opening ratio;
camera device. 19. The method of claim 18,
In the mask array, the masking pattern of the group area is repeated,
camera device. 19. The method of claim 18,
The sensing information sensed by the plurality of sensing elements is converted into a frequency domain to generate frequency information, and the frequency information is divided by a frequency conversion result of a blur kernel corresponding to the masking pattern of the mask array to deblur. A processor that generates high-resolution frequency information and restores a high-resolution image by inversely transforming the deblurred frequency information into a time domain.
A camera device further comprising a.

Images