JP2012231377A - Imaging apparatus and image generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus which is capable of capturing a high-resolution image from a low-resolution motion picture through simple processing, and an image generation method or the like.SOLUTION: An imaging apparatus comprises an image capturing section, an added image generating section, a compression processing section, an expansion processing section, an estimate arithmetic section, and an image output section. The added image generating section successively pixel-shifts an addition unit in a captured image fx, and acquires added images Ato Aby weight-adding pixel values included in the addition unit. The compression processing section determines a mean of the added images Ato Aas a mean image M, determines a differential between the mean image M and an added image Aas a differential image D, and compresses the determined mean image M and the differential image D. The expansion processing section expands the compressed mean image M and differential image Dand determines the added images Ato A. The estimate arithmetic section estimates a pixel value vof the captured image fx on the basis of the added images Ato A. The image output section outputs a high-resolution image based on the pixel value v.

Description

  The present invention relates to an imaging apparatus, an image generation method, and the like.

  Some recent digital cameras and video cameras can be used by switching between a still image shooting mode and a moving image shooting mode. For example, there is one that can shoot a still image with a higher resolution than a moving image by a user operating a button during moving image shooting.

JP 2009-124621 A JP 2008-243037 A

  However, the method of switching between the still image shooting mode and the moving image shooting mode has a problem that when the user notices a photo opportunity, a decisive moment is often already missed.

  The present inventor considers generating a high-resolution still image at an arbitrary timing from a low-resolution moving image in order to realize photographing at this decisive moment. For example, Patent Documents 1 and 2 disclose a technique for synthesizing a high-resolution image from a low-resolution image acquired by pixel shift. However, this method requires imaging by pixel shift, which complicates the camera configuration. In addition, there is a problem that the load of high resolution processing is large and pixel value estimation may be difficult.

  According to some aspects of the present invention, it is possible to provide an imaging device, an image generation method, and the like that can acquire a high-resolution image from a low-resolution moving image with simple processing.

  According to one aspect of the present invention, an image acquisition unit that acquires a captured image, an addition unit that is a unit for acquiring an added pixel value is set for each of a plurality of pixels of the captured image, and a pixel value included in the addition unit is set. An addition image generation unit that obtains the addition pixel value by weighted addition, obtains an addition image based on the obtained addition pixel value, a compression processing unit that compresses the addition image, and a decompression that decompresses the compressed addition image A processing unit; an estimation calculation unit that estimates a pixel value of the captured image based on the expanded added image; and an image output unit that outputs a high-resolution image based on the pixel value estimated by the estimation calculation unit; The addition image generation unit obtains first to nth addition images (n is a natural number equal to or greater than 2) as the addition image by sequentially shifting the addition unit, and the compression processing unit includes: The first to nth addition images Is obtained as an average image, and a difference between the average image and the m-th added image (m is a natural number equal to or less than n) of the first to n-th added images is obtained as an m-th difference image. The obtained average image and the m-th difference image are compressed, and the decompression processing unit decompresses the compressed average image and the m-th difference image to obtain the first to n-th added images. It relates to the desired imaging device.

  According to one aspect of the present invention, the addition units are sequentially pixel-shifted to obtain the first to nth added images, and the average image of the first to nth added images is obtained. Then, an mth difference image that is a difference between the average image and the mth addition image is obtained, and the average image and the mth difference image are compressed. Then, the compressed images are expanded to obtain first to nth added images, and a captured image is estimated based on the first to nth added images.

  Accordingly, for example, by applying an estimation process described later, it is possible to restore a high-resolution captured image from a low-resolution added image with a simple process. Since moving image shooting and still image shooting are not switched, a high-resolution still image at an arbitrary timing can be acquired from moving image data. Further, by obtaining the difference image, the image data can be compressed at a high compression rate.

  In the aspect of the invention, the addition image generation unit may sequentially shift the addition unit horizontally or vertically one pixel at a time to set the first to nth positions, and the first to nth positions. The first to n-th added images may be acquired in step S1, and the addition unit of the m-th position and the (m + 1) -th position among the first to n-th positions may include a common pixel.

  In this way, the addition unit value is obtained by sequentially shifting the addition unit while including the common pixels, and the first to nth addition images can be configured by the addition pixel value.

  In the aspect of the invention, the compression processing unit may obtain an average value of the addition pixel values at the first to n-th positions as a pixel value of the average image, A difference image generation unit that obtains a difference value between the pixel value and the added pixel value at the m-th position as a pixel value of the m-th difference image.

  In this way, the pixel value of the average image can be obtained by averaging the pixel values of the first to nth added images. Further, the pixel value of the mth difference image can be obtained from the difference value between the pixel value of the average image and the pixel value of the mth addition image.

  In one aspect of the present invention, a low-resolution moving image generation unit that generates a low-resolution moving image is included, the captured image is an image of an RGB Bayer array, and the addition unit is four adjacent pixels. The addition image generation unit acquires first to fourth addition images in which RGB is mixed by the weighted addition at the first to fourth positions (n = 4), and the low-resolution moving image generation unit The RGB pixel values of the low-resolution moving image may be obtained by estimation based on the first to third addition images among the first to fourth addition images.

  In this way, an RGB color image can be generated from an added image in which RGB is mixed by addition. Thereby, it is possible to easily obtain a color moving image without restoring the captured image.

  In the aspect of the invention, the compression processing unit may reversibly compress the m-th difference image by entropy coding.

  Since the difference image is a difference between the average image and the added image, the entropy of the pixel value can be made smaller than that of the captured image. Therefore, the difference image can be compressed at a high compression rate by entropy encoding.

  In one aspect of the present invention, when the captured image is an image of an RGB Bayer array, the estimation calculation unit obtains a pixel value of the RGB Bayer array of the captured image by estimation, and the image output unit includes: A high-resolution moving image may be output based on an image obtained by demosaicing the estimated pixel values of the RGB Bayer array.

  In this way, a high-resolution moving image having a resolution corresponding to the captured image can be restored from the image data compressed at a high compression rate. This makes it possible to achieve both a high compression rate and high image quality for moving image data.

  In one aspect of the present invention, the image processing apparatus may further include a noise reduction processing unit that performs adaptive noise reduction processing that adaptively adjusts the moving average range according to the change width of the pixel value with respect to the mth difference image. Good.

  Since the difference image is a difference between the average image and the added image, the amplitude of the pixel value is smaller than that of the captured image. Therefore, noise can be effectively reduced by performing adaptive noise reduction processing on the difference image.

  In one aspect of the present invention, the image acquisition unit acquires a captured image of the first frame as a reference captured image, and a difference between the captured image of the second frame before or after the first frame and the reference captured image. And the addition image generation unit acquires the first to nth addition images based on the reference captured image and the first to nth addition images based on the difference captured image. May be.

  In this way, the inter-frame difference image becomes a difference between the added images having high similarity. Thereby, the compression rate of the average image produced | generated from the difference image between frames can be improved rather than the average image produced | generated from the normal addition image.

  In the aspect of the invention, when the addition unit set in the first position and the addition unit set in the second position next to the first position overlap, the estimation calculation unit Is a difference value between the added pixel value of the first position and the added pixel value of the second position, and the added pixel value of the first area excluding the overlapping area from the addition unit of the first position. A relational expression between the first intermediate pixel value that is and the second intermediate pixel value that is the addition pixel value of the second region excluding the overlap region from the addition unit of the second position, and the difference value The first intermediate pixel value is estimated using the relational expression, and the pixel value of each pixel included in the addition unit is obtained using the estimated first intermediate pixel value. May be.

  In this way, it is possible to estimate the intermediate pixel value from the first to n-th added images that are sequentially pixel-shifted while the addition unit is superimposed, and obtain the final estimated pixel value from the estimated intermediate pixel value. . Thereby, a captured image can be estimated by simple processing.

  In the aspect of the invention, the estimation calculation unit may include an intermediate included in the intermediate pixel value pattern when successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern. A relational expression between pixel values is represented using the addition pixel value, the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values is compared with the addition pixel value, and similarity is evaluated, Based on the similarity evaluation result, an intermediate pixel value included in the intermediate pixel value pattern may be determined so that the similarity is the highest.

  In this way, an intermediate pixel value can be obtained by estimation based on a plurality of added pixel values acquired by pixel shifting while the addition unit is superimposed.

  In another aspect of the present invention, a captured image is acquired, an addition unit, which is a unit for acquiring an added pixel value, is set for each of a plurality of pixels of the captured image, and a pixel value included in the added unit is weighted. Adding to obtain the added pixel value, obtaining an added image based on the obtained added pixel value, compressing the added image, decompressing the compressed added image, and based on the decompressed added image When estimating the pixel value of the captured image and outputting a high-resolution image based on the estimated pixel value, the addition units are sequentially pixel-shifted to obtain first to nth added images (n is 2 or more). (Natural number) is obtained as the added image, the average of the first to n-th added images is obtained as an average image, and the average image and the m-th added image among the first to n-th added images (I is a natural number less than or equal to n) The difference image of m is obtained, the obtained average image and the mth difference image are compressed, the compressed average image and the mth difference image are expanded, and the first to nth added images are obtained. It relates to the desired image generation method.

Explanatory drawing about the 1st data compression method. An example of a Huffman code table. 1 is a first configuration example of an imaging apparatus. The 1st code allocation example of fusion image data. The 2nd example of code allocation of fusion image data. 2 shows a second configuration example of an imaging apparatus. Fig. 7 shows an example of code assignment of fusion image data in the second configuration example of the imaging apparatus. The modification structural example of an imaging device. Explanatory drawing about the production method of a low resolution animation. Explanatory drawing about the production method of a low resolution animation. Explanatory drawing about the production method of a low resolution animation. 3rd structural example of an imaging device. An example of an imaging process for simultaneously acquiring a high-resolution still image and a low-resolution still image. The sequence example of the imaging process which acquires a high-resolution still image and a low-resolution still image simultaneously. An example of data reconstruction before data compression. An example of a data unit set in data reconstruction. The example of the superimposition shift addition value of a reconstruction data unit. The structural example of the imaging part in the case of adding 4 pixels of the same color. Explanatory drawing about adaptive noise reduction processing. FIG. 20A and FIG. 20B are explanatory diagrams of adaptive noise reduction processing. Explanatory drawing about adaptive noise reduction processing. Explanatory drawing about adaptive noise reduction processing. Explanatory drawing about adaptive noise reduction processing. 4 shows a fourth configuration example of an imaging apparatus. The schematic diagram of the process which the 4th structural example of an imaging device performs. The modification structural example of the imaging part in the 4th structural example of an imaging device. 27A and 27B are explanatory diagrams of estimated pixel values and intermediate pixel values. Explanatory drawing about a restoration estimation process. Explanatory drawing about a restoration estimation process. Explanatory drawing about a restoration estimation process.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Outline of this embodiment Digital camera and video camera products have a digital camera that mainly shoots still images with a video shooting function, or a video camera that mainly shoots video with a still image shooting function. There is something. If such a device is used, there is the convenience that a still image and a moving image can be shot with a single device.

  However, with this method, there is a problem that it is difficult to obtain a high-quality still image without missing the shutter chance that many people demand. For example, in a method of instantaneously switching to a mode for shooting a high-quality still image during moving image shooting, there is a problem that the moving image is interrupted or a decisive moment is already missed when the user notices.

  In order to obtain a still image at a decisive moment, a method of acquiring a high-resolution moving image with a high frame rate and extracting a still image at an arbitrary timing from the high-resolution moving image can be considered. However, this method has a problem that the amount of data increases in order to capture a high-resolution moving image.

In this embodiment, as shown in FIG. 1, the captured image by pixel addition to obtain a plurality of low-resolution images A ₁ to A _4, the average image of the plurality of low-resolution images A ₁ to A ₄ Find M. Then, the difference image _D 1 to D ₄ of the average image M and the low-resolution image _A 1 to A ₄ asked, compressing the difference image _D 1 to D _4, for example, by entropy coding. In the difference images D _{1 to} D ₄ , since the pixel values are considered to be unevenly distributed near zero, the compression rate can be improved.

  As a technique for obtaining a high-resolution image from a low-resolution image, a technique for performing so-called super-resolution processing on a low-resolution image captured by pixel shift is conceivable. For example, in this method, addition reading is performed while sequentially shifting the position, and a high-definition image is temporarily assumed based on the plurality of position-shifted images. Then, the assumed image is degraded to generate a low resolution image, which is compared with the original low resolution image, and the high definition image is estimated so that the difference is minimized. As this super-resolution processing, an ML (Maximum-Likelihood) method, a MAP (Maximum A Postterior) method, a POCS (Projection Onto Convex Set) method, an IBP (Iterative Back Projection) method, and the like are known.

  For example, in Patent Document 1 described above, low-resolution images that have been pixel-shifted during moving image shooting are sequentially captured in time series, and a plurality of low-resolution images are combined to assume a high-resolution image. A technique for performing the above-described super-resolution processing on a high-resolution image and estimating a high-resolution image with high likelihood is disclosed.

  However, this technique uses a general super-resolution process in which the estimation accuracy is increased by iterative calculations that frequently use a two-dimensional filter. For this reason, the scale of processing becomes very large and the processing time increases, and there is a problem that it is difficult to apply to a device such as a digital camera with processing capacity and cost restrictions.

  In Patent Document 2 described above, a plurality of pixel-shifted low-resolution images are captured, and temporary pixels constituting the high-resolution image to be obtained are set as sub-pixels. A technique for estimating the pixel value of the sub-pixel so as to match the pixel value of the low-resolution image that has been made is disclosed. In this method, initial values of a plurality of sub-pixels are set, pixel values of sub-pixels excluding sub-pixels to be calculated are subtracted from pixel values of the low-resolution image, and pixel values are sequentially obtained for adjacent pixels. Apply.

  However, with this method, there is a problem that the estimation error becomes very large unless the initial value is specified. In this method, an area satisfying a predetermined condition is found from an image in order to set an initial value. Therefore, if an appropriate area cannot be found from the captured image, it is difficult to estimate the initial value. In addition, it is necessary to search for an appropriate area for setting the initial value.

In this regard, in the present embodiment, as described later in FIG. 1, and reproduces the low-resolution image _A 1 to A ₄ from the average image M and the difference image _D 1 to D _4, the low-resolution image _A 1 to A ₄ , the captured image is restored by estimation. The added pixel values a ⁽¹⁾ _{00 to} a ⁽⁴⁾ ₀₁ of the low-resolution images A _{1 to} A ₄ are shifted by overlapping pixels, and adjacent added pixel values include common pixels. As will be described later with reference to FIGS. 27A to 30, the process of estimating a high-resolution image can be simplified by using the added pixel value that is shifted by the superimposed pixel.

2. First Data Compression Method Next, this embodiment will be described in detail. First, a method for compressing captured image data will be described. In the following description, the case where the image sensor is an RGB Bayer array will be described as an example. However, the present embodiment is not limited to this, and may be, for example, an image sensor of a complementary color filter. In the following, a case where 4-pixel addition is performed will be described as an example. However, the present embodiment is not limited to this, and another pixel number may be added. Hereinafter, a captured image of the frame fx will be described as an example, but the same applies to other frames.

  Here, the frame is, for example, a timing at which an image is captured by an image sensor or a timing at which one captured image is processed in image processing. Alternatively, one image in the image data is also referred to as a frame as appropriate.

As shown in FIG. 1, a captured image of an RGB Bayer array (hereinafter referred to as “high-definition frame image fx”) is acquired by reading all pixels in a frame fx. An addition unit (a pixel group to be added) composed of four pixels is set in the high-definition frame image fx, and the pixel values of the addition unit are weighted and added. At this time, the addition unit is shifted horizontally or vertically while superimposing one pixel, and four pixel addition images A _{1 to} A ₄ are generated.

The 4-pixel addition value a _ij is expressed by the following expression (1), and the addition pixel values constituting the pixel addition images A _{1 to} A ₄ are {a _ij , a _{(i + 1) j} , a _{(i + 1) (j + 1), respectively.} , A _{i (j + 1)} }.
a _ij = v _ij + (1 / r) v _{(i + 1) j} +
(1 / r) v _{i (j + 1)} + (1 / r ² ) v _{(i + 1) (j + 1)} (1)

Here, r is a weighting parameter, and 1 ≦ r. V _ij is the pixel value of the address (i, j) in the high-definition frame image.

Next, the addition positions of the generated pixel addition images A _{1 to} A ₄ are matched, and an average image M is generated by superimposing the four pixel addition images and taking the addition average of the values at the same position. . That is, the 4-pixel addition values of the pixel addition images A _{1 to} A ₄ are respectively represented by {a ⁽¹⁾ _ij , a ⁽²⁾ _{(i + 1) j} , a ⁽³⁾ _{(i + 1) (j + 1)} , a ⁽⁴⁾ _{i ( j + 1)} }, the pixel value a ^M _ij of the average image M is expressed by the following equation (2).
a ^M _ij = [a ⁽¹⁾ _ij + a ⁽²⁾ _{(i + 1) j} +
a ⁽³⁾ _{i (j + 1)} + a ⁽⁴⁾ _{(i + 1) (j + 1)} ] / 4 (2)

Using the generated average image M as a reference, the difference (projection to the direction vector (1, −1)) between the average image M and the pixel addition images A _{1 to} A ₄ is set as difference images D _{1 to} D ₄ , respectively. The difference images D _{1 to} D ₄ and the average image M are combined to obtain fused image data F (M, D _{1 to} D ₄ ). If the 4-pixel addition difference values constituting the difference images D _{1 to} D ₄ are represented as {a ^D1 _ij , a ^D2 _ij , a ^D3 _ij , a ^D4 _ij }, they can be expressed as the following expression (3).
a ^D1 _ij = a ^M _ij −a ⁽¹⁾ _ij ,
a ^D2 _ij = a ^M _ij −a ⁽²⁾ _{(i + 1) j} ,
a ^D3 _ij = a ^M _ij −a ⁽³⁾ _{(i + 1) (j + 1)} ,
a ^D4 _ij = a ^M _ij −a ⁽⁴⁾ _{i (j + 1)} (3)

In this way, the still image fusion is fused to the moving image data _{F (M, D 1 ~D 4} ) generates, may be recorded it, the recorded F _(M, D 1 ~D ₄ ) In the subsequent stage, "high-definition still image data" or "moving image data" can be appropriately generated.

In this example, the difference (that is, projection onto the direction vector (1, −1)) is taken, but the present embodiment is not limited to this. For example, in accordance with the correlation distribution between the pixel values of the average image M and the pixel values of the pixel addition images A _{1 to} A ₄ , the principal component axis (straight line with the largest variance) is obtained, and the axis orthogonal to the principal component axis is obtained. It goes without saying that the data can be further compressed by using the projected value.

3. Entropy Coding Next, a method for compressing the above-described average image M and difference images D _{1 to} D ₄ will be described.

In the pixel value data of the average image M and the difference images D _{1 to} D ₄ , the bias of the occurrence frequency distribution becomes remarkable, so that entropy coding (for example, Huffman code, LZH, etc.) is effective as data compression. Entropy coding is a compression technique in which the occurrence probability of a pixel value is obtained and a short code length is assigned from the one with the highest occurrence probability.

For example, encoding in which the shortest code length is assigned to the average value of the pixel value data of the average image M and the code length is increased as the value is farther from the average value is appropriate. That is, since the pixel value data of the difference images D _{1 to} D ₄ has an average of zero and is distributed with the occurrence probability of zero as a peak, the shortest code may be assigned to the zero value. It is also effective to perform nonlinear quantization on the pixel value data of the average image M and the difference images D _{1 to} D ₄ according to the probability density.

On the other hand, in the average image M, the deviation of the occurrence frequency distribution becomes remarkable because the pixel value data is certainly the average of the addition, but the range that the pixel addition value can take is compared with the pixel value data of the difference images D _{1 to} D _4. , Get pretty big. For this reason, application of an irreversible compression method (JPEG, MPEG, etc.) that can generally be expected to have a high compression rate is effective as data compression.

Therefore, in the average image M, the number of pixels is reduced and irreversible high compression is applied, and in the difference images D _{1 to} D ₄ , high-speed entropy coding using a characteristic exhibiting a similar occurrence distribution bias is applied. It is possible.

  Next, an example of applying the simplest Huffman code is shown. If one Huffman code table is generated based on the pixel value distribution of a large number of images, the application range becomes wide and the compression effect becomes image-dependent. However, it is not effective to generate the Huffman table for each image or image range every time.

ａ’^Ｍ _ｉｊ＝ａ^Ｍ _ｉｊ−α^Ｍ _ｉｊ （５） Therefore, when applied to the above average image M, the average value α ^M _ij is obtained for each predetermined image area by the following equation (4), and the difference between the average value α ^M _ij and each added average value {a ^M _ij }. The value {a ′ ^M _ij } is obtained by the following equation (5). Then, the difference value {a ′ ^M _ij } is used as the data of the average image M again, and Huffman coding may be applied. Thus, the difference value {a ′ ^M _ij } can form an occurrence distribution of values universally centered on zero. M in the fusion image data F (M, D _{1 to} D ₄ ) is considered to be replaced with M = {α ^M _ij , a ′ ^M _ij }.

a ′ ^M _ij = a ^M _ij −α ^M _ij (5)

Here, (h0, v0) is the start point of the area where the average value α ^M _ij is calculated, and (h, v) is the end point of the area where the average value α ^M _ij is calculated. For example, the area for calculating the average value α ^M _ij is the entire average image M. The area for calculating the average value α ^M _ij and the number of average values a ^M _ij used for calculating the average value α ^M _ij may be determined by evaluating the balance with the data compression rate.

Since the difference value {a ′ ^M _ij } and the 4-pixel addition difference value {a ^D1 _ij , a ^D2 _ij , a ^D3 _ij , a ^D4 _ij } are themselves composed of a value of average zero, FIG. The code assignment shown in FIG. In this way, the Huffman code table can be a fixed type.

Note that, as the above-described average value α ^M _ij , the fixed value m _AVR shown in the following equation (6) may be applied, and may be handled as look-ahead information without being treated as fluctuation data. In this case, the average value m _AVR need not be recorded as data. M in the fusion image data F (M, D _{1 to} D ₄ ) is considered to be replaced with M = {m _AVR , a ′ ^M _ij }. The following expression (6) is applied to the case of the weighted 4-pixel addition value defined by the above expression (1), and it should be noted that an appropriate value is appropriately set depending on the number of added pixels and the weighting method. is necessary.
m _AVR = v _max · {1+ (1 / r) + (1 / r) + (1 / r ² )} / 2 (6)

Here, r is a weighting parameter, and 1 ≦ r. v _max is a maximum value that defines the pixel value.

4). Second Data Compression Method Next, a second data compression method for further reducing the data amount of the difference images D _{1 to} D ₄ described above will be described.

There is a negative correlation between the combination of the difference image {D ₁ , D ₃ } and the combination of the difference image {D ₂ , D ₄ }. That is, {a ^D1 _ij , a ^D3 _ij } is a pattern in which the weight of R pixel or B pixel is r = 1, and {a ^D2 _ij , a ^D4 _ij } is weight of Gr pixel or Gb pixel is r = 1. Are the patterns. Since these pixel values are the difference values between the average value and the added pixel value, they have a negative correlation with each other.

Therefore, in the two-dimensional coordinate space in which the generation of the 4-pixel addition difference values {a ^D1 _ij , a ^D3 _ij }, {a ^D2 _ij , a ^D4 _ij } is defined, an appropriate axis that minimizes the occurrence distribution variance is selected. If the value obtained by projecting the difference value on the axis is obtained, the data can be further compressed. The axis with the smallest variance of the occurrence distribution is, for example, the axis (1, 1) orthogonal to the axis (1, -1) along the occurrence distribution having a negative correlation.

Here, an example in which the above two sets of 4-pixel addition difference values are simply added is shown. That is, as shown in the following equation (7), the data is compressed by projecting the direction vector (1, 1) onto the axis.
φ ¹ = (a ^D1 _ij + a ^D3 _ij ) / 2
φ ² = (a ^D2 _ij + a ^D4 _ij ) / 2 (7)

In this case, the recording data may be the difference addition value {φ ¹ , φ ² } instead of the 4-pixel addition difference value {a ^D3 _ij , a ^D4 _ij }. If the image data composed of the difference addition values {φ ¹ , φ ² } is expressed as {Φ ₁ , Φ ₂ }, the fusion image data to be finally recorded is F (M, a ^D1 _ij , a ^D2 _ij , Φ ₁ , Φ ₂ ).

5. In the high-definition still picture playback method first data compression techniques described above, the fusion image data _{F (M, D 1 ~D 4} ) is obtained. A technique for restoring and estimating a high-definition still image (frame image fx in FIG. 1) from F (M, D _{1 to} D ₄ ) will be described.

When the above equation (3) is transformed, the following equation (8) is obtained. As shown in the following equation (8), the 4-pixel addition value {a ^M _ij } constituting the average image M and the 4-pixel addition values {a ^D1 _ij , a ^D2 _ij , constituting the difference images D _{1 to} D ₄ , From the a ^D3 _ij , a ^D4 _ij }, the 4-pixel addition values {a ⁽¹⁾ _ij , a ⁽²⁾ _{(i + 1) j} , a ⁽³⁾ _{(i + 1) (j + 1) of} the pixel addition images A _{1 to} A _{4 ^{), a (4) i (}} j + 1)} is obtained.
a ⁽¹⁾ _ij = a ^M _ij −a ^D1 _ij ,
a ⁽²⁾ _{(i + 1) j} = a ^M _ij −a ^D2 _ij ,
a ⁽³⁾ _{(i + 1) (j + 1)} = a ^M _ij −a ^D3 _ij ,
a ⁽⁴⁾ _{i (j + 1)} = a ^M _ij −a ^D4 _ij (8)

The obtained 4-pixel addition value {a ⁽¹⁾ _ij , a ⁽²⁾ _{(i + 1) j} , a ⁽³⁾ _{(i + 1) (j + 1)} , a ⁽⁴⁾ _{i (j + 1)} } is one high definition This is image data composed of an added value of four pixels obtained by superimposing and shifting an image horizontally or vertically one pixel at a time. The pixel value of the original high-definition frame image fx is obtained by applying this 4-pixel addition value to the restoration estimation process described later with reference to FIGS.

In the second data compression method described above, the fused image data F (M, a ^D1 _ij , a ^D2 _ij , Φ ₁ , Φ ₂ ) is obtained. A method for restoring and estimating a high-definition still image from F (M, a ^D1 _ij , a ^D2 _ij , Φ ₁ , Φ ₂ ) will be described.

When the above equation (7) is transformed, the following equation (9) is obtained. Difference images D _{1 to} D ₄ are obtained from the following equation (9). If the difference images D _{1 to} D ₄ are obtained, the final high-definition frame image fx is restored by performing the same processing as described above.
a ^D3 _ij = a ^D1 _ij -2φ ¹ ,
a ^D4 _ij = a ^D2 _ij -2φ ² (9)

In the above embodiment, as shown in FIG. 1, the addition unit is shifted four times to obtain the addition images A _{1 to} A ₄ and the average image M and the difference images D _{1 to} D ₄ are obtained. The form is not limited to this. For example, the addition unit may be shifted nine times to obtain the added images A _{1 to} A ₉ and the average image M and the difference images D _{1 to} D ₉ may be obtained.

6). Imaging Device FIG. 3 shows a configuration example of an imaging device that performs the above-described data compression processing and high-definition frame image fx restoration processing.

  The imaging apparatus includes an imaging unit 10 that performs imaging and data compression processing, and an image processing unit 20 that performs restoration processing of a high-definition image. The image processing unit 20 may be built in the camera body or may be configured by an external information processing apparatus such as a PC.

  Specifically, the imaging unit 10 includes a lens 110, an imaging element 120 (imaging sensor), an addition image generation unit 130 (superimposition shift weighted addition image generation unit), a compression processing unit 140, and a compressed data recording unit 150 (fusion compression data). Recording unit), a low-resolution moving image generation unit 160 (standard moving image generation unit), a moving image data compression unit 170, a moving image data recording unit 180, and a monitor display unit 190.

  The lens 110 forms an image of the subject 100. The image sensor 120 captures the formed subject image. An analog signal obtained by imaging is converted into a digital signal by an A / D converter (not shown).

As described above, the addition image generation unit 130 adds the pixel values of the captured image while shifting the pixels, and generates pixel addition images A _{1 to} A ₄ from the captured image.

The compression processing unit 140 generates an average image M and difference images D _{1 to} D ₄ from the pixel addition images A _{1 to} A _4, and performs a process of compressing the average image M and the difference images D _{1 to} D ₄ . Specifically, the compression processing unit 140 includes a difference image generation unit 141, an average image generation unit 142, an entropy encoding unit 143, and a data compression unit 144.

The average image generation unit 142 generates an average image M from the pixel addition images A _{1 to} A ₄ . The difference image generation unit 141 generates difference images D _{1 to} D ₄ from the pixel addition images A _{1 to} A ₄ and the average image M. The entropy encoding unit 143 compresses the difference images D _{1 to} D ₄ by lossless compression such as the above-described entropy encoding. The data compression unit 144 compresses the average image M by irreversible compression such as M-JPEG or JPEG-XR.

The compressed data recording unit 150 records the compressed image data. Specifically, the compressed data recording unit 150 includes a difference data recording unit 151 that records the compressed difference images D _{1 to} D ₄ and an average image data recording unit 152 that records the compressed average image M.

  The low-resolution moving image generation unit 160 generates a real-time low-resolution moving image by processing described later with reference to FIG. The moving image data compression unit 170 compresses a low resolution moving image by using, for example, AVCHD (H.264), M-JPEG, JPEG-XR, or the like. The moving image data recording unit 180 records a compressed low resolution moving image. The monitor display unit 190 displays a real-time or reproduced low-resolution moving image.

  The image processing unit 20 includes an expansion processing unit 205, an added image noise reduction processing unit 220, an estimation calculation unit 230 (high-definition image restoration estimation unit), a high-definition still image generation unit 240, a high-definition moving image generation unit 250, and a high-quality standard. A moving image generation unit 260, a moving image compressed data decompression unit 270, a standard moving image generation unit 280, an image output unit 290, and an image selection unit 295 are included.

The decompression processing unit 205 decompresses the compressed data and reproduces the added images A _{1 to} A ₄ . Specifically, the expansion processing unit 205 includes a compressed data expansion unit 200 and an addition image reproduction unit 210 (weighted addition image reproduction unit).

The compressed data decompression unit 200 performs a process of decompressing the compressed average image M and the difference images D _{1 to} D ₄ . The addition image reproduction unit 210 performs a process of reproducing the addition images A _{1 to} A ₄ from the average image M and the difference images D _{1 to} D ₄ . The added image noise reduction processing unit 220 performs adaptive noise reduction processing, which will be described later with reference to FIG. 19 and the like, on the added images A _{1 to} A ₄ .

The estimation calculation unit 230 restores the high-definition frame image fx by estimation based on the added images A _{1 to} A ₄ . The restored image is a Bayer array RAW image. This estimation calculation will be described later with reference to FIG.

  The high-definition still image generation unit 240 performs demosaicing processing on the restored image with the Bayer arrangement, and performs image processing such as gradation correction processing on the image to generate a high-definition still image. At this time, a still image at the timing selected by the image selection unit 295 is generated. The timing is selected by a user instruction. The user may select the timing by viewing the moving image displayed on the monitor display unit 190, or may select the timing by viewing the moving image output by the image output unit 290.

  The high-definition moving image generation unit 250 performs demosaicing processing on the restored moving image having the Bayer arrangement, performs image processing such as gradation correction processing on the moving image, and generates a high-definition moving image. The high-quality standard video generation unit 260 downsamples the high-definition video and generates, for example, a video with the number of high-definition pixels as a high-quality standard video.

The standard moving image generation unit 280 generates a low resolution moving image by processing described later with reference to FIG. 9 and the like using the addition images A _{1 to} A ₄ reproduced by the addition image reproduction unit 210, and the low resolution moving image is standardized. Output as a movie. The standard moving image is, for example, a moving image having a high-definition pixel number. Further, the moving image compression data expansion unit 270 may expand the compressed low resolution moving image from the moving image data recording unit 180, and the standard moving image generation unit 280 may output the expanded low resolution moving image as a standard moving image. .

  The image output unit 290 outputs a high-definition still image, a high-definition moving image, a high-quality standard moving image, and a standard moving image, for example, to a display device or a printer.

  7). Code assignment example of fusion image data

The code assignment when the above-described compression processing unit 140 compresses the fused image data F (M, D _{1 to} D ₄ ) will be described.

FIG. 4 shows a first code allocation example. As shown in FIG. 4, 24 bits are assigned to the pixel value of the coordinates (i, j) of the fused image data F (M, D _{1 to} D ₄ ). Specifically, 8 bits are assigned to the pixel value a ^M _ij of the average image M, and 4 bits are assigned to the pixel values a ^D1 _{ij to} a ^D4 _ij of the difference images D _{1 to} D ₄ , respectively. For example, if the number of pixels of the captured image (original image) is N pixels and the pixel value data is 12 bits, the data amount of the compressed data is 1/2 of the data amount of the captured image = (N / 4 pixels × 24 bits) / (N pixels × 12 bits). Then, lossless or lossy compression is performed on the data after the code assignment.

FIG. 5 shows a second code allocation example. As shown in FIG. 5, 8 bits are assigned to each pixel value of the average image M, and the data is irreversibly compressed. Also, 4 bits are assigned to each pixel value of the difference images D _{1 to} D ₄ and the data is reversibly compressed. In Figure 5 shows a code assignment of the difference image D _1, it is the same for code assignment of the other of the difference image D ₂ to D _4. In FIG. 5, d _ij = a ^D1 _ij .

In this example, it is conceivable that the average image is somewhat degraded when the data is expanded again, but as shown in the above equation (3), the difference pixel values {a ^D1 _ij , a ^D2 _ij , a ^D3 _ij , a ^D4 _ij } has a common average value a ^M _ij . In the restoration estimation process described later, estimation is performed using a difference value between adjacent pixel values (for example, a ⁽²⁾ _{(i + 1) j} −a ⁽¹⁾ _ij = a ^D2 _ij −a ^D1 _ij ). At this time, since the average value a ^M _ij is canceled, the difference value between adjacent pixel values can be completely restored even if the average image is degraded. Thus, it can be said that the method of specifying and estimating the original pixel value relationship based on the difference value of the adjacent addition values is a restoration estimation process that is very convenient for the compression method of the present method.

8). Second Configuration Example of Imaging Device FIG. 6 shows a second configuration example of the imaging device as a configuration example in the case of performing reversible compression using the average value m _AVR described in the above equation (6). The imaging device includes an imaging unit 10 and an image processing unit 20. In the following description, the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The imaging unit 10 includes a lens 110, an image sensor 120, an addition image generation unit 130, a difference image generation unit 141, an average image generation unit 142, an average value calculation unit 145, a subtraction unit 146, a lossless compression encoding unit 147, and compressed data recording. Unit 150, low-resolution moving image generation unit 160, moving image data compression unit 170, moving image data recording unit 180, and monitor display unit 190. The components of the image processing unit 20 are the same as those of the image processing unit 20 in FIG.

The average value calculation unit 145 calculates the average value m _AVR described in the above equation (6) based on the average image M. The subtracting unit 146 subtracts the average value m _AVR from the pixel value a ^M _ij of the average image M, and outputs the pixel value md _ij after the subtraction. The lossless compression encoding unit 147 performs lossless compression on the data of the difference images D _{1 to} D _{4 and} the data of the pixel value md _ij . The compressed data recording unit 150 records the compressed data and the average value m _AVR .

The compressed data decompression unit 200 decompresses the difference images D _{1 to} D ₄ , the pixel value md _ij, and the average value m _AVR from the compressed data. The addition image reproduction unit 210 reproduces the average image M from the pixel value md _ij and the average value m _AVR, and reproduces the addition images A _{1 to} A ₄ from the average image M and the difference images D _{1 to} D ₄ .

FIG. 7 shows an example of code assignment of the fused image data F (M, D _{1 to} D ₄ ) in the second configuration example. As shown in FIG. 7, the average value _{m AVR} is arranged at the head of the pixel values _{md ij,} 8 bits each are assigned to the mean value _{m AVR} and pixel values _{md ij.} Also, 4 bits are assigned to each of the pixel values d _ij of the difference images D _{1 to} D ₄ . These data are reversibly compressed.

According to the second configuration example, even when the occurrence distribution probability of the pixel value md _ij of the average image M is not biased, it is possible to reduce the entropy because the bias can be created by converting to the 4-pixel different color addition image. A compression effect in lossless compression can be expected. Further, by adopting the difference between the average value m _AVR and the pixel value md _ij as data, a stable data value can be obtained without greatly depending on the position (average value) of the occurrence probability distribution, and a compression effect is expected. it can. Also, since all are lossless compression, there is no data degradation before restoration estimation.

9. Modified Configuration Example of Imaging Device In the above embodiment, the case where the fusion image data F (M, D _{1 to} D ₄ ) is compressed has been described as an example. However, the present embodiment is not limited to this, and the added image A ₁ to A ₁ to it may be compressed the a _4. FIG. 8 shows a modified configuration example of the imaging device when the added images A _{1 to} A ₄ are compressed. The imaging device includes an imaging unit 10 and an image processing unit 20.

  The imaging unit 10 includes a lens 110, an imaging element 120, an added image generation unit 130, a data compression unit 148, a data recording unit 155, a low resolution moving image generation unit 160, a moving image data compression unit 170, and a monitor display unit 190. The components of the image processing unit 20 are the same as those of the image processing unit 20 in FIG.

The data compression unit 148 compresses the added images A _{1 to} A ₄ by irreversible compression such as M-JPEG or JPEG-XR. The data recording unit 155 records the added images A _{1 to} A ₄ compressed by the data compression unit 148 and the low resolution moving image compressed by the moving image data compression unit 170.

According to this modified configuration example, by compressing the added images A _{1 to} A ₄ by superposition shift weighted addition, it is possible to directly compress the Bayer image without performing normal demosaicing processing or YCbCr conversion.

  According to the above embodiment, as illustrated in FIG. 3, the imaging apparatus includes an image acquisition unit (for example, the imaging device 120), the addition image generation unit 130, the compression processing unit 140, the expansion processing unit 205, and the estimation calculation unit 230. An image output unit 290 is included.

As illustrated in FIG. 1, the image acquisition unit acquires a captured image fx (high-definition frame image). The addition image generation unit 130 sets an addition unit, which is a unit for acquiring an addition pixel value (for example, a ⁽¹⁾ _ij ), for each of a plurality of pixels (for example, every four pixels) of the captured image, and is included in the addition unit. The pixel values are weighted and added (the above formula (1)) to obtain the added pixel value, and the added images A _{1 to} A _{4 based} on the obtained added pixel value are obtained. The compression processing unit 140 compresses the added images A _{1 to} A ₄ . The decompression processing unit 205 decompresses the compressed addition images A _{1 to} A ₄ . The estimation calculation unit 230 estimates the pixel value v _ij of the captured image fx based on the expanded added images A _{1 to} A ₄ . The image output unit 290 outputs a high resolution image based on the estimated pixel value v _ij .

In this case, the addition image generation unit 130 sequentially shifts the addition unit by pixels to obtain first to fourth (first to nth in a broad sense) addition images A _{1 to} A ₄ . The compression processing unit 140 obtains the average of the _first to fourth addition images A _{1 to} A ₄ as the average image M (the above expression (2)), and the average image M and the m-th addition image A _m (m Is a natural number equal to or less than n) as the m-th difference image D _m (the above equation (3)), and the obtained average image M and difference image D _m are compressed. Decompression processing unit 205 decompresses the compressed average image M and the difference image _{D m} by determining the added image _A 1 to A _4.

In this way, the captured image fx can be efficiently compressed. That is, although the captured image fx and the four added images A _{1 to} A ₄ have the same number of pixels, the entropy of the pixel value of the difference image D _m can be made smaller than that of the captured image fx as described above. The rate can be improved. As a result, the captured image fx can be restored from the high-compression-rate image data to extract a high-definition still image at an arbitrary timing. Further, since the added images A _{1 to} A ₄ are generated from one captured image fx, a restored image with a high temporal resolution with little blur can be obtained even with moving objects.

  In addition, since the data can be reduced by being able to reduce the resolution at high speed immediately after the sensor, the data processing load in the subsequent stage can be reduced, and a high-speed image processing system for high-definition images without high-speed data processing. Can be built. In addition, it is possible to provide a technology that can be easily applied to existing systems.

Further, according to the present embodiment, the captured image fx can be reproduced from the compressed data with a simple process. That is, the addition images A _{1 to} A ₄ are image data obtained by superposition shift addition, and a restoration estimation process described later can be applied. This restoration estimation process can simplify the process of estimating a high-resolution image from a low-resolution image as compared with Patent Documents 1 and 2 described above.

In this embodiment, as shown in FIG. 1, the addition image generation unit 130 sequentially shifts the addition unit one pixel at a time horizontally or vertically (i-axis direction or j-axis direction). The position (for example, coordinates (0,0), (1,0), (1,1), (0,1)) is set, and the added images A _{1 to} A ₄ are respectively displayed at the first to fourth positions. get. The addition unit of the m-th position and the m + 1-th position (for example, (0, 0), (1, 0)) includes a common pixel (v ₁₀ , v ₁₁ ).

In the present embodiment, as illustrated in FIG. 3, the compression processing unit 140 includes an average image generation unit 142 and a difference image generation unit 141. As shown in the above equation (2), the average image generation unit 142 obtains the average value a ^M _ij of the added pixel values at the first to fourth positions as the pixel value of the average image. The difference image generation unit 141 uses the pixel value a ^M _ij of the average image and the difference value (a ^D1 _ij ) of the added pixel value (for example, a ⁽¹⁾ _ij ) of the m-th position as the pixel of the m-th difference image. Calculate as a value.

By doing so, it is possible to perform superposition shift addition for obtaining an added pixel value by sequentially shifting the addition unit while including a common pixel. Further, by averaging the added pixel value of the added image A ₁ to A _4, may be obtained pixel values of the average image M. Further, the pixel values of the difference images D _{1 to} D ₄ can be obtained from the difference value between the pixel value of the average image and the addition pixel values of the addition images A _{1 to} A ₄ .

In this embodiment, as described with reference to FIG. 3 and the like, the compression processing unit 140 losslessly compresses the mth difference image _Dm by entropy coding.

In this way, it is possible to compress the difference image D _m entropy is reduced as described above at a high compression ratio, it is possible to improve the compression ratio of the image data. Thereby, the capacity of the storage for recording the compressed data can be reduced, and the communication load in data communication can be reduced.

In the present embodiment, the captured image fx is an RGB Bayer array image. In this case, the estimation calculation unit 230 obtains the pixel value v _ij of the RGB Bayer array of the captured image fx by estimation. The image output unit 290 outputs a high-resolution moving image based on an image obtained by demosaicing the estimated RGB Bayer array pixel values v _ij .

  For example, in the present embodiment, as described with reference to FIG. 3, the high-definition moving image generation unit 250 performs demosaicing processing, performs image processing such as gradation conversion on the image, generates a high-resolution moving image, The image output unit 290 outputs the high resolution moving image to a display device or the like. For example, a high-resolution moving image is a moving image having the same number of pixels as the number of pixels of the captured image.

  In this way, a high-resolution moving image having a resolution corresponding to the captured image fx can be restored from image data compressed at a high compression rate. This makes it possible to achieve both a high compression rate and high image quality for moving image data.

10. Method for Generating Low-Resolution Video As described above with reference to FIG. 3, the low-resolution video generation unit 160 and the standard video generation unit 280 simply generate a low-resolution video from the added images A _{1 to} A ₄ . A method for generating this low-resolution moving image will be described in detail.

In this method, low-resolution moving image data is generated using three addition images A 1 to A 3 among the addition images A 1 to A 4 . First, as shown in FIG. 9, using 4 addition pixel value of the added image A 1 {a (1) 22 }, 4 -pixel sum values undetected in addition image A 1 and {a (1) 12} { It is calculated as a relational expression of G 12 + G 23 }.

Specifically, the added image A _1, color relationship of weighted using four pixel sum value near that matches the non-detection value a ⁽¹⁾ _12, the estimated value of the undetected value a ⁽¹⁾ ₁₂ Perform approximate calculation. This estimated value a ⁽¹⁾ ₁₂ is obtained as a 4-pixel addition value on the assumption that the weight of the R color pixel is set to the reference “1”.

The pixel values constituting the non-detection value {a (1) 12}, expressed as in correspondence with colors {R 22, G 12, G 23, B 13}, non-detection value {a (1) 12} is It is represented by the following formula (10).
a (1) 12 = 1 · R 22 + (1 / r) · G 12 +
(1 / r) · G 23 + (1 / r 2 ) · B 13 (10)

Similarly, the 4-pixel addition value {a ⁽¹⁾ ₂₂ } adjacent to the right of the undetected value {a ⁽¹⁾ ₁₂ } is expressed by the following expression (11).
a ⁽¹⁾ ₂₂ = 1 · R ₂₂ + (1 / r) · G ₃₂ +
(1 / r) · G ₂₃ + (1 / r ² ) · B ₃₃ (11)

In the undetected value {a ⁽¹⁾ ₁₂ }, since the weighting of the {R ₂₂ } component and the {G ₂₃ } component is strongly dominant, a ⁽¹⁾ ₁₂ ≈ a ⁽¹⁾ ₂₂ is considered approximately. Assume that the following equation (12) is satisfied.
a ⁽¹⁾ ₁₂ = 1 · R ₂₂ + (1 / r) · G ₁₂ +
(1 / r) · G ₂₃ + (1 / r ² ) · B ₁₃
= A ⁽¹⁾ ₂₂ (12)

It should be noted that a ⁽¹⁾ ₁₂ may be obtained as an interpolated value (average, etc.) of a ⁽¹⁾ ₀₂ and a ⁽¹⁾ ₂₂ adjacent to the left and right of a ⁽¹⁾ ₁₂ , A ⁽¹⁾ ₁₂ may be obtained by interpolation from the surrounding pixel addition values having the same weighting.

Next, as shown in FIG. 10, using 4 addition pixel value of the added image ^{_{A 3 {a (3) 13}} }, the 4-pixel addition value of the undetected in addition image ^{_{A 3 {a (3) 12}} } Calculated as a relational expression of {G ₁₂ + G ₂₃ }.

Specifically, in addition image A _3, using 4-pixel sum values near the color relationship of the weighting to match the non-detection value a ⁽³⁾ _12, estimate undetected value a ⁽³⁾ ₁₂ Perform approximate calculation. The estimated value a ⁽³⁾ ₁₂ is obtained as a 4-pixel addition value on the assumption that the weight of the B color pixel is set to the reference “1”.

When the pixel values constituting the undetected value {a ⁽³⁾ ₁₂ } are expressed as {R ₂₂ , G ₁₂ , G ₂₃ , B ₁₃ }, the undetected value {a ⁽³⁾ ₁₂ } is expressed by the following equation (13). expressed.
a ⁽³⁾ ₁₂ = (1 / r ² ) · R ₂₂ + (1 / r) · G ₁₂ +
(1 / r) · G ₂₃ + 1 · B ₁₃ (13)

Similarly, the 4-pixel addition value {a ⁽³⁾ ₁₃ } adjacent to the lower side of the undetected value {a ⁽³⁾ ₁₂ } is expressed by the following expression (14).
a ⁽³⁾ ₁₃ = (1 / r ² ) · R ₂₄ + (1 / r) · G ₁₄ +
(1 / r) · G ₂₃ + 1 · B ₁₃ (14)

Since the {B ₁₃ } component and the {G ₂₃ } component are strongly dominant with respect to the undetected value {a ⁽³⁾ ₁₂ }, a ⁽³⁾ ₁₂ ≈ a ⁽³⁾ ₁₃ Consider the following equation (15).
a ⁽³⁾ ₁₂ = (1 / r ² ) · R ₂₂ + (1 / r) · G ₁₂ +
(1 / r) · G ₂₃ + 1 · B ₁₃
= A ⁽³⁾ ₁₃ (15)

It should be noted that a ⁽³⁾ ₁₂ may be obtained as an interpolated value (average, etc.) of a ⁽³⁾ ₁₁ and a ⁽³⁾ ₁₃ adjacent to the upper and lower sides of a ⁽³⁾ _12. It is also possible to obtain a ⁽³⁾ ₁₂ by interpolation from surrounding pixel addition values with equal weighting.

Now, since G ₁₂ and G ₂₃ in a ⁽¹⁾ ₁₂ and a ⁽³⁾ ₁₂ are the same pixel and are adjacent pixels, as shown in the following equation (16), if it is equal to the average value G _12/23 Handle with approximation.
_{G 12 ≒ G 23 ≒ G 12/23} = (G 12 + G 23) / 2 (16)

When the above equation (16) is applied to the above equations (12) and (15), the following equation (17) is obtained.
a ⁽¹⁾ ₁₂ = R ₂₂ + (1 / r) · 2G _12/23 + (1 / r ² ) · B ₁₃ ,
a ⁽³⁾ ₁₂ = (1 / r ² ) · R ₂₂ + (1 / r) · 2G _{12 /} G ₂₃ + B ₁₃ (17)

The above equation (17) is solved to obtain a relational expression between R ₂₂ , B ₁₃ and G _12/23 as shown in the following equation (18). In this way, it can be seen that R ₂₂ and B ₁₃ constituting the undetected value a ⁽¹⁾ ₁₂ or a ⁽³⁾ ₁₂ are obtained by G _12/23 .

Next, as shown in FIG. 11, the above R ₂₂ , B ₁₃ , and G _12/23 are obtained by using the 4-pixel addition value {a ⁽²⁾ ₁₂ } detected in the addition image A ₂ .

Specifically, the 4-pixel addition value {a ⁽²⁾ ₁₂ } is expressed by the following equation (19) using the above equation (16).
a ⁽²⁾ ₁₂ = (1 / r) · R ₂₂ + 1 · G ₁₂ +
(1 / r ² ) · G ₂₃ + (1 / r) · B ₁₃
≒ (1 / r) · R ₂₂ + [1 + (1 / r ² )] · G _12/23 +
(1 / r) ・ B ₁₃ (19)

Substituting R ₂₂ and B _{13 in} the above equation (18) into the above equation (19), G _12/23 is derived as shown in the following equation (20). By substituting G _12/23 derived by the following equation (20) into the above equation (18), R ₂₂ and B ₁₃ are obtained. A frame image of a low-resolution moving image is configured with R ₂₂ , G _12/23 and B ₁₃ obtained in this way as pixel values having the same coordinates.

When the values of each {R, G, B} constituting the 4-pixel addition value {a ⁽²⁾ _ij } are obtained in the same manner as described above, the original high-definition image fx is 1 / as shown in FIG. It is possible to obtain an RGB three-primary color image with a resolution reduced to 4 pixels.

  According to the method described above, a low-resolution three-plate color image can be obtained by simple arithmetic processing, which is effective as a moving image generation process that requires real-time processing. Further, in this method, as compared with the conventional four-pixel addition method of the same color, a shift in the center of gravity position of the G color when the Bayer is formed does not occur, and higher image quality can be obtained.

According to the above embodiment, as illustrated in FIG. 3, the imaging apparatus includes the low-resolution moving image generation unit 160 that generates a low-resolution moving image. As illustrated in FIG. 1, the captured image fx is an RGB Bayer array image, and the addition unit is four adjacent pixels (for example, four pixels of R, Gr, Gb, and B). In this case, the addition image generation unit 130 acquires _first to fourth addition images A _{1 to} A _{4 in} which RGB is mixed by weighted addition at the first to fourth positions. As described with reference to FIGS. 9 to 11, the low-resolution moving image generation unit 160 uses the RGB pixel values (for example, FIG. 11) of the low-resolution moving image based on the added images of the _{first to third} A _{1 to} A ₃ . R ₂₂ , G _12/23 , B ₁₃ ) are obtained by estimation.

In this way, a low-resolution three-plate color image can be obtained by simple arithmetic processing, which is effective as a moving image generation process that requires real-time processing such as live view, for example. In addition, since the RGB pixel value at the same position (for example, a ₁₂ ) is estimated, the center of gravity of G color does not shift when the Bayer method is used, and higher image quality can be obtained as compared with the conventional 4-pixel addition method of the same color. Can be obtained.

11. Third Configuration Example of Imaging Device In the above embodiment, the fusion image data F (M, D _{1 to} D ₄ ) is compressed, but in this embodiment, after the difference images D _{1 to} D ₄ are entropy-encoded. Data reconstruction may be performed and the reconstructed data may be compressed.

  FIG. 12 shows a third configuration example of the imaging apparatus as a configuration example when data reconstruction is performed. The imaging device includes an imaging unit 10 and an image processing unit 20. In the following description, the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The imaging unit 10 includes a lens 110, an image sensor 120, a low resolution image generation unit 125 (a moving image low resolution Bayer image generation unit), a first addition image generation unit 130, a second addition image generation unit 135, and a difference. Image generation unit 141, average image generation unit 142, entropy encoding unit 143, compressed data recording unit 150, low-resolution moving image generation unit 160, pixel data reconstruction unit 165, third addition image generation unit 175, data compression unit 185 (fusion data compression unit) and monitor display unit 190.

  The image processing unit 20 includes a compressed data decompression unit 200, an addition image reproduction unit 210, a low resolution image estimation calculation unit 215 (a low resolution Bayer image generation unit for moving images), an estimation calculation unit 230, and a high-definition still image generation unit 240. , A high-definition video generation unit 250, a high-quality standard video generation unit 260, a standard video generation unit 280, an image output unit 290, and an image selection unit 295.

The low-resolution image generation unit 125 newly generates a low-resolution image having a Bayer array with a ¼ pixel number, for example, by adding four pixels of the same proximity and the same color. The second addition image generation unit 135 performs addition processing described with reference to FIG. 1 on the low resolution image to generate addition images B _{1 to} B ₄ .

The average image generation unit 142 generates an average image of the addition images A _{1 to} A _{4 and} an average image of the addition images B _{1 to} B ₄ . The difference image generation unit 141 generates a difference image between the addition images A _{1 to} A ₄ and the average image, and a difference image between the addition images B _{1 to} B ₄ and the average image. The entropy encoding unit 143 performs entropy encoding on the difference image from the difference image generation unit 141.

The pixel data reconstruction unit 165 reconstructs the pixel data of the average image and the pixel data of the difference image subjected to entropy coding by a method described later with reference to FIG. 16, and outputs reconstruction data u _ij . The third addition image generation unit 175 generates an addition value ξ _ij by performing addition processing described later with reference to FIG. 17 on the reconstructed pixel data. The data compression unit 185 compresses the addition value ξ _ij by irreversible compression such as M-JPEG or JPEG-XR.

The compressed data decompression unit 200 decompresses the added value ξ _ij from the compressed data. Adding image reproduction unit 210 reproduces the reconstructed data _{u ij} from the extended summation value xi] _ij, to reproduce the added image _{A 1 ~A 4, B 1 ~B} 4 from the reconstructed data _{u ij.} The low-resolution image estimation calculation unit 215 performs estimation processing, which will be described later with reference to FIGS. 27A to 30, on the added images B _{1 to} B ₄ to restore a Bayer-array low-resolution image. The standard moving image generation unit 280 performs demosaicing on the Bayer array low resolution image to generate a low resolution moving image (standard moving image).

12 Operation and Processing of Third Configuration Example Next, operations and processing performed by the third configuration example of the imaging apparatus will be described.

  FIG. 13 conceptually shows an example of an imaging process for simultaneously acquiring a high resolution still image and a low resolution still image. In the following, it is assumed that the imaging frame from the imaging device is high resolution (for example, 12 megapixels) and the frame rate is high speed (for example, 60 fps).

The even frames f ₀ , f ₂ ,... Are converted from a 12-megapixel Bayer image into four 3-megapixel addition images by the addition processing described with reference to FIG. The odd-numbered frames f ₁ , f ₃ ,... Are converted into a 3 megapixel Bayer array image by, for example, general addition of four pixels of similar colors. The converted image is converted into four 0.75 megapixel added images by the addition processing described in FIG.

  An average image and a difference image are generated from these converted images, and data of the difference image is compressed by entropy coding. The compressed difference image is subjected to pixel bit reconstruction together with the average image. Pixel addition is performed on the reconstructed pixel data, and normal image data compression processing is performed on the data after the addition.

Here, pixel addition is performed on the reconstructed pixel data for the following reason. That is, as described above with reference to FIG. 12, an average image and a difference image are generated from the added images A _{1 to} A ₄ and B _{1 to} B ₄ , and the difference image is entropy-coded. The average image data and the encoded difference image data are no longer pixel value sets having image characteristics, and are merely an array of bits with high randomness. Therefore, it cannot be recognized as data having a bias in a specific code. That is, it is safer to consider that the entropy is high. The same applies to the data group obtained by integrating the difference image data and the average image data into one by the pixel data reconstruction unit 165.

  Therefore, it is necessary to rewrite the data group once having increased entropy to data having lower entropy. For this purpose, the entropy is reduced by using the addition effect in the third addition image generation unit 175. By reducing the entropy of reconstructed data in this way, the effect of subsequent data compression is enhanced.

FIG. 14 shows a sequence example of the imaging process. As shown in FIG. 14, first, the high-resolution image of the even frame f ₀ is exposed and read in the period T ₀ , and the pre-encoding process (PE 0) and the normal image data compression process (compression 0) are performed in the period T ₁ + T ₂ . I do. In parallel with this pre-encoding processing and the normal image data compression processing, exposure and read-out of the high resolution image of the odd-numbered frames f ₁ in the period T _1, the low-resolution process in the period T _{2 +} T _3/2 and the pre Encoding processing (PE1) and normal image data compression processing (compression 1) are performed. In this processing sequence, the normal image data compression processing of the even frame f ₀ and odd frames f ₁ consecutive (compression 0, compression 1) is processed within the period 2T serially.

  Next, the reconstruction of the pixel value data will be described. FIG. 15 shows an example of data reconstruction before data compression. This example is a data reconstruction example in the case where the data before data compression is formed with 12 bits per pixel, but the data compression process can be processed only with 8 bits per pixel.

As shown in FIG. 15, the pixel value bit string of the average image is represented by m _{10 to} m ₁₆ and m _{20 to} m ₂₆ , and the pixel value bit string of the difference image after entropy coding is represented by d ₁₀ to d ₁₄ and d _{20 to} d. ₂₄ . When reconstructing the sequence data of the 12-bit to 8-bit array data, arranged in the upper bits of the reconstructed data bit stream _{d 10 ~d 14, d 20 ~d} 24, the bit string _m 10 _{~m 16, m 20} The upper bits in .about.m ₂₆ are arranged in the upper bits of the reconstructed data as much as possible. This is to prevent the pixel value after entropy coding and the pixel value of the average image from being deteriorated by data compression decoding as much as possible.

FIG. 16 shows an example of a data unit set in data reconstruction. As shown in FIG. 16, the difference value data {d _k } of the difference image after entropy coding is added to the pixel value data {a ^M _ij } (fixed bit width) of the average image, and the data is re-synthesized. Configure. That is, the pair {a ^M _ij } + {d _k } of the difference value data after entropy encoding and the pixel value data of the average image is reconstructed as a data unit {u _ij } by _regarding it as a virtual pixel value.

In this data unit {u _ij }, the average image pixel value {a ^M _ij } is arranged on the lower bit side, and the difference value data {d _k } is arranged on the upper bit side. In this way, when the reconstructed data is subjected to lossy compression / decompression such as JPEG, the decoding error of the data unit {u _ij } can affect the average image pixel value having a fixed bit width. That is, it is possible to minimize the influence of the decoding error on the difference value data. This is because the difference value data is entropy-encoded data, so that if there is a decoding error, it is reproduced as a value with a large error, so that it is avoided.

Since the reconstructed data unit {u _ij } is treated as a fixed length as the code bit length N, the data {d _k } after the entropy encoding is calculated from the code bit length N to the pixel value {a ^M _ij } of the average image. The code bit length n _M0 is subtracted from the code bit length [N−N _M0 ] so as to be appropriately connected. Therefore, the data {d _k } may be divided and formed across a plurality of reconstruction units {u _ij }.

FIG. 17 shows an example of the superimposed shift addition value of the reconstruction data unit {u _ij }. Since {u _ij }, which is a virtual pixel, is a value obtained as a result of the average image bit addition and arbitrarily bit-separated, it is not possible to expect bias in the distribution of distribution or high correlation between neighboring pixels. Therefore, as shown in FIG. 17, again performs superimposition shift weighted addition processing on the reconstructed data {u _ij}, to generate a sum value {ξ _ij}. The added value {ξ _ij } is obtained by the following equation (21).

ξ _ij = [u _ij + (1 / r) u _{(i + 1) j} +
(1 / r) u _{i (j + 1)} + (1 / r ² ) u _{(i + 1) (j + 1)} ] / 4
(21)

In this way, it is possible to increase the bias of the generation distribution of the added value {ξ _ij } and increase the correlation between neighboring pixels (reducing entropy). As a result, the compression effect in the subsequent image data compression can be enhanced.

13. Same Color Addition In the above embodiment, the case where four adjacent pixels of different colors in the Bayer array are added has been described, but in the present embodiment, four pixels of the same color in the Bayer array may be added. FIG. 18 shows a configuration example of the imaging unit when adding four pixels of the same color.

  The imaging unit illustrated in FIG. 18 includes a lens 110, an imaging element 120, addition image generation units 301 to 304, difference image generation units 311 to 314, average image generation units 321 to 324, entropy encoding units 331 to 334, and Bayer image generation. Section 340, moving image data compression section 350, and compressed data recording section 150. In the following description, the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

The added image generation units 301 to 304 perform pixel shift addition of R pixels, Gr pixels, Gb pixels, and B pixels, respectively, in the Bayer array captured image. Taking the added pixel values {a ^R1 ₀₀ , a ^R2 ₂₀ , a ^R3 ₂₂ , a ^R4 ₀₂ } of the R added images A _{R1 to} A _R4 as an example, they are represented by the following expression (22).
a ^R1 ₀₀ = R ₀₀ + R ₂₀ + R ₂₂ + R ₀₂ ,
a ^R2 ₂₀ = R ₂₀ + R ₄₀ + R ₄₂ + R ₂₂ ,
a ^R3 ₂₂ = R ₂₂ + R ₄₂ + R ₄₄ + R ₂₄ ,
a ^R4 ₀₂ = R ₀₂ + R ₂₂ + R ₂₄ + R ₀₄ (22)

  The average image generation units 321 to 324 generate average images based on the added images of the respective colors. The difference image generation units 311 to 314 generate difference images between the average image of each color and the added image, respectively. The entropy encoding units 331 to 334 perform entropy encoding of the difference images for each color.

  The Bayer image generation unit 340 arranges the pixel values of the average image of the four colors in the Bayer array, and generates one Bayer array image. The moving image data compression unit 350 irreversibly compresses moving image data based on the Bayer array image using, for example, AVCHD (H.246), M-JPEG, JPEG-XR, or the like.

14 Adaptive Noise Reduction Processing The adaptive noise reduction processing performed by the above-described added image noise reduction processing unit 220 will be described in detail with reference to FIG.

  In this embodiment, adaptive noise reduction (hereinafter abbreviated as ANR) is applied to the 4-pixel addition value. ANR refers to adaptive moving average filtering that performs pixel value averaging while adaptively varying the target range along the spatial axis. In ANR, when the range of change of the sampling value within the target range is small, the range of the moving average is widened, and when the range of change of the sampling value within the target range is large, the range of the moving average is narrowed. In order to realize this, the average of the sampling values included in a predetermined pixel value width centered on the pixel value at the processing position (target position) is calculated, and is set as the pixel value at the processing position again.

  In the range where the change in the sampling value is small, even if the change is a high-frequency component, the amplitude is small, so rounding is considered to have little effect. In the case of a low-frequency component, information is maintained even if a moving average is taken. In this case, since the moving average range is wide, more sampling values are averaged, and the noise reduction effect due to the addition effect is large.

  On the other hand, in the range where the change range of the sampling value is large, information is almost maintained to narrow the moving average range regardless of whether the change is a high-frequency component or a low-frequency component. In this case, since the average of a small number of sampling values is taken, the noise reduction effect due to the addition effect is small.

Therefore, the case where this method is applied has a large effect when the amplitude change width of the sampling value is small. As illustrated in FIG. 19, the pixel value a ^M _ij of the average image and the pixel value (for example, a ⁽¹⁾ _ij ) of the added image have high pixel value correlation. Since these difference values a ^D1 _ij are projections onto the vector (1, −1), the occurrence probabilities of the difference values a ^D1 _ij are distributed in a narrow range, and the amplitude of the sampling value of the difference image than the original captured image Change can be reduced. Therefore, if ANR is applied to this difference image, it is considered that noise in the difference image can be effectively reduced.

Next, an ANR application method and a calculation method will be described. As shown in FIG. 20 (A), assuming that the 4-pixel addition value (sampling value) to be processed is a _k , the 4-pixel addition value in the vicinity that falls within the range of width ± Δ / 2 around a _k is specified. For example, in the case of FIG. 20A, there is no 4-pixel addition value that falls within the range of width ± Δ / 2. In this case, as shown in the following equation (23), the moving average value a _k ′ is _{obtained with} d1 = d2 = 0.

In the case of FIG. 20B, the 4-pixel addition values that fall within the range of the width ± Δ / 2 are a _k−2 , a _k−1 , and a _{k + 1} . In this case, d1 = 2 and d2 = 1 in the above equation (23). Note that d1 represents a range of 4-pixel addition values that take a moving average in the left direction (negative direction of the k-axis) of the 4-pixel addition values _{ak to be} processed. d2 represents the range of the 4-pixel addition value that takes a moving average in the right direction (the positive direction of the k-axis) of the 4-pixel addition value _{ak to be} processed. It is assumed that d1 and d2 satisfy the condition shown in the following formula (24).
a _k −Δ / 2 ≦ a _k−d1 and a _{k + d2} ≦ a _k + Δ / 2
Among d1 and d2 that satisfy d1, d1 + d2 is the maximum,
d1 and d2 are integers of 0 or more,
0 <Δ (24)

Next, taking the above-described difference images D _{1 to} D ₄ as examples in FIG. 3, an application example of the ANR will be described. As shown in FIG. 21, the difference images D _{1 to} D ₄ are each two-dimensional array data. In order to simplify the calculation, the ANR is applied separately in the horizontal direction and the vertical direction.

Specifically, as shown in FIG. 21, when the ANR is sequentially applied to the 4-pixel addition value a _ij in the horizontal direction (i-axis direction), the horizontal moving average value a ^h _ij is obtained. As shown in FIG. 22, when the ANR is sequentially applied in the vertical direction (j-axis direction) to the 4-pixel addition value a _ij , the moving average value a ^v _{ij in the} horizontal direction is obtained. As shown in FIG. 23, an average value a _ij ′ (for example, a _ij ′ = (a ^h _ij + a ^v _ij ) / 2) of the moving average values a ^h _ij and a ^v _ij is calculated, and the average value a _ij ′ is calculated. Is the final moving average value.

According to the above embodiment, as illustrated in FIG. 3, the imaging device includes a noise reduction processing unit (added image noise reduction processing unit 220). As described with reference to FIG. 20A and the like, the noise reduction processing unit performs adaptive noise reduction processing for adaptively adjusting the moving average range d1 + d2 according to the change width of the pixel value, and performs the m-th difference image D. _{to m} .

For example, in the present embodiment, the range d1 + d2 is adjusted according to the number of pixel values that fall within the range of width ± Δ / 2 from the pixel value _ak to be processed, and the average value of the pixel values in the range d1 + d2 is determined as the pixel value. a _Reset as _k .

  In this way, as described above, it is possible to effectively reduce the noise of the difference image in which the occurrence distribution of the pixel value is narrow and the amplitude of the pixel value is small. That is, in the adaptive noise reduction processing, when the amplitude of the pixel value is small, the moving average range d1 + d2 is widened, so that a noise component having a higher frequency than that range is effectively reduced.

15. Data compression method using inter-frame difference image In the above embodiment, the average image and the difference image are generated directly from the captured image. However, in this embodiment, the difference between the captured images is obtained between frames, and the inter-frame difference is obtained. An average image and a difference image may be generated from the images.

  FIG. 24 shows a fourth configuration example of the imaging apparatus as a configuration example when generating an average image and a difference image from the inter-frame difference image. This imaging apparatus includes an imaging unit 10 and an image processing unit 20. In the following description, the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The imaging unit 10 includes a lens 110, a subtraction unit 115, an imaging element 120, an addition image generation unit 130, a difference image generation unit 141, an average image generation unit 142, an average value calculation unit 145, a subtraction unit 146, and a lossless compression encoding unit 147. , A compressed data recording unit 150, a low resolution moving image generating unit 160, a moving image data compressing unit 170, a moving image data recording unit 180, and a monitor display unit 190. The components of the image processing unit 20 are the same as those of the image processing unit 20 in FIG.

Assume that the image sensor 120 captures captured images V _t−1 , V _t , and V _{t + 1} in frames fx−1, fx, and fx + 1. The subtractor 115 uses the captured image V _t as a reference image, and obtains a difference image V ′ _t−1 = V _t −V _t−1 and V ′ _{t + 1} = V _t −V _{t + 1} between the reference image and another captured image. . The added image generation unit 130 obtains an added image from the reference image V _t and the difference images V ′ _t−1 and V ′ _{t + 1} . The subsequent processing is the same as the processing described above with reference to FIG.

Estimation when calculating unit 230 restores the original captured image, restores the reference image V _t and the difference image _{V 't-1, V'} t + 1 from the first addition image, captured image V _t-1 from the images = determine the _{V t -V 't-1,} V t + 1 = V t -V' t + 1.

FIG. 25 schematically shows processing performed by the imaging apparatus. As shown in FIG. 25, RAW image data of frames f ₀ , f ₁ ,... Taken at a predetermined time interval (for example, 1/60 sec.) Are represented as V ₀ , V ₁ ,. The captured image of the reference frame f _{3k + 1} is V _{3k + 1,} and the captured images for generating the difference frames f _3k and f _{3k + 2} are V _3k and V _{3k + 2} . Here, k is an integer of 0 or more.

The reference captured image V _{3k + 1} is high-definition RAW image data itself, and superimposition shift weighted addition is performed on V _{3k + 1} to generate four four-pixel added images A ^{(3k + 1)} _{1 to} A ^{(3k + 1)} ₄ . Also, the difference between corresponding pixel values of the images V _3k and V _{3k + 2 in the} frames before and after the reference image and the reference image V _{3k + 1} is _calculated , and difference frame images V ′ _3k and V ′ _{3k + 2} are generated. These difference frame images V ′ _3k and V ′ _{3k + 2} are respectively subjected to superposition shift weighted addition, and four 4-pixel addition images A ^(3k) _{1 to} A ^(3k) ₄ , A ^{(3k + 2)} _{1, respectively.} ~ A ^{(3k + 2)} ₄ is generated.

  Next, in the same manner as in the method described above with reference to FIG. 1, an average image and a difference image are generated from the four-pixel addition image, and the average image and the difference image are subjected to compression processing (particularly lossless compression processing). Is recorded as fused image data.

  According to the above embodiment, since the close frame is generally a similar image, if the inter-frame difference is used, the compression effect as moving image data can be increased as compared with the method described above with reference to FIG. Note that entropy reduction in dissimilar portions cannot be expected with simple RAW data differences, and therefore, subsequent processing becomes effective when lossless compression is considered.

  FIG. 26 shows a modified configuration example of the imaging unit in the fourth configuration example. The imaging unit includes a lens 110, a subtraction unit 116, an imaging element 120, an addition image generation unit 130, a difference image generation unit 141, an average image generation unit 142, an average value calculation unit 145, a subtraction unit 146, and a lossless compression encoding unit 147. , A compressed data recording unit 150, a low resolution moving image generating unit 160, a moving image data compressing unit 170, a moving image data recording unit 180, and a monitor display unit 190. In the following description, the same components as those described above with reference to FIG. 3 and the like are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

In this modified configuration example, the inter-frame difference image of the added image is obtained instead of the inter-frame difference image of the captured image. Specifically, the added image generation unit 130 performs superposition shift weighted addition on the captured images V _3k , V _{3k + 1} , and V _{3k + 2} , and adds the added images A ^(3k) _{1 to} A ^(3k) ₄ , A ^{(3k + 1).} _{1 to} A ^{(3k + 1)} ₄ and A ^{(3k + 2)} _{1 to} A ^{(3k + 2)} ₄ are generated. The subtracting unit 116 adds the reference image addition images A ^{(3k + 1)} _{1 to} A ^{(3k + 1)} ₄ and the addition images A ^(3k) _{1 to} A ^(3k) ₄ and A ^{(3k + 2)} _{1 to} A ^{(3k + 2).} ₄ ⁾ _Find the difference of ₄ and output it as an added image of the difference frame. Then, in the same manner as in the above-described method in FIG. 1, an average image and a difference image are generated from the addition image of the reference frame and the addition image of the difference frame, and a reversible compression process is performed on the average image and the difference image. The data is recorded as fused image data.

According to the above embodiment, as shown in FIG. 25, the image acquisition unit (e.g., image sensor 120), the captured image _{V 3k + 1} of the first frame _{f 3k + 1} obtained as a reference photographed image, the first frame _{f 3k + 1} The difference between the captured image V _3k of the previous or subsequent second frame (for example, the previous frame f _3k ) and the reference captured image V _{3k + 1} is acquired as the differential captured image V ′ _3k . The addition image generation unit 130 includes first to fourth (first to nth in a broad sense) addition images A ^{(3k + 1)} _{1 to} A ^{(3k + 1)} ₄ based on the reference captured image V _{3k + 1} , and the difference captured image V ′. obtaining first to n-th addition image ^{_{a (3k) 1 ~A (3k}} ) 4 based on _3k.

The second frame may be a frame f _{3k + 2} after the first frame f _{3k + 1} . In this case, the differential captured image V ′ _{3k + 2} and the first to nth added images A ^{(3k + 2)} _{1 to} A ^{(3k + 2)} ₄ are acquired. Alternatively, the difference image and the added image may be obtained using both the frames before and after the first frame f _{3k + 1} .

  In this way, the inter-pixel correlation in the frame is increased by the pixel addition in the frame, and the inter-frame difference image can be obtained by subtraction of the added images. Thereby, it can be set as the subtraction of images with higher similarity, and the difference effect can be expected. For example, an average image generated from an inter-frame difference image is considered to have lower entropy than an average image generated from a normal addition image. Therefore, it is considered that the compression rate by entropy coding of the average image can be improved.

16. Next, the estimation process performed by the above-described estimation calculation unit 230 will be described in detail with reference to FIG. In the following, the addition pixel values {a ₀₀ , a ₁₀ , a ₁₁ , a ₀₁ } will be described as an example (i and j are integers of 0 or more), but the same applies to other addition pixel values. Further, the case where the addition unit is set for every 2 × 2 pixels will be described as an example, but the present invention is not limited to this, and may be, for example, every 3 × 3 pixels.

27A and 27B are explanatory diagrams of the estimated pixel value and the intermediate pixel value. Summing the pixel values shown in FIG. _{27 (A) {a 00,} a 10, a 11, a 01} is added pixel value of the added image _A 1 to A ₄ described in FIG. _{^{1 {a (1) 00,}} a ( ²⁾ Corresponds to ₁₀ , a ⁽³⁾ ₁₁ , a ⁽⁴⁾ ₀₁ }. In the estimation process, final estimated pixel values v _{00 to} v ₂₂ are estimated using the added pixel values. The estimated pixel value v _ij corresponds to the pixel value of the captured image fx described in FIG.

As shown in FIG. 27B, first, intermediate pixel values b _{00 to} b ₂₁ (intermediate estimated pixel values) are estimated from the added pixel values a _{00 to} a ₁₁ . Intermediate pixel value corresponds to 2-pixel sum values, for example, _{b 00} corresponds to the sum of the pixel values _{v 00} and _{v 01.} The final pixel values v _{00 to} v ₂₂ are estimated from these intermediate pixel values b _{00 to} b ₂₁ .

First, a process for estimating the intermediate pixel value will be described. Hereinafter, a case where the intermediate pixel values b _{00 to} b ₂₀ of the first row in the horizontal direction are estimated will be described as an example. The intermediate pixel values b _{01 to} b _{21 in the} next row are estimated by the same method.

As shown in FIG. 28, the intermediate pixel values b _{00 to} b ₂₀ are estimated based on the added pixel values a ₀₀ and a ₁₀ in the first row in the horizontal direction. For the sake of simplicity, for example, assuming that the weighting factor r = 2, the addition pixel values a ₀₀ and a ₁₀ are expressed by the following expression (25).
a ₀₀ = v ₀₀ + (1/2) v ₀₁ + (1/2) v ₁₀ + (1/4) v ₁₁ ,
_{a 10 = v 10 + (1/2} ) v 11 + (1/2) v 20 + (1/4) v 21
(25)

B ₀₀ , b ₁₀ , and b ₂₀ are defined as shown in the following formula (26).
b ₀₀ = v ₀₀ + (1 / r) v ₀₁ = v ₀₀ + (1/2) v ₀₁ ,
b ₁₀ = v ₁₀ + (1 / r) v ₁₁ = v ₁₀ + (1/2) v ₁₁ ,
b ₂₀ = v ₂₀ + (1 / r) v ₂₁ = v ₂₀ + (1/2) v ₂₁ (26)

Next, when the above equation (25) is transformed using the above equation (26), the following equation (27) is established.
a ₀₀ = b ₀₀ + (1/2) b ₁₀ ,
a ₁₀ = b ₁₀ + (1/2) b ₂₀ (27)

In the above equation (27), a difference δi ₀ is obtained by multiplying a ₀₀ , a ₁₀ by a predetermined weighting factor, and transformed using the above equation (26), the following equation (28) is established.
δi ₀ = a ₁₀ -2a ₀₀
= (1/2) b ₂₀ -2b ₀₀ (28)

If b ₀₀ is an unknown number (initial variable), intermediate pixel values b ₁₀ and b ₂₀ can be obtained as a function of b ₀₀ as shown in the following equation (29). In this way, a high-definition combination pattern of intermediate pixel values {b ₀₀ , b ₁₀ , b ₂₀ } is obtained with b ₀₀ as an unknown.
b ₀₀ = (unknown number),
b ₁₀ = 2 (a ₀₀ −b ₀₀ ),
b ₂₀ = 4b ₀₀ + 2δi ₀ = 4b ₀₀ +2 (a ₁₀ −2a ₀₀ ) (29)

Next, a description will be given of a method of obtaining the unknown _{b 00.} As shown in FIG. 29, the pattern {a ₀₀ , a ₁₀ } of the added pixel value is compared with the pattern {b ₀₀ , b ₁₀ , b ₂₀ } of the intermediate pixel value. Then, an unknown number b ₀₀ that minimizes the error E is derived and set as the intermediate pixel value b ₀₀ .

Specifically, the relationship of the following expression (30) is established between the added pixel value {a _ij } and the intermediate pixel value {b _ij , b _{(i + 1) j} }. Considering the weighting by the following equation (30), the evaluation function Ej shown in the following equation (31) is obtained. Then, with this evaluation function Ej, similarity evaluation between the pattern {a ₀₀ , a ₁₀ } and the pattern {b ₀₀ , b ₁₀ , b ₂₀ } is performed.
a _ij = b _ij + (1/2) b _{(i + 1) j} (30)

As shown in FIG. 30, an unknown b ₀₀ (= α) that minimizes Ej is obtained, and the value of b ₀₀ can be determined. Then, the estimated value of b ₀₀ is substituted into the above equation (29) to obtain b ₁₀ and b ₂₀ .

Next, a method for _{obtaining the} final estimated pixel value v _ij using the obtained intermediate pixel value b _ij will be described. In the following, description will be given taking the leftmost vertical column (i = 0 column) as an example. The final estimated pixel value v _ij is obtained in the same manner as the method for _obtaining the intermediate pixel value b _ij . That is, if the above equation (27) is replaced with the following equation (32), the subsequent processing is the same.
b ₀₀ = v ₀₀ + (1/2) v ₀₁ ,
b ₀₁ = v ₀₁ + (1/2) v ₀₂ (32)

According to the above embodiment, as shown in FIG. 27A, the addition unit (for example, a ₀₀ ) set to the first position and the second position subsequent to the first position are set. The addition unit (for example, a ₁₀ ) is overlapped. As shown in the above equation (28), the estimation calculation unit 230 obtains a difference value δi ₀ between the added pixel values a ₀₀ and a ₁₀ at the first and second positions. As shown in FIG. 27B, the first intermediate pixel value b ₀₀ is the addition of the first area (v ₀₀ , v ₀₁ ) obtained by removing the overlapping area (v ₁₀ , v ₁₁ ) from the addition unit a _00. It is a pixel value. The second intermediate pixel value b ₂₀ is an addition pixel value of the second region (v ₂₀ , v ₂₁ ) obtained by removing the overlap region (v ₁₀ , v ₁₁ ) from the addition unit a ₁₀ . As shown in the above equation (29), a relational expression between the first and second intermediate pixel values b ₀₀ and b ₂₀ is expressed using a difference value δi ₀ . As shown in FIG. 29 and the like, the first and second intermediate pixel values b ₀₀ and b ₂₀ are estimated using the relational expression. Using the estimated first intermediate pixel value b ₀₀ , pixel values (v ₀₀ , v ₁₀ , v ₁₁ , v ₀₁ ) of each pixel included in the addition unit are obtained.

  In this way, it is possible to simplify the estimation process of the high-resolution image by once estimating the intermediate pixel value from the addition pixel value subjected to the superposition shift and obtaining the estimation pixel value from the intermediate pixel value subjected to the superposition shift. . For example, complicated processing such as iterative calculation of a two-dimensional filter (Patent Document 1) and searching for a part suitable for setting an initial value (Patent Document 2) becomes unnecessary.

Here, superimposing means having an area where the addition unit and the addition unit overlap. For example, as shown in FIG. 27A, the addition unit a ₀₀ and the addition unit a ₁₀ are two estimated pixels v _10. is to share the _{v 11.}

  Further, the position of the addition unit is the position and coordinates of the addition unit in the captured image, or the position and coordinates of the addition unit on the estimated pixel value data (image data) in the estimation process. The next position is a position shifted from the original position by a pixel, and is a position where the position and coordinates do not coincide with the original position.

In this embodiment, continuous intermediate pixel values including the first and second intermediate pixel values (for example, b ₀₀ , b ₂₀ ) are defined as intermediate pixel value patterns ({b ₀₀ , b ₁₀ , b ₂₀ }). As shown in the above equation (29), the estimation calculation unit 230 represents the relational expression between the intermediate pixel values included in the intermediate pixel value pattern using the added pixel values a ₀₀ and a ₁₀ . As shown in FIG. 29, the similarity is evaluated by comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values with the added pixel value. Based on the similarity evaluation result, the intermediate pixel values b ₀₀ , b ₁₀ , and b ₂₀ included in the intermediate pixel value pattern are determined so that the similarity is the highest.

  In this way, the intermediate pixel value can be estimated based on a plurality of added pixel values acquired by pixel shifting while the addition unit is superimposed.

  Here, the intermediate pixel value pattern is a data string (a set of data) of intermediate pixel values in a range used for estimation processing. The addition pixel value pattern is a data string of addition pixel values in a range used for the estimation process.

Further, in the present embodiment, as shown in the above equation (31), the estimation calculation unit 230 has an intermediate pixel value pattern ({b ₀₀ , b ₁₀ , b ₂₀ }) represented by a relational expression between intermediate pixel values. And an evaluation function Ej representing an error between the pixel value (a ₀₀ and a ₁₀ ). The intermediate pixel values b ₀₀ , b ₁₀ and b ₂₀ included in the intermediate pixel value pattern are determined so that the value of the evaluation function Ej is minimized.

  In this way, the value of the intermediate pixel value can be estimated by expressing the error by the evaluation function and obtaining the intermediate pixel value corresponding to the minimum value of the evaluation function. For example, as described above, the initial value of the intermediate pixel estimation can be set with a simple process by obtaining the unknown using the least square method. For example, searching for an image portion suitable for initial value setting (Patent Document 2) is unnecessary.

In the present embodiment, as shown in the above equation (25), each pixel value (for example, v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ ) of the addition unit is weighted and added to the added pixel value (a ₀₀ ). get. Based on the obtained addition pixel value (a ₀₀ , a ₁₀ ) of the addition unit, the pixel value (v ₀₀ , v ₁₀ , v ₀₁ , v ₁₁ ) of each pixel of the addition unit is estimated.

  If it does in this way, each pixel value of an addition unit will carry out weighted addition, an addition image will be acquired, and the pixel value of a high-resolution image can be estimated from the acquired addition image. Thereby, in the estimation process, the reproducibility of the high-frequency component of the subject can be improved. That is, when the pixel values of the addition unit are simply added, a rectangular window function is convoluted for imaging. On the other hand, when the pixel value of the addition unit is weighted and added, a window function containing more high frequency components than the rectangle is convoluted for imaging. Therefore, it is possible to acquire an added image that includes more high-frequency components of the subject, and to improve the reproducibility of the high-frequency components in the estimated image.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configurations and operations of the imaging unit, the image processing unit, the imaging device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

10 imaging unit, 20 image processing unit, 100 subject, 110 lens,
115, 116 subtraction unit, 120 image sensor, 125 low-resolution image generation unit,
130, 135 addition image generation unit, 140 compression processing unit,
141 difference image generation unit, 142 average image generation unit,
143 Entropy encoding unit, 144 data compression unit, 145 average value calculation unit,
146 subtraction unit, 147 lossless compression encoding unit, 148 data compression unit,
150 compressed data recording unit, 151 differential data recording unit,
152 average image data recording unit, 155 data recording unit,
160 low-resolution video generation unit, 165 pixel data reconstruction unit,
170 video data compression unit, 175 addition image generation unit,
180 video data recording unit, 185 data compression unit, 190 monitor display unit,
200 compressed data decompression unit, 205 decompression processing unit, 210 addition image reproduction unit,
215 low-resolution image estimation calculation unit, 220 addition image noise reduction processing unit,
230 estimation calculation unit, 240 high definition still image generation unit, 250 high definition video generation unit,
260 high-quality standard video generation unit, 270 video compression data decompression unit,
280 standard video generation unit, 290 image output unit, 295 image selection unit,
301 to 304 addition image generation unit, 311 to 314 difference image generation unit,
321 to 324 average image generation unit, 331 to 334 entropy encoding unit,
340 Bayer image generation unit, 350 video data compression unit,
A ₁ to A ₄ adds the _image, D 1 to D ₄ differential image, M average _image, T 0 through T ₂ period,
V _{3k + 1} reference captured image, V _3k , V _{3k + 2} differential captured image,
a ⁽¹⁾ ₀₀ , a ⁽²⁾ ₁₀ , a ⁽³⁾ ₁₁ , a ⁽⁴⁾ ₀₁ addition pixel value,
a pixel value of a ^D1 _{ij to} a ^D4 _ij difference image, pixel value of a ^M _ij average image,
a ^h _ij , a ^v _ij , a ′ _ij moving average value, b ₀₀ -b ₂₀ intermediate pixel value,
d1, d2 range, fx captured image, m _AVR average value, r weighting coefficient,
u _ij reconstruction data, v ₀₀ -v ₂₂ estimated pixel value, α ^M _ij average value,
δi ₀ difference value, addition value of ξ _ij reconstruction data

Claims (11)

An image acquisition unit for acquiring a captured image;
An addition unit, which is a unit for obtaining an addition pixel value, is set for each of a plurality of pixels of the captured image, and the addition pixel value is obtained by weighted addition of pixel values included in the addition unit, and the obtained addition pixel value An addition image generation unit for acquiring an addition image by;
A compression processing unit for compressing the added image;
A decompression processing unit for decompressing the compressed added image;
An estimation calculator that estimates a pixel value of the captured image based on the expanded added image;
An image output unit that outputs a high-resolution image based on the pixel value estimated by the estimation calculation unit;
Including
The addition image generation unit
The first to nth added images (n is a natural number greater than or equal to 2) are obtained as the added images by sequentially shifting the addition units.
The compression processing unit
An average of the first to n-th added images is obtained as an average image, and a difference between the average image and the m-th added image (m is a natural number equal to or less than n) of the first to n-th added images. , As the m-th difference image, compress the obtained average image and the m-th difference image,
The decompression processing unit
An imaging apparatus, wherein the compressed average image and the m-th difference image are expanded to obtain the first to n-th added images. In claim 1,
The addition image generation unit
The addition unit is sequentially shifted one pixel at a time horizontally or vertically to set the first to n-th positions, and the first to n-th added images are obtained at the first to n-th positions, respectively.
The imaging apparatus, wherein the addition unit of the m-th position and the (m + 1) -th position among the first to n-th positions includes a common pixel. In claim 2,
The compression processing unit
An average image generation unit for obtaining an average value of the addition pixel values of the first to n-th positions as a pixel value of the average image;
A difference image generation unit for obtaining a difference value between the pixel value of the average image and the addition pixel value of the m-th position as a pixel value of the m-th difference image;
An imaging device comprising: In claim 2 or 3,
Including a low-resolution video generation unit for generating a low-resolution video,
When the captured image is an image of an RGB Bayer array and the addition unit is four adjacent pixels,
The addition image generation unit
In the first to fourth positions (n = 4), the first to fourth added images in which RGB are mixed by the weighted addition are acquired,
The low-resolution video generation unit
An image pickup apparatus that obtains RGB pixel values of the low-resolution moving image by estimation based on first to third addition images of the first to fourth addition images. In any one of Claims 1 thru | or 4,
The compression processing unit
An imaging apparatus, wherein the m-th difference image is reversibly compressed by entropy coding. In any one of Claims 1 thru | or 5,
When the captured image is an RGB Bayer array image,
The estimation calculation unit includes:
Obtain the pixel value of the RGB Bayer array of the captured image by estimation,
The image output unit includes:
A high-resolution moving image is output based on an image obtained by demosaicing the estimated pixel values of the RGB Bayer array. In any one of Claims 1 thru | or 6.
An image pickup apparatus comprising: a noise reduction processing unit that performs adaptive noise reduction processing for adaptively adjusting a moving average range according to a change width of a pixel value for the m-th difference image. In any one of Claims 1 thru | or 7,
The image acquisition unit
Acquiring a captured image of the first frame as a reference captured image, obtaining a difference between the captured image of the second frame before or after the first frame and the reference captured image as a differential captured image;
The addition image generation unit
An imaging apparatus, wherein the first to nth added images based on the reference captured image and the first to nth added images based on the differential captured image are acquired. In any one of Claims 1 thru | or 8.
When the addition unit set in the first position and the addition unit set in the second position next to the first position overlap,
The estimation calculation unit includes:
A difference value between the added pixel value of the first position and the added pixel value of the second position;
A first intermediate pixel value that is an addition pixel value of the first area obtained by removing the overlap area from the addition unit of the first position, and a second that is obtained by removing the overlap area from the addition unit of the second position. A relational expression with the second intermediate pixel value that is the added pixel value of the region is expressed using the difference value,
The first and second intermediate pixel values are estimated using the relational expression, and the pixel value of each pixel included in the addition unit is obtained using the estimated first intermediate pixel value. Imaging device. In claim 9,
The estimation calculation unit includes:
When successive intermediate pixel values including the first and second intermediate pixel values are used as an intermediate pixel value pattern, a relational expression between intermediate pixel values included in the intermediate pixel value pattern is obtained using the added pixel value. Represent,
Comparing the intermediate pixel value pattern represented by the relational expression between the intermediate pixel values and the added pixel value to evaluate similarity;
An image pickup apparatus, comprising: determining an intermediate pixel value included in the intermediate pixel value pattern based on the similarity evaluation result so that the similarity becomes the highest. Acquire the captured image,
An addition unit, which is a unit for obtaining an addition pixel value, is set for each of a plurality of pixels of the captured image, and the addition pixel value is obtained by weighted addition of pixel values included in the addition unit, and the obtained addition pixel value Acquire the addition image by
Compress the added image;
Decompressing the compressed added image,
Estimating the pixel value of the captured image based on the expanded added image,
When outputting a high-resolution image based on the estimated pixel value,
The first to nth added images (n is a natural number greater than or equal to 2) are obtained as the added images by sequentially shifting the addition units.
An average of the first to nth addition images is obtained as an average image, and a difference between the average image and the mth addition image (i is a natural number equal to or less than n) of the first to nth addition images. , As the m-th difference image, compress the obtained average image and the m-th difference image,
A method of generating an image, comprising: decompressing the compressed average image and the m-th difference image to obtain the first to n-th added images.

Images