JP2005057529A - Method and apparatus for image processing, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To maintain image quality after processing while evading arithmetic redundancy such that data are temporarily increased by color interpolation and decreased again by resizing when resizing processing is for reduction in color imaging by a digital camera etc. <P>SOLUTION: Provided are an apparatus, a method, and a program for image processing, for evading the arithmetic redundancy accompanying a conventional way of individually performing color interpolation processing and resizing processing and avoiding low data loss even when resizing is reduction to hold an image of high quality. For the purpose, the color interpolation processing and resizing processing are integrated with image conversion including a filter as shown in Fig.; and up-sampling processing is moved to behind the integrated image conversion filter 502 and down-sampling processing is moved to before it, so as to eliminate the arithmetic redundancy. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

The present invention relates to an image processing apparatus, method, and program for efficiently converting captured image data.

  When a subject is color-captured as a moving image or a still image using a digital camera device such as a digital camera or a camera attached to a mobile phone, and is recorded and displayed, usually the captured data is reduced, that is, the imaging device has Recording or preview display is performed after reducing the data amount to a lower resolution than the data captured at the resolution. As shown in FIG. 12, the number of pixels for display or the number of pixels for moving image recording is usually considerably smaller than about 1 in comparison with the number of pixels that can be imaged by a camera device. In addition, since image data obtained from a color image sensor (pixel) can only obtain the brightness value of the color filter arranged in the pixel, color interpolation processing is required to obtain other color values.

Therefore, in digital color image processing, color interpolation and reduction / resize processing are required during the processing from imaging to recording or display.
FIG. 13 shows a configuration example of a conventional general image processing apparatus. In the imaging means 1301 in the figure, a color filter, for example, a primary color filter having an arrangement of R, G, and B shown in FIG. 14A, or an array of Ye, Cy, Mg, and Gr shown in FIG. Since the complementary color filter covers an image pickup device composed of a CCD (Charge-Coupled Devices), the luminance value of the color arranged in the image pickup device is obtained as the color value of the pixel. This is called raw data or original image data. Here, for convenience of explanation, it is assumed that the color filter is a primary color filter and the color arrangement is a widely used Bayer arrangement shown in FIG. In the Bayer array, the G color is different from the other R colors and B colors, and occupies 2 out of 4 pixels constituting the array unit, and is arranged in a checkered pattern.

  The original image data captured in this way is subjected to color interpolation by the color interpolation means 1303 in the imaging processing portion 1302 of FIG. 13 and the resulting image is resized by the resizing means 1304. The original image data is image data composed of one color of each pixel, and arithmetic processing for estimating and calculating the color value assuming that the value of the other color of the pixel is 0 is a narrowly defined color interpolation process. As shown in FIG. 15, in terms of spatial processing, color interpolation is to obtain image data of the same size as the original image for each color of R, B, and G on a real storage area or a virtual data plane.

  Further, as shown in FIG. 16, in terms of signal processing, color interpolation is performed by sequentially inputting raw data in the horizontal or vertical direction, which is the original image, and specifying the specified color separation by downsampling of ↓ 2 for the data. This is a process of performing thinning for extraction (the data amount is halved) and one zero value insertion (data amount is doubled) between the data by upsampling of ↑ 2. In the example of this figure, the image data in the horizontal direction is thinned out, and instead, 0-value data is inserted as a dummy and an interpolation value is given later. Reference numeral 1701 in FIG. 17A schematically shows this color interpolation processing. Here, Hc is a color interpolation filter. The process of color separation and zero value insertion for the original image data is considered to be a process that should be originally performed by the imaging unit 1301, but is regarded as a part of the color interpolation process here. However, data after specific color separation from the original image data may also be referred to as original image data. Since color separation is not necessary for the original image data in this case, color separation is not included in the color interpolation. Unless otherwise specified, the original image data refers to image data before color separation.

  Next, the resizing means 1304 in FIG. 13 performs resizing (size changing) which is a size change of the color-interpolated image data. A schematic representation of resizing is resizing processing 1702 in FIG. This represents a resizing processing portion when the horizontal and vertical resizing of the image is U / D. ↑ U represents upsampling that increases the data by U times, and ↓ D represents downsampling that thins out to 1 / D. More specifically, ↑ U inserts U-1 0-value data immediately after the data as in the case of color interpolation. Also, ↓ D performs a process of extracting only the first one of the D data and discarding the others.

  Now, paying attention to the data amount, the color interpolation is the process of increasing the data amount of the color twice at ↑ 2 after extracting only the color value from the raw data at ↓ 2, while the reduction resizing is This is processing for reducing the data amount of the color. Thus, reducing the data once increased by the color interpolation process by resizing means that these processes have redundancy.

The prior art for eliminating the redundancy of color interpolation and resizing in the prior art is disclosed in Patent Document 1. In this prior art, resizing is performed first while maintaining the pattern of the pixel Bayer array, and then color interpolation is performed. Therefore, even in the case of reduced resizing, there is no redundancy as described above, and the amount of computation is greatly reduced. it can.
JP 2002-84547 A

However, in Patent Document 1, since the resizing process in the reduction is performed before the color interpolation process, a loss of raw data due to the reduction of the original image data occurs, and the restoration quality of the image is lowered by the color interpolation process performed thereafter. Will come. This reduction in restoration quality due to the loss of raw data did not occur in the prior art in which color interpolation is performed first.
In view of the above-described problems, the present invention eliminates the operational redundancy associated with performing color interpolation processing and resizing processing as separate processing as in the prior art, and does not deteriorate image quality due to loss of raw data. , Provide methods and programs.

To solve the above problem,
(1) An image processing apparatus according to the present invention includes color interpolation that interpolates color values other than colors arranged in an image sensor with respect to original image data captured by a plurality of color image sensors including color filters, and color An image conversion unit that collectively performs resizing for changing the image size with respect to the image data interpolated by the interpolation is provided.

  (2) The color interpolation is data conversion corresponding to color separation for the original image data, first upsampling for data expansion for color interpolation, and color interpolation filter processing for calculating an interpolation value. The resizing is a data conversion corresponding to a second upsampling that performs data expansion on the image data after the color interpolation process, a resizing filter process for calculating image size change data, and a downsampling that performs data reduction. The image conversion means includes an image conversion filter that collectively processes the color interpolation filter processing and the resizing filter processing, one or more first upsamplers that perform the first upsampling, and the second One or more second upsamplers that perform upsampling and one or more downsamplers that perform the downsampling Comprising a Npura, and the first up-sampler and the second up-sampler disposed downstream of said image transform filter, the down-sampler is characterized by placing in front of the image transform filter.

If there is no color separation processing unit in this configuration, the processing target is original image data after color separation, and if the color separation processing unit is in the configuration, the processing target is original image data before color separation.
(3) Further, the color of the color filter is a Bayer array, and the image conversion means regards the G colors arranged in two of the four pixels in the array unit of the Bayer array as different colors G1 and G2. The image conversion is performed on the original image data of four colors for each color, and the image processing apparatus further converts the low-frequency G color and the low-frequency G color based on the G1 color and G2 color image conversion data obtained by the image conversion. Band division means for generating high-frequency G color image data, and high-frequency compensation for high-frequency compensation of the R, low-G color, and B-color image conversion data based on the high-frequency G color It is good also as providing an area compensation means.

  (4) Further, the color of the color filter is a Bayer array, and the image converting means processes the two colors R and B by the image conversion, and the G color is 2 of 4 elements in the Bayer array. Considering the G color arranged in the pixel as G1 and G2 of different colors, convert the original image data of four colors in either horizontal or vertical direction, and merge the G1 and G2 data that have been converted The processed data may be processed only by resizing in the other direction.

  (5) In order to solve the above problem, an image conversion method according to the present invention is a method of color interpolation and resizing processing performed on original image data captured by an image sensor provided with a color filter. Interpolation and data conversion of the resizing are collectively performed as image conversion, and the image conversion is an image in which a color interpolation filter that is a filter for the color interpolation and a resizing filter that is a filter for the resizing are integrated A transform filter processing step; a first upsampling step for the color interpolation; and a second upsampling step and a downsampling step for the resizing, the first upsampling step and the second upsampling step. A sampling step is performed after the processing step of the image conversion filter, and the downsampling step is performed on the image conversion filter. And performing the previous filter processing steps.

  (6) In order to solve the above problem, the image conversion program of the present invention is a program that causes a computer to execute color interpolation and resizing processing performed on original image data captured by an image sensor having a color filter. The color interpolation and the resizing data conversion are collectively performed as image conversion, and the image conversion includes a color interpolation filter that is a filter for the color interpolation, and a resizing filter that is a filter for the resizing. Including a first upsampling process for the color interpolation, a second upsampling process for the resizing, and a downsampling process. Processing and second upsampling processing are performed after the image conversion filter processing, and the downsampling processing is performed on the image. And performing before treatment conversion filter.

The image processing apparatus of the present invention
Color interpolation for interpolating color values other than colors arranged in the image sensor with respect to original image data captured by a plurality of color image sensors having color filters;
Image conversion means for collectively performing resizing to change the image size for the image data interpolated by color interpolation is provided.

With this configuration, the color interpolation and resizing processes have been processed by the respective filter operation circuits in separate hardware modules in the prior art, but with this configuration of the present invention, the two processing modules of color interpolation and resizing can be performed on an image. One conversion module is simplified.
Furthermore, the preconditions for eliminating the redundancy by optimizing the operation of both the color interpolation and resizing processes are established.

  In the image processing apparatus of the present invention, the image conversion means includes one or more image conversion filters that collectively perform color interpolation filter processing and resizing filter processing, and one or more that performs first upsampling for color interpolation. The first upsampler and the second upsampler include a first upsampler, one or more second upsamplers that perform second upsampling for resizing, and one or more downsamplers that perform downsampling for resizing. The sampler is disposed in the subsequent stage of the image conversion filter, and the down sampler is disposed in the preceding stage of the image conversion filter.

This configuration is functionally equivalent to the case of the prior art except for the technology disclosed in Document 1 regarding the color interpolation and resizing processing, and is an optimized configuration with no operational redundancy. Therefore, the image quality is not deteriorated, and the calculation amount can be greatly reduced as compared with the conventional case.
Furthermore, in the image processing apparatus of the present invention, the color filter of the image sensor is a Bayer array, and the G colors arranged in two pixels of the four pixels in the array unit are regarded as different colors G1 and G2, and the four colors. Performs image conversion for each color of the original image data, and generates low-frequency G color and high-frequency G color image data based on the G1 and G2 color image conversion data obtained by the image conversion Band dividing means, and high-frequency compensation means for performing high-frequency compensation of each image conversion data of the R color, the low-frequency G color, and the B color based on the high-frequency G color,
It is good.

According to this configuration, since the processing for increasing the frequency of the image data is further performed after the image conversion, it is possible to cope with a request for a later recording / reproduction level.
In the image processing apparatus of the present invention, the color filter of the image sensor is a Bayer array, and the image conversion means processes the two colors R and B by the image conversion and the Bayer array 4 for the G color. Considering the G colors arranged in two pixels of the elements as G1 and G2 of different colors, image conversion is performed in either the horizontal or vertical direction with respect to the original image data of four colors, and the image converted G1, The data obtained by merging the G2 data may be processed by only resizing in the other direction.

According to this configuration, the generation of image data is limited to a low frequency range. Therefore, high-frequency correction calculation is unnecessary, and calculation of G1 and G2 data generation is about half that in the above case. It can also be used when performing original high-frequency compensation in a later process or preview display with low-frequency data.
The image conversion method of the present invention is a method of color interpolation and resizing processing performed on original image data captured by an image sensor having a color filter, and each color conversion and resizing data conversion is performed on an image. The image conversion is performed collectively as a conversion, and the image conversion is performed by integrating a color interpolation filter that is a filter for the color interpolation and a resize filter that is a filter for the resizing; A first upsampling step for the resizing, a second upsampling step and a downsampling step for the resizing, the first upsampling step and the second upsampling step of the image conversion filter After the processing step, the downsampling step is performed before the processing step of the image conversion filter It is characterized by that.

  The image conversion program of the present invention is a program for causing a computer to perform color interpolation and resizing processing performed on original image data captured by an image sensor having a color filter, and each of the color interpolation and resizing Data conversion is performed collectively as image conversion, and the image conversion is performed by an image conversion filter that integrates a color interpolation filter that is a filter for color interpolation and a resize filter that is a filter for resizing; A first upsampling process for the color interpolation, a second upsampling process and a downsampling process for the resizing, the first upsampling process and the second upsampling process It is performed after the image conversion filter processing, and the downsampling processing is performed before the image conversion filter processing. Run on a computer.

  By executing the program of the present invention, the camera device can be operated under a microprocessor instead of a logic circuit dedicated to image processing. It can also be used for high-speed processing of pre-recorded image data on a general-purpose computer.

(Embodiment 1)
FIG. 1 shows the overall configuration of the image processing apparatus according to the first embodiment. Characteristic parts of the first embodiment are the image conversion means 103 in the figure, a band dividing means 104, an edge detection means 105, and a high frequency compensation means 106 for performing subsequent processing. Next, these will be described for each means.
First, since the image conversion unit 103 is obtained by integrating the color interpolation unit 1303 and the resizing unit 1304 of FIG. 13 of the prior art, the configuration of the image conversion unit is defined by the configuration of the color interpolation unit and the resizing unit. Therefore, not only the result of the integration of the color interpolation means and the resizing means but also the process will be described.

The color interpolation filter Hc and the resizing filter HR that are the basis of the image conversion filter are both linear and separable in the horizontal and vertical directions. These filters can be expressed by transfer functions. Examples of transfer functions are shown in FIGS. 19A and 19B for the case where the number of taps of the color interpolation filter is 7 and the number of taps of the resize filter is 12, respectively.
Unless otherwise noted, all filters will be in the horizontal or vertical direction. Note that the horizontal or vertical direction is convenient, and the arrangement of the pixels of the image data is a matrix, so that either the row or the column may be regarded as horizontal and the other as the vertical direction.

  As preprocessing for creating an image conversion filter, the resize filter HR is divided into U pieces by a known polyphase decomposition method, and the upsampler (↑ U) is moved from the front stage to the rear stage of the resize filter. Here, the former stage means the upstream side in the processing flow, and the latter stage means the downstream side. The movement result is shown in FIG. Reference numeral 402 in the figure denotes a resize filter. Then, the color interpolation filter 401 and the resizing filter 402 are integrated to create an image conversion filter 403. Since the color interpolation filter, the resizing filter, and its division filter are linear, the U division filter (U number of partial filters Hi) of the image conversion filter is given as a product of them as shown in the following equation. This state is shown in FIG.

Hi = Hc · RRi where i = 0,1, ..., U-1
Here, the number of taps, which is the number of terms in the filter operation, will be described. If the number of taps of the resize filter HR before integration is M, the number of taps of each of the U RRIs is M / U. Here, when M / U is not divisible and the quotient is m and the remainder is n, according to the definition of polyphase decomposition, up to n division filters Rri in the order of increasing i are (m + 1) taps, otherwise m taps. Become. Therefore, even if polyphase decomposition is performed, the total number of taps remains the same as before division. If the number of taps in Hc is N, the number of taps in Rri is M / U, so the number of taps in the Hi partial filter is (N-1 + M / U).

  Next, the optimum configuration of the image conversion means in FIG. For that purpose, I would like to see the general behavior of upsamplers and downsamplers. Consider a case where there is an upsampler ↑ n in front of the filter. The upsampler ↑ n generates (n−1) 0-value data for one piece of input data, adds it, and outputs it. In other words, the data is increased n times by the upsampler, but most of them are 0 values. When these data enter the filter, the filter performs a product-sum operation corresponding to the transfer function, but the operation for 0-value data is wasted. Therefore, if there is an upsampler in front of the filter, it can be moved to the subsequent stage of the filter in order to avoid redundancy of operation.

  Next, consider the case where there is a downsampler ↓ n in the subsequent stage of the filter. The downsampler ↓ n adopts and outputs only one of n pieces of input data, and performs a thinning process that discards the rest. That is, most of the data obtained as a result of calculation by the filter is wasted. Therefore, if there is a downsampler after the filter, it can be moved to the previous stage of the filter in order to avoid redundancy of operation.

In the configuration before and after the movement of the up sampler and the down sampler, if the input data is the same, the output data is also the same.
For the above reasons, the color interpolation upsampler ↑ 2 is moved to the rear stage of the image conversion filter 403 and the resized downsampler ↓ D is moved to the front stage in order to optimize the configuration of the image conversion means in FIG. . After polyphase decomposition (two-division filter Hi, j (j = 0,1) can be performed) for each two U-division filters Hi for image conversion, the upsampler ↑ 2 for color interpolation is added to each partial filter. Move to the next stage. Then, 2U Hi, j filters are further polyphase-decomposed into D pieces (D-division filters Hi, j, k (k = 0,1, ..., D-1) can be generated), and then the resized downsampler ↓ Move D to the previous stage. In this way, an image conversion filter comprising 2DU partial filters Hi, j, k for performing horizontal or vertical image conversion is obtained. This is shown at 502 in FIG. The number of taps of this partial filter Hi, j, k is (N-1 + M / U) / 2D on average.

Now, as a result of the movement of the upsampler and downsampler, the calculation performed by the image conversion filter is performed in spite of outputting the same image data as when performing the color interpolation and resizing processing order as before. Only necessary or sufficient processing or calculation is required to generate the output data.
Therefore, as in the prior art, an arithmetically optimized configuration is obtained in which there is no redundancy in “enlargement by color interpolation and then reduction by resizing” or image quality deterioration in “color interpolation after reduction of raw data”. . The configuration in FIG. 5 is the configuration in the image conversion means of the part that performs horizontal or vertical image conversion in the first embodiment in which the resizing is U / D times. Even if the transfer functions of the color interpolation filter and the resizing filter are fixed, the configurations shown in FIG. 5 are different if the resizing parameters U and D are different. Therefore, in practice, the image conversion configuration of FIG. 5 is prepared for each of several pairs of U and D in advance. Note that there are two types of image conversion units 501 in FIG. 5, horizontal and vertical.

  As an example of the configuration of image conversion, the resizing filter has a transfer function of 12 taps (the number of computation terms) and a resizing of 2/3 times, that is, resizing with upsampling ↑ 2 and downsampling ↓ 3, as shown in FIG. Each transfer function of the two-divided filter of the resize filter 402 of FIG. FIG. 7B shows the transfer function of each D-divided filter in the 2-divided filter 503 in FIG. 5 under the same conditions.

  Since the design method of the arithmetic circuit for realizing the filter is the same as that of the prior art, it will be briefly described here. For example, an example of a circuit diagram corresponding to the two-divided filter 503 in FIG. 5 when the transfer function of each partial filter is in FIG. 7B is shown at 604 in FIG. 6A. 601 in FIG. 6 is an arithmetic circuit of the D-division filter H0,0,0 in FIG. 5, 602 is H0,0,1 and 603 is H0,0,2. For example, 601 stores data input one data clock before in register 0 and outputs the sum of the product of it and the coefficient a7 and the product of the latest input data and the coefficient a1. 602 and 603 are calculated and output in the same manner, and the total sum of the output data is the output value of 604. In the figure, DFF (D-flip-flop) is a delay element.

  In FIG. 6 (a), one arithmetic circuit corresponds to one partial filter, but actually it does not necessarily have a one-to-one correspondence. For example, FIG. 6 (b) is equivalent to FIG. 6 (a) by using the arithmetic circuit for each partial filter in FIG. 6 (a) as one arithmetic circuit 611 and sharing it among the three partial filters, and using a selector and a register. Is performing an arithmetic operation. The selectors 613, 614, and 615 control the select signal Sel in the order of calculation cycles of Sel = 2, 1, 0 to switch the filter coefficient and select the data storage destination register. The result of sum of products specified by one Sel is summed by the cumulative register adder 612. When the Sel in the third cycle is 0 and the operation is completed, the output enable signal oe of the cumulative register is turned on and output. , Clear the cumulative register.

  In this way, the calculation for the above three cycles is equivalent to calculating the 6-tap filter once every three cycles, and the number of 6 multipliers required for the original 2-divided filter 6-tap is reduced to 2, Five adders can be reduced to two. The example of the circuit that executes the two-divided filter 503 in FIG. 5 has been described so far. However, paying attention to the fact that the data input of the same two-divided filter 504 is the same as 503, both share the circuit. By putting the variable part data and coefficients in the register, the operation for each partial filter can be performed by replacing the register contents. Further, the circuit can be simplified and saved, but circuit optimization is not the main subject. Therefore, further explanation is omitted.

The image conversion means 103 treats the G colors in the Bayer arrangement as G1 on the upper side and G2 on the lower side, and treats them as different colors. For each of the four colors R, B, G1, and G2, the horizontal direction in the following order Alternatively, R, B, G1, and G2 image conversion data are generated in the vertical direction.
When it is desired to perform image conversion for a certain color, since the color exists every other row or every other column in the original image data before color separation, the image conversion means 103 every other row or column where the color exists. select. The image conversion means 103 first processes the data row of the color in the horizontal direction (row) or vertical direction (column) of the original image with the configuration shown in FIG. 5, but the downsampler 511 indicated by ↓ 2 in FIG. To separate only that color from the data sequence. If the original image data to be processed by the image conversion means has already been color-separated, the color separation down-sampler 511 of FIG. 5 is not necessary.

  A sequence of the color-separated data of the color is sequentially input to the image conversion unit 501 in FIG. 5 at a reference timing (clock), and enters a partial filter for image conversion following several routes. One route is thinned to 1 / D. If it is extracted here, it is immediately applied to the partial filter, and another route is the same data in the case of 1 delay (symbol part of z minus 1 in the figure). It passes at the time of data timing and is thinned out to 1 / D in the same way, but when extracted here, it is input to the relevant partial filter. In these partial filters, the sum of products of the filter coefficient determined by the transfer function and the filter input (or already input) data is calculated.

  In the numerical example described above, it is the same that both the resized divided partial filter in the prior art and the image conversion partial filter in the first embodiment are 2-tap operations. The image conversion unit 501 in FIG. 5 outputs calculation data in units of the two-divided filter Hi, j. That is, the sum of the calculated values of the D-divided filters Hi, j, k for k = 0, 1,..., D−1 is one of the output data of the image conversion unit.

  As described above, by performing image conversion on all the row data of the original image data having the color to be processed, the horizontal image conversion of the original image data is performed, and the vertical image conversion is further performed on the data subjected to the horizontal image conversion. Do. The state of this image conversion is shown in FIG. In FIG. 8, first, by horizontal image conversion, original image data having a horizontal size (number of pixels) of H and a vertical size of V is input, and after color separation of the color, the horizontal conversion configuration of FIG. Perform horizontal image conversion as a whole. For the color data obtained in this way with the horizontal size of H (U / D) and the vertical size of V / 2, color interpolation and resizing only in the vertical direction are performed as shown in FIG. This is performed by vertical image conversion by the configuration of the conversion unit 501.

Note that the same result can be obtained even if the horizontal and vertical conversion orders are reversed, but in this case as well, in the vertical image conversion, if the original image data is before color separation, the vertical configuration including the color separation downsampler 511 of FIG. If the entire image is color-separated, the vertical image conversion unit 501 shown in FIG. 5 is used. In the horizontal image conversion, the horizontal image conversion unit 501 shown in FIG. 5 is used.
Since both color interpolation and resizing filters have LPF (Low Pass Filter) characteristics, the image conversion filter 502 inherits the same characteristics. For this reason, the image conversion data that has been subjected to the horizontal / vertical image conversion processing by the above-described image conversion means also lacks the high-frequency portion, so here it is referred to as low-frequency color data, referred to as RL, BL, G1, G2 data. To do.

In the first embodiment, first, the above-described image conversion is performed on the R and B colors by the image conversion means to obtain low-frequency RL data and BL data. Next, G1 and G2 are also considered as different colors, and image conversion is performed in the same manner as R and B data, respectively. Here, G1 image conversion data is referred to as G1 data, and G2 conversion data is referred to as G2 data.
G1 data which is G1 image conversion data is shown in FIG. 9A, and G2 data is shown in FIG. 9B.
Next, the band dividing means 104 in FIG. 1 generates GL data that is a low band G and GH data that is a high band G based on the G1 and G2 data.
GL data is
GL = (G1 + G2) / 2
Calculate by the following formula.

Here, the above formula defines the GL value of a certain pixel as the average value of the color values of G1 and G2 at the same pixel position. In this way, R, G, and B raw data were all converted to low-frequency data.
Next, high-frequency compensation is performed by the high-frequency compensation means 106 in FIG. 1 as post-processing of the image conversion data. First, GH, which is G high-frequency data, is obtained from the converted data of G1 and G2 using the same pixel position values as in GL, using the following equation.

GH = G1-G2 or G2-G1
Which of the right side is selected is determined by calculating an edge by the edge detecting means 105 in FIG. As shown in FIG. 10A, the horizontal edge ΔH and the vertical edge ΔV are calculated at the position of the pixel A1 for which GH is to be obtained. Here, the meaning of A1, A2, etc. in FIG. 10 means that A is an arbitrary one of the predetermined colors among RL, GL, BL, G1, and G2, and the subscript numbers are relative to the pixels. It indicates the position, 1 means the pixel for edge calculation, 2 means it, 3 means left and so on. If △ H> △ V, G1-G2 is adopted so that horizontal emphasis is applied. Otherwise, G2-G1 of vertical emphasis is adopted.

Next, high frequency compensation is performed by the high frequency compensation means 106 of FIG. If GH is determined for each pixel, as shown in FIG. 10 (b), high frequency compensation is performed by adding a correction value (product of color coefficient and GH) for each color by the following formula, and R Wide color data Rα, Gα, and Bα data that are G and B are obtained.
Wide-area color data = Low-pass color data + Color coefficient / GH
Here, the color coefficient is a gain of GH data defined for each color, and a fixed value may be prepared or may be adjusted according to edge information.

  The above is the part unique to the first embodiment, and the remaining means in the configuration of FIG. 1 are the same as in the prior art. For convenience of explanation, the image conversion is a one-dimensional image conversion in the horizontal or vertical direction. However, the horizontal / vertical image conversion can be obtained by a two-dimensional method because the same result can be obtained by a two-dimensional image conversion equivalent thereto. . In that case, the filter calculation is also an operation for two-dimensional (area) data. Other means shown in FIG. 1 are the same as those of the prior art. The above is the first embodiment of the present invention.

Next, a description will be given of how much the calculation can be reduced by integration with image conversion when the same image data is processed, compared to the conventional technology.
For comparison, an optimized configuration in the prior art is obtained. First, in order to shift ↑ 2 and ↑ U, which are upsamplers of color interpolation processing and resizing processing, to each corresponding filter, polyphase decomposition is performed on two color interpolation filters and U resize filters. The result is shown in FIG. Furthermore, each partial filter of the resize filter is divided into D parts, and the upsampler ↓ D that is the resize process is shifted to before the resize filter. The result of the shift is a configuration that is computationally optimized as the prior art, and this is shown in FIG. In the case of the numerical example of the image conversion described above, the number of taps of each partial filter obtained by dividing the resize filter into D is the same, and the resize transfer function is different from the image conversion filter, such as coefficients, but the D division of the image conversion filter. This is a binary sum-of-products operation similar to a filter.

  Next, the data rate, which is an important factor for the amount of calculation, will be described. Generally, when the downsampler ↓ n is placed before the filter or after the upsampler ↑ n during the polyphase decomposition, the input data rate to the filter becomes 1 / n. A data rate of 1 / n means that the amount of data per unit time or the total number of input data is 1 / n, and that the amount of operations (multiplication and addition) is also proportionally 1 / n. means.

The data rate magnification is set to reference 1 immediately after color separation ↓ 2. The number of operations is indicated as a value per clock (inter-data cycle) of input data at the reference data rate.
The number of filter taps is N taps for the color interpolation filter, M taps for the resizing filter, and U / D times for resizing.
The number of multiplications is determined by the product of the number of divided filters, the number of taps thereof, and the data rate magnification. The number of additions is equal to the value obtained by subtracting 1 from the total number of filter taps multiplied by the data rate magnification.
For conventional general technology,
Number of division filters in color interpolation processing: 2
Number of division filter taps: N / 2
Data rate magnification: 1
Number of multiplications: 2 (N / 2)
Number of additions: N-1
Resize processing: Number of division filters: DU
Number of division filter taps: M / (DU)
Data rate magnification: 2 / D
Number of multiplications: 2 / D (M / (DU)) DU = 2M / D
Number of additions: (2 / D) ((DU) M / (DU) -1) = 2 (M-1) / D
Therefore, the total number of multiplications: N + 2M / D
Number of additions: N-1 + 2 (M-1) / D
On the other hand, in the present invention, the number of division filters: 2DU
Number of division filter taps: (N-1 + M / U) / (2D)
Data rate magnification: 1 / D
Number of multiplications: 2DU (N-1 + M / U) / (2D) (/ D) = (N-1) U / D + M / D
Number of additions: (1 / D) ((2DU) (N-1 + M / U) / (2D) -1) = (N-1) U / D + (M-1) / D
From the corresponding terms, it can be seen that both the multiplication and addition decrease to nearly half.

As an example of numerical values, the following is a comparison between N = 7, M = 12, U = 2, and D = 3.
In the prior art, the number of multiplications: 15
Number of additions: 14
In the present invention, the number of multiplications: 8
Number of additions: 8
Such a single filter not only simplifies the arithmetic circuit, but also facilitates movement of the upsampler and downsampler, and as a result, an image conversion filter can be easily constructed. In addition, since the data rate of the filter input can be greatly reduced, the arithmetic circuit can also be reduced proportionally, thereby reducing the circuit scale and power consumption.

(Embodiment 2)
The configuration of the image processing apparatus according to the second embodiment is shown in FIG. 2, and the details of the image conversion means are shown in FIG. In the second embodiment, the low-frequency R and B data are obtained by performing image conversion by horizontal / vertical image conversion 301 in the same manner as the low-frequency R and B described in the first embodiment, but G1 and G2 are used. There is no high frequency compensation. For G color, G1 and G2 are not treated as different colors. In order to create one plane of G data, first, an edge is calculated by the edge detection unit 302 in FIG. 3 for an arbitrarily selected pixel near the center of the image based on low-frequency R or B conversion data, The following processing is performed depending on the direction.

When the edge is in the horizontal direction, image conversion in the horizontal (row) direction of G1 and G2 is performed, and each row of the image conversion data is merged by interleaving. To merge with this interleaving, the first row of G1 data, then the first row of G2 data, the second row of G1 data, then the second row of G2 data, and the rows of G1 and G2 data in turn from the top alternately It is to select and synthesize.
Next, only the resizing process in the vertical (column) direction is performed on the merged data. This is shown in FIGS. 3 (a) and 3 (b).

When the edge is in the vertical direction, vertical image conversion is performed, and only the resizing process in the horizontal direction is performed on the data obtained by merging the columns of the image conversion data by interleaving.
In the second embodiment, it is not necessary to perform both horizontal and vertical image conversion of G1 and G2 as in the first embodiment, and GL that is low-frequency G data and GH that is high-frequency G data are generated. In addition, since none of the high-frequency compensation by the GH data is performed, the amount of calculation can be further reduced as compared with the first embodiment. The above is a description of the means unique to the second embodiment, and the remaining means in the configuration of FIG. 1 are the same as in the prior art.

(Other variations)
Although the above description has been made based on two embodiments relating to the present invention, the present invention is of course not limited to the above-described embodiments. The following cases are also included in the present invention.
In the present invention, the primary color array or the complementary color array may be used instead of the Bayer array. In these arrays, the image conversion means is the same as in the Bayer array.

The invention may be a method including the steps described in the embodiments.
Further, these methods may be a computer program for realizing the method using a computer system, or may be a digital signal representing the program.
FIG. 11 shows a flow of processing for realizing the image conversion means 103 in FIG. 1 by an image conversion program. FIG. 11A shows the main part of the image conversion program. Image conversion is performed by horizontal (S102) and vertical (S103) image conversion for each of the four colors (S101 in the figure).

The horizontal image conversion routine in FIG. 5B performs horizontal image conversion of a specified color. A data string in which only the color data is thinned out is created for each row including the data of the designated color (S201) (S202), a data conversion routine is called (S203), image conversion of this data string is performed, and converted data is output.
FIG. 6C shows a data conversion routine in which the operation values of the two-divided filter Hi, j divided in order of U division (S302) and two divisions (S303) as shown in FIG. 5 are output as image conversion data (S307). ) Here, since the two-divided filter Hi, j is a set of D-divided filters Hi, j, k (k = U-1, U-2,..., 0), the calculated value is the sum of these D-divided filter calculated values. (S304, S305, S306).

The meaning of the downsample data cycle (S301) in FIG. 6C is that when the color data string is processed, the cycle 0 is started at the beginning of the 0th data, and the 0th data is the 2-part filter of FIG. Calculated values of all D-divided filters of the two-divided filter Hi, j in order from the bottom D-divided filter in FIG. 5 at the timing input to the D-divided filter Hi, j, U-1 at the bottom of And outputs it as the operation value of the above-described two-divided filter Hi, j. Next, advance the piece and perform the same processing in cycle 1 (data string Dth), cycle 2 (2Dth in the same), and so on.
The same processing is repeated until the horizontal image conversion data is sequentially output. The processing of the vertical image conversion routine is not shown because it is the same as the processing of horizontal image conversion except that the input is data of all columns of the horizontal image conversion data. The above is the embodiment according to the program of the present invention.

Further, the present invention may be the program or the digital signal transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, or the like.
Further, the present invention is a computer system including a CPU and a memory, wherein the memory stores the program, and the CPU realizes the method by operating according to the program stored in the memory. You may do that.

  The program may be recorded on the recording medium and transferred, or transferred via the network or the like, and executed in another independent computer system.

  The present invention can be used for color digital camera devices, and can also be used for processing original image data previously captured by a digital camera as a program.

It is a block diagram which shows the structure of Embodiment 1 of this invention. It is a block diagram which shows the structure of Embodiment 2 of this invention. It is a detailed block diagram of the image conversion means of Embodiment 2 of this invention. It is a figure which shows the mode of integration of the image conversion filter of this invention from the color interpolation filter and resizing filter of a prior art. It is a block diagram of the image conversion means of this invention. It is an example of the circuit which implement | achieves the image conversion filter of this invention. It is an example of the transfer function of the image conversion filter of this invention. It is a figure which shows the mode of the data conversion by the horizontal / vertical image conversion filter of this invention.

It is a figure which shows each of G1, G2 data, G data of a low region and a high region. It is a figure which shows the mode of an edge calculation and a high region compensation process. It is a flowchart of the process of the program of this invention. It is a figure which shows each example of image size at the time of camera imaging, at the time of recording, and a display. It is a block diagram which shows the structure of the conventional general image processing. It is a figure which shows a color filter arrangement | sequence. It is a two-dimensional processing diagram of color interpolation in a primary color filter. It is a figure which shows the flow of color interpolation and resizing in signal processing. It is a figure which shows the structure of the color interpolation and resizing process in a prior art. FIG. 10 is a diagram showing an optimized configuration of each of color interpolation and resizing in the prior art. It is an example of the transfer function of the color interpolation and resizing filter in the prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Imaging means 102 Imaging processing part 103 Image conversion means 104 Band division means 105 Edge detection means 106 High frequency compensation means 201 Image processing part 202 Image conversion means 301 Horizontal / vertical image conversion 302 Edge detection part 401 Color interpolation filter 402 Resize filter 403 Image conversion filter 501 Image conversion unit 502 Image conversion filter 503 Two-part filter 504 Two-part filter 511 Color separation downsampler 601 D-partition filter H0,0,0 implementation circuit example 602 D-partition filter H0,0,1 implementation circuit example 603 D-division filter H0,0,2 implementation circuit example 611 2-division filter H0,0 implementation circuit example 612 2-division filter H0,0 calculation result 613 First term coefficient selector 614 Data register selector 615 Second term coefficient selector 1301 Imaging means 1302 Imaging processing unit 1303 color interpolation means 1304 Size 1701 color interpolation processing unit 1702 resizing section 1801 color interpolation processing unit 1802 resizing section

Claims (6)

Color interpolation for interpolating color values other than colors arranged in the image sensor with respect to original image data captured by a plurality of color image sensors having color filters;
An image processing apparatus comprising: image conversion means for collectively performing resizing for changing the image size of image data interpolated by color interpolation. The color interpolation is color separation for the original image data;
A first upsampling for data expansion for color interpolation;
Data conversion equivalent to color interpolation filter processing for interpolation value calculation,
The resizing is a second upsampling that performs data expansion on the image data after the color interpolation process;
Resize filter processing for calculating image size change data,
Data conversion equivalent to downsampling to reduce data,
The image conversion means includes
An image conversion filter that collectively processes the color interpolation filter processing and the resizing filter processing;
One or more first upsamplers for performing the first upsampling and one or more second upsamplers for performing the second upsampling;
One or more downsamplers for performing the downsampling, and
The first upsampler and the second upsampler are arranged after the image conversion filter,
The image processing apparatus according to claim 1, wherein the down sampler is arranged in front of the image conversion filter. The color of the color filter is a Bayer array,
The image conversion means regards the G color arranged in two of the four pixels in the Bayer arrangement unit as G1 and G2 of different colors, and performs the image conversion for each of the four colors of the original image data,
The image processing apparatus further includes:
Band division means for generating low-frequency G color and high-frequency G color image data based on the G1 color and G2 color image conversion data obtained by the image conversion,
High-frequency compensation means for performing high-frequency compensation of each image conversion data of the R color, the low-frequency G color, and the B color based on the high-frequency G color,
The image processing apparatus according to claim 2, further comprising: The color of the color filter is a Bayer array,
The image conversion means includes
The two colors R and B are processed by the image conversion,
Regarding the G color, the G color arranged in 2 pixels of the four elements of the Bayer array is regarded as different colors G1 and G2, and the image is converted in either the horizontal or vertical direction with respect to the original image data of the four colors. Then, the data obtained by merging the image-converted G1 and G2 data is processed by performing only resizing in the other direction.
The image processing apparatus according to claim 2. A method of color interpolation and resizing processing performed on original image data captured by an image sensor having a color filter,
The color interpolation and the resizing data conversion are collectively performed as an image conversion.
An image conversion filter processing step in which a color interpolation filter that is a filter for color interpolation and a resize filter that is a filter for resizing are integrated;
A first upsampling step for the color interpolation;
A second upsampling step and a downsampling step for the resizing;
Including
Performing the first upsampling step and the second upsampling step after the image conversion filter processing step;
The image conversion method, wherein the downsampling step is performed before the processing step of the image conversion filter. A program for causing a computer to perform color interpolation and resizing processing performed on original image data captured by an image sensor having a color filter,
The color interpolation and the resizing data conversion are collectively performed as image conversion,
The image conversion includes processing of an image conversion filter that integrates a color interpolation filter that is a filter for color interpolation and a resize filter that is a filter for resizing;
A first upsampling process for the color interpolation;
A second upsampling process and a downsampling process for the resizing;
Including
The first upsampling process and the second upsampling process are performed after the image conversion filter process,
An image conversion program, wherein the downsampling process is performed before the image conversion filter process.

Images