JP2015039049A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology to suppress redundancy in use of a plural frame image to maintain high resolution.SOLUTION: An image processing apparatus includes a super-resolution conversion unit that converts an image to have high resolution by using a plurality of image frames, a frame memory that stores the image frames to be input to the super-resolution conversion unit, and an identical frame detection unit that detects if identical frames are included in the image frames to be input. When the identical frame detection unit detects the identical frames, the image processing apparatus switches a control method of the frame memory so that the identical frames are not stored in the frame memory.

Description

  The present invention relates to an image processing apparatus, and more particularly to an image processing technique for performing high resolution processing.

  With the widespread use of high-resolution televisions and displays, the resolution of video signals is increasing. In particular, super-resolution that realizes high resolution while maintaining the sharpness of the video signal by restoring the high-resolution video signal by estimating the original pixel value from the low-resolution video signal and increasing the number of pixels An image processing technique called conversion (super resolution conversion processing) has appeared (for example, see Patent Document 1).

  Also, super-resolution conversion processing using mapping between a plurality of frame images disclosed in Patent Document 2 can be used. However, there has been a problem in that the use of a plurality of frames may hinder high resolution due to the redundancy of handling frames having the same content in an overlapping manner.

JP 2007-336239 A JP 2000-188680 A

  An object of the present invention is to provide a technique for suppressing redundancy in use of a plurality of frame images and maintaining high resolution.

  In order to solve the above problems, an image processing apparatus according to the present invention stores a super-resolution conversion unit that increases the resolution of an image using a plurality of image frames, and an image frame that is input to the super-resolution conversion unit. And the same frame detection unit that detects whether the same frame is included in the input image frame, and the same frame detection method is used when the same frame is detected by the same frame detection unit. It is characterized by switching so that the frame is not saved.

  According to the present invention, it is possible to obtain an image processing apparatus and an image processing method in which redundancy in use of a plurality of frame images is suppressed and high resolution is maintained.

The schematic diagram which shows the flame | frame structure of a moving image. Explanatory drawing which shows the high resolution process using the some frame information of one Embodiment of this invention. Explanatory drawing which shows the interpolation pixel production | generation of the embodiment. The block block diagram of the broadcast receiving apparatus concerning the embodiment. The block diagram of the high resolution image processing apparatus of the embodiment. Explanatory drawing which shows the frame after IP conversion of the embodiment. Explanatory drawing which shows the flame | frame input into the super-resolution conversion part of the embodiment. The block block diagram which shows the IP conversion part with the 2-3 pulldown detection function of the embodiment. Explanatory drawing which shows the IP conversion image | video at the time of the pull-down detection of the embodiment. The flowchart which shows the process sequence of the embodiment. Explanatory drawing which shows the flame | frame input into the super-resolution conversion part of the embodiment. The block diagram which shows another embodiment of this invention. Explanatory drawing which shows the flame | frame input into a super-resolution conversion part at the time of suspension of embodiment. The block diagram which shows the conventional video processing part. Explanatory drawing which shows the frame input into the conventional super-resolution conversion part at the time of cinema mode IP conversion. Explanatory drawing which shows the field comparison pattern of embodiment.

Embodiments of the present invention will be described below.
(Embodiment 1)
A first embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a schematic diagram showing a frame configuration of a moving image. With respect to the video of the car at the time Ta in the sequence of continuous frames of the moving image, the car progresses such that Tb after 7 frames and Tc after 7 frames.

  FIG. 2 is an explanatory diagram illustrating high resolution processing using a plurality of frame information according to the embodiment. FIG. 2 shows a mechanism for generating an image having a higher resolution than the input image using a plurality of pieces of frame information in the moving image as shown in FIG.

  Now, assuming that pixels are arranged at the intersections of the solid lines shaded in FIG. 2, an interpolation pixel located at the intersection of the dotted lines of the Nth frame is generated and converted into an image having twice the vertical and horizontal pixel counts. Assume a case. When estimating the interpolated pixels, using only the pixels in the Nth frame (indicated by black circles) increases the resolution to restore the original image of the subject such as so-called super-resolution due to lack of information. I can't do it. Therefore, by bringing pixel information indicated by triangles and diamonds from frames before and after the Nth frame, it is possible to estimate an interpolation pixel using more information as shown in FIG. And higher resolution is possible. In FIG. 2, information for each frame before and after is used, but the accuracy of interpolation pixel estimation can be increased by using more frame information.

  FIG. 3 is an explanatory diagram illustrating interpolation pixel generation according to the embodiment. FIG. 3 corresponds to the second region from the left and the second region from the top surrounded by the solid line in FIG. That is, in order to generate an interpolated pixel indicated by a white circle, pixel information indicated by triangles and diamonds from the preceding and following frames is used in addition to the pixel information indicated by black circles in the own frame.

  FIG. 4 shows a block diagram when the image processing apparatus according to the embodiment is applied to a broadcast receiving apparatus. As shown in the figure, the broadcast receiving apparatus includes an antenna 41, a tuner / demodulator 43, an MPEG demultiplexer 44, a video decoder 45, a video processing unit 46 corresponding to the image processing apparatus, and a display panel 47. And an audio decoder 48 and a speaker 49.

  The antenna 41 is an antenna for receiving digital broadcasts such as BS, CS, and terrestrial waves. The RF signal of the digital broadcast received by the antenna 41 is guided to the subsequent tuner / demodulator 43, selected and demodulated, and output to the MPEG demultiplexer 44 as a digital video signal and audio signal.

  The MPEG demultiplexer 44 separates the video signal and the audio signal input from the tuner / demodulator 43 into a video signal and an audio signal, and the video decoder 45 performs signal processing (to be described later) on the video signal. The decoder 45 outputs to the video processing unit 46. Here, as the signal processing performed by the video decoder 45, decoding processing of an input video signal compressed by a compression method such as MPEG2 or resolution of an input video signal is set to a predetermined resolution (for example, display resolution). For example, a scaling process for conversion to 1280 × 720, etc.). On the other hand, the audio decoder 48 performs predetermined signal processing on the audio signal from the MPEG demultiplexer 44 and then outputs it to the speaker 49.

  FIG. 5 is a configuration diagram of the high-resolution image processing apparatus according to the embodiment. Assume that the video processing unit 46 shown in FIG. 4 includes an IP conversion unit 51 and a super-resolution conversion unit 52 as shown in FIG. 5 as an image processing apparatus.

  For the super-resolution conversion processing, it is possible to use a publicly known / public technique disclosed in Japanese Unexamined Patent Application Publication No. 2007-310837 and Japanese Unexamined Patent Application Publication No. 2008-98803. As a technique of super-resolution conversion processing of the present embodiment, for example, a technique of restoring an image having a frequency component higher than the Nyquist frequency determined by the sampling period of the input image is used.

  For example, when using the super-resolution conversion process disclosed in Japanese Patent Application Laid-Open No. 2007-310837, a target image area including a target pixel in a frame for each of a plurality of low-resolution video signals (low-resolution frames). A plurality of corresponding points corresponding to a plurality of target image areas closest to the change pattern of the pixel value in the middle are selected from the reference frame, and the luminance sample values at the corresponding points of the target pixel corresponding to the corresponding points are selected. A high-resolution frame corresponding to the reference frame that is set to the pixel value and has a larger number of pixels than the reference frame based on the size of the plurality of sample values and the arrangement of the corresponding points. By calculating the pixel value, the original pixel value is estimated from the low resolution video signal and the number of pixels is increased, thereby restoring the high resolution video signal.

  In addition, when using the super-resolution conversion process using self-congruent position search in the same frame image disclosed in Japanese Patent Application Laid-Open No. 2008-98803, the error of each pixel in the search area of the low-resolution frame is compared. The minimum first pixel position is calculated, and based on the first pixel position and the first error, the second pixel position around the first pixel, and the second error, The position where the error is minimized is calculated with decimal precision. Then, a decimal precision vector with this position as the end point and the target pixel as the start point is calculated, and an extrapolation vector of the decimal precision vector with the pixel on the screen not included in the search region as the end point is calculated using the decimal precision vector. calculate. Then, based on the decimal precision vector, the extrapolation vector, and the pixel value acquired from the image data, the pixel value of the high-resolution image having a larger number of pixels than the number of pixels included in the image data is calculated. By performing such processing, the super-resolution conversion unit 52 estimates the original pixel value from the low-resolution video signal and increases the number of pixels, thereby restoring the high-resolution video signal.

  However, the method of super-resolution conversion processing in the super-resolution conversion unit 52 is not limited to the above, and by estimating the original pixel value from the low-resolution video signal and increasing the number of pixels, Any method can be applied as long as it restores the video signal.

  In FIG. 5, the interlaced video signal input to the video processing unit 46 is IP-converted by the IP conversion unit 51, is then super-resolution converted by the super-resolution conversion unit 52, and is output as a high-resolution progressive video.

  Here, FIG. 6 is an explanatory diagram illustrating a plurality of frames after IP conversion according to the embodiment, and FIG. 7 is an explanatory diagram illustrating a plurality of frames input to the super-resolution conversion unit 52 according to the embodiment.

  If the 60p frames output from the IP converter 51 are arranged as shown in FIG. 6, the frames input to the super-resolution converter 52 are as shown in FIG. F4 is a 4-frame delay signal (C signal before 4 frame periods), F3 is a 3-frame delay signal, F2 is a 2-frame delay signal, F1 is a 1-frame delay signal, and C is a frame currently output from the IP conversion unit 51 Signal. The super-resolution conversion unit 52 uses these five frames of data to increase the resolution of the F2 frame.

  That is, the high resolution image processing apparatus of FIG. 5 further includes a frame memory 53 and a memory controller 54. The frame memory 53 stores a signal obtained by delaying the input signal by several frame periods as described above, and supplies the signal to the super-resolution conversion unit 52 through the memory controller 54.

  FIG. 8 is a block diagram for explaining the IP conversion unit 51 having the 2-3 pull-down detection function in more detail. FIG. 9 is an explanatory diagram showing an IP converted video when pull-down is detected.

  In normal 60i broadcast content, the frames input to the super-resolution conversion unit 52 are all different in this way. For example, in 60i broadcast content that has been subjected to 2-3 pulldown processing, the exact same frame is super-resolution. The image may be input to the image conversion unit 52. As shown in FIG. 8, the IP conversion unit 51 in FIG. 5 includes a motion adaptive IP conversion unit 81, a cinema mode IP conversion unit 82, a 2-3 pull-down detection unit 83, and the like. If a 60i signal obtained by pulling down 2-3p of 24p cinema content is input to this IP conversion unit, the 2-3 pulldown detection unit 83 detects that it is a pulldown video and generates the same frame information SF. , 60p video as shown in FIG. 9 is output. At this time, not the motion adaptive IP conversion output but the cinema mode IP conversion output by the complete still image processing specified by the selector 84 controlled by the 2-3 pull-down detection unit 83, and clear stillness equivalent to the original 24p progressive video. Image quality is improved by generating image images at 60p. In the progressive video processed in this way, two or three identical frames continue.

  FIG. 16 is an explanatory diagram showing a field comparison pattern related to the 2-3 pull-down detection unit 83. Although there are various methods for detecting 2-3 pull-down, the simple method is to use the difference between fields.

  For example, the 2-3 pull-down detection unit 83 is configured to receive field data with a 2-field delay from a field memory (not shown). When the current field data and this field data are compared as top fields or bottom fields, as shown by a dotted line in FIG. 16, a repetitive pattern of “moving and moving” appears. If this pattern is repeated many times, the 2-3 pulldown detection unit 83 determines that the input is a 2-3 pulldown signal and generates the same frame information SF. In order to detect pull-down more accurately, information such as 1-field delay field data may be used.

  FIG. 14 is a block diagram showing a conventional video processing unit. FIG. 15 shows a frame input to the conventional super-resolution conversion unit during cinema mode IP conversion. As can be seen from the figure, the same frame is input several times. In the multi-frame super-resolution conversion, the resolution is increased by using information of different frames. Therefore, even if there are a plurality of the same frames, the information does not increase. Accordingly, the frame indicated by hatching in FIG. 15 becomes useless information.

  Therefore, in this embodiment, in order to prevent only such a useless frame from accumulating in the frame memory, in the case of the same frame, the frame can be discarded. The configuration diagram of this embodiment is shown in FIG. 5 as described above. In this embodiment, when cinema mode IP conversion is executed, the same frame information SF is input to the memory controller 54 as binary information indicating whether or not the frame is the same as the immediately preceding frame. .

  FIG. 10 is a flowchart showing the execution procedure of the processing of the video processing unit 46 of this embodiment. As shown in FIG. 10A, as a whole, for example, IP conversion processing is first performed (step S10), and then super-resolution conversion processing is performed (step S20). Finally, the frame memory is updated (step S30), and the process returns to step S10 to repeat these processes.

  In the frame memory update routine in step S30, as shown in FIG. 10B, first, it is determined whether the current 1-frame delay signal F1 and the 2-frame delay signal F2 are the same (step S31), and it is determined that they are the same. If so, F2 is discarded (step S32), and the process proceeds to step S34. If it is determined in step S31 that F1 and F2 are not the same, F3 is set to the next F4 and F2 is set to the next F3 (step S33), F1 is set to the next F2 and F0 is set to the next F1 (step S34).

  By updating the frame memory in this way, the frame input to the super-resolution conversion unit 52 becomes as shown in FIG. In FIG. 11, the number of different frames is increased from that in FIG. 15, and the effect of super-resolution conversion can be improved. In FIG. 11, the F2 frame with high resolution is output from the third line from the top.

(Embodiment 2)
A second embodiment of the present invention will be described with reference to FIGS. 1 to 4 and FIGS. 6 to 13. Description of the parts common to the first embodiment is omitted.
Another embodiment of the present invention is shown in the block diagram of FIG. As shown in FIG. 12, the IP conversion unit 51 may be an IP conversion unit 121 that does not have a determination output as to whether or not the frames are the same. Instead, the frame difference detection unit 122 may determine whether or not the frames are the same. In this case, even if the content is not 2-3 pulled down, for example, when the external input video is paused, the same frame is detected so that only the same frame is not accumulated in the frame memory. At this time, the frame input to the super-resolution conversion unit 52 is as shown in FIG. 13, and the effect of the super-resolution conversion can be obtained even during the pause.

  In the above embodiment, as an outline, in the image processing apparatus, a frame memory is used so that the same frame in the input image does not continue to the frame memory of the super-resolution conversion unit that super-resolutions a moving image using a plurality of frames. The control method was switched.

  As this effect, even when the same frame continues, the effect of super-resolution conversion can be improved. Even when only the same frame is input due to the temporary stop, the effect of the super-resolution conversion can be obtained.

  In addition, this invention is not limited to the said embodiment, In the range which does not deviate from the summary, it can implement in various modifications. For example, in addition to the tuner, the video signal may receive IP broadcasts transmitted via a dedicated IP network, or may receive data (still images or moving images) transmitted via an IP network such as the Internet. It may be in the form obtained. Further, the portion corresponding to the IP conversion unit 121 may be in the preceding stage of the video processing unit 46. In other words, the video processing unit 46 may input a progressive video.

  Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements according to different embodiments may be appropriately combined.

  DESCRIPTION OF SYMBOLS 41 ... Antenna, 43 ... Tuner / demodulator, 44 ... MPEG demultiplexer, 45 ... Video decoder, 46 ... Video processing part, 47 ... Display panel, 48 ... Audio decoder, 49 ... Speaker, 51 ... IP conversion part, 52 ... Super-resolution conversion unit, 53 ... frame memory, 54 ... memory controller, 81 ... motion adaptive IP conversion unit, 82 ... cinema mode IP conversion unit, 83 ... 2-3 pull-down detection unit, 84 ... selector, 121 ... IP conversion unit 122... Frame difference detection unit.

Claims (5)

A frame memory for holding the image frame input from the outside and outputting the image frame delayed by a frame period;
A super-resolution conversion unit that increases the resolution of an image input from the outside using the plurality of delayed image frames in which the frame period units are different from each other;
The same frame detection unit for detecting whether the delayed image frame is the same as the previous image frame;
A memory control unit that controls reading and writing of the image frame to the frame memory;
The image processing apparatus according to claim 1, wherein the memory control unit performs control so that the same frame is not held in the frame memory when the same frame detection unit determines that the same frame is the same.   The image processing apparatus according to claim 1, wherein the same frame detection unit includes a 2-3 pull-down detection circuit.   The image processing apparatus according to claim 1, wherein the same frame detection unit includes a frame difference detection circuit that detects a difference by comparing frames.   The image processing apparatus according to claim 1, further comprising a display panel that displays an output of the super-resolution conversion unit. Increase the resolution of this image using multiple image frames,
Holds the image frame to be input for the high resolution,
Detecting whether the same frame is included in the input image frame,
An image processing method comprising controlling the held image frames so that the same frames are not stored when the same frames are detected.

Images