JP6579816B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  In order to perform camera shake correction on video shot using an imaging device such as a digital still camera or a digital video camera, the amount of motion between frame images is detected and alignment is performed on a plurality of images. There is a need. As a method for detecting the amount of motion between frame images, there are a method of using information of an external device such as a gyro sensor, a method of estimating the amount of motion from a captured frame image, and the like.

  Various methods for estimating the amount of motion using a frame image have been proposed in the past. A typical example is motion vector detection by template matching. In template matching, first, one of two frame images in a video is set as a standard image, and the other is set as a reference image. Then, a rectangular area having a predetermined size arranged on the standard image is used as a template block, and a correlation with a distribution of pixel values in the template block is obtained at each position of the reference image. At this time, the position where the correlation is highest in the reference image is the destination of the template block, and the direction and amount of movement to the destination when the position of the template block on the reference image is the reference is the motion vector. .

  When parallax occurs in an image captured due to the difference in perspective of a subject, particularly when parallax occurs in a template, the motion vector may not be calculated correctly. Therefore, in Patent Document 1, when parallax occurs in the template, the parallax in the template is reduced by reducing the template size.

  In addition, there is a technique for extracting feature points, placing template blocks at the extracted feature points, and performing template matching with frame images in order to improve the detection rate of motion vectors. In this case, the extracted feature points move greatly between the frame images due to factors such as the background, the direction of the subject's movement, and noise, so that the quality of the vector changes for each frame image. There is. Therefore, in Patent Document 2, feature point tracking is performed on a frame image, the reliability of a feature point is determined according to the number of successes, and a feature point with high reliability is deleted by deleting feature points with low reliability. Is detected.

Japanese Patent Laid-Open No. 10-021403 JP 2006-336235 A

  However, if the template size is reduced according to the parallax in the template as in Patent Document 1, it reacts sensitively to changes in the template, is greatly affected by camera shake and noise, etc., and the motion vector detection result is unsatisfactory. It becomes stable. Also, the feature point tracking disclosed in Patent Document 2 has a high calculation cost, and the feature points move sensitively and become unstable.

  In view of the above problems, the present invention provides an image processing apparatus capable of stably calculating a motion vector by extracting feature points with high reliability using depth information in template matching. With the goal.

  In order to solve the above problems, an image processing apparatus according to the present invention is an image processing apparatus that detects a motion vector between a plurality of images, and inputs the image and depth information corresponding to the image. Means, storage means for storing the image and depth information input by the input means, extraction means for extracting feature points of the first image stored in the storage means, and feature points extracted by the extraction means A similarity determination means for determining the similarity of a second area corresponding to the first area in the second image input by the input means, and the similarity determination First detection means for detecting a motion vector of the feature point between the first image and the second image from the determination result of the means, and the first image stored in the storage means The corresponding first depth information and the second Based on second depth information corresponding to the image, reliability determination means for determining the reliability of the feature point, and based on the determination result of the reliability by the reliability determination means, the first image and the And second detection means for detecting a motion vector of the entire image between the second image and the second image.

  According to the present invention, image processing that can stably calculate a motion vector by performing reliability evaluation of feature points using depth information and extracting feature points with high reliability in template matching. An apparatus can be provided.

It is a block diagram which shows the structure of the whole image processing apparatus. It is a block diagram which shows the structure of a feature point extraction part. It is a block diagram which shows the structure of a template matching part. It is a block diagram which shows the structure of a reliability determination part. It is a flowchart which shows the process of a motion vector detection. It is a flowchart which shows the process of a feature point extraction. It is a flowchart which shows the process of template matching. It is a schematic diagram of template matching. It is a flowchart which shows the process of reliability determination. It is a graph which shows the change of the reliability in reliability determination. It is a block diagram which shows the structure of the whole image processing apparatus which concerns on 2nd Embodiment. It is a block diagram which shows the structure of the motion vector adjustment part which concerns on 2nd Embodiment. It is a flowchart which shows the process of the motion vector detection which concerns on 2nd Embodiment. It is a flowchart which shows the process of the motion vector adjustment which concerns on 2nd Embodiment. It is a schematic diagram of the motion vector adjustment which concerns on 2nd Embodiment. It is a block diagram which shows the structure of the whole image processing apparatus which concerns on 3rd Embodiment. It is a block diagram which shows the structure of the inter-image reliability determination part which concerns on 3rd Embodiment. It is a flowchart which shows the process of the motion vector detection which concerns on 3rd Embodiment. It is a flowchart which shows the process of the reliability determination between images which concerns on 3rd Embodiment.

  The best mode for carrying out the present invention will be described below with reference to the drawings. The present invention can be applied to an imaging apparatus such as a digital single-lens reflex camera, a digital still camera, or a digital video camera.

(First embodiment)
First, the configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. The image processing apparatus 100 includes an image input unit 101, an image memory 102, a feature point extraction unit 103, a feature point memory 104, and a template matching unit 105. In addition, a motion vector detection unit 106, a depth information input unit 107, a depth information memory 108, a feature point reliability determination unit 109, and an entire image motion vector detection unit 110 are provided. In addition, you may provide the control part not shown which controls each component. The image input unit 101 and the depth information input unit 107 function as input means for inputting each information, and the image memory 102, the feature point memory 104, and the depth information memory 108 are storage means for storing each information. Function as.

  A feature point extraction unit (extraction unit) 103 extracts feature points from the image input by the image input unit 101. The image memory 102 temporarily stores and holds an image of one frame or a plurality of frames input by the image input unit 101. In the present embodiment, the image stored and held in the image memory 102 is a standard image (first image), and the image before being stored and held in the image memory 102 is a reference image (second image). However, without being limited thereto, for example, an image before being stored and held in the image memory 102 may be used as a standard image, and an image stored and held in the image memory 102 may be used as a reference image.

  The feature point memory 104 temporarily stores and holds the feature point coordinates of at least one frame extracted by the feature point extraction unit 103. In this embodiment, the feature points stored and held in the feature point memory 104 are set as reference image feature points, and the feature points before being stored and held in the feature point memory 104 are set as reference image feature points. However, conversely, the feature point before being stored in the feature point memory 104 may be used as the reference image feature point, and the feature point stored in the feature point memory 104 may be used as the reference image feature point.

  The template matching unit 105 arranges template blocks at the standard image feature points stored and held in the feature point memory 104, and uses the reference image input by the image input unit 101 and the standard image stored and held in the image memory 102. The correlation value is calculated. The motion vector detection unit (first detection means) 106 detects a motion vector from the correlation value input from the template matching unit 105. The depth information input unit 107 inputs depth information corresponding to an image input by the image processing apparatus 100.

  The depth information memory 108 temporarily stores and holds the depth information of at least one frame input by the depth information input unit 107. In the present embodiment, the depth information stored and held in the depth information memory 108 is set as the depth information (first depth information) of the reference image, and the depth information before being stored and held in the depth information memory 108 is the reference image. Depth information (second depth information) is used. However, conversely, the depth information before being stored in the depth information memory 108 may be used as the reference image depth information, and the depth information stored in the depth information memory 108 may be used as the reference image depth information.

  The feature point reliability determination unit (reliability determination unit) 109 determines the reliability of the feature points used in the template matching unit 105 from the template block, the motion vector, the depth information of the reference image, and the depth information of the reference image. I do. The entire image motion vector detection unit (second detection unit) 110 uses the motion vector detected by the motion vector detection unit 106 and the reliability of the feature point determined by the feature point reliability determination unit 109 to detect the entire image. Detect motion vectors.

  Next, the configuration of the feature point extraction unit 103 will be described with reference to FIG. The feature point extraction unit 103 includes a feature filter unit 1031, a feature evaluation unit 1032, and a feature point determination unit 1033. The feature filter unit 1031 includes, for example, a horizontal differential filter, a vertical differential filter, a smoothing filter, and the like, and performs filter processing on the image input by the image input unit 101. The feature evaluation unit 1032 calculates a corner point such as an intersection of two edges or a curved point having a maximum curvature for each pixel as a feature value with respect to the image filtered by the feature filter unit 1031. To do. The feature point determination unit 1033 determines a feature point from the feature value calculated for each pixel by the feature evaluation unit 1032. Note that another method may be used as long as feature points can be extracted without being limited to the method in the present embodiment.

  Next, the configuration of the template matching unit (similarity determination unit) 105 will be described with reference to FIG. The template matching unit 105 includes a correlation value calculation unit 1051 and a correlation value determination unit 1052. The correlation value calculation unit 1051 sets the template block arrangement and the template matching area on the reference image feature points stored and held in the feature point memory 104. Then, the correlation value is calculated by template matching using the reference image input by the image input unit 101 and the standard image stored and held in the image memory 102 (determining the similarity of images). To do). The correlation value determination unit 1052 calculates the maximum value of the correlation value input from the correlation value calculation unit 1051, and selects the minimum value to be output to the subsequent processing based on the position information.

  Next, the configuration of the feature point reliability determination unit 109 will be described with reference to FIG. The feature point reliability determination unit 109 includes an in-template depth evaluation unit 1091, a motion vector depth evaluation unit 1092, and a reliability determination unit 1093. The in-template depth evaluation unit 1091 calculates the depth information of the template block set by the template matching unit 105 from the depth information of the reference image stored in the depth information memory 108, and determines the change in the depth in the template. I do.

  When there is a change in depth in the template, the motion vector depth evaluation unit 1092 stores the depth information of the start point of the motion vector detected by the motion vector detection unit 106 and the depth of the reference image stored in the depth information memory 108. Calculate from information. Then, the depth information of the end point of the motion vector is calculated from the depth information of the reference image input by the depth information input unit 107, and the depth difference between the start point and the end point of the motion vector is calculated. The reliability determination unit 1093 determines the reliability of the feature points used for the placement of the template block by the template matching unit 105 from the depth difference between the start point and end point of the motion vector calculated by the motion vector depth evaluation unit 1092.

  Next, the motion vector detection process will be described with reference to the flowchart shown in FIG. In this embodiment, motion vector detection is performed between frames that are continuous in the time axis direction, and the base frame and the reference frame are frames that are related to the time axis direction, but they are not necessarily adjacent frames. Good.

  First, in step S <b> 501, a photographed image is input to the image processing apparatus 100 via the image input unit 101 and stored and held in the image memory 102. In the present embodiment, an image stored and held in the image memory 102 is a standard image, and an image before being stored and held in the image memory 102 is a reference image.

  Next, in step S <b> 502, depth information is input to the image processing apparatus via the depth information input unit 107. The input depth information is stored and held in the depth information memory 108. In this embodiment, the depth information stored and held in the depth information memory 108 is the depth information of the reference image, and the depth information before being stored and held in the depth information memory 108 is the depth information of the reference image. The depth information input method is, for example, a TOF (Time of Flight) method in which a depth image is taken by measuring the time required for reading after irradiating infrared rays, or parallax of a plurality of images taken with a multi-lens lens. May be used to input depth information. Any method may be used as long as the depth information associated with the image input in step S501 can be input.

  In step S503, the feature point extraction unit 103 extracts feature points from the image input in step S501. The extracted feature points are stored and held in the feature point memory 104. In the present embodiment, the feature points stored and held in the feature point memory 104 are set as reference image feature points, and the feature points before being stored and held in the feature point memory 104 are set as reference image feature points.

  Here, the feature point extraction processing performed by the feature point extraction unit 103 will be described with reference to the flowchart shown in FIG. In the present embodiment, feature points are extracted by a Harris Corner detector. First, in step S601, the feature filter unit 1031 performs filter processing on the image input in step S501 of FIG. In this embodiment, a first-order differential filter is applied horizontally, vertically, and diagonally to an input image, and a Gaussian filter is applied to the result of applying the first-order differential filter.

式１においてＩｘは、水平微分フィルタを施した結果を示し、Ｉｙは、垂直微分フィルタを施した結果を示しており、ガウシアンフィルタＧを畳み込む。 Next, in step S602, the feature evaluation unit 1032 performs, on the image filtered in step S601, a differential value around the pixel such as an intersection of two edges or a curved point having a maximum curvature for each pixel. A point that is large in multiple directions is calculated as a feature value using a feature evaluation formula. Note that an autocorrelation matrix H is created from the result of applying the horizontal differential filter and the vertical differential filter in step S601. Formula 1 of the autocorrelation matrix H is shown in Formula 1.

In Expression 1, Ix indicates the result of applying the horizontal differential filter, and Iy indicates the result of applying the vertical differential filter, and convolves the Gaussian filter G.

式２において、αは定数であり、実験的に０．０４〜０．１５の値が良いとされている。ｄｅｔは、自己相関行列Ｈの行列式を示し、ｔｒは、自己相関行列Ｈの主対角成分の和を示す。 Next, the characteristic evaluation formula of the Harris detector is shown in Formula 2.

In Expression 2, α is a constant, and a value of 0.04 to 0.15 is experimentally good. det represents the determinant of the autocorrelation matrix H, and tr represents the sum of the main diagonal components of the autocorrelation matrix H.

  Next, in step S603, the feature point determination unit 1033 determines a feature point based on the feature value calculated for each pixel in step S602. In the present embodiment, a predetermined number is extracted as a feature point in order from the highest feature value. Then, the feature point extraction process ends. At this time, grid division processing or the like may be performed in order to prevent uneven distribution of feature points. In the present embodiment, the Harris corner detector is used for feature point extraction. However, the present invention is not limited to this, and other feature point extraction methods such as the Shi and Tomasi method may be used.

  Returning to FIG. 5, in step S <b> 504, the template matching unit 105 arranges the template block on the reference image feature points extracted in step S <b> 503 and stored in the feature point memory 104. Then, a correlation value is calculated by template matching between the reference image input in step S501 and stored and held in the image memory 102 and the reference image input in step S501.

  Here, the template matching process performed by the template matching unit 105 will be described with reference to the flowchart shown in FIG. First, in step S701, the correlation value calculation unit 1051 calculates a correlation value for an arbitrary region between frame images. A method of calculating the correlation value at this time will be described with reference to FIG. FIG. 8A shows a standard image, and FIG. 8B shows a reference image.

  In the present embodiment, the template block 802 is arranged on the reference image feature point 801 extracted in step S502 and stored in the feature point memory 104. A template block 802 is arranged in the reference image shown in FIG. 8A, and a correlation value between the template block 802 and each region (correlation value calculation region 804) of the reference image shown in FIG. 8B is calculated. To do. At this time, calculating the correlation value for the entire region of the reference image requires an enormous amount of computation. In practice, as shown in FIG. 8B, the correlation value on the reference image is calculated. A rectangular area to be set is set as a search range 803. The position and size of the search range 803 are not particularly limited, and may be any area including the correlation value calculation area 804, for example.

式３において、ｆ（ｉ，ｊ）は、テンプレートブロック８０２内の座標（ｉ，ｊ）における輝度値を示し、ｇ（ｉ，ｊ）は、サーチ範囲８０３において相関値算出の対象となる相関値算出領域８０４内の各輝度値を示す。そして、ＳＡＤでは、両ブロック内の各輝度値ｆ（ｉ，ｊ）およびｇ（ｉ，ｊ）について差の絶対値を計算し、その総和を求めることで相関値Ｓ＿ＳＡＤを得ることができる。従って、相関値Ｓ＿ＳＡＤの値が小さいほど両ブロック間の輝度値の差分が小さい、つまりテンプレートブロック８０２と相関値算出領域８０４のブロック内のテクスチャが類似していることを示す。 In the present embodiment, a sum of absolute difference (hereinafter abbreviated as SAD) will be described as an example of a correlation value calculation method. The calculation formula of SAD is shown in Formula 3.

In Equation 3, f (i, j) represents the luminance value at the coordinates (i, j) in the template block 802, and g (i, j) represents the correlation value for which the correlation value is calculated in the search range 803. Each luminance value in the calculation area 804 is shown. In SAD, the absolute value of the difference is calculated for each luminance value f (i, j) and g (i, j) in both blocks, and the correlation value S_SAD can be obtained by obtaining the sum. Therefore, the smaller the correlation value S_SAD, the smaller the difference in luminance value between the two blocks, that is, the texture in the block of the template block 802 and the correlation value calculation area 804 is similar.

  In this embodiment, SAD is used as an example of the correlation value. However, the present invention is not limited to this. For example, other correlation values such as sum of squares of differences (SSD) and normalized cross-correlation (NCC) are used. It may be used. Next, in step S702, the correlation value determination unit 1052 detects the position information of the minimum value from the correlation value calculated in step S701. Then, the template matching process ends.

  Returning to FIG. 5, next, in step S505, the motion vector detection unit 106 detects a motion vector from the position of the minimum value of the correlation value in the template matching result (similarity determination result) in step S504. Specifically, a motion vector indicating which position the template block on the base image has moved in the reference image is detected from the motion vector between images, that is, the position of the minimum correlation value. In this embodiment, the start point of the motion vector is set to the coordinates of the reference image feature point used when the template block is arranged, and the end point of the motion vector is calculated by adding the calculated motion between the images to the start point coordinates of the motion vector. To calculate.

  Next, in step S506, the feature point reliability determination unit 109 determines the reliability of the feature point of the motion vector that is the start point of the motion vector from the result of the processing in steps S502 to S505. That is, using the depth information of the reference image input in step S502, the depth information of the reference image held in the depth information memory 108, the motion vector detected in step S505 as a result of template matching in step S504, and the feature points The reliability of is determined.

  Here, the reliability determination process performed by the feature point reliability determination unit 109 will be described with reference to the flowchart shown in FIG. First, in step S901, the in-template depth evaluation unit 1091 obtains the depth information in the template block from the depth information of the reference image input in step S502 and the template block arranged on the feature point of the reference image in step S504. get. Then, it is determined whether or not the depth information in the template has changed. If the change in the depth information in the template is equal to or greater than a certain value (Yes), the process proceeds to step S902. If the change is less than the certain value (No), the process proceeds to step S903.

  Next, in step S902, the depth information of the start point of the motion vector from the depth information of the reference image input in step S502 and stored and held in the depth information memory 108, and the coordinates of the start point of the motion vector detected in step S505. Is calculated. Then, the depth information of the end point of the motion vector is calculated from the depth information of the reference image input in step S502 and the coordinates of the end point of the motion vector detected in step S505. The calculated motion vector start point depth information is compared with the motion vector end point depth information to calculate the depth difference. Next, in step S903, the reliability of the feature point used when placing the template block in step S504 is determined from the difference between the depth of the start point and end point of the motion vector calculated in step S902.

  Here, the relationship between the depth difference and the reliability is shown in FIG. FIG. 10A shows a relationship diagram when the depth difference and the reliability are changed linearly. FIG. 10B shows that the reliability does not change until the depth difference reaches a predetermined value, and if the predetermined value is far from the predetermined value, the reliability is drastically decreased. When the depth difference decreases to a certain level, the reliability is gradually changed. The relationship diagram when going is shown. FIG. 10C shows a relationship diagram when the reliability is gradually lowered to a certain depth difference, and the reliability is drastically lowered when a certain depth difference is reached. If it is determined in step S901 that the depth change is less than a certain value in the template, the reliability is not lowered. In the present embodiment, any one of FIGS. 10A to 10C is applied as the reliability evaluation. However, the present invention is not limited to this, and the relationship indicating that the reliability decreases as the depth difference increases. As long as it is a diagram, it may be another relationship diagram. In the present embodiment, the reliability indicates the degree of motion vector detection between images for determining the use of a motion vector at each feature point, the magnitude of influence, and the like.

  Returning to FIG. 5, next, in step S507, the entire image motion vector detection unit 110 uses the motion vector detected for each feature point in step S505 and the reliability of the feature point determined in step S506. Thus, the motion vector of the entire image is detected. At this time, the motion vector of the entire image is detected by determining whether or not to use the motion vector of each feature point and the weighting (magnitude of influence) according to the reliability determined in step S506. For example, in the case of a feature point with low reliability, the motion vector of the feature point is not used or the influence upon detection is reduced. The motion vector of the entire image is detected, and the process ends.

  As described above, according to the present embodiment, it is possible to stably calculate a motion vector by performing reliability evaluation of feature points using depth information and extracting feature points with high reliability in template matching. An image processing apparatus can be provided. That is, even if the depth change in the image is large and the motion vector changes greatly between the frame images due to the movement of the feature point, the reliability of the feature point is determined from the depth information, and the motion vector detection process By reflecting the above, motion vector detection can be performed with high accuracy.

(Second Embodiment)
Next, an image processing apparatus 1100 according to the second embodiment will be described with reference to FIG. In FIG. 11, the same components as those shown in FIG. 1 are denoted by the same reference numerals as those in FIG. An image processing apparatus 1100 according to the present embodiment includes a motion vector adjustment unit (vector adjustment unit) 1101 in addition to the components shown in FIG. The motion vector adjustment unit 1101 detects the motion vector detected by the motion vector detection unit 106 from the reference image depth information input by the depth information input unit 107 and the depth information of the reference image stored in the depth information memory 108. Adjust.

  Next, the configuration of the motion vector adjustment unit 1101 will be described with reference to FIG. The motion vector adjustment unit 1101 includes a reference depth setting unit 11011, a motion vector direction adjustment unit 11012, and a motion vector size adjustment unit 11013. The reference depth setting unit 11011 sets the reference depth by using the depth information of the reference image input by the depth information input unit 107 or the depth information of the reference image stored and held in the depth information memory 108.

  The motion vector direction adjustment unit 11012 uses the depth information of the reference image input by the depth information input unit 107 and the depth information of the reference image held in the depth information memory 108, and starts and ends of detected motion vectors. Is adjusted, and the direction of the motion vector is adjusted. The motion vector size adjustment unit 11013 adjusts the size of the motion vector by matching the motion vector adjusted by the motion vector direction adjustment unit 11012 with the reference depth set by the reference depth setting unit 11011.

  Next, a motion vector detection process according to this embodiment will be described with reference to the flowchart shown in FIG. In FIG. 13, the same processes as those shown in FIG. 5 are denoted by the same reference numerals as those in FIG. After step S506, in step S1301, the motion vector adjustment unit 1101 uses the depth information of the reference image and the depth information of the reference image stored in the depth information memory 108, and uses the motion vector calculated in step S505. adjust.

  Next, a motion vector adjustment process performed by the motion vector adjustment unit 1101 will be described with reference to a flowchart shown in FIG. 14 and a schematic diagram of motion vector adjustment shown in FIG. 15A shows a standard image, FIG. 15B shows a reference image, and FIGS. 15C, 15D, and 15E show the depth direction with respect to FIG. 15B. It is a figure shown on a vertical axis.

  In each figure, reference depths 1501, 1502, 1505, 1508, and 1511 are lines indicating the reference depth, and are all set to the same depth information regardless of the image. Motion vectors 1503, 1504, 1506, and 1507 are motion vectors calculated in motion vector calculation. Motion vectors 1509 and 1510 are motion vectors whose directions are adjusted in the motion vector direction adjustment. Motion vectors 1512 and 1513 are motion vectors that have undergone size adjustment in motion vector size adjustment.

  First, in step S1401, the reference depth setting unit 11011 uses the reference image depth information input in step S502 or the reference image depth information input in step S502 and stored in the depth information memory 108 as a reference. Set the depth. Note that once the reference depth is set, there is no need to reset it even if step S1401 is repeated a plurality of times. As for the method for setting the standard depth, for example, the focus position of the standard image or the reference image may be set as the standard depth, or the depth at the center of the standard image or the reference image may be set as the standard depth. Further, the distribution of the entire depth may be calculated, and the intermediate depth may be set as the reference depth. Any method can be used as long as the reference depth can be set.

  Next, in step S1402, the motion vector direction adjustment unit 11012 determines the start point of the motion vector from the depth information of the reference image stored and held in the depth information memory 108 and the coordinates of the start point of the motion vector detected in step S505. Depth information is calculated. Then, the depth information of the end point of the motion vector is calculated from the reference image depth information input in step S502 and the coordinates of the end point of the motion vector detected in step S505. From the calculated motion vector start point depth information and motion vector end point depth information, the vector is rotated in the depth direction and the direction is adjusted to make the start point and end point depth the same.

  In FIG. 15B, motion vectors 1503 and 1504 are motion vectors calculated in step S505 from the reference image shown in FIG. 15A and the reference image shown in FIG. In addition, motion vectors 1506 and 1507 shown in FIG. 15C are obtained by setting each motion vector to the vertical axis in the depth direction. The motion vectors having different depths at the start and end points of the motion vectors 1506 and 1507 are rotated in the depth direction, so that the vectors become the same vectors at the start and end points as shown by the motion vectors 1509 and 1510 shown in FIG. Adjust.

  Next, in step S1403, the motion vector size adjustment unit 11013 adjusts the size of the motion vector using the reference depth set in step S1401 and the motion vector whose direction is adjusted in step S1402. The motion vectors 1509 and 1510 shown in FIG. 15D whose orientation has been adjusted in step S1402 are adjusted in size according to the reference depth and converted into motion vectors 1512 and 1513 shown in FIG. 15E, respectively. In the present embodiment, the motion vectors 1509 and 1510 are projected onto the reference depth and converted into motion vectors 1512 and 1513, respectively.

  Next, the motion vector detected in step S505 is adjusted in step S1301, and the adjusted motion vector and the reliability of each feature point calculated in step S506 are used. In step S507, the motion vector of the entire image is calculated. calculate. Then, the motion vector process ends. It is possible to perform motion vector detection with higher accuracy than in the first embodiment.

  As described above, according to the present embodiment, in template matching, an image processing apparatus capable of calculating a motion vector by performing reliability evaluation of feature points using depth information and extracting feature points with high reliability. Can be provided. Further, according to the present embodiment, by adjusting the motion vector, it is possible to detect the motion vector with higher accuracy than in the first embodiment.

(Third embodiment)
Next, an image processing apparatus 1600 according to the third embodiment will be described with reference to FIG. In FIG. 16, the same components as those shown in FIG. 1 are denoted by the same reference numerals as those in FIG. The image processing apparatus 1100 according to the present embodiment includes an inter-image reliability determination unit 1601 in addition to the components illustrated in FIG. The inter-image reliability determination unit 1601 uses the depth information of the reference image, the depth information of the base image, the template block on the base image, and the template block on the reference image to compare the depth information between the images. The reliability between images is determined according to the depth difference.

  Next, the configuration of the inter-image reliability determination unit 1601 will be described with reference to FIG. The inter-image reliability determination unit 1601 includes an inter-image depth evaluation unit 16011 and a reliability determination unit 16012. The inter-image depth evaluation unit 16011 calculates the depth difference between images using the depth information of the reference image, the depth information of the standard image, the template block on the standard image, and the template block on the reference image. The reliability determination unit 16012 determines the reliability between images according to the depth difference between images calculated by the inter-image depth evaluation unit 16011.

  Next, a motion vector detection process according to the present embodiment will be described with reference to the flowchart shown in FIG. In FIG. 18, the same processes as those shown in FIG. 5 are denoted by the same reference numerals as those in FIG. After step S505, in step S1801, the inter-image reliability determination unit 1601 determines the inter-image reliability based on the reference image depth information, the base image depth information, and the template matching result in step S504, which are input in step S502. Determine the reliability of.

  Here, the inter-image reliability determination will be described with reference to the flowchart shown in FIG. First, in step S1901, the inter-image depth evaluation unit 16011 determines the depth information in the template block on the reference image and the reference image from the depth information of the reference image, the depth information of the reference image, and the result of the process in step S504. And the depth difference is calculated.

  Next, in step S1902, the reliability determination unit 16012 determines the reliability between images in accordance with the depth difference between images calculated in step S1901. As for the relationship between the depth difference and the reliability, FIG. 10 is applied, and the reliability decreases as the depth difference increases. Then, the inter-image reliability determination ends. Returning to FIG. 18, the reliability between images determined in step S1801 is reflected in the feature point reliability in the subsequent stage, and the reliability determination of the feature points is performed in step S506. Then, the motion vector detection process ends.

  As described above, according to the present embodiment, in template matching, an image processing apparatus capable of calculating a motion vector by performing reliability evaluation of feature points using depth information and extracting feature points with high reliability. Can be provided. In addition, according to the present embodiment, before determining the reliability of each feature point, the reliability between images is determined and reflected in the reliability of each feature point, so that the accuracy is higher than in the first embodiment. A motion vector can be detected.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Moreover, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Image input part 102 Image memory 103 Feature point extraction part 105 Template matching part 106 Motion vector detection part 107 Depth information input part 109 Feature point reliability determination part 110 Whole image motion vector detection part

Claims (9)

An image processing apparatus for detecting a motion vector between a plurality of images,
Input means for inputting the image and depth information corresponding to the image;
Storage means for storing the image and depth information input by the input means;
Extraction means for extracting feature points of the first image stored in the storage means;
Similarity for determining similarity between a first region including the feature point extracted by the extraction unit and a second region corresponding to the first region in the second image input by the input unit A determination means;
First detection means for detecting a motion vector of the feature point between the first image and the second image from the determination result of the similarity determination means;
Reliability for determining reliability of the feature point based on first depth information corresponding to the first image stored in the storage unit and second depth information corresponding to the second image A determination means;
And a second detection unit that detects a motion vector of the entire image between the first image and the second image based on a determination result of the reliability by the reliability determination unit. An image processing apparatus. The reliability determination means determines a difference in depth between the feature point of the first image and the feature point of the second image corresponding to the feature point from the first depth information and the second depth information. The image processing apparatus according to claim 1, wherein the reliability of the feature point of the first image is lowered as the depth difference is calculated. The reliability determination unit determines a reliability by calculating a difference between the depths when the depth is changed in the first region from the first depth information and the second depth information. The image processing apparatus according to claim 2. The image processing apparatus according to claim 1, wherein the extraction unit extracts a predetermined number of feature points. Vector adjustment means for adjusting the motion vector detected by the first detection means;
5. The vector adjustment unit sets a reference depth serving as a depth reference from the depth information, and adjusts the motion vector based on the set reference depth. 6. An image processing apparatus according to 1. The image processing apparatus according to claim 5, wherein the vector adjustment unit adjusts a direction and a magnitude of the motion vector according to the set reference depth. The reliability determination means determines a reliability between the images of the first image and the second image based on the first depth information and the second depth information,
5. The second detection unit detects a motion vector of the entire image based on a reliability of the feature point and a reliability between the images. 6. Image processing device. An image processing method for detecting a motion vector between a plurality of images,
An input step of inputting the image and depth information corresponding to the image;
A storage step of storing the image and depth information input in the input step;
An extraction step of extracting feature points of the first image stored in the storage step;
A degree of similarity for determining a degree of similarity between the first region including the feature point extracted in the extraction step and the second region corresponding to the first region in the second image input in the input step A determination process;
A first detection step of detecting a motion vector of the feature point between the first image and the second image from a determination result in the similarity determination step;
Reliability for determining reliability of the feature points based on first depth information corresponding to the first image stored in the storing step and second depth information corresponding to the second image A determination process;
A second detection step of detecting a motion vector of the entire image between the first image and the second image based on a determination result of the reliability in the reliability determination step. Image processing method.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-7.

Images