TW201514926A - Disparity calculating method and stereo matching system thereof - Google Patents

Abstract

A disparity calculating method, for a stereo matching system, includes calculating a global energy matrix according to a first image, a second image and a previous disparity matrix of a previous frame; calculating a global disparity matrix according to the global energy matrix and a first matching algorithm; calculating a first image block of the first image, a second image block of the second image, a previous block disparity matrix between the first image and the second image and the global disparity matrix; and calculating a local disparity matrix between the first image block and the second image block according to the local energy matrix and a second matching algorithm.

Description

Parallax calculation method and stereo matching system thereof

The present invention relates to a parallax calculation method and a stereo matching system thereof, and more particularly to a parallax calculation method capable of generating a parallax map with temporal and spatial stability and a stereo matching system thereof.

With the continuous advancement of imaging technology, the size and function of display devices that consumers can choose are increasingly diverse, and in order to meet the needs of different users, manufacturers of display devices have also managed to provide new products with better output performance and better resolution. One of the most interesting products is a display device having a three-dimensional display. The commonly used three-dimensional display technology has a display mode such as Polarized, Interlaced or Anaglyph. The principle is to project different perspective images related to depth information by special optical design. The left and right eyes of the person can make the left and right eyes of the person see the images of different viewing angles respectively, and after the synthesis of the human brain, the three-dimensional depth information is generated, and the three-dimensional image is perceived.

When the source image to be displayed by the display device having the three-dimensional display function is a two-dimensional image without depth information, since the source image lacks depth information, the display device cannot generate various multi-views related to depth information (multi-view) ) Image to project to the left and right eyes of the person. In this case, the display device needs to analyze the two-dimensional image to obtain the depth information, so as to display the multi-view image. In the prior art, if at least two different viewing angle images are obtained by using multiple image capturing devices at different positions, the depth information can be analyzed, and the two different viewing angle images are analyzed to obtain the depth information. The process is called stereo matching. The principle is to search for matching objects between two different perspective images (or The difference between the position values of the matching objects in the two different viewing angle images can be obtained as the disparity information (the so-called disparity map), and the matching object can be calculated through the parallax information. In-depth information.

However, when stereoscopic matching is performed between two different viewing angle images to obtain depth information, since the landscapes of the images are not the same, and the position distances between the two image capturing devices for obtaining two different viewing angle images are different, how to Properly searching for matching objects between two different perspective images and obtaining the correct parallax information will affect the accuracy of the depth information. For example, when an object of the two images has a matching error, for example, the A object of the left view image matches the B object of the right view image instead of the A object, the parallax information of the object may generate an error, so that the error depth information is obtained. Make the object show the wrong depth, and make the object invisible or make the object deform. Therefore, when performing stereo matching between two-view images, how to accurately obtain the stereo matching result of each object in the image has become one of the most important topics of stereoscopic vision technology.

In order to solve the above problems, the present invention provides a parallax calculation method capable of generating a parallax map having temporal and spatial stability and a stereo matching system thereof.

In one aspect, a disparity calculation method is disclosed for a stereo matching system, where the disparity calculation method includes calculating a global domain according to a previous disparity matrix of a first image, a second image, and a previous image. An energy matrix; calculating a global parallax matrix according to the global energy matrix and a first matching algorithm; a first image block of the first image, a second image block of the second image, the first Calculating a region energy matrix between the first image block and the second image block by using a previous block disparity matrix between the image and the second image and the global disparity matrix; and calculating an energy matrix according to the region And a matching algorithm, calculating a regional parallax matrix between the first image block and the second image block.

In another aspect, a stereo matching system is disclosed, including a global parallax storage module for storing a previous parallax matrix of a previous image and a global parallax matrix; and a previous block parallax storage module for storing a first a previous block disparity matrix between an image and a second image; a current block disparity storage module for storing a regional disparity matrix; a global disparity computing module coupled to the global disparity storage module, And calculating a global energy matrix according to the first image, the second image, and the previous parallax matrix; and calculating the global parallax matrix according to the global energy matrix and a first matching algorithm; and a regional parallax computing module The first parallax storage module and the current block parallax storage module are coupled to the first image block and the second image of the first image. a second image block, the previous block disparity matrix, and the global disparity matrix, and calculating an area energy matrix between the first image block and the second image block; And a second matrix region energy matching algorithm, calculates a disparity in the region between the matrix block of the first image and the second image block.

10‧‧‧ Stereo Matching System

100‧‧‧Global parallax storage module

102‧‧‧Previous block parallax storage module

104‧‧‧ Current block parallax storage module

106‧‧‧Global Parallax Computing Module

108‧‧‧Regional Parallax Calculation Module

110‧‧‧resolution adjustment unit

112‧‧‧Global Disparity Computing Unit

30‧‧‧ Parallax calculation method

300~312‧‧‧Steps

B1, B2‧‧‧ image blocks

CF‧‧‧ current image

GDM‧‧‧ global parallax matrix

GEM‧‧‧Global Energy Matrix

IL, ILL‧‧‧ left eye image

IMG1, IMG2‧‧‧ images

IR, IRL‧‧‧ right eye image

LDM‧‧‧Regional Parallax Matrix

LEM‧‧‧Regional Energy Matrix

PBDM‧‧‧Previous block parallax matrix

PDM‧‧‧Previous Parallax Matrix

PF‧‧‧ previous image

SM1, SM2‧‧‧ matching algorithm

FIG. 1 is a schematic diagram of a stereo matching system according to an embodiment of the present invention.

2 is a schematic diagram of a nonlinear equation according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of a parallax calculation method according to an embodiment of the present invention.

In an exemplary embodiment of the present invention, the stereo matching system generates an energy matrix for determining a disparity of a current region of the current image according to the disparity information of a previous image and the disparity information of the previous region of the current image. Accordingly, the parallax map output by the stereo matching system can have both temporal and spatial continuity, thereby obtaining a precise and stable parallax map. In order to more clearly understand the present invention, the following detailed description will be described in the accompanying drawings. In addition, the connection terms mentioned in the following embodiments, such as couplings or connections, are merely exemplified with reference to the additional drawings, and are not intended to limit the connection between the two elements. Or indirectly coupled.

Please refer to FIG. 1. FIG. 1 is a schematic diagram of a stereo matching system 10 according to an embodiment of the present invention. The stereo matching system 10 is used to calculate the disparity of each pixel in the stereoscopic image. As shown in FIG. 1 , the stereo matching system 10 includes a global disparity storage module 100 , a previous block disparity storage module 102 , a current block disparity storage module 104 , a global disparity computing module 106 , and an area . Parallax calculation module 108. The global disparity storage module 100 is configured to store a previous disparity matrix PDM of a previous image PF and a global disparity matrix GDM. The previous block disparity storage module 102 is configured to store a previous block disparity matrix PBDM between an image IMG1 and an image IMG2. The current block disparity storage module 104 is configured to store an area parallax matrix LDM. The global disparity computing module 106 includes a resolution adjusting unit 110 and a global disparity computing unit 112, and is coupled to the global disparity storage module 100. The global disparity calculation module 106 is configured to calculate a global energy matrix GEM based on the images IMG1, IMG2 and the previous parallax matrix PDM, and calculate the global disparity matrix GDM according to the global energy matrix GEM and a matching algorithm SM1. The regional disparity computing module 108 is coupled to the global disparity storage module 100, the previous block disparity storage module 102, and the current regional disparity storage module 104, and is configured to correspond to an image block B1 and an image IMG2 in the image IMG1. Calculating a region energy matrix LEM between the image block B1 and the image block B2 in an image block B2, a previous block disparity matrix PBDM, and a global disparity matrix GDM of the image block B1; the regional disparity calculation module 108 is based on The regional energy matrix LEM and a matching algorithm SM2 calculate the regional parallax matrix LDM between the image blocks B1 and B2. In short, since the global disparity matrix GDM generated from the previous disparity matrix PDM of the previous image PF has a correlation with the disparity information of the previous image PF, when the stereo matching system 10 generates the regional disparity matrix LDM according to the global disparity matrix GDM The regional parallax matrix LDM has a correlation with the disparity information of the previous image PF (ie, the regional disparity matrix LDM has temporal continuity). Further, since the previous block disparity matrix PBDM includes disparity information of the blocks around the image blocks B1, B2, when the stereo matching system 10 generates the regional disparity matrix LDM according to the previous block disparity matrix PBDM, the regional disparity matrix LDM and The disparity information of the blocks around blocks B1 and B2 is related (ie, regional view) The difference matrix LDM has spatial continuity). As a result, the regional parallax matrix LDM will have both temporal and spatial continuity. Accordingly, the stereo matching system 10 can produce a stable and accurate parallax map.

In detail, the images IMG1 and IMG2 are respectively a left eye image IL and a right eye image IR of a current image CF of a stereoscopic image I3D. After receiving the left-eye image IL and the right-eye image IR, the global disparity calculation module 106 first uses the resolution adjustment unit 110 to reduce the resolution of the left-eye image IL and the right-eye image IR to generate a low-resolution left-eye image. ILL and right eye image IRL. The global disparity calculation unit 112 can calculate the global energy matrix according to the low-resolution left-eye image ILL, the right-eye image IRL, and the previous disparity matrix PDM stored in the global disparity storage module 100 corresponding to the disparity map of the previous image PF. GEM. The previous image PF may be the previous picture of the current image CF in the stereo image I3D, and the formula for calculating the accumulative energy of a pixel x in the previous parallax matrix PDM corresponding to the plurality of disparity candidates may be expressed as:

Where E is the cumulative energy, C ( x , k )=| I _l ( x )- I _r ( x + k )|, TC ( x , k )= f ₁ (| k - dt _{n -1} ( x ) |), k is a disparity candidate within a specific search range, I _l is the pixel value of the left-eye image ILL, I _r is the pixel value of the left-eye image IRL, and dt _{n -1} ( x ) is the previous parallax matrix PDM Corresponding to the parallax of pixel x, f () is a nonlinear mapping equation. For example, f ₁ () may be a nonlinear equation as shown in FIG. 2, but is not limited thereto.

Through the formula (1), the global disparity calculation module 106 can calculate the global energy matrix GEM of the current image CF, and obtain the global disparity matrix GDM according to the matching algorithm SM1. It is worth noting that when the difference between the disparity candidate k and the disparity dt _{n -1} ( x ) corresponding to the same pixel x in the previous image PF is larger, the value of TC ( x , k ) is larger (ie, TC ( x , k ) ) is proportional to the difference between the parallax candidate k and the parallax dt _{n -1} ( x )). Since TC ( x , k ) is added to the formula (1), the global disparity matrix GEM has a correlation with the disparity information of the previous image PF. In other words, after increasing TC ( x , k ), the global disparity matrix GEM may have temporal continuity.

Next, the regional disparity calculation module 108 is based on the image block B1 of the left-eye image IL, the image block B2 corresponding to the image block B1, the previous block disparity matrix PBDM, and the global disparity matrix GDM in the right-eye image IR. The regional energy matrix LEM between the image block B1 and the image block B2 is calculated. In this embodiment, the image blocks B1 and B2 are pixel regions of the same row in the left eye image IL and the right eye image IR, respectively, and the previous block parallax matrix PBDM corresponds to the image blocks B1 and B2. The parallax matrix of the previous column of pixels, but is not limited to this. The regional disparity calculation module 108 obtains a formula for calculating the energy of a pixel x corresponding to a plurality of disparities in the previous parallax matrix PDM, which can be expressed as:

Where E is the cumulative energy, C ( x , k )=| I _l ( x )- I _r ( x + k )|, TC ( x , k )= f ₂ (| k - dt _{n -1} ( x ) |), SC (x, k ) = 2 (f | k - ds (x) |), k is a disparity candidate within a specified search range, I _l is the pixel value of left-eye image IL, I _r to the left The pixel value of the eye image IR, dt ( x ) is the parallax corresponding to the pixel x in the global disparity matrix GDM, and ds ( x ) is the disparity of the same column in the previous block disparity matrix PBDM. f ₂ () and f ₃ () are nonlinear mapping equations similar to f ₁ ().

Through the formula (2), the regional disparity calculation module 108 can obtain the regional energy matrix LEM of the image block B1 and the image block B2, and obtain the regional parallax matrix LDM according to the matching algorithm SM2. The matching algorithm SM2 can be a dynamic programming algorithm. It is worth noting that when the difference between the parallax candidate k and the parallax dt ( x ) corresponding to the same pixel x in the global disparity matrix GDM is larger, the value of TC ( x , k ) is larger (ie, TC ( x , k ) is proportional The difference between the disparity candidate k and the disparity dt ( x )). Since TC ( x , k ) is added to the formula (1) and the global parallax matrix GDM has temporal continuity, the correlation between the regional parallax matrix LDM and the parallax of the previous image PF is generated. In other words, after increasing TC ( x , k ), the regional parallax matrix LDM also has an association with the disparity information of the previous image PF, thereby having temporal continuity. On the other hand, when the disparity candidate k is larger in difference from the disparity ds ( x ) of the same column in the previous block disparity matrix PBDM, the value of SC ( x , k ) is larger (ie, SC ( x , k ) ) is proportional to the difference between the disparity candidate k and the disparity ds ( x )). By adding SC ( x , k ) to equation (1), the regional parallax matrix LDM has an association with the disparity information of the current image CF located in the previous column of image blocks B1 and B2. In other words, after increasing SC ( x , k ), the regional parallax matrix LDM may have spatial continuity.

In addition, observing the left eye image IL and the right eye image IR, the temporal correlation of the parallax of the same pixel in the left eye image IL and the right eye image IR is more correlated than the surrounding pixels in the left eye image IL and the right eye image IR. low. Therefore, in the formula (2), the gain of f ₂ () is larger than the gain of f ₃ () to ensure that the accuracy of the parallax and the stability in time can be maintained while calculating the parallax matrix LDM.

After the regional parallax matrix LDM is calculated, the stereo matching system 10 can store the global parallax matrix GDM and the regional parallax matrix LDM as the previous parallax matrix PDM and the previous block parallax matrix PBDM, thereby calculating the left eye image IL and the right eye image. The parallax of the next column of pixels in the image blocks B1 and B2 in the IR. By repeating the above steps, the stereo matching system 10 can obtain a disparity map of the current image CF.

It should be noted that, after adding the parameters related to the parallax information of the surrounding pixels of the previous image and the current image in the formula for calculating the cumulative energy, the parallax calculated by the stereo matching system disclosed in the above embodiment will have time and space. The stability, which in turn produces a stable and accurate parallax map. Depending on the application, those of ordinary skill in the art should be able to implement appropriate changes and modifications. For example, the stereo matching system 10 can also utilize a scan-line optimization algorithm as the matching algorithm SM2, and is not limited thereto.

The manner in which the stereo matching system 10 calculates the regional parallax matrix LDM can be further improved. Step by step into a parallax calculation method 30, please refer to Figure 3. It should be noted that, if substantially the same result is obtained, the order of steps of the parallax calculation method 30 is not limited to the sequence of steps shown in FIG. As shown in FIG. 3, the parallax calculation method 30 can be applied to a stereo matching system and includes the following steps:

Step 300: Start.

Step 302: Adjust the resolution of the first image and the second image to generate a first low-resolution image and a second low-resolution image, respectively.

Step 304: Calculate a global energy matrix according to the first low-resolution image, the second low-resolution image, and the previous parallax matrix.

Step 306: Calculate a global parallax matrix according to the global energy matrix and a first matching algorithm.

Step 308: A first image block of the first image, a second image block of the second image, a previous block disparity matrix between the first image and the second image, and the global disparity matrix And calculating an area energy matrix between the first image block and the second image block.

Step 310: Calculate a regional parallax matrix between the first image block and the second image block according to the area energy matrix and a second matching algorithm.

Step 312: End.

According to the parallax calculation method 30, the stereo matching system can generate a stable and accurate parallax map. It should be noted that since the accuracy of the global parallax matrix does not need to be too high, the parallax calculation method 30 lowers the resolution of the first image and the second image in step 302, and uses the first low resolution obtained in step 302. The degree image and the second low-resolution image generate a global energy matrix, thereby reducing the resources used to calculate the global energy matrix and the global parallax matrix. However, the global energy matrix can also be generated directly from the first image and the second image. That is, step 302 can be omitted, and the disparity calculation method 30 can instead generate a global energy matrix directly from the first image, the second image, and the previous parallax matrix.

In summary, the parallax calculation method and the stereo matching system provided by the above embodiments increase the parallax information related to the previous image and the parallax information of the previous block of the current image by adding a formula for calculating the accumulated energy. The calculated disparity information has temporal and spatial continuity. Accordingly, the parallax calculation method and the stereo matching system provided by the above embodiments can generate a stable and accurate disparity map.

30‧‧‧ Parallax calculation method

300~312‧‧‧Steps

Claims (14)

A disparity calculation method for a stereo matching system, the parallax calculation method includes: calculating a global energy matrix according to a first image, a second image, and a previous parallax matrix of a previous image; The global energy matrix and a first matching algorithm calculate a global parallax matrix; a first image block of the first image, a second image block of the second image, the first image, and the first image Calculating a region energy matrix between the first image block and the second image block by using a previous block disparity matrix between the two images and the global parallax matrix; and calculating an energy matrix and a second matching algorithm according to the region And calculating a regional parallax matrix between the first image block and the second image block. The disparity calculation method of claim 1, wherein the calculating the global energy matrix according to the first image, the second image, and the previous parallax matrix of the previous image comprises: adjusting the first image and the first The resolution of the two images to generate a first low resolution image and a second low resolution image respectively; and calculating the first low resolution image, the second low resolution image, and the previous parallax matrix Global energy matrix. The disparity calculation method of claim 1, wherein a first global energy system corresponding to a first position and a first difference value in the global energy matrix is proportional to the first difference value and corresponding to the previous disparity matrix A difference between a first previous disparity at the first location. The parallax calculation method of claim 1, wherein a first energy system corresponding to a first position and a first difference value in the regional energy matrix is proportional to the first difference value and the global disparity A difference in a matrix corresponding to a first previous disparity of the first location. The disparity calculation method of claim 1, wherein a first energy system corresponding to a first position and a first difference value in the regional energy matrix is proportional to the first difference value and the previous block disparity matrix Corresponding to the difference between the disparity of a first previous block of the first location. The disparity calculation method of claim 1, wherein the second matching algorithm comprises a dynamic programming algorithm. The disparity calculation method of claim 1, wherein the second matching algorithm comprises a scan-line optimization algorithm. A stereo matching system includes: a global parallax storage module for storing a previous parallax matrix of a previous image and a global parallax matrix; and a previous block parallax storage module for storing a first image and a a previous block disparity matrix between the second images; a current block disparity storage module for storing a regional disparity matrix; a global disparity computing module coupled to the global disparity storage module for Calculating a global energy matrix according to the first image, the second image, and the previous parallax matrix; and calculating the global parallax matrix according to the global energy matrix and a first matching algorithm; and a regional parallax computing module coupled to The global parallax storage module, the previous block disparity storage module, and the current block disparity storage module are configured to use a first image block of the first image and a second image area of the second image. a block, the previous block disparity matrix, and the global disparity matrix, and calculating a region energy matrix between the first image block and the second image block; A second matching algorithm, the calculation of a parallax matrix of the region between an image block and the second image block. The stereo matching system of claim 8, wherein the global disparity calculation module comprises: a resolution adjustment unit, configured to adjust resolutions of the first image and the second image to respectively generate a first low a resolution image and a second low-resolution image; and a global disparity calculation unit configured to calculate the global energy matrix according to the first low-resolution image, the second low-resolution image, and the previous parallax matrix. The stereo matching system according to claim 8, wherein a first global energy system corresponding to a first position and a first difference value in the global energy matrix is proportional to the first difference value and corresponding to the previous parallax matrix A difference between a first previous disparity at the first location. The stereo matching system according to claim 8, wherein a first energy system corresponding to a first position and a first difference value in the region energy matrix is proportional to the first difference value and the global disparity matrix corresponds to The difference between a first previous disparity of the first location. The stereo matching system of claim 8, wherein a first energy system corresponding to a first position and a first difference value in the region energy matrix is proportional to the first difference value and the previous block disparity matrix Corresponding to the difference between the disparity of a first previous block of the first location. The stereo matching system of claim 8, wherein the second matching algorithm comprises a dynamic programming algorithm. The stereo matching system of claim 8, wherein the second matching algorithm comprises a scan-line optimization algorithm.