FR2973980A1 - Method for detection of composite format utilized to format sequence of three-dimensional video, received by decoder, involves utilizing format to format sequence of images if comparison between threshold and correlation satisfies condition - Google Patents

Abstract

The method involves providing a left image and a right image that are relative to same image in a sequence of stereoscopic images i.e. three-dimensional video. A set of half-images is obtained from the image in the sequence of images according to a combination defined by a composite format. The composite format is determined (6) by determining maximum correlation between one of the half-images and another half-image. The composite format is utilized to format the sequence of stereoscopic images if a comparison between a maximum threshold and a correlation satisfies a condition. Independent claims are also included for the following: (1) a device for detection of a composite format (2) a computer program comprising instructions for performing a method for detection of the composite format (3) a storage unit comprising instructions for performing a method for detection of the composite format.

Description

The present invention relates to a method and device for detecting the composite format used to format a sequence of stereoscopic images. Such a method and device is used, in particular, to obtain a good positioning of the left and right images of each of the stereoscopic images.

The invention is adapted, in particular, to video decoding equipment equipped with HDMI 1.3 sockets which are widely deployed to date. More generally, it is suitable for any equipment that is not intended to process the signaling relating to stereoscopic image sequences that is transmitted during the broadcast of these sequences or to transmission conditions that would not allow the transmission of these sequences. correct reception of this signaling by equipment. A sequence of stereoscopic images, also called 3D video, provides a user with a depth effect of the objects of a scene when the images of this sequence are viewed on a viewing screen. To obtain this effect, two images, called left and right, are needed per image of the sequence. These right and left images of the scene are acquired by two cameras that are spatially offset from each other by a few centimeters to recreate the stereoscopic effect of human vision. A 3D video can be coded according to the MPEG4 / H264 standard and be high definition (1920x1080 resolution). We are talking about 3D 1080i video. A 3D video can be broadcast by satellite, cable or by ADSL according to a broadcasting chain shown schematically in FIG. 1. This broadcast system comprises a 3D video COD coder, a C broadcast channel and a DEC decoder which is connected via, for example, a link conforming to an HDMI standard, to a 3DTV viewing screen provided for viewing videos. 3D.

The user then has to put on a pair of shutters and active glasses to recreate the stereoscopic effect of the objects of the scene when the right image and the left image relating to the same image of the 3D video are visualized on this screen by alternately. A 3D video is encoded by the coder COD so that the resulting stream can be broadcast in a broadcast channel C whose bandwidth does not exceed that commonly used to broadcast conventional (non-3D) video streams. Since it is necessary to broadcast twice as many images for a 3D video as for a conventional video for the same image quality, the COD encoder formats the 3D video in a so-called composite format as illustrated in FIG. 2.

Firstly, the right image and the full resolution left image that are related to the same image of the 3D video are each subsampled (step 1) to obtain two half-images Id and Ig. It is called half-image because the sampling rate is equal to 2. This sub-sampling is, in most cases, preceded by a low-pass filtering of these images to avoid anti-aliasing problems that could occur. manifest on contours and textures that contain high frequencies and affect the visual quality of these images when viewed. These two half-images thus subsampled are then combined (step 2) with each other to obtain a full-resolution composite image, which image is then broadcast as would be an image of a conventional (non-3D) video. Several composite formats are commonly used. One of these formats, which is usually called side-by-side, is illustrated in FIG. 2. A composite image in side-by-side format is defined as a full-resolution image which has the half-image Ig in its left-hand side and the half-image Id in its right-hand side, the two half-images Id and Ig being the results of a horizontal sub-sampling of left and right full resolution images. Another composite format is called up-and-down (Top-and-Bottom). A full-resolution high-resolution image is defined as a full-resolution image which has the half-image Ig in its upper part and the half-image Id in its lower part, the two half-images Id and Ig. being then the results of vertical subsampling of left and right full resolution images. Other composite formats are of course possible in which the two half-images Ig and Id, once subsampled, are mixed together to form a mosaic. The decoder DEC is intended to receive a formatted 3D video stream. The decoder extracts (step 3) the two images Ig and Id from the received composite image, extrapolates (step 4) these two half-images to obtain a right image and a full resolution left image, which images are then displayed on the image. 30 3DTV display screen (Fig. 3). Some broadcast video streams include information that allows the decoder DEC to be notified of the format that was used to format the 3D video. However, it happens that video streams are broadcast without this information which causes a decoder dysfunction DEC if the format that was used to format the 3D video is not that which the decoder DEC expects. Furthermore, it also happens that 3D videos are broadcast in a format identical to that expected by the decoder DEC but that the coder COD has reversed the right images with the left images during the formation of the composite image thus causing a malfunction DEC decoder and important artifacts on the visualized 3D image. The problem solved by the present invention is to overcome the drawbacks raised.

For this purpose, the present invention relates to a method for detecting the composite format that has been used to format a sequence of stereoscopic images received by a decoder, the format combining a filtered and undersampled version of a right image with each other. a filtered and subsampled version of a left image according to a combinatorial specific to this format, said left and right images being relative to the same image of the sequence of stereoscopic images, characterized in that it comprises the following steps: - Obtain, according to a combinatorics defined by a composite format expected by the decoder, a first half-image and a second half-image from the image, - detect the composite format that has been used to format the image sequence stereoscopic, by determining the maximum correlation between the first half-image shifted and the second half-image, the composite format that has been used se to format the sequence of stereoscopic images then being the format that was used to obtain the first and second half-image if the comparison between this correlation maximum and a threshold satisfies a condition.

According to an embodiment of the detection step, the minimum of the absolute difference between the first offset half-image and the half-image is determined, and this minimum is then compared to a threshold, if this minimum is less than threshold then the composite format in which is formatted the sequence of stereoscopic images is the format that was used to obtain the first and the second half-image, and if this minimum is greater than the threshold then another composite format expected by the decoder is considered and the steps to obtain the two half-images and to detect the composite format are repeated until all the composite formats expected by the decoder are considered.

This minimum of the absolute difference between the first offset half-image and the half-image corresponds to the maximum correlation between the two half-images which occurs when the relative offset to this minimum is equal to (or very close to) the distance that separates the optical axes of the two cameras used for the acquisition of left and right images. The use of the absolute difference between the two half-images and the introduction of the hypothesis that the two cameras are perfectly aligned horizontally (the first half-image is shifted only in one dimension) are advantageous, in terms of calculation cost, compared to a conventional correlation of these two half-images which requires an exhaustive search for a displacement expressed in the plane of the image (two-dimensional offset). In addition, because of the small ranges of values of the vertical or horizontal offsets, the method facilitates the identification of a maximum correlation unlike a conventional correlation which has several correlation peaks around the maximum. According to one variant, the threshold is adaptive and, preferably, is equal to the signal-to-noise ratio PSNR weighted by a coefficient. According to one embodiment, the first and second half-images are subsampled prior to their correlation.

This embodiment is advantageous because it makes it possible to reduce the cost of calculating the method in order to establish the maximum correlation. According to one embodiment, the minimum is calculated from the pixels belonging to a sliding window of dimension less than that of these half-images.

According to one embodiment, in the case where one of the formats expected by the decoder has been considered to be the format that was used to form the image, the method also includes a step of detecting the inversion of the first and second half-frames during which if the sign of the relative offset to the minimum is negative then the first half-image is the left half-image and the second half-image is the right half-image, otherwise the first half-image is the right half-image and the second half-image is the left half-image. According to one embodiment, the minimum relative offset calculated for an image of the stereoscopic image sequence is smoothed temporally by a low-pass filter.

According to its material aspect, the present invention also relates to a device comprising means for implementing the method above. The present invention also relates to a computer program, which can be stored on a medium and / or downloaded from a communication network, in order to be read by a computer system or a processor. This computer program includes instructions for implementing the method mentioned above, when said program is executed by the computer system or the processor. The invention also relates to storage means comprising such a computer program.

The characteristics of the invention mentioned above, as well as others, will emerge more clearly on reading the following description of an exemplary embodiment, said description being given in relation to the attached drawings, among which: FIG. . 1 schematically represents a broadcast chain of a sequence of stereoscopic images. Fig. Figure 2 illustrates the creation of a composite image from a side-by-side format. Fig. Figure 3 illustrates the creation of a composite image from a top-and-bottom format. FIG. 4 is a diagram of the steps of the composite format detection method used to format a sequence of stereoscopic images. Fig. 5 illustrates the implementation of the method for two particular formats. Fig. 6 shows the architecture of a composite format sensing device used to format a sequence of stereoscopic images. FIG. 7 schematically represents an example of parallelization of the calculation of the absolute difference between two half-images. Fig. 4 is a diagram of the steps of the composite format detection method used to format a sequence of stereoscopic images. In the following of this method, it is considered that several composite formats Fk 30 are likely to have been used by the coder COD to format the 3D video and that the decoder expects to receive a 3D video according to one of these Fk formats. Of course, this general case includes the case where the decoder expects to receive only 3D videos formatted in a single format.

Each format Fk combines a filtered and subsampled version, denoted Id, of a right image and a filtered and subsampled version, denoted Ig, of a left image according to a specific combinatorics. The left and right images are relative to an I-frame of the formatted 3D video. The

method comprises a step for obtaining, according to the combinatorics defined by one of the composite formats Fk expected by the decoder DEC, a first half-image I1 and a second half-image I2 from the image I of the 3D video.

The method also includes a step (5) of detecting the composite format that has been used to format the 3D video, during which the maximum of

correlation between the imaged half-image I1 and the half-image I2 is determined. The composite format that was used to format the 3D video is then the Fk format that was used to obtain the first and second half-image if the comparison between this correlation maximum and a threshold satisfies a condition.

According to an embodiment of the detection step, the minimum of the absolute difference between the offset half-image I1 and the half-image I2, denoted SADmin, is determined. In mathematical terms, this minimum is given by: S4D. = min S4D (v) 11 - D <v <D with D an integer value in which, if the format Fk is of type side-by-side, NN N S4D (v) = 1 11 (i + v, j) -1z (i + Nc, j) with N, the number of columns (N ~ -V) NI i = 1 1 = 1 of the image I, NI the number of lines of the image I and v the offset of the half-image I1 with respect to the half-image I2, and if the format Fk is of up-and-down type, N0 N ~ S4D (v) = 1E Il (i, j + v) -I2 ( i, j + NI). (Nc-V) N, i = 1 J = 1 The minimum SADmin is then compared to a threshold SS4D and if S4Dmin <SS4D then the composite format in which the 3D video is formatted is the format Fk which has been used to obtain the first and the second half-image. In the case where the minimum SADmin is greater than the threshold, then another composite format Fk waited by the decoder DEC is considered and the steps to obtain the two half-images and to detect the composite format are repeated until all the Composite formats Fk expected by decoder DEC are considered.

According to one variant, the threshold SS4D is adaptive and, preferably, is equal to the weighted signal-to-noise ratio PSNR (peak of signal to noise ratio) weighted by a coefficient: SS4D = a xPSNR The PSNR ratio is an average value one-pixel deviations from its neighbors over the entire full-resolution image of 3D video and represents the average dynamic or contrast of that image.

The shift v which corresponds to the minimum SADmin is a displacement vector signed between the half-image I1 and the half-image I2. This displacement vector is noted later vmin. According to one embodiment, the first and second half-images are subsampled prior to their correlation.

According to one embodiment, the minimum SADmin is calculated from the pixels belonging to a sliding window of dimension less than that of these half-images. In the case where one of the formats Fk expected by the decoder DEC has been considered to be the format that has been used to form the image I, the method also comprises, according to one embodiment, a step (6) of detection of reversal of the two half-images I1 and I2 during which if the sign of the shift vmin is negative then the first half-image I1 is the left half-image Ig and the second half-image I2 is the right half-image Id, if not the first half-image I1 is the right half-image Id and the second half-image I2 is the left half-image Ig.

According to an embodiment, the venom offset calculated for an image of the 3D video is smoothed temporally by a low-pass filter. In mathematical terms, the venom shift for an image of 3D video is smoothed by a first-order low-pass filter. In mathematical terms, Venin (t + 1) = fi x Venin (t) + (1-13) xVmin (t + 1) with Venom (t + 1) the shift for an image following temporally the image I, vmin ( t) the shift for the image I and f3 a real value varying between 0 and 1 and which is worth O when a change of scene is detected in the 3D video. For example, this scene change is information that comes from the decoder DEC, information that gives or allows to deduce the value of f3.

Fig. 5a illustrates the implementation of this method when the format Fk presaged as having been used to format the 3D video is of side-by-side type with the left half-image Ig on the left and the half-image Id on the right. The process extracts the two half-images I1 and I2, calculates the minimum SADmin and then obtains a vvin displacement vector. If, venom <0, there was no inversion

8 of the two half-images I1 and I2 with respect to the expected format Fk and the half-image I1 is the left half-image and the half-image I2 is the right half-image. If, venom> 0, an inversion has occurred and the half-image I1 is, this time, the right half-image and the half-image I2 is the left half-image.

Fig. 5b illustrates the implementation of this method when the format Fk presaged as having been used to format the 3D video is of up-and-down type with the left half-image Ig at the top and the half-image Id at the bottom. The process extracts the two half-images I1 and I2, calculates the minimum S-IDmin and then obtains a displacement vector vn, in. If vn, ~ n <0, there has been no inversion of the two half-images I1 and I2 with respect to the expected format Fk and the half-image I1 is indeed the left and half half-image. image I2 is the right half-image. If venom> 0, an inversion has occurred and the half-image I1 is, this time, the right half-image and the half-image I2 is the left half-image. The above method is integrated in the conventional decoding method described in the introductory part to replace step 3. According to its hardware aspect, the present invention relates to a composite format detection device used to format a sequence of stereoscopic images. Fig. 6 schematically illustrates the architecture of such a device 600. The device 600 comprises, connected by a communication bus 601: - a processor, microprocessor, microcontroller (noted μc) or CPU (Central Processing Unit or Unit) Central Processing Station in French) 602; a random access memory RAM (Random Access Memory in English or Random Access Memory in French) 603; A ROM (Read Only Memory in English or Memory Read Only in French) 604; a reader 605 for storage medium, such as a SD card reader (Secure Digital Card in English or Secured Digital Card in French); interface means 606 with a communication network, for example a satellite network; and microcontroller 602 is capable of executing instructions loaded into RAM 603 from ROM 604, external memory (not shown), storage medium such as SD card or the like, or a communication network. When the device 600 is turned on, the microcontroller 602 is able to read instructions from RAM 603 and execute them. These instructions form a computer program which causes the microcontroller 602 to implement all or some of the algorithms described below in connection with FIG. 4. All or part of the algorithms described below in connection with FIG. 4 can be implemented in software form by executing a set of instructions by a programmable machine, such as a DSP (Digital Signal Processor in English or a Digital Signal Processing Unit in French) or a microcontroller, such as the microcontroller 602, or be implemented in hardware form by a machine or a dedicated component, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application-Specific Integrated Circuit). or Integrated Circuit Specific to an Application in French). According to an embodiment illustrated in FIG. 7, the device comprises means for calculating the S4D (v) which exploits data flow computation architectures of SIMD (Single Instruction Multiple Data) type. The elementary operator OP (on the left of Fig. 7) realizes the difference between the pixels of the half-images I1 and I2, the absolute value and the sum by accumulation. On the right of this same fig. 7, a parallelization of block calculations of 16 pixels is performed by duplication of the operator OP.

This parallelization of the calculations on the data stream makes it possible to accelerate the correlation calculations by similarity and thus allows the implementation of this device for the real-time processing of high-definition 3D videos. The device can be integrated into the decoder DEC which includes a TNT / SAT / CABLE tuner for receiving a broadcast 3D signal carrying a sequence of stereoscopic images and for digitally demodulating this signal to obtain a formatted 3D video. This decoder DEC may also include a dedicated circuit compliant with the MPEG4 standard for decoding this sequence of images. The device is then connected to this circuit to receive the scene changes used by the method. Stereoscopic image sequences can be in color. In this case, the method is applied only to the luminance component of this image sequence.

Claims (11)

CLAIMS1) A method for detecting the composite format that has been used to format in format a sequence of stereoscopic images received by a decoder (DEC), the composite format combining a filtered and a subsampled version of a right image ( Id) and a filtered and undersampled version of a left image (Ig) according to a combinatorial specific to that format, said left and right images being relative to the same image (I) of the stereoscopic image sequence, characterized in it comprises the following steps: - obtaining, according to a combinatorics defined by a composite format (Fk) expected by the decoder (DEC), a first half-image (I1) and a second half-image (I2) from of the image (I), - detecting (5) the composite format that has been used to format the stereoscopic image sequence, by determining the maximum correlation between the first half-image (I1) shifted and the second half-image (I2), the composite format that was used to format the stereoscopic image sequence then being the format (Fk) that was used to obtain the first and the second half-image if the comparison between this maximum of correlation and a threshold checks a condition. 2) The method according to claim 1, wherein a minimum (S4Dmin) of the absolute difference between the first half-image (I1) shifted and the half-image (I2) is determined, and this minimum (SADmin) is then compared. at a threshold (SSAD), if this minimum is below the threshold then the composite format in which the stereoscopic image sequence is formatted is the format (Fk) which has been used to obtain the first and second half-images, and if this minimum is greater than the threshold then another composite format (Fk) expected by the decoder (DEC) is considered and the steps to obtain the two half-images and to detect the composite format are repeated until all the formats composite (Fk) expected by decoder DEC are considered. 3) The method of claim 2, wherein the threshold (SSAD) is adaptive and, preferably, is equal to the factor-weighted PSNR signal-to-noise ratio. 4) Method according to one of claims 2 to 3, wherein the minimum (SADm; n) is calculated from the pixels belonging to a sliding window of dimension less than that of these half-images. 5) Method according to one of claims 1 to 4, the first and second half-images being subsampled prior to their correlation. 6) Method according to one of claims 2 to 5, which also comprises, in the case where one of the formats (Fk) expected by the decoder (DEC) was considered to be the format that was used to form the image (I), an inversion detection step of the first and second half-images (I1 and I2) during which if the sign of the minimum relative shift (vn, in) is negative then the first half-image (I1) is the left half-image (Ig) and the second half-image (I2) is the right half-image (Id), otherwise the first half-image (I1) is the right half-image (Id) and the second half-image (I2) is the left half-image (Ig). 7) Method according to one of claims 2 to 6, wherein the shift (wine) relative to the minimum and calculated for an image of the sequence of stereoscopic images is smoothed temporally by a low-pass filter. 8) A composite format detection device that has been used to format a sequence of stereoscopic images received by a decoder (DEC) in a format, the composite format combining a sub-sampled version of a right image (Id) with one another and a subsampled version of a left image (Ig) according to a combinatorial specific to this format, said left and right images being relative to the same image (I) of the stereoscopic image sequence, characterized in that comprises means for: - obtaining, according to a combinatorial defined by a composite format (Fk) expected by the decoder (DEC), a first half-image (I1) and a second half-image (I2) from the image (I), - detecting the composite format that has been used to format the stereoscopic image sequence, determining the maximum correlation between the first half-image (I1) shifted and the second half-image (I2), the format composite that was used to format the stereoscopic image sequence then being the form (Fk) that was used to obtain the first and second half-images if the comparison between this correlation maximum and a threshold satisfies a condition. 9) Device according to claim 8, which also comprises means for determining the correlation maximum which exploits data flow computation architectures of the SIMD type. 10) Computer program, characterized in that it comprises instructions for implementing, by a device, the method of claim 1 when said program is executed by a processor of the device. 11) Storage means, characterized in that they store a computer program comprising instructions for implementing, by a device, the method of claim 1, when said program is executed by a processor of the device.