KR100496513B1 - Image conversion method and image conversion system, encoding method and encoding system - Google Patents

Abstract

The present invention is a method for generating an image through the left and right eyes for stereoscopic display of the original 2D image,

a) identifying at least one object in the original image,

b) outlining the or each object;

c) defining a depth characteristic for the or each object;

d) forming two elongated images for viewing with the viewer's left and right eyes, and displacing the images or their respective selected regions by an amount determined in the lateral direction as a function of the depth characteristic of the or each object, respectively. Provide a way to.

Description

Method and system for generating stereoscopic display image, method and system for encoding video signal, and decoder

The present invention generally relates to stereoscopic image synthesis, specifically 2D encoding, transmission and decoding for the purpose of displaying stereoscopic images on a 2D or 3D stereoscopic display system. The present invention relates to a video conversion method.

Until recently, advances in the fields of small, high-performance video projection systems, image processing, digital video and liquid crystal panels have led to both single and multiple active and passive polarizing glasses. The emergence of many practical 3D display systems using auto stereoscopic displays and the like has been made possible.

3D display systems have now moved away from their technical curiosities and have been used as practical display systems in entertainment, commercial and scientific fields, and the demand for 3D media to be displayed on these devices has emerged. Traditionally, there are only two methods for creating 3D media (ie, media containing image information for at least two separate views of the same scene from different views). This includes:

1) Create two different city halls (usually in real time) using a computer.

2) Video or film with both cameras.

In the case of computer-generated images used in computer aided design (CAD), simulators, and video game machines, obtaining two separate images from different perspectives is not a complicated process.

Filmmaking of movies using cameras placed on both sides to generate 3D has been well understood for many years. However, there are many problems with this method. For example, filming or video becomes very difficult in 3D compared to 2D because of the limitations on the allowable distance between the closest object and the farthest object in the scene, and the framing problem (e.g. only on one camera). Visible problems) and inaccuracies in 3D image generation. Another problem is to maintain a smooth pan without causing error 3D artifacts due to latency between images from two cameras.

Due to this complexity and the high production and implementation costs, and the fact that very few 3D display systems are created for the domestic market and for commercial purposes yet, the production of 3D media in major large film and video producers is not very attractive. However, if the technology for reprocessing 2D films up to now in 3D is developed, not only will it be possible to make 3D at a very low cost compared to the cost of producing a new film in 3D from scratch? The vast amount of 2D film currently in storage can be re-edited for the film or video market.

However, it is advantageous to convert existing 2D images to be viewed as 3D images. One way to do this is to convert a 2D image into two left and right images using the "Cut and Paste" technique. This technique cuts out an object from the image, places it on the left and right sides at different angles, and then re-attaches it to the original image, creating the required separate image. However, this results in a blank area in the area previously occupied by the object in the image.

Reference to the drawings with reference to the description of the invention will be more readily understood.

Other implementations of the invention will be better understood with reference to the accompanying drawings.

1 illustrates an original image and a conventional left and right image for providing a 3D or stereoscopic image.

2 illustrates original and left and right images for providing a 3D image generated using a "cut & passt" technique.

3 illustrates an image and an original image generated by DDC (dynamic depth cueing) according to the present invention.

4 shows a left and right image and a resultant 3D image according to the present invention.

5 shows an image discontinuously distorted by a distortion mesh.

6 shows an image continuously distorted by the distortion mesh.

7 shows an example of mesh spatial displacement (MSD) data for left and right meshes.

8 shows a flowchart illustrating how MSD data is added to a video image in accordance with the present invention.

9 is a block diagram illustrating a method for integrating a DDC decoder into a video chain according to the present invention.

10 is a block diagram illustrating a possible implementation of a DDC decoder unit in accordance with the present invention for providing field sequential composite video output.

11 is a block diagram of another possible embodiment of a DDC decoder unit according to the present invention for providing field parallel composite video output.

12 is a block diagram of one type of MSD decoder according to the present invention.

13 shows how MSD data is encoded into a composite video signal.

14 shows a block diagram of an arrangement for providing a real-time generated DDC encoded video image.

15 shows a block diagram of an alternative arrangement for providing a real-time generated DDC encoded video image.

16 illustrates the principle of operation of a multiviewer 3D system.

FIG. 17 shows a block diagram showing the operating principle of a lenticular lens based 3D system.

18 and 19 respectively show multiple projector systems using biconvex lens assemblies.

20 illustrates a multiple projector system incorporated into a DDC decoder in accordance with the present invention.

It is therefore an object of the present invention to somehow overcome or minimize this problem.

With this in mind, the present invention is a method for generating an image of left and right eyes for stereoscopic display from an original 2D image, wherein the selected region of the original image is displaced by a predetermined amount to form a left and right eye image. A method of generating a stretched image is provided.

When two transformed images are respectively seen by the viewer's left and right eyes, these images can provide a 3D image without any blank area, such as would be produced by a "cut and attach" technique.

This document covers some of the key algorithmic processes involved in the conversion of 2D media into 3D format, a new synthetic data format suitable for communication and storage of new 3D media, and encodes this new 3D format in real time, Some hardware implementations for transmitting and decoding are also described.

The biggest advantage of this technology is the huge cost savings and media delivery. For example, only one camera is used for filming. The 2D to 3D conversion process makes it possible to package and deliver the virtually unchanged image medium except for the addition of small packets of 3D data that do not in any way interfere with the final 2D presentation process. That is, it allows viewing of images in 2D or 3D (with the use of shutter glasses or the like) without degrading the quality of a standard 2D TV, and can be displayed in 3D on 3D TV or on another display.

The final stage of the 2D to 3D conversion process is completed at the receiver in real time, and the increased amplification requirements for displaying 3D images are limited to TV decoders and do not adversely affect the channel handling capacity of the TV carrier.

According to another aspect of the present invention, there is provided a method for explaining a change made to an original 2D image for converting a 2D image into an extended image for stereoscopic display.

According to still another aspect of the present invention, there is provided a method for encoding a video signal of a 2D video to enable conversion of a 2D video video to an extended video for stereoscopic display.

According to still another aspect of the present invention, a video signal of a 2D video includes encoding data and extracting encoded data from the video signal to enable conversion of the 2D video to an extended video for stereoscopic display. A method of receiving is provided.

According to still another aspect of the present invention, there is provided a method of providing a stretched image in stereoscopic display by manipulating a 2D video image with encoded data.

According to another aspect of the invention, as a method for generating an image through the left and right eyes for stereoscopic display of the original 2D image,

a) identifying at least one object in the original image,

b) outlining the or each object;

c) defining a depth characteristic for the or each object;

d) forming two elongated images for viewing with the viewer's left and right eyes, and displacing the images or their respective selected regions by an amount determined in the lateral direction as a function of the depth characteristic of the or each object, respectively. do.

The pair of images may be mirrored or similar to each other so that stereoscopic 3D effects are optimal.

The image includes a plurality of objects each having the respective depth characteristics. The image can be converted to individual criteria. Alternatively, the series of related images in the video or film can be converted.

The image is digitized and stretched or transformed electronically by temporarily placing a mesh in the image, which initially includes a plurality of parallel transverse mesh lines and a parallel longitudinal mesh perpendicular to the transverse mesh line. With lines. The intersection of the mesh lines on the mesh provides a mesh subpoint. The image can move with this mesh, resulting in stretching of the underlying image due to distortion by the mesh. These mesh lines remain to provide a smooth stretch of the image. The displacement amount of each mesh subpoint from the initial position provides the transformed data of the original image. Submesh points may be laterally displaced.

The displacement of the mesh subpoint can also be determined by a mathematical algorithm to provide automatic conversion of the image. It is a further improvement over the method that the shadow, blurring and motion interpolation data can be added to the transform data including the direction for forced displacement information, field delay and motion displacement delay.

It is an advantage to be able to use existing video transmission systems in order to transmit video that can be viewed as 3D video. The present invention can be applied to the use in the image transmission for transmitting a video signal for providing a 2D image.

According to another aspect of the invention, from the transformation / extension process describing which objects in the image are selected for processing, how these objects are processed and their priorities or other objects and their depth characteristics, A method of generating a set of 'object scripting' data is provided. This scripting data is stored in the computer's memory for later use and sent to another site (assuming the other site has the same 2D image) for reprocessing the original 2D image or for playing back the 3D image.

Therefore, according to another aspect of the invention, an encoder for encoding a video signal for providing a 2D video image,

An encoder is provided that provides an encoded signal by adding transform data, which is data defining a displacement of each selected point of the video image, for converting the video image into a stretched image for stereoscopic display.

By adding this transformed data to the video signal, existing transmission systems can be used to transmit the encoded signal. Various structures may be provided for adding the transform data to the video signal. For example, such data is included in a blank line of a video image transmitted in the up and down, horizontal sync period or horizontal overscan region of each line of the image.

The present invention is not limited to the conversion of existing 2D video images. Rather, the process can be easily used to generate transformed data while simultaneously generating a 2D video image.

Therefore, according to another aspect of the present invention, as a method for generating a 2D video image encoded with 3D converted data,

Capturing video images from a plurality of video cameras,

Comparing the video images from each video camera to produce transform data, the transform data defining displacements of respective points of the video image for converting the video image into a stretched image for stereoscopic display;

And combining the video signal from one of the video cameras with the transformed data to produce an encoded video signal.

In still another aspect of the present invention, there is provided a method of generating a 2D video image encoded with 3D converted data.

Capturing video images of left and right eyes from a stereoscopic video camera,

Comparing left and right eye video images from a stereoscopic video camera to generate converted data, wherein the converted data define displacements of respective points of the video image for converting the video image into an extended image for stereoscopic display;

And combining the video signal from the video camera with the transformed data to produce an encoded video signal.

In still another aspect of the present invention, there is provided a system for generating a 2D video signal encoded with 3D converted data.

At least first and second video cameras disposed laterally from each other,

To generate converted data, to receive data from the video camera and to compare the data, to generate converted data, and to convert the video image into an extended image for stereoscopic display. Conversion means for defining the displacement of each point,

A system is provided comprising an encoder for combining an video signal from said one video camera with transform data from said conversion means to produce an encoded video signal.

If a 2D video image encoded with 3D converted data is required for only a single viewer, only two cameras are required, and each camera represents the viewing seen by the viewer's left and right eyes.

In still another aspect of the present invention, there is provided a system for generating a 2D video signal encoded with 3D transform data.

Stereoscopic video camera,

Generating transform data, receiving data from the video camera and comparing the data, generating transform data, and defining displacements of each point of the video image for converting the video image to an extended image for stereoscopic display. Conversion means,

A system is provided comprising an encoder for combining the video signal with transform data from the conversion means to produce an encoded video signal.

According to another aspect of the invention, a decoder for decoding a video signal to provide stereoscopic display, the signal further comprising transformed data for providing a 2D video image and for converting the video image, wherein the converted data is video Define a displacement of each point of the video image for converting the image into an extended image for stereoscopic display,

a) means for receiving a video signal,

b) A decoder is provided that includes decoding means for reading the converted data and controlling the video signal to provide the converted video signal.

Decoder is

a) an RGB or component video converter for converting the video signal into a separate video component,

b) an analog-to-digital converter for converting each video component to each digital signal;

c) digital storage means for storing said digital signal.

The decoding means controls the variable frequency clock means, which controls the read out rate of the digital storage means, causing the storage means to be read at a variable rate. This results in a video image that is stretched or compressed according to the transform data.

Alternatively, RGB or video components are read into the storage means at a variable rate and from the storage means at a fixed rate.

Decoder processes single video lines or multiple lines such as complex fields or frames. In this case, the full mesh from the transform data is restored to pixel distortion (lateral shifts) calculated over the complete field or frame.

The storage means may be in the form of a dual port RAM store.

The digital-to-analog converting means converts the read digital signal into a converted video signal for viewing on the viewing means. The viewing means comprises a television or other screen for watching the converted video image. The viewing means may further include shutter glasses controlled by the decoder to allow the converted video image to be viewed as a stereoscopic image.

Alternatively, the decoder may comprise parallel storage means for storing a digital signal for each of the converted left and right video images. The viewing means comprises a display unit for simultaneously projecting left and right video images.

The decoder means comprises separating means for separating the transformed data from the video signal.

According to another feature of the present invention, a three-dimensional image display system,

a) an encoder for providing a conversion signal to a video signal to encode the video signal, the conversion data defining a displacement of each point of the video image for converting the video image into an extended image for stereoscopic display;

b) A system is provided that includes a decoder that separates transform data from a video signal and converts the video signal as a function of the transform data.

According to still another aspect of the present invention, there is provided a multiviewer stereoscopic display system.

a) a decoder for decoding a video signal to provide a stereoscopic display, the signal further comprising transform data for providing a 2D video image and transforming the video image, wherein the converted data is extended for stereoscopic display of the video image; A decoder defining a displacement of each point of a video image for conversion to an image, the decoder comprising means for receiving a video signal and decoding means for reading the converted data and controlling the video signal to provide a converted video signal A system is provided.

The method according to the invention which can convert a 2D or "monoscopic" video signal into a 3D or "stereo" video signal is referred to as the following DDC but this technique is not limited thereto.

a) 3D generation-techniques and procedures for converting 2D images into 3D stereoscopic pairs and generating 3D transformed data.

b) 3D Scripting-A technique that describes the changes required in a 2D image to convert a 2D image into a 3D stereoscopic image pair. It describes which objects are selected and how they are processed and provides a means of storing 3D data.

c) 3D data encoding-a technique for adding information to a 2D video image in a defined format. The resulting modified video is compatible with existing video recording, editing, transmitting and receiving systems.

d) 3D standardized protocol-3D converted data is added to 2D video using a defined data format or standardized protocol. This protocol is a world standard for adding 3D transformed data to 2D transmission.

e) 3D data encoding-A technology that can receive 3D video images and transformed data, and extract 3D stereoscopic images by extracting information added to 2D video images.

f) 3D synthesis-A technique for adjusting 2D video images using transformed data to synthesize 3D stereoscopic image pairs.

In order to convert a 2D image into a simulated 3D image, it is necessary to modify the original image to generate two slightly different images and to present these separated images independently to each of the left and right eyes.

The correction of the original image consists of a lateral shift of the objects in the image plane (located on the projection or viewing screen) to effect the depth.

In order for an object in the image to appear farther from the viewer with respect to the image plane, it is necessary to shift the object in the image slightly to the left to the left eye and slightly to the right to the right eye. This is shown in FIG. In order for the object to appear closer to the viewer, it is necessary to shift the object in the image of the left eye to the right and the object in the image of the right eye to the left. In order to place an object on the image plane, the object is placed on the image at the same position of both eyes.

In the real world, when viewing an object, the viewer uses focus information. However, in the simulated 3D, this information is not presented, and if there are too many lateral shifts, especially in order to bring the object closer to the viewer, the object appears broken into two separate images and the 3D effect is lost.

The left and right images may be generated using a computer. The image is first digitized with a video digitizer and the final data is stored in memory. Two new images can then be generated.

The easiest way to create a new left and right image with the desired side shift is to "cut" the object from the image and "attach" back to the required lateral displacement, which is known as a "cut and attach" technique. This can be done by first defining the position of the object as moved by identification, then "cutting" the object from the image and moving it to the side.

As shown in Fig. 2, the problem with this simple technique is that once the selected object is moved, the background is removed and there is also a blank area in the background.

According to the invention, the objects in the image are " extended " to provide the necessary side shifts and to maintain the original background details. The final side distortions of the image are mathematically smoothed and consequently perceived as "real" 3D with no visible artifacts.

To better visualize the effect of this stretching on the image, imagine that the image to be converted is printed on a thin rubber sheet. It is possible to pick up a point on the image surface in the vicinity of an object and stretch it to a new position, for example to the right of the original position. Therefore, as shown in FIG. 3, the image portion on the right side of the object is compressed and stretched to the left side. For the viewer, the object now appears distorted if seen with both eyes.

As shown in FIG. 4, however, if a similar but opposing image is presented to the other eye, the viewer does not see the distorted image, but rather sees an object with 3D characteristics.

The "extension" of the object in the image can also be done electronically. The objects in each video frame are identified by first outlining them. In each object, the depth or mesh distortion characteristic is also defined. Stretching can be done by allowing the operator to stretch the image and see the final 3D image effect in real time. Operator skill and artistic adjustments can be used to determine the 3D impact of the final image and the resulting video sequence. While individual video frames can be converted manually (i.e. not in real time), one can expect that a series of associated frames that form a video "clip" can be converted automatically (i.e. in real time). have.

The operator defines the start and end frames of the video clip to be converted. They are also determined by the relative depth of each object with respect to the image plane in the start and end frames. Video clips can be processed using the start and end positions and depths of each object in the clip to interpolate the desired stretch or manipulation of the intermediate frame.

In the case of multiple overlapping objects with different depths, the foreground objects are given priority. Because raw 2D images are captured by a single camera, pixel information is automatically prioritized in the foreground.

The "height" of such an image can be performed electronically by manipulating the digitized image. The mesh (grid) is temporarily placed on the image to be distorted, for example, as the common coordinates of each row and column of the mesh become zero before distortion. The X-coordinate of the mesh is changed, resulting in distortion of the mesh base image. Rather than moving only the image area directly below the line, resulting in discontinuities (FIG. 5), adjacent mesh lines may also be moved to cause smooth distortion (FIG. 6).

Roughness of the distortion mesh determines the degree of 3D effect. The coarser the mesh, the more other objects that are adjacent to the object spin. This results in a bad 3D effect on the final image. A denser mesh will result in sharper edges of the object, better 3D image effects, but greater edge discontinuities. For the purpose of explanation, the size of the distortion mesh may be assumed to be 16 × 16. Information about each subpoint on the mesh (ie post-distorted coordinate position) is encoded to create background and foreground subpoints. For example, 4 bits can be used for subpoint encoding resulting in 16 different levels, 4 backgrounds and 12 foregrounds. The format of the subpoint encoding is determined by experiment and can be adjusted to suit the application.

Alternatively, this mesh distortion process may be defined by a mathematical algorithm that enables automatic processing of the image.

Once the mesh distortion of the left eye is determined, the coordinates of the right eye's distortion are easily obtained by scalar multiplication of -1 (i.e. shifted in opposite lateral directions by the same amount) to the matrixTM. It is calculated automatically. This is shown in FIG.

A matrix formed from the relative horizontal deviation of each intersection of the distorted meshes defines mesh space displacement (MSD) data.

What is needed to fully define and reproduce the final 3D image is to provide the original unchanged 2D image and MSD data. Thus, 3D images are stored, transmitted, generated, edited and calculated in consideration of MSD data files related to 2D images.

Therefore, it is possible to encode MSD data into each video frame to store and transmit 3D video through a conventional 2D video system. Since the original 2D video image can be stored and transmitted unchanged, the final video can be fully used in existing video and television systems. Existing 2D TV receivers can also display a normal picture.

Many existing technologies can be used to add MSD data to 2D images so that they are not detected by the viewer and are compatible with existing video standards. These techniques are not limiting,

a) on the extra lines above and below the picture set to black level in a manner similar to the addition of "teletext data,"

b) invisible overscan areas to the left and right of each image, and

c) in a horizontal sync cycle in accordance with the British Broadcasting Union "sound in sync" system.

Inserting MSD information.

In the future, with the introduction of digital HDTV, extra digital data frames will be available for inserting MSD data.

The process of adding MSD data to a 2D video image to form a DDC encoded video frame is shown in FIG. 8.

The amount of MSD data is small enough to be estimated at about 100 bytes per frame. It can be further compressed for storage and transmission using standard data compression techniques such as run length or differential encoding as needed.

Since the amount of data is small, the desired data rate is also lowered. In addition, since MSD data does not easily change over multiple frames, it is possible to further reduce the required data using spatial and temporal compression. The exact time relationship between the MSD data and its associated frame is not important and the displacement error of one frame is probably acceptable.

Due to the small amount of data, low data rate, and non-deterministic alignment, MSD data is sent over multiple frames, that is, four frames with one quarter of the information in each frame.

A block diagram illustrating how the DDC decoder is incorporated into the video chain is shown in FIG. 9. Any existing video source, i.e., DDC encoded video obtained via the earth, satellite, etc., is applied to the input of the DDC decoder. One output of the decoder is a standard video waveform (or video modulated wireless signal) that can drive a standard TV display and show 3D video to a viewer wearing shutter glasses synchronized by a DDC decoder.

Additionally outputs are available from DDC decoders for driving other 3D displays, such as virtual reality headsets or autostereoscopic displays as described in Australian application 6667/94.

One possible embodiment of a DDC decoder is shown in FIG. 10. Incoming video, which can be either PAL or NTSC in the S-Video format, can be applied to a composite RGB or component video converter. Each of the RGB or component video outputs is fed to an analog-to-digital converter and the digital outputs are sent to the input port of a dual-port RAM line storage. Each line of digital video data enters RAM at a constant rate. Data is read from line storage at a rate determined by a variable frequency clock controlled by the output of the MSD decoder.

The effect of reading line data from RAM at variable rates causes the final video to be stretched or compressed depending on the MSD data.

The converted data is then applied to a digital / analog converter and a PAL / NTSC encoder. The final 3D field sequential composite video signal is applied to the display. (Note: this process can be operated with a video signal that is read at a variable rate and read at a fixed rate. The output from the line storage is variable rate. It is necessary to convert the incoming composite video signal to RGB or composite video because the chrominance frequency changes when read as, causing display error.)

In particular, the converted data can be transmitted at the vertical blanking interval of the television signal.

DDC decoding can be realized using field or frame storage. In this case, the entire mesh from the MSD data is reconstructed with pixel distortion (lateral shift) calculated over the entire field or frame.

The 3D stereoscopic image pair is then displayed from the final RGB or component video output.

The shutter eyeglasses controller provides an ultraviolet light source that provides timing pulses to the shutter eyeglasses. The controller is synchronized by the PAL / NTSC encoder. Additionally, the controller instructs the shutter glasses to open during scenes that are not 3D encoded and not suitable for 3D encoding, providing improved image quality during these portions of the video sequence.

11 shows a block diagram of a DDC decoder that produces a 3D field parallel composite video output. The MSD decoder generates two variable rate clocks, one for the left and one for the right RAM line storage. This type of decoder is suitable for replacing field storage in our existing stereoscopic 3D displays. This technique provides left and right video sources that are at the same field rate as the original 2D video source, ie field sequential video output is generated.

Alternatively, non-field video output is produced at an output of a scanning rate of higher resolution than 2D video.

12 shows a block diagram of one type of MSD decoder. In this case, assume that the MSD data is encoded into the composite video signal on the first 16 lines of the video signal of FIG. Incoming composite video is sent to a sink separator that provides vertical and horizontal timing signals for the microprocessor. In addition, video is fed to black level clamp circuits, comparators and level shifters. The output from the level shifter is a TTL level signal with serially encoded MSD data in lines 1 to 16 of the video signal. After the microprocessor determines the horizontal sync pulse on line 1 that reads the next 16 bytes, the microprocessor loop waits for the horizontal sync pulse. A similar processor is repeated on the next 15 lines until the MSD data is read. Based on the received MSD data, the microprocessor provides a variable speed clock to digital video line storage on each sequential video line. The microprocessor maintains the index at which the video line is being processed by counting video line sync pulses.

Depth perception of 3D images depends on the viewer. Also, when viewing 3D images with shutter glasses, the "strength" of the 3D images requires adjustment of the viewing distance. The intensity of the 3D image may be changed by a remote control device that causes the intensity of the 3D image to be changed by the viewer by an algorithm applied by the microprocessor. The algorithm changes the magnitude of each element in the MSD matrix to change the intensity of the 3D effect. This preference is expected to be maintained by the decoder unit once the preferences of a particular viewer are entered.

There are many techniques for real-time generation of DDC encoded video images. In one technique the distance between the camera lens and the array of additional Charge Coupled Devices (CCD) is varied (FIG. 14). This produces a series of frames of each object in the image at the variable focus stage. A sharpness detection algorithm is then run through a series of frames, and the sharpness index of each object in the image is determined. Determines in which frame each object is the sharpest, which indicates which focal plane the object is on. This information is used to form the MSD data.

15 illustrates another technique where two video cameras are used to create separate left and right eye images. The light emission information from each camera is digitized and transferred to the line storage device. An auto correlator or similar operation finds a match by comparing the bit patterns in two line storages (16 elements left and right). The difference (distance) between video patterns representing objects in an image is used to generate MSD data. One of the camera outputs combines MSD data in real time to produce DDC encoded video.

Alternatively, a stereoscopic video camera may be used instead of two video cameras.

DDC is used to overcome the serious problems of the existing non eye-tracking autostereoscopic 3D multi-viewer system. Such a system creates a repeated sequence of left and right images shown in FIG. 16 and provides 3D images. The distance between each successive image is 65 mm equal to the distance between viewer eyes. Therefore, the viewer at position A can see the correctly processed 3D image.

However, when the viewer moves 32mm to the side or is in the B position, the left image is visible on the right and the right image is visible on the left. In other words, the viewer sees an inverse 3D image. Inverse 3D images are very inconvenient to see and cause viewer headaches after a short time.

Most multi-viewer autostereoscopic systems have this drawback. In particular, they are based on biconvex lenses and grid type image separators. Figure 17 illustrates a multi-viewer, biconvex lens based automated stereoscopic imaging system. The image from the left projector passes through the first biconvex lens and is focused on the surface of the matt screen. The second biconvex lens focuses the image again to form a vertical strip of light at the viewing distance. The second projector with the right image illuminates the first biconvex lens but the final right eye image on the viewer's surface is shifted 65 mm from the left image by lateral displacement between the two projectors. This sequence of alternating left and right images is repeated 65 mm apart.

The viewer in the correct position can see the 3D image, but the reverse 3D image appears when the viewer moves or becomes incorrect at the position described above.

Indeed, the first time a user sits to view a biconvex lens based 3D system, it is difficult for the viewer to determine whether an accurate or inverted 3D image is seen. Only after the viewer feels uncomfortable does he know that he is inaccurate.

In addition, it is difficult for a viewer to maintain an accurate viewing position for a long time. In addition, it is necessary for the viewer to be positioned at the correct distance from the second biconvex lens, because if it is viewed at an incorrect distance, a Morae effect or a crosstalk phenomenon occurs.

Another problem with biconvex lens based systems is the resolution. The resolution is limited by the pitch of each "lens-let" in the total biconvex lens in 1 mm increments.

Instead of just projecting the left and right images, consider a biconvex lens based system for generating a series of images, 1, 2, 3 and 4 images, each 65 mm apart, using multiple (4 as an example) projectors (FIG. 18). . The original scene is recorded using four cameras arranged in the same sequence and interval. A. The viewer at position B, or D sees the correct 3D image, while the viewer at position C sees the reverse 3D image.

This is a substantial improvement over conventional left and right systems, where an acceptable 3D image can now be viewed at three times the lateral distance. It should be noted that, as in conventional left and right systems, the viewer does not know that the D position provides an inverted 3D image until it feels uncomfortable.

If four projectors were replaced with a "null" image (black) as in Figure 19, then positions A and B would be the same as before. The position C shows a flat image in the right eye and a black image in the left eye so that the viewer can see it without inconvenience. Similarly, location D produces a planar image but no inverse 3D imaging effect. Thus, 50% of the viewer's position produces an accurate 3D image, while the other 50% has a system that produces slightly less image, but unless objectioned, the image and inverse 3D effects are eliminated.

By increasing the number of projectors and including null images, the lateral distance for viewing accurate 3D images is extended and the inverse 3D effect is excluded.

However, since the transmission / recording bandwidth required to provide a video image to each projector becomes impractical as the number of projectors increases, it is considered that an embodiment of such a multiple projector system is not practical.

The limitation of this approach can be overcome by transmitting DDC encoded 2D images and synthesizing the desired number of projection images using DDC. While wide bandwidth is required for the DDC decoder, the original 2D bandwidth is maintained at the transmitter and / or recorder.

A multiviewer 3D biconvex lens based system using a DDC decoder is shown in FIG. 20.

DDC encoding / decoding enables the generation of a sequence of video images representing the range of possible images from the leftmost to the rightmost of the original image as follows.

(L, L1, L2 ……… R2, R1, R)

In summary, some of the applications are described as follows:

DDC is a term for the type of data derived from the 2D to 3D transformation process that is in the middle of the transformation. At this stage, the data consists of the original video signal and the data packet (encoded in digital or analog form) so that this additional data is all that is needed to direct the particular electronic hardware and / or software to fulfill the conversion purpose. The resulting 3D information can be in the form of a field sequential (ie left or right) video format, two separate video streams, a line-to-line system (ie one line from the left field, one line from the right field) or one of the other advantageous formats. Take

With careful design of the format of the transform data packet, it is possible to have this additional data so that it is unknown when displayed on a standard TV. This enables 3D television transmission without overturning the infrastructure of existing televisions. The decoder can be located near the viewing device (eg TV) and essentially be a "black box" that intercepts the transmitted signal that is encoded and output to the TV for viewing. Therefore, upgrading an existing 2D TV or television network structure is simply accomplished by adding a "black box" to each TV.

When providing a medium of a multi-image type autostereoscopic 3D display system, these systems rely on providing a somewhat different perspective to the multi-image. If there are many different images (eg 8-16) it can be very efficient in terms of allowing true multi-viewer capability. The big disadvantage is that the provision of the medium is very difficult even with complex video compression techniques because it requires many different views of everything available at the same time. However, if a DDC decoder is used to create a 3D medium, it becomes possible to generate as many discrete perspective views as required and such imaging equipment, namely a TV and a video recorder, as normal 2D images. The viewer will not be aware of any deformation of the transmitted image.

DDC encoded standard 2D video images have the following characteristics:

DDC-encoded 2D video can be received as normal video on standard video equipment, ie TVs and video recorders. The viewer does not know any modifications to the transmitted image.

DDC encoded video is fully compatible with existing video, editing, recording, receiving and tactical systems and technologies. So DDC-encoded 2D video images fit all existing analog video and television technologies.

DDC encoded 2D video can be introduced to the market in a similar way to the introduction of color TV and stereo sound. However, a TV set suitable for a DDC decoder (the viewer wears appropriate viewing glasses), or a 3D TV displays a 3D image.

DDC encoding allows for continuous transmission between scenes, which is beneficial for 3D encoding and can be displayed even more advantageously in 2D. This transition is not known to the viewer.

-DDC encoded video can be displayed on all existing 3D displays and is suitable for multi-viewer systems.

DDC encoding maintains the line and field standards of the original video source.

DDC encoding does not reduce the image update frequency as in the case of encoding a 3D video image in left and right field continuous format.

Claims (59)

A method of generating a left and right eye image for stereoscopic display from an original 2D image, Generating a stretched image by displacing the selected region of the original image by a determined amount and direction, The stretched image forms the left and right eye images, And each of the left and right eye images includes a portion of the original 2D image to be compressed and a portion of the original 2D image to be stretched. A method of generating a left and right eye image for stereoscopic display from an original 2D image, a) identifying at least one object in the original image; b) outlining the or each object; c) defining depth characteristics of the or each object; d) displacing the selected area of each or each object as a function of the depth characteristic of each object by an amount determined in the lateral direction to form two extended images for viewing by the viewer's left and right eyes; , The or each object in the stretched image for viewing with the viewer's left eye includes a compressed portion and a stretched portion, and the or each object in the stretched image for viewing with the viewer's right eye is compressed And a portion and an elongated portion. The method according to claim 2, Displacing the original 2D image while avoiding any blank areas in the image. The method according to claim 2, One of the stretched images is a mirror image of the other stretched image. The method according to claim 2, And each depth object is provided with an individual depth characteristic. The method according to claim 2, Wherein said steps are performed on a plurality of 2D images. A method of generating a stereoscopic left and right eye image from a digitized 2D image, Forming a mesh through the digitized image, wherein the mesh has a plurality of parallel transverse mesh lines and a plurality of parallel longitudinal meshes from the beginning, the transverse lines being positioned perpendicular to the longitudinal lines Intersecting to form a plurality of sub-points, Extending the underlying image by distorting the mesh by moving the subpoints Method comprising a. The method according to claim 7, And wherein said mesh lines between adjacent subpoints remain continuously in the result of any distortion. The method according to claim 7, The distortion distorting the subpoint transversely to distort the mesh. The method according to claim 7, The amount of distortion of each subpoint is used to generate data that can convert the original 2D image into a left and right eye image for stereoscopic display. And the data describes which objects are processed in the image, how the objects are processed, priorities for other objects of the object, and their depth characteristics. The method according to claim 10, Wherein the distortion required by each subpoint is defined by a mathematical algorithm. The method according to claim 10, The raw 2D image and the transformed data can be transmitted according to the standard 2D technology. A system for generating a left and right eye image for stereoscopic display from an original 2D image, Means for selecting an area of the original image; Means for displacing the selected area by a determined amount and direction to produce an extended image, The stretched image forms the left and right eye images, Each of the left and right eye images includes a portion of the compressed original 2D image and a portion of the expanded original 2D image. A system for generating a left and right eye image for stereoscopic display from an original 2D image, Means for identifying an object in the original image; Means for defining depth characteristics for each object, Means for displacing the selected area of each object only as a function of the depth characteristic of each object to form two elongated images for viewing with the viewer's left and right eyes, in the transverse direction only; Wherein each object in each of the two stretched images includes a portion to be compressed and a portion to be stretched. The method according to claim 14, Means for generating a mirror image of one stretched image. The method according to claim 14, Said means for defining depth characteristics may define individual depth characteristics for each object in said image. The method according to claim 14, The system is capable of converting a plurality of 2D images. A method of generating a left and right eye image for stereoscopic display from an original 2D image, Compressing the first selection region of the 2D image by the determined amount and extending the second selection region of the 2D image by the determined amount to form an extended left eye image; Compressing the third selection region of the 2D image by the determined amount and compressing the fourth selection region of the 2D image by the determined amount to form an extended right eye image. A method of generating a left and right eye image for stereoscopic display from an original 2D image, a) identifying an object in the original image; b) outlining the object; c) defining a depth characteristic for the object; d) compressing and stretching each of the first and second selection regions of the object as a function of the depth characteristic of the object by an amount determined in a transverse direction to form an extended image for viewing by the viewer's left eye; , e) compressing and stretching the third and fourth selected regions of the object by the determined amount in the lateral direction as a function of the depth characteristic of the object, respectively, to form an extended image for viewing by the viewer's right eye And comprising a step. In the method of encoding the video signal with the conversion data to help convert the video signal of the 2D image to the left and right eye image for stereoscopic display, Adding converted data to a video signal of an image to provide an encoded signal, And the converted data defines a displacement of each selected point of the image for converting the image into a format suitable for displaying a stereoscopic image. The method of claim 20, And said transform data is transmitted on a blank line at the top and / or bottom of a standard 2D image to be transmitted. The method of claim 20, And the converted data is transmitted in a horizontal synchronization period of a standard 2D image to be transmitted. The method of claim 20, And the converted data is transmitted over a horizontal overscan area of each line of a standard 2D image to be transmitted. The method of claim 20, And said transform data is transmitted at a vertical blanking interval of a television signal. A method for generating a 2D video image encoded with 3D converted data, Capturing video images from a plurality of video cameras, Comparing the video images from individual video cameras to generate conversion data defining displacements of individual points of the video image for converting the video image into a stereoscopic left and right eye image; Combining the video image from one of the video cameras with the converted data to generate an encoded video signal Video image generation method comprising a. A method of decoding a video signal for providing stereoscopic display, Receiving a video signal comprising a 2D image and the conversion data for the conversion of the 2D image, the conversion data defining the displacement of each point of the 2D image for converting the 2D image into a format suitable for stereoscopic display Steps, Reading the converted data from the video signal; Generating left and right eye images from the 2D image for display by displacing an object in the 2D image according to the converted data. And a video signal encoding method. The method of claim 26, Converting the 2D video image into an RGB component, Converting each component into a respective digital signal, And storing the digital signals before generating the left and right eye images. The method of claim 27, And said digital signal is read out at a variable rate (speed) from a storage as a function of said converted data and outputted. The method of claim 27, And said digital signal is input with a variable rate (speed) read out to a storage device as a function of said converted data. The method of claim 27, And said digital signal is converted to analog for viewing on an analog system. The method of claim 26, Separating the converted data from the video signal. A system for generating a 2D video signal encoded with 3D converted data, At least first and second video cameras which are laterally displaced relative to one another, A conversion circuit for generating converted data, receiving data from the video camera, and comparing the data to generate the converted data, wherein the converted data converts the video image into an extended image for stereoscopic image display. A conversion circuit defining a displacement of an individual point of the video image from one of the cameras, An encoder that combines the video signal from the one video camera with the transform data from the conversion circuit to produce an encoded video signal Video signal generation system comprising a. A decoder for decoding a video signal that provides stereoscopic display and further includes converted data for providing a 2D video image and converting the 2D video image, wherein the converted data converts the 2D video image into a format suitable for stereoscopic display. A decoder for defining a displacement of an individual point of the 2D video image for A receiver for receiving the video signal; And a decoder circuit for reading the converted data and controlling the video signal to produce a converted video signal. The method of claim 33, wherein the decoder circuit, An RGB or component video converter for converting the video signal into individual video components; An analog-to-digital converter for converting each video component into a respective digital signal, And a digital storage device for storing each digital signal. The method of claim 34, wherein Further comprising circuitry suitable for controlling a variable frequency clock, The variable frequency clock controls the read rate of the digital storage device, The digital signal reads the digital storage device at a variable rate. The method of claim 34, wherein Further comprising circuitry suitable for controlling a variable frequency clock, The variable frequency clock controls the read at the rate of the digital storage device, The separated video component is read into the digital storage device at a variable rate The method of claim 34, wherein And the digital storage device is a dual port RAM line store. The method according to claim 33, Wherein said decoder processes a single video line. The method according to claim 33, Wherein said decoder processes a plurality of video lines. The method according to claim 33, And the decoder circuitry comprises an analog to digital converter for converting the video signal into a converted video signal for viewing on a display. The method according to claim 33, The decoder circuit comprises a parallel storage device for storing the digital signals for the left and right images of the converted video signal, respectively. The method according to claim 33, And a separating circuit for separating the converted data from the video signal. In the stereoscopic image display system, An encoder for encoding a video signal for providing the video image with converted data defining a displacement of individual points of the video image for converting the video image into an extended image suitable for stereoscopic display; A decoder that separates the transform data from the video signal and converts the video signal as a function of the transform data Stereoscopic image display system comprising a. In a multi-viewer stereoscopic display system, A decoder for decoding the video signal for providing stereoscopic display, The video signal further comprises conversion data for providing and converting a 2D video image, wherein the conversion data defines a displacement of individual points of the video image for converting the video signal into an extended image for stereoscopic display. , And the decoder comprises a receiver for receiving the video signal and a decoding circuit for reading the converted data and controlling the video signal to provide a converted video signal. A method for generating a 2D video image encoded with 3D converted data, Capturing left and right eye video images from a stereoscopic video camera; Comparing the left and right eye video images from the stereoscopic video camera to generate transformed data defining displacements of individual points of the video image for converting the video image into a left and right eye image for stereoscopic image display; And combining the video signal from the video camera with the converted data to produce an encoded video signal. A system for generating a 2D video signal encoded with 3D converted data, A stereoscopic video camera for capturing video images, A converter for generating converted data, receiving data from the video camera, and comparing the data to generate the converted data, wherein the converted data converts the video image into an extended image for stereoscopic display. A transducer defining the displacement of the individual points of And an encoder for combining the converted data from the converter with the video image to produce an encoded video signal. A method of decoding a video signal for providing stereoscopic display, Receiving a video signal including a 2D video image; Receiving conversion data for 2D video signal conversion, the conversion data defining a displacement of each point of the 2D video image for converting the 2D video image into a format suitable for stereoscopic display; Generating left and right eye images from the 2D video image for display by displacing the object in the 2D video image according to the converted data. Video signal decoding method comprising a. A decoder for decoding a video signal for providing stereoscopic display, A receiver for receiving a 2D video image and a video signal for supplying converted data for converting the 2D video image, wherein the converted data is used to convert the 2D video image into a format suitable for stereoscopic display. A receiver defining the displacement of each point, A decoder that reads the converted data and controls the video signal to provide a converted video signal Decoder comprising a. A system for generating a 2D video signal encoded with 3D converted data, At least first and second video cameras which are laterally displaced relative to one another, A processor for generating converted data and receiving the data from the video camera and comparing the data to generate the converted data, wherein the converted data converts the video image into an extended image for stereoscopic display. A processor defining a displacement of an individual point of the video image from one of the An encoder that combines the video signal from the one video camera with the transform data from the processor to produce an encoded video signal Video signal generation system comprising a. A decoder for decoding a video signal for providing stereoscopic display, the video signal further comprising conversion data for providing a 2D video image and converting the 2D video image, wherein the converted data stereoscopically displays the 2D video image. A decoder for defining the displacement of individual points of the 2D video image for conversion into a format suitable for A receiver for receiving the video signal; A decoder circuit for reading the converted data and controlling the video signal to provide a converted video signal Decoder comprising a. The method of claim 50, wherein the decoder circuit, An RGB or component video converter for converting the video signal into a separate video component; An analog-to-digital converter for converting each video component into its own digital signal, And a digital storage device for storing each digital signal. The method of claim 51, Further comprising circuitry suitable for controlling a variable frequency clock, The variable frequency clock controls the read rate of the digital storage device, The digital signal reads the digital storage device at a variable rate. The method of claim 51, Further comprising circuitry suitable for controlling a variable frequency clock, The variable frequency clock controls the read at the rate of the digital storage device, And the separated video component is read into the digital storage device at a variable rate. The method of claim 51, And the digital storage device is a dual port RAM line store. The method of claim 50, Wherein said decoder processes a single video line. The method of claim 50, Wherein said decoder processes a plurality of video lines. The method of claim 50, And the decoder circuitry comprises an analog to digital converter for converting the video signal into a converted video signal for viewing on a display. The method of claim 50, The decoder circuit comprises a parallel storage device for storing the digital signals for the left and right images of the converted video signal, respectively. The method of claim 50, And a separating circuit for separating the converted data from the video signal.