WO2021031455A1 - System, method and device for realizing three-dimensional augmented reality of multi-channel video fusion - Google Patents

Abstract

Disclosed are a system, method and device for realizing three-dimensional augmented reality of multi-channel video fusion. Video streams are generated for images photographed by a multi-channel video acquisition device; data synchronization is carried out on the returned video streams; after the video frame texture acquired by calculating the video stream is mapped to the three-dimensional scene, texture reconstruction is carried out on the texture of the overlapping region of the texture mapping according to the weight condition contributed by the texture of projectors, so that the overlapping condition of the projection regions generated among the plurality of projectors is reduced, and the display effect of the rendered three-dimensional scene is improved.

Description

System, method and equipment for realizing three-dimensional augmented reality of multi-channel video fusion Technical field

The embodiments of the present application relate to the field of computer graphics, and in particular to a system, method, and device for realizing three-dimensional augmented reality of multi-channel video fusion.

Background technique

The Virtual Reality Fusion (MR) technology matches and synthesizes the virtual environment and the real environment, reduces the workload of 3D modeling, and improves the user experience and credibility with the help of real scenes and objects. With the current popularity of video images, the discussion and research of MR technology has attracted more attention.

Video fusion technology utilizes existing video images and merges them into a three-dimensional virtual environment, which can realize unified and deep video integration.

When fusing a video into a three-dimensional scene, in the prior art, there is a situation in which projection areas overlap between multiple projectors, which will adversely affect the display of the scene.

Summary of the invention

The embodiments of the application provide a system, method, and device for realizing three-dimensional augmented reality of multi-channel video fusion, which reconstruct the texture of the overlapped area of the texture mapping, thereby fusing the texture mapping, and reducing the occurrence of projection areas between multiple projectors. Overlap, and adversely affect the display of the scene

In the first aspect, the embodiments of the present application provide a three-dimensional augmented reality system that realizes multi-channel video fusion, including a three-dimensional scene system, a video real-time solution system, an image projection system, an image fusion system, and a virtual three-dimensional rendering system, in which:

Three-dimensional scene system, save the three-dimensional scene of the scene;

Video real-time solution system, which performs real-time solution on the received video stream to obtain video frames;

The image projection system is used to determine the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional scene, and perform texture mapping on the video frame in the three-dimensional scene according to the mapping relationship to complete the image projection of the video frame;

The image fusion system is used to determine the texture value corresponding to each three-dimensional point in the overlap area of the texture mapping, and reconstruct the texture of the texture mapping overlap area according to the texture value to complete the fusion of the texture mapping, wherein the texture value is based on the overlap area of the texture mapping The weight of the texture contribution of the corresponding projector is determined;

The virtual 3D rendering system renders the fused texture and 3D scene.

Further, the system also includes a data synchronization system, and the video stream is generated by a multi-channel image acquisition system collecting images at multiple locations on the spot, and the video generated by the multi-channel image acquisition system flows through the multi-channel image in real time. The backhaul control system performs backhaul; the data synchronization system performs data synchronization on the returned video stream, and the data synchronization is specifically time synchronization, so that the returned video streams of the same batch are located in the same time slice space.

Further, the video real-time solution system includes a video frame extraction module and a hardware decoder, wherein:

The video frame extraction module uses FFMPEG library to extract frame data from the video stream;

The hardware decoder is used to calculate the frame data to obtain the video frame.

Further, the formula for determining the weight of the texture contribution of the projector corresponding to the texture mapping overlap region is r=p/(α×d);

Among them, r is the weight of the texture contribution of the projector, p is the pixel resolution of the projector image, α is the angle between the two straight lines, and d is the distance from the projector position to the corresponding three-dimensional point.

Further, the formula for determining the texture values of each three-dimensional point corresponding to the overlapping region of texture mapping is _{T = (ΣI i × r i} ) / Σr i.

Further, the image fusion system is also used to determine a dividing line, and the dividing line is used to intercept video frames of different paths, and to fuse the intercepted video frames. The texture is obtained by weighting the weight of the texture contribution of the projector corresponding to the intercepted video frame.

Further, the method for determining the dividing line is:

Transform the video frames corresponding to the overlapping area to the same viewpoint, and use the GraphCut method to obtain the dividing lines that merge the video frames corresponding to the overlapping area.

Further, when the image fusion system intercepts video frames of different channels, it back-projects the dividing line into the video frame, and intercepts the actual use area of each video frame to obtain the corresponding video frame part.

In the second aspect, the embodiments of the present application provide a three-dimensional augmented reality method for multi-channel video fusion, including:

The 3D scene system saves the 3D scene on site;

The video real-time solution system performs real-time solution on the received video stream to obtain the video frame;

The image projection system determines the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional scene, and performs texture mapping on the video frame in the three-dimensional scene according to the mapping relationship to complete the image projection of the video frame;

The image fusion system determines the texture value corresponding to each three-dimensional point in the overlap area of the texture mapping, and reconstructs the texture of the texture mapping overlap area according to the texture value to complete the texture mapping fusion, wherein the texture value is based on the projection corresponding to the texture mapping overlap area The weight of the texture contribution of the machine is determined;

The virtual 3D rendering system renders the fused texture and 3D scene.

In the third aspect, an embodiment of the present application provides a device, including: a display screen, a memory, and one or more processors;

The display screen is used to display a three-dimensional scene fused with multiple channels of video;

The memory is used to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional augmented reality method for multi-channel video fusion as described in the second aspect.

In the embodiment of this application, a multi-channel image acquisition system captures on-site images and generates a video stream. The multi-channel image real-time return control system and the data synchronization system return the video stream and time synchronization, so that the returned video of the same batch The stream is located in the same time slice space, and the returned video is solved by the video real-time solution system to obtain the video frame, and the image projection system maps the video frame texture obtained by the video stream solution to the 3D scene, where the texture mapping The texture of the overlapped area is reconstructed by the image fusion system according to the weight of the texture contribution of the projector, reducing the overlap of the projection area between multiple projectors, and then through the virtual 3D rendering system, the fused texture and 3D The scene is rendered to improve the display effect of the rendered 3D scene.

Description of the drawings

FIG. 1 is a schematic structural diagram of a three-dimensional augmented reality system for realizing multi-channel video fusion provided by an embodiment of the present application;

2 is a schematic structural diagram of another three-dimensional augmented reality system that realizes multi-channel video fusion provided by an embodiment of the present application;

FIG. 3 is a flowchart of a method for realizing 3D augmented reality of multi-channel video fusion provided by an embodiment of the present application;

Fig. 4 is a schematic structural diagram of a device provided by an embodiment of the present application.

detailed description

In order to make the objectives, technical solutions, and advantages of the present application clearer, specific embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It is understandable that the specific embodiments described here are only used to explain the application, but not to limit the application. In addition, it should be noted that, for ease of description, the drawings only show part but not all of the content related to the present application. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowchart describes various operations (or steps) as sequential processing, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of various operations can be rearranged. The processing may be terminated when its operation is completed, but may also have additional steps not included in the drawings. The processing may correspond to methods, functions, procedures, subroutines, subroutines, and so on.

FIG. 1 shows a schematic structural diagram of a three-dimensional augmented reality system for realizing multi-channel video fusion provided by an embodiment of the present application. 1, the 3D augmented reality system for realizing multi-channel video fusion includes a 3D scene system 110, a real-time video resolution system 140, an image projection system 150, an image fusion system 160, and a virtual 3D rendering system 170. among them:

The three-dimensional scene system 110 saves a three-dimensional scene on site, and uses the three-dimensional scene as a base map for digital fusion. The source of the 3D scene can be added from an external server, or it can be obtained from local 3D modeling. After the 3D scene is obtained, it is saved locally, and the 3D scene is used as the base map of digital fusion as the basic analysis Starting point.

The video real-time solution system 140 is used to perform real-time solution on the received video stream to obtain video frames.

The image projection system 150 is used to determine the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional scene, and perform texture mapping on the video frame in the three-dimensional scene according to the mapping relationship to complete the image projection of the video frame.

The image fusion system 160 is used to determine the texture value corresponding to each three-dimensional point in the overlap area of the texture mapping, and reconstruct the texture of the texture mapping overlap area according to the texture value to complete the fusion of the texture mapping, wherein the texture value is based on the overlap area of the texture mapping The weight of the texture contribution of the corresponding projector is determined. Specifically, the image fusion system 160 cuts the video frame according to the dividing line obtained by the GraphCut method. When video frames of different channels are intercepted, the image fusion system 160 back-projects the dividing line into the video frame, and intercepts the actual use area of each video frame to obtain the corresponding video frame part. Among them, the projector should be understood as a representation of a video capture device (such as a camera, a camera, etc.) in a virtual scene.

The virtual 3D rendering system 170 renders the merged texture and 3D scene. Exemplarily, the virtual three-dimensional rendering system 170 obtains a three-dimensional scene from the three-dimensional scene system 110, uses the three-dimensional scene as a base map, fuse the fused texture in the three-dimensional scene according to the mapping result frame by frame, and integrates the fused three-dimensional scene Render for visual display.

As mentioned above, the video real-time solution system 140 performs real-time solution to the received video stream to obtain the video frame, the image projection system 150 determines the mapping relationship of the pixels of the video frame in the 3D scene, and the image fusion system 160 uses the segmentation Line-pair video frames are cut in the texture overlapping area, and the actual use area of each video frame is intercepted. Then, the image fusion system 160 splices and fuses the cut video frames, and finally is rendered by the virtual 3D rendering system 170 to complete the visualization and intuitiveness Show.

FIG. 2 shows a schematic structural diagram of another three-dimensional augmented reality system for multi-channel video fusion provided by an embodiment of the present application. 2, the 3D augmented reality system that realizes multi-channel video fusion includes a 3D scene system 110, a data synchronization system 120, a video real-time solution system 140, an image projection system 150, an image fusion system 160, and a virtual 3D rendering system 170. The data synchronization system 120 is connected to a multi-channel image real-time return control system 130, and the multi-channel image real-time return control system 130 is connected to a multi-channel image acquisition system 180.

The multi-channel images collected by the multi-channel image acquisition system 180 are sent back to the data synchronization system 120 in real time via the multi-channel image real-time return control system 130 for synchronization. The synchronized video stream is solved in real time by the video real-time solution system 140 Calculation, the solution result is directly displayed in the virtual three-dimensional rendering system 170 and the three-dimensional scene through the image projection system 150 and the image fusion system 160.

Specifically, the three-dimensional scene system 110 saves a three-dimensional scene on site, and uses the three-dimensional scene as a base map for digital fusion. The source of the 3D scene can be added from an external server, or it can be obtained from local 3D modeling. After the 3D scene is obtained, it is saved locally, and the 3D scene is used as the base map of digital fusion as the basic analysis Starting point.

Further, the 3D scene system 110 divides the 3D data of the 3D scene into blocks, and when the 3D scene on the scene is updated, the 3D scene system 110 receives the 3D update data packet of the corresponding block, and the 3D update data packet should contain the pointed to The block of is used for the updated three-dimensional data, and the three-dimensional scene system 110 replaces the three-dimensional data of the corresponding block with the three-dimensional data in the three-dimensional update data package to ensure the timeliness of the three-dimensional scene.

Specifically, the multi-channel image acquisition system 180 includes a multi-channel video acquisition device, which is used to collect images from multiple locations on the scene and generate a video stream.

In this embodiment, the multi-channel video capture device should include a video capture device (such as a camera, a camera, etc.) that supports a maximum number of not less than 100. Among them, each video capture device is not less than 2 million pixels, and the resolution is 1920X1080. The following functions can also be selected according to actual needs: integrated ICR dual filter day and night switching, fog penetration function, electronic anti-shake, multiple white balances Mode switching, video automatic iris, support for H.264 encoding, etc.

Each video capture device monitors different areas of the site, and the monitoring range of the multi-channel video capture device should cover the range of the site corresponding to the three-dimensional scene, that is, the range of interest on the site should be monitored.

Further, the multi-channel image real-time return control system 130 is used to return the video stream generated by the multi-channel image acquisition system 180.

In this embodiment, the effective transmission distance of the multi-channel image real-time return control system 130 should be no less than 3KM, the video code stream should be no less than 8Mpbs, and the delay should be no more than 80ms to ensure the timeliness of the display effect.

Exemplarily, an access switch is set on the side of the multi-channel image acquisition system 180 to collect the video streams generated by the multi-channel image acquisition system 180, and aggregate the collected video streams to the aggregation switch or the middle station, the aggregation switch or the middle station The video stream is preprocessed and sent to the multi-channel image real-time return control system 130, and the multi-channel image real-time return control system 130 returns the video stream to the data synchronization system 120 for synchronization processing.

Optionally, the connection between the aggregation switch or the middle station and the access switches on both sides can be connected through wired and/or wireless communication. When connected via wired, it can be connected via RS232, RS458, RJ45, bus, etc. When connected via wireless, if the distance between each other is close, wireless communication can be performed through WiFi, ZigBee, Bluetooth and other near field communication modules. When the distance is far, the long-distance wireless communication connection can be carried out through the wireless bridge, 4G module, 5G module, etc.

The data synchronization system 120 receives the video stream returned by the multi-channel image real-time return control system 130 and performs data synchronization on the returned video stream. The synchronized video stream is sent to the video real-time solution system 140 for solution. The data synchronization is specifically time synchronization, so that the returned video streams of the same batch are located in the same time slice space. The data synchronization system 120 in this embodiment should include data synchronization that supports a maximum number of not less than 100 video capture devices to return video streams. The time slice space can be understood as the abstraction of several fixed-size real time intervals.

Specifically, the video real-time solution system 140 is configured to perform real-time solution on the video stream to obtain video frames.

Further, the real-time video resolution system 140 includes a video frame extraction module 141 and a hardware decoder 142, wherein:

The video frame extraction module 141 uses the FFMPEG library to extract frame data from the video stream. The FFMPEG library is a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams, which can meet the requirements of extracting frame data in this embodiment.

The hardware decoder 142 is used to resolve the frame data to obtain a video frame. In this embodiment, the hardware decoder 142 is an independent video decoding module built in the NVIDIA graphics card, supports H.264 and H.265 decoding, and has a maximum resolution of 8K.

Specifically, the image projection system 150 is used to determine the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional scene, and perform texture mapping on the video frame in the three-dimensional scene according to the mapping relationship to complete the video frame Image projection.

Exemplarily, the posture information of the camera needs to be solved, so as to determine the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional virtual environment. The registration of the video frame and the virtual scene is essentially a problem of camera calibration. It is necessary to know the internal and external parameters of the camera. The position of the camera is clear when the camera is shooting video. According to the spatial position and posture of the camera, the information of the 2D picture in the 3D scene can be obtained, and the corresponding 3D point can be obtained according to the corresponding information of the 2D picture in the 3D scene. . Given a point (x _i , y _i ) on a video frame and the corresponding three-dimensional point (X _i , Y _i , Z _i ), there is a 3×4 matrix M, which has the following relationship:

M=K[RT],

Among them, w is a scaling factor, and M can be decomposed into a 3×3 camera internal parameter matrix K, a 3×3 rotation matrix R and a translation vector T. Z _i is the depth value of the three-dimensional point, which is calculated through the reconstruction process, that is, a triangle is formed by using the different orientations of the camera, and the depth information is inversely calculated as the value of _{Z i.} The scaling factor w is generally given in combination with experience. The translation vector T is the true absolute position of the camera, which is the coordinate representation of the camera in the scene coordinate system, f is the focal length of the camera, and s is the offset of the camera, usually 0, x ₀ And y ₀ represents the main point of the video frame, which is generally the center point of the video frame. The 3×4 matrix M is the posture and position transformation of the physical camera itself. Generally, these parameters of the camera can be obtained through the SDK provided by the camera manufacturer. Theoretically, the matrix M can be calculated by 6 sets of corresponding points, but due to the error in matching, it is necessary to find more corresponding points to calibrate the camera.

Since the internal and external parameters of the camera are calibrated, the corresponding relationship between the 3D points in the 3D scene and the pixels in the video frame can be obtained. This mapping relationship is essentially determined by a 4×4 matrix N, and the 4×4 matrix N represents graphics The transformation of virtual camera in learning. The projection texture mapping in this system is implemented based on OpenGL, and N can be decomposed into a 4×4 view matrix V and a 4×4 projection matrix P. Given a three-dimensional point in space, its texture coordinates are calculated as follows:

Among them, (s, t) represents the texture coordinates, u is used to determine whether the 3D point is in front of the camera (>0) or behind (<0), q represents the depth value of the 3D point, and the range of these values is (-1, 1) Between, it needs to be normalized to (0,1). Since the depth value is considered, the camera internal parameter matrix K needs to be changed into a 4×4 matrix. The calculation methods of P and V are as follows:

Among them, F is the distance from the camera to the far clipping plane, N is the distance from the camera to the near clipping plane, and W and H are the width and height of the video frame.

Specifically, the image fusion system 160 is used to determine the texture value corresponding to each three-dimensional point in the overlap region of the texture mapping, and reconstruct the texture of the texture mapping overlap region according to the texture value to complete the fusion of the texture mapping, wherein the texture value is based on the texture mapping The weight of the texture contribution of the projector corresponding to the overlapping area is determined.

Further, the formula for determining the weight of the texture contribution of the projector corresponding to the texture mapping overlap area is r=p/(α×d); where r is the weight of the texture contribution of the projector, and p is the pixel resolution of the projector image Rate, α is the angle between the two straight lines, d is the distance from the position of the projector to the corresponding three-dimensional point, where the angle between the two straight lines refers to the angle formed by the two opposite hypotenuses of the cone formed by the projector projection .

Further, after the obtained weights, based on the determined texture value calculated texture value of the formula, formula for determining the texture values of the respective three-dimensional point corresponding to the overlapping region of texture mapping is _{T = (ΣI i × r i} ) / Σr i. Where I _i is the original color value of the texture of the corresponding projector, and r _i is the weight value contributed by the texture of the corresponding projector.

It should be noted that the image fusion system 160 is also used to determine the dividing line, where the dividing line is used to intercept the video frames of different paths, and to fuse the intercepted video frames. The texture of the three-dimensional points around the dividing line is determined by The weight of the texture contribution of the projector corresponding to the intercepted video frame is weighted and obtained.

Exemplarily, the method for determining the dividing line is to select the image closest to the virtual viewpoint as the main projection source, and project the main projection source first, and then project other projection sources. The determination of the main projection source is mainly determined according to the difference between the position and the angle of view of the projector and the position and direction of the virtual viewpoint. If the difference is within the threshold, the main projection source is determined. If there is no main projection source, in other words, the contribution rates of multiple videos are similar, the video frames corresponding to the overlapping area are transformed to the same viewpoint, and the dividing line of the video frames corresponding to the overlapping area is obtained by using the GraphCut method.

Optionally, the image fusion system 160 cuts the video frame according to the dividing line obtained by the GraphCut method. When capturing video frames of different channels, the image fusion system 160 back-projects the dividing line into the video frame, and intercepts the actual use area of each video frame to obtain the corresponding video frame part. In the subsequent fusion process, it is only necessary to obtain the use area of the video frame according to the previously obtained segmentation line for fusion, which reduces the calculation amount of the fusion process and improves the real-time performance.

Specifically, the virtual 3D rendering system 170 renders the merged texture and 3D scene. Exemplarily, the virtual three-dimensional rendering system 170 obtains a three-dimensional scene from the three-dimensional scene system 110, uses the three-dimensional scene as a base map, fuse the fused texture in the three-dimensional scene according to the mapping result frame by frame, and integrates the fused three-dimensional scene Render for visual display.

As mentioned above, the multi-channel images collected by the multi-channel image acquisition system 180 are returned in real time via the multi-channel image real-time return control system 130 and are time synchronized by the data synchronization system 120. The video real-time resolution system 140 then compares the synchronized The video stream is calculated in real time to obtain the video frame. The image projection system 150 determines the mapping relationship of the pixels of the video frame in the three-dimensional scene, and the image fusion system 160 uses the dividing line to cut the video frame in the texture overlap area, and intercept each Then, the image fusion system 160 splicing and fusion of the cut video frames, and finally the virtual three-dimensional rendering system 170 performs rendering to complete the visual display.

Figure 3 shows a flow chart of the method for realizing multi-channel video fusion in 3D augmented reality provided by an embodiment of the application. The method of realizing multi-channel video fusion in 3D augmented reality provided by this embodiment can be implemented by realizing multi-channel video fusion. A three-dimensional augmented reality system is implemented. The three-dimensional augmented reality system that realizes multi-channel video fusion can be implemented by hardware and/or software, and integrated in a computer and other equipment. Referring to Figure 3, the method for realizing multi-channel video fusion 3D augmented reality includes:

S101: The three-dimensional scene system saves the three-dimensional scene of the scene.

Specifically, the source of the 3D scene can be added from an external server, or can be obtained by local 3D modeling. After the 3D scene is obtained, it is saved locally, and the 3D scene is used as a base map for digital fusion. As a starting point for fundamental analysis.

Further, the 3D scene system divides the 3D data of the 3D scene into blocks, and when the 3D scene is updated on site, the 3D scene system receives the 3D update data packet of the corresponding block, and the 3D update data packet should contain the pointed area. The block is used for updated 3D data, and the 3D scene system replaces the 3D data of the corresponding block with the 3D data in the 3D update data package to ensure the timeliness of the 3D scene.

S102: The video real-time calculation system performs real-time calculation on the received video stream to obtain a video frame.

Exemplarily, the video stream is generated by the multi-channel image acquisition system collecting images at multiple locations on the scene, and the video stream generated by the multi-channel image acquisition system is returned by the multi-channel image real-time return control system and is transmitted by the data synchronization system Perform data synchronization.

S103: The image projection system determines the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional scene, and performs texture mapping on the video frame in the three-dimensional scene according to the mapping relationship to complete image projection of the video frame.

S104: The image fusion system determines the texture value corresponding to each three-dimensional point in the overlap area of the texture mapping, and reconstructs the texture of the texture mapping overlap area according to the texture value to complete the fusion of the texture mapping, wherein the texture value corresponds to the overlap area of the texture mapping The weight of the texture contribution of the projector is determined.

S105: The virtual 3D rendering system renders the merged texture and 3D scene.

Exemplarily, the virtual three-dimensional rendering system obtains the three-dimensional scene from the three-dimensional scene system, uses the three-dimensional scene as the base map, fuse the fused texture in the three-dimensional scene according to the mapping result frame by frame, and renders the fused three-dimensional scene Visual display.

As mentioned above, the received video stream is calculated in real time by the video real-time solution system to obtain the video frame, the image projection system determines the mapping relationship of the pixel points of the video frame in the 3D scene, and the image fusion system uses the dividing line to divide the video Frames are cut in the texture overlapping area, and the actual use area of each video frame is intercepted. Then, the image fusion system splices and fuses the cut video frames, and finally is rendered by the virtual 3D rendering system to complete the visual display.

On the basis of the foregoing embodiment, FIG. 4 is a schematic structural diagram of a device provided by an embodiment of this application. 4, the device provided in this embodiment may be a computer, which includes: a display screen 24, a memory 22, and one or more processors 21; the display screen 24 is used to display a three-dimensional scene fused with multiple channels of video The memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 realize the implementation of this application The method provided by the example to achieve multi-channel video fusion 3D augmented reality.

As a computer-readable storage medium, the memory 22 can be used to store software programs, computer-executable programs, and modules, such as the method for realizing multi-channel video fusion three-dimensional augmented reality described in any embodiment of the present application. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device and the like. In addition, the memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some examples, the memory 22 may further include a memory remotely provided with respect to the processor 21, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

Further, the computer device further includes a communication module 23, which is used to establish wired and/or wireless connections with other devices and perform data transmission.

The processor 21 executes various functional applications and data processing of the device by running software programs, instructions, and modules stored in the memory 22, that is, realizing the above-mentioned three-dimensional augmented reality method of multi-channel video fusion.

The computer equipment provided above can be used to implement the three-dimensional augmented reality method for multi-channel video fusion provided in the above embodiments, and has corresponding functions and beneficial effects.

An embodiment of the present application also provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions are used to execute the implementation of multi-channel video fusion as provided in the embodiments of the present application when being executed by a computer processor. The three-dimensional augmented reality method for realizing multi-channel video fusion includes: the three-dimensional scene system saves the on-site three-dimensional scene; the video real-time solution system performs real-time solution on the received video stream to obtain the video frame; The image projection system determines the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional scene, and performs texture mapping on the video frame in the three-dimensional scene according to the mapping relationship to complete the image projection of the video frame; image fusion system Determine the texture value corresponding to each three-dimensional point in the overlap area of the texture mapping, and reconstruct the texture of the texture mapping overlap area according to the texture value to complete the fusion of the texture mapping, wherein the texture value is based on the texture of the projector corresponding to the overlap area of the texture mapping The weight of the contribution is determined; the virtual 3D rendering system renders the fused texture and 3D scene.

Storage medium-any of various types of storage devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc. ; Non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements. The storage medium may further include other types of memory or a combination thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the Internet). The second computer system may provide program instructions to the first computer for execution. The term "storage media" may include two or more storage media that may reside in different locations (for example, in different computer systems connected through a network). The storage medium may store program instructions executable by one or more processors 21 (for example, embodied as a computer program).

Of course, the storage medium containing computer-executable instructions provided by the embodiments of the present application is not limited to the above-mentioned method for realizing multi-channel video fusion of three-dimensional augmented reality, and can also perform any implementation of the present application. The relevant operations in the three-dimensional augmented reality method for multi-channel video fusion provided by the example.

The system and equipment for realizing 3D augmented reality of multi-channel video fusion provided in the above embodiment can execute the method of realizing 3D augmented reality of multi-channel video fusion provided in any embodiment of this application, which is not described in detail in the above embodiment For technical details, please refer to the three-dimensional augmented reality system and method for multi-channel video fusion provided by any embodiment of this application.

The foregoing are only the preferred embodiments of the present application and the technical principles used. The application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not depart from the protection scope of the application. Therefore, although the application has been described in more detail through the above embodiments, the application is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the application. The scope of is determined by the scope of the claims.

Claims (10)

1. The three-dimensional augmented reality system that realizes multi-channel video fusion is characterized by including a three-dimensional scene system, a real-time video resolution system, an image projection system, an image fusion system, and a virtual three-dimensional rendering system. Among them:

   Three-dimensional scene system, save the three-dimensional scene of the scene;

   Video real-time solution system, which performs real-time solution on the received video stream to obtain video frames;

   The image projection system is used to determine the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional scene, and perform texture mapping on the video frame in the three-dimensional scene according to the mapping relationship to complete the image projection of the video frame;

   The image fusion system is used to determine the texture value corresponding to each three-dimensional point in the overlap area of the texture mapping, and reconstruct the texture of the texture mapping overlap area according to the texture value to complete the fusion of the texture mapping, wherein the texture value is based on the overlap area of the texture mapping The weight of the texture contribution of the corresponding projector is determined;

   The virtual 3D rendering system renders the fused texture and 3D scene.
2. The three-dimensional augmented reality system for realizing multi-channel video fusion according to claim 1, wherein the system further comprises a data synchronization system, and the video stream is processed by a multi-channel image acquisition system on images at multiple locations on the scene. The video stream generated by the multi-channel image acquisition system is returned by the multi-channel image real-time return control system; the data synchronization system performs data synchronization on the returned video stream, and the data synchronization is specifically time Synchronize so that the returned video streams of the same batch are located in the same time slice space.
3. The three-dimensional augmented reality system for realizing multi-channel video fusion according to claim 1, wherein the video real-time resolution system includes a video frame extraction module and a hardware decoder, wherein:

   The video frame extraction module uses FFMPEG library to extract frame data from the video stream;

   The hardware decoder is used to calculate the frame data to obtain the video frame.
4. The three-dimensional augmented reality system for realizing multi-channel video fusion according to claim 1, wherein the formula for determining the weight of the texture contribution of the projector corresponding to the texture mapping overlap area is r=p/(α×d );

   Among them, r is the weight of the texture contribution of the projector, p is the pixel resolution of the projector image, α is the angle between the two straight lines, and d is the distance from the projector position to the corresponding three-dimensional point.
5. Realized according to claim 4, wherein the three-dimensional video multiplex fusion augmented reality system, wherein the determined texture value for each formula texture mapping three-dimensional point corresponding to the overlapping region of _{T = (ΣI i × r i} ) / Σ r _i .
6. The three-dimensional augmented reality system for realizing multi-channel video fusion according to claim 5, wherein the image fusion system is also used to determine a dividing line, and the dividing line is used to intercept video frames of different channels, The intercepted video frames are merged, and the texture of the three-dimensional points around the dividing line is weighted by the weight of the texture contribution of the projector corresponding to the intercepted video frame.
7. The 3D augmented reality system for realizing multi-channel video fusion according to claim 6, wherein the method for determining the dividing line is:

   Transform the video frames corresponding to the overlapping area to the same viewpoint, and use the GraphCut method to obtain the dividing lines that merge the video frames corresponding to the overlapping area.
8. The three-dimensional augmented reality system for realizing multi-channel video fusion according to claim 7, wherein when the image fusion system intercepts video frames of different channels, it back-projects the dividing line into the video frame , Intercept the actual use area of each video frame to obtain the corresponding video frame part.
9. The three-dimensional augmented reality method for multi-channel video fusion is characterized in that it includes:

   The 3D scene system saves the 3D scene on site;

   The video real-time solution system performs real-time solution on the received video stream to obtain the video frame;

   The image projection system determines the mapping relationship between the pixels in the video frame and the three-dimensional points in the three-dimensional scene, and performs texture mapping on the video frame in the three-dimensional scene according to the mapping relationship to complete the image projection of the video frame;

   The image fusion system determines the texture value corresponding to each three-dimensional point in the overlap area of the texture mapping, and reconstructs the texture of the texture mapping overlap area according to the texture value to complete the texture mapping fusion, wherein the texture value is based on the projection corresponding to the texture mapping overlap area The weight of the texture contribution of the machine is determined;

   The virtual 3D rendering system renders the fused texture and 3D scene.
10. A device, characterized by comprising: a display screen, a memory, and one or more processors;

    The display screen is used to display a three-dimensional scene fused with multiple channels of video;

    The memory is used to store one or more programs;

    When the one or more programs are executed by the one or more processors, the one or more processors implement the three-dimensional augmented reality method for realizing multi-channel video fusion according to claim 9.

Images