WO2021031454A1

WO2021031454A1 - Digital twinning system and method and computer device

Info

Publication number: WO2021031454A1
Application number: PCT/CN2019/123194
Authority: WO
Inventors: 石立阳; 程远初; 徐建明; 陈奇毅; 高星; 朱文辉; 赵康嘉; 秦伟
Original assignee: 佳都新太科技股份有限公司
Priority date: 2019-08-21
Filing date: 2019-12-05
Publication date: 2021-02-25
Also published as: CN110505464A; CN110753218B; CN110753218A

Abstract

Disclosed are a digital twinning system and method and a computer device. According to the technical solution provided in the embodiments of the present application, multi-channel images are acquired by a multi-channel image acquisition system; a multi-channel image real-time return control system and a data synchronization system return the multi-channel images to a video real-time calculation system in real time for real-time calculation; a calculation result, a three-dimensional scene and multi-source data are mapped, fused and visually and intuitively displayed through a virtual three-dimensional rendering system, interactive operation can be conducted on virtual-real interactive middleware in an interactive interface, and accordingly the entity control system is controlled and control over a field device is achieved.

Description

Digital twin system, method and computer equipment

Technical field

The embodiments of the present application relate to the field of computer technology, and in particular to a digital twin system, method, and computer equipment.

Background technique

At present, the digital wave represented by new technologies such as the Internet of Things, big data, and artificial intelligence is sweeping the world. The physical world and the corresponding digital world are forming two systems that develop and interact in parallel. The digital world exists to serve the physical world. Because the digital world has become efficient and orderly, the digital twin technology has emerged as the times require. It has gradually expanded from the manufacturing industry to the urban space, and has profoundly affected urban planning, construction and development.

The urban information model based on multi-source data fusion is the core, and the intelligent facilities and perception systems deployed across the city are the prerequisites. The intelligent private network that supports the efficient operation of twin cities is the guarantee. The digital twin cities can have a relatively high level of digitalization and require operating mechanisms. Provide support for modeling, realizing collaborative optimization of virtual and real space, and highlighting multi-dimensional intelligent decision support.

At present, the digital twin system only displays the three-dimensional model in the three-dimensional display interface, and cannot control on-site equipment based on real-time detection.

Summary of the invention

The embodiments of the application provide a digital twin system, method, and computer equipment, so that while constructing and displaying data scenes, it can also control on-site equipment based on real-time detection conditions, so that scene elements can be known, measured, and controlled. .

In the first aspect, the embodiments of the present application provide a digital twin system, including a scene overlay and analysis system, a video real-time solution system, a multi-source data acquisition and processing analysis system, a virtual 3D rendering system, and a virtual-real interaction middleware. :

The scene overlay and analysis system saves the three-dimensional scene of the scene and uses the three-dimensional scene as the base map;

Video real-time solution system, which performs real-time solution on the received video stream to obtain video frames;

Multi-source data acquisition and processing analysis system, receiving and storing multi-source data, and performing basic analysis and processing, and converting multi-source data into a three-dimensional representation format;

The virtual three-dimensional rendering system uses the three-dimensional scene as a base map, maps and merges the video frames in the three-dimensional scene, and performs position matching and fusion of the multi-source data in the three-dimensional scene, and combines the virtual and real The interactive middleware performs mapping in the three-dimensional scene, renders and interacts with the fused three-dimensional scene, generates an interactive instruction in response to the interactive operation of the virtual-real interaction middleware, and sends it to the virtual-real interaction middleware;

The virtual and real interaction middleware is used to send out the interaction instructions issued by the virtual 3D rendering system.

Further, the video stream is generated by a multi-channel image acquisition system collecting images at multiple locations on the spot, and the video stream generated by the multi-channel image acquisition system is returned through a multi-channel image real-time return control system.

Further, the system also includes a data synchronization system for data synchronization of the video streams returned by the multi-channel image real-time return control system. The data synchronization is specifically time synchronization, so that the returned videos of the same batch The stream is in the same time slice space.

Further, the video real-time solution system includes a video frame extraction module and a hardware decoder, wherein:

The video frame extraction module uses FFMPEG library to extract frame data from the video stream;

The hardware decoder is used to calculate the frame data to obtain the video frame.

Further, the multi-source data acquisition and processing analysis system includes a multi-source data acquisition system and a multi-source data analysis system, wherein:

Multi-source data acquisition system for receiving and storing multi-source data returned by multi-source sensors;

Multi-source data analysis system, used for basic analysis and processing of multi-source data, and convert multi-source data into three-dimensional representation format.

Further, the virtual-real interaction middleware includes an instruction receiving module and an instruction transmission module, wherein:

The instruction receiving module is used to receive interactive instructions issued by the virtual 3D rendering system;

The instruction transmission module is used to transmit the interactive instruction to the entity control system pointed to by the interactive instruction.

Further, according to the position correspondence between the multi-source sensor used to return the multi-source data and the virtual-real interaction middleware, the rendering position of the multi-source data in the three-dimensional scene corresponds to the position of the virtual-real interaction middleware.

Further, the virtual three-dimensional rendering system also responds to changes in multi-source data to change the rendering state of the virtual-real interactive middleware in the three-dimensional scene.

In the second aspect, an embodiment of the present application provides a digital twin method, including:

The scene overlay and analysis system saves the on-site three-dimensional scene, and uses the three-dimensional scene as a base map;

The video real-time solution system performs real-time solution on the received video stream to obtain the video frame;

The multi-source data acquisition and processing analysis system accepts and stores multi-source data, performs basic analysis and processing, and converts the multi-source data into a three-dimensional representation format;

The virtual three-dimensional rendering system uses the three-dimensional scene as the base map, maps and fuses the video frames in the three-dimensional scene, matches and fuses the multi-source data in the three-dimensional scene, and integrates the virtual and real interaction middleware Performing mapping in the three-dimensional scene, and rendering and interacting with the merged three-dimensional scene;

The virtual three-dimensional rendering system generates and sends interactive instructions to the virtual-real interactive middleware in response to the interactive operation of the virtual-real interactive middleware;

The virtual-real interaction middleware sends out the interaction instructions issued by the virtual 3D rendering system.

In the third aspect, embodiments of the present application provide a computer device, including: a display screen, an input device, a memory, and one or more processors;

The display screen is used to display virtual and real interactive interfaces;

The input device is used to receive interactive operations;

The memory is used to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the digital twin method as described in the second aspect.

In the fourth aspect, an embodiment of the present application provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions are used to execute the digital data described in the second aspect when the computer-executable instructions are executed by a computer Twin method.

The embodiment of the application collects multiple images on site through a multiple image acquisition system, and returns the video stream in real time via the multi-channel image real-time return control system and synchronizes the time of the video stream. The synchronized video stream is calculated by the video in real time The system performs real-time calculation to obtain the video frame; the multi-source sensor detects the on-site environment and generates the corresponding multi-source data, and transmits it to the multi-source data acquisition and processing analysis system for analysis and processing, and converts it into a three-dimensional scene that can be displayed Three-dimensional representation format; then the on-site three-dimensional scene is used as the base map, and the virtual three-dimensional rendering system will match, map, and merge the video frame and multi-source data in the three-dimensional scene, and render the fused three-dimensional scene, and at the same time, the fusion The latter three-dimensional scene can be visualized and intuitively displayed, and the fused three-dimensional scene can be interacted through the virtual three-dimensional rendering system. The interactive instructions generated by the interactive operation are sent to the entity control system through the virtual-real interaction middleware, and the entity control system responds to the interactive instructions Control field equipment. Through the embodiments of this application, the digital twin system maps, merges, and visualizes the three-dimensional scene, real-time video frames, and on-site multi-source data. The three-dimensional display interface is more realistic and comprehensive, and the virtual three-dimensional rendering system realizes the integration When the field equipment needs to be controlled, the virtual-real interaction middleware sends interactive instructions to the physical control system used to control the field equipment, so as to realize the control of the field equipment and truly realize the knowable, Measurable and controllable.

Description of the drawings

FIG. 1 is a schematic structural diagram of a digital twin system provided by an embodiment of the present application;

Figure 2 is a schematic structural diagram of another digital twin system provided by an embodiment of the present application;

FIG. 3 is a schematic structural diagram of another digital twin system provided by an embodiment of the present application;

FIG. 4 is a schematic flowchart of a digital twin method provided by an embodiment of the present application;

Fig. 5 is a schematic structural diagram of a computer device provided by an embodiment of the present application.

detailed description

In order to make the objectives, technical solutions, and advantages of the present application clearer, specific embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It is understandable that the specific embodiments described here are only used to explain the application, but not to limit the application. In addition, it should be noted that, for ease of description, the drawings only show part but not all of the content related to the present application. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowchart describes various operations (or steps) as sequential processing, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of various operations can be rearranged. The processing may be terminated when its operation is completed, but may also have additional steps not included in the drawings. The processing may correspond to methods, functions, procedures, subroutines, subroutines, and so on.

Figure 1 shows a schematic structural diagram of a digital twin system provided by an embodiment of the present application. 1, the digital twin system includes a scene overlay and analysis system 110, a video real-time solution system 140, a multi-source data acquisition and processing analysis system 150, a virtual three-dimensional rendering system 160, and a virtual-real interaction middleware 170. among them:

The scene overlay and analysis system 110 stores a three-dimensional scene on the spot, and uses the three-dimensional scene as a base map. Specifically, the source of the 3D scene can be added from an external server, or it can be obtained by 3D modeling locally. After the 3D scene is obtained, it is saved locally, and the 3D scene is used as the base map for other concerns. The data in the three-dimensional scene is fused, and the three-dimensional scene is used as the starting point for basic analysis.

The video real-time solution system 140 performs real-time solution on the received video stream to obtain video frames.

The multi-source data acquisition and processing analysis system 150 receives and stores multi-source data, performs basic analysis and processing, and converts the multi-source data into a three-dimensional representation format.

The virtual three-dimensional rendering system 160 uses the three-dimensional scene as the base map to map and fuse video frames in the three-dimensional scene, position matching and fusion of multi-source data in the three-dimensional scene, and implement the virtual-real interaction middleware 170 in the three-dimensional scene Mapping, rendering and interacting with the fused three-dimensional scene, and generating interaction instructions in response to the interactive operation of the virtual-real interaction middleware 170 and sending them to the virtual-real interaction middleware 170.

Specifically, when the virtual three-dimensional rendering system 160 maps and merges video frames in a three-dimensional scene, it determines the mapping relationship between pixels in the video frame and three-dimensional points in the three-dimensional scene, and maps the video frame in the three-dimensional scene according to the mapping relationship. Perform texture mapping in the scene, and perform smooth transition processing on the overlapped area of the texture mapping, thereby fusing the video frame into the three-dimensional scene.

Further, when the virtual 3D rendering system 160 performs position matching and fusion of multi-source data in the 3D scene, it is based on the position correspondence between the multi-source sensor and the virtual-real interaction middleware 170, that is, according to the position information or the position information carried in the multi-source data. The device identification number determines the position of the multi-source data corresponding to the 3D scene, and maps the multi-source data in the 3D scene according to the target expression form, so that the position of the multi-source data rendering in the 3D scene corresponds to the position of the virtual-real interaction middleware 170 , So as to complete the position matching and fusion of multi-source data in the 3D scene.

The virtual-real interaction middleware 170 is used to send out the interaction instructions issued by the virtual 3D rendering system 160. The interactive instructions point to the equipment to be controlled and are used to instruct the field equipment to perform corresponding actions.

As mentioned above, the video real-time solution system 140 performs real-time solution to the received video stream and obtains the video frame, while the multi-source data collection and processing analysis system 150 processes the received multi-source data and converts it into a three-dimensional representation format. Then the video frame, multi-source data and 3D scene are mapped, fused and visualized and displayed intuitively through the virtual 3D rendering system 160, and the rendered 3D scene can be interactively operated. The interactive commands generated by the interaction are sent through the virtual and real interaction middleware 170 In the control module of the field device, the field device can execute corresponding actions in response to interactive instructions, thereby realizing the control of the field device.

Figure 2 shows a schematic structural diagram of another digital twin system provided by an embodiment of the present application. Referring to FIG. 2, the digital twin system includes a scene overlay and analysis system 110, a video real-time solution system 140, a multi-source data acquisition and processing analysis system 150, a virtual three-dimensional rendering system 160 and a virtual-real interaction middleware 170. among them:

The scene overlay and analysis system 110 stores a three-dimensional scene on the spot, and uses the three-dimensional scene as a base map.

Specifically, the video real-time resolution system 140 includes a video frame extraction module 141 and a hardware decoder 142, where:

The video frame extraction module 141 uses the FFMPEG library to extract frame data from the video stream. The FFMPEG library is a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams, which can meet the requirements of extracting frame data in this embodiment.

The hardware decoder 142 is used to resolve the frame data to obtain a video frame. In this embodiment, the hardware decoder 142 is an independent video decoding module built in the NVIDIA graphics card.

Further, the video stream is generated by the multi-channel image acquisition system 190 (consisting of a multi-channel video acquisition device) collecting images at multiple locations on the scene.

Specifically, the multi-source data acquisition and processing analysis system 150 includes a multi-source data acquisition system 151 and a multi-source data analysis system 152, among which:

The multi-source data collection system 151 is used to receive and store multi-source data returned by multi-source sensors.

The selection of multi-source sensors is based on the target, equipment and actual conditions of the scene, and is installed in the corresponding position. Install a multi-source data access switch on site to access and aggregate monitoring data from multi-source sensors, and transmit it to the multi-source data access switch set on the side of the multi-source data acquisition system 151 via wired or wireless means. The source data access switch sends the received multi-source data to the multi-source data collection system 151, and the multi-source data collection system 151 then sends the received multi-source data to the multi-source data analysis system 152.

The multi-source data analysis system 152 is used to perform basic analysis and processing on multi-source data and convert the multi-source data into a three-dimensional representation format.

Exemplarily, after the multi-source data collection system 151 outputs multi-source data reflecting the condition of the monitored equipment, the multi-source data analysis system 152 receives the multi-source data according to the demand and performs basic analysis and processing of the multi-source data, such as AD conversion, Threshold analysis, trend analysis, early warning analysis, numerical range, working status, etc., its three-dimensional representation format should be understood as the format corresponding to the target expression in the virtual three-dimensional rendering system 160, and its target expression can be the real-time value of the monitored data , Real-time status, data table, color and other forms of one or more combinations.

The virtual-real interaction middleware 170 is used to send out the interaction instructions issued by the virtual 3D rendering system 160.

Specifically, the virtual-real interaction middleware 170 includes an instruction receiving module 171 and an instruction transmission module 172. The instruction receiving module 171 is used to receive interactive instructions issued by the virtual 3D rendering system 160; the instruction transmission module 172 is used to transmit the interactive instructions to The device to which the interactive command points.

As described above, the video frame extraction module 141 and the hardware decoder 142 perform real-time calculation of the received video stream to obtain the video frame. At the same time, the multi-source data acquisition system 151 receives the multi-source data, which is processed by the multi-source data analysis system 152. Converted into a three-dimensional representation format, and then map, merge and visualize the video frame, multi-source data and three-dimensional scene through the virtual three-dimensional rendering system 160 for intuitive display, and can perform interactive operations on the rendered three-dimensional scene, and send the interactive instructions generated by the interaction To the instruction receiving module 171 and sent by the instruction transmission module 172 to the control module of the field device, the field device can perform corresponding actions in response to the interactive instruction, thereby realizing the control of the field device.

Fig. 3 shows a schematic structural diagram of another digital twin system provided by an embodiment of the present application. 3, the digital twin system includes a scene superimposition and analysis system 110, a data synchronization system 120, a video real-time solution system 140, a multi-source data acquisition and processing analysis system 150, a virtual 3D rendering system 160, and a virtual-real interaction middleware 170, The data synchronization system 120 is connected to the multi-channel image real-time return control system 130, the multi-channel image real-time return control system 130 is connected to the multi-channel image acquisition system 190, and the virtual-real interaction middleware 170 is connected to the physical control system 180.

Specifically, the scene superimposition and analysis system 110 stores a three-dimensional scene on site, and uses the three-dimensional scene as a base map. The source of the 3D scene can be added from an external server, or it can be obtained from local 3D modeling. After obtaining the 3D scene, save it locally and use the 3D scene as the base map. Fusion on the three-dimensional scene, the three-dimensional scene as the starting point of the basic analysis.

Further, the scene overlay and analysis system 110 divides the three-dimensional data of the three-dimensional scene into blocks, and when the on-site three-dimensional scene is updated, the scene overlay and analysis system 110 receives the three-dimensional update data packet of the corresponding block, and the three-dimensional update data packet The three-dimensional data for the updated block should be included. The scene overlay and analysis system 110 replaces the three-dimensional data of the corresponding block with the three-dimensional data in the three-dimensional update data package to ensure the timeliness of the three-dimensional scene.

Specifically, the multi-channel image acquisition system 190 includes a multi-channel video acquisition device, which is used to collect images from multiple locations on the scene and generate a video stream.

In this embodiment, the multi-channel video capture device should include a video capture device (such as a camera) that supports a maximum number of not less than 100. Among them, each video capture device is not less than 2 million pixels, and the resolution is 1920X1080. The following functions can also be selected according to actual needs: integrated ICR dual filter day and night switching, fog function, electronic anti-shake, multiple white balances Mode switching, video automatic iris, support for H.264 encoding, etc.

Each video capture device monitors different areas of the site, and the monitoring range of the multi-channel video capture device should cover the range of the site corresponding to the three-dimensional scene, that is, the range of interest on the site should be monitored.

Further, the multi-channel image real-time return control system 130 is used to return the video stream generated by the multi-channel image acquisition system 190.

In this embodiment, the effective transmission distance of the multi-channel image real-time return control system 130 should be no less than 3KM, the video code stream should be no less than 8Mpbs, and the delay should be no more than 80ms to ensure the timeliness of the display effect.

Exemplarily, an access switch is set on the side of the multi-channel image acquisition system 190 to collect the video streams generated by the multi-channel image acquisition system 190, and aggregate the collected video streams to the aggregation switch or the middle station, the aggregation switch or the middle station The video stream is preprocessed and sent to the multi-channel image real-time return control system 130, and the multi-channel image real-time return control system 130 returns the video stream to the data synchronization system 120 for synchronization processing.

Optionally, the connection between the aggregation switch and the access switches on both sides can be connected in a wired and/or wireless manner. When connected via wired, it can be connected via RS232, RS458, RJ45, bus, etc. When connected via wireless, if the distance between each other is close, wireless communication can be performed through WiFi, ZigBee, Bluetooth and other near field communication modules. When the distance is far, the long-distance wireless communication connection can be carried out through the wireless bridge, 4G module, 5G module, etc.

The data synchronization system 120 receives the video stream returned by the multi-channel image real-time return control system 130 and is used for data synchronization of the returned video stream. The synchronized video stream is sent to the video real-time solution system 140 for solution. The data synchronization is specifically time synchronization, so that the returned video streams of the same batch are located in the same time slice space. In this embodiment, the data synchronization system 120 should support the data synchronization of the video stream returned by the maximum number of not less than 100 video capture devices. The time slice space can be understood as the abstraction of several fixed-size real time intervals.

Specifically, the video real-time solution system 140 is configured to perform real-time solution on the video stream to obtain video frames.

Further, the real-time video resolution system 140 includes a video frame extraction module 141 and a hardware decoder 142, wherein:

The hardware decoder 142 is used to resolve the frame data to obtain a video frame. In this embodiment, the hardware decoder 142 is an independent video decoding module built in the NVIDIA graphics card, supports H.264 and H.265 decoding, and has a maximum resolution of 8K.

Specifically, the multi-source data acquisition and processing analysis system 150 receives and stores the multi-source data returned by the multi-source sensor, performs basic analysis and processing, and converts the multi-source data into a three-dimensional representation format.

Further, the multi-source data acquisition and processing analysis system 150 includes a multi-source data acquisition system 151 and a multi-source data analysis system 152, in which:

Exemplarily, the multi-source sensor includes at least resistive sensors, capacitive sensors, inductive sensors, voltage sensors, pyroelectric sensors, impedance sensors, magnetoelectric sensors, photoelectric sensors, resonance sensors, and Hall sensors , Ultrasonic sensor, isotope sensor, electrochemical sensor, microwave sensor, etc. one or more.

Specifically, the multi-source data analysis system 152 is used to perform basic analysis and processing on multi-source data and convert the multi-source data into a three-dimensional representation format.

Specifically, the virtual three-dimensional rendering system 160 uses the three-dimensional scene as the base map to map and fuse video frames in the three-dimensional scene, match and fuse multi-source data in the three-dimensional scene, and integrate the virtual-real interaction middleware 170 in the three-dimensional scene. Mapping is performed in the scene, and the merged three-dimensional scene is rendered and interacted. In response to the interactive operation of the virtual-real interaction middleware 170, an interaction instruction is generated and sent to the virtual-real interaction middleware 170.

Specifically, when the video frame is mapped and merged in the 3D scene, the mapping relationship between the pixels in the video frame and the 3D points in the 3D scene is determined, and the texture mapping of the video frame in the 3D scene is performed according to the mapping relationship. The overlapped area of the texture mapping is smoothly transitioned, so that the video frame is merged into the three-dimensional scene.

Further, when the position matching and fusion of the multi-source data in the three-dimensional scene is performed, the position correspondence between the multi-source sensor and the virtual-real interaction middleware 170 is determined according to the position information or the device identification number carried in the multi-source data. The source data corresponds to the position in the 3D scene, and the multi-source data is mapped in the 3D scene according to the target expression form, so that the position where the multi-source data is rendered in the 3D scene corresponds to the position of the virtual-real interaction middleware 170, thereby completing the multi-source The data is matched and fused in the 3D scene.

Further, the virtual three-dimensional rendering system 160 also changes the rendering state of the virtual-real interaction middleware 170 in the three-dimensional scene in response to changes in multi-source data. Exemplarily, the color or performance state of the virtual and real interaction middleware 170 in the three-dimensional scene can be correspondingly changed according to the value range or working status of the multi-source data of the corresponding device, such as distinguishing different value ranges with different colors, Different working states are expressed in the form of switch states.

Specifically, the virtual-real interaction middleware 170 is used to implement the transmission of interaction instructions between the virtual three-dimensional rendering system 160 and the entity control system 180.

Specifically, the entity control system 180 receives an interactive instruction from the virtual-real interactive middleware 170, and performs corresponding control on the field device in response to the interactive instruction.

Specifically, the virtual-real interaction middleware 170 includes an instruction receiving module 171 and an instruction transmission module 172. The instruction receiving module 171 is used to receive interactive instructions issued by the virtual 3D rendering system 160; the instruction transmission module 172 is used to transmit the interactive instructions to The entity control system 180 pointed to by the interactive instruction.

Exemplarily, according to the needs of multi-source data interaction in the three-dimensional scene, the instruction receiving module 171 of the virtual-real interaction middleware 170 may be displayed in a position corresponding to the multi-source data in the three-dimensional scene. The expression of the instruction receiving module 171 may be a physical button The form of the three-dimensional model may also be the form of the three-dimensional model of the corresponding device.

Further, the physical control system 180 includes a controller for controlling the device set on site, and the controller can control the device in response to an interactive instruction. The command transmission module 172 and the controller may be connected to each other in a wired and/or wireless manner. When connected via wired, it can be connected via RS232, RS458, RJ45, bus, etc. When connected via wireless, it can be connected via WiFi, ZigBee, Bluetooth, wireless bridge, 4G module, 5G module, etc., in the control When the number of devices is large, data can be collected and distributed through the switch.

When the user selects the instruction receiving module 171 in the 3D scene for interactive operation, the virtual 3D rendering system 160 generates a corresponding interactive instruction according to the multi-source data of the corresponding device and the preset interactive response mode and sends it to the instruction receiving module 171. Instructions include control instructions and location information or device identification numbers for the device. The instruction transmission module 172 sends the interactive instruction to the corresponding controller according to the location information or the device identification number, and the controller controls the field device in response to the control instruction.

As mentioned above, the multi-channel images collected by the multi-channel image acquisition system 190 are sent back to the data synchronization system 120 through the multi-channel image real-time return control system 130 in real time for time synchronization. The video real-time resolution system 140 performs time synchronization on the synchronized The video stream is calculated in real time to obtain the video frame. At the same time, the multi-source data acquisition system 151 receives the multi-source data, which is processed by the multi-source data analysis system 152 and converted into a three-dimensional representation format, and then the video frame, multi-source data and three-dimensional scene Through the virtual three-dimensional rendering system 160 mapping, fusion and visualization and intuitive display, the interactive instructions generated by the interaction are sent to the instruction receiving module 171 in the virtual interactive middleware, and the instruction transmission module 172 is sent to the entity control system 180, and the entity control system 180 The central controller controls the device to perform corresponding actions according to the location information of the device pointed to by the interactive instruction and the control instruction, so as to realize the control of the field device.

Figure 4 shows a schematic flow chart of a digital twin method provided by an embodiment of the present application. The digital twin method provided in this embodiment can be executed by a digital twin system, which can be implemented by hardware and/or software. , And integrated in the computer. Referring to Figure 4, the digital twin method includes:

S201: The scene overlay and analysis system saves the three-dimensional scene on site, and uses the three-dimensional scene as a base map.

Specifically, the source of the 3D scene can be added from an external server, or it can be obtained by 3D modeling locally. After the 3D scene is obtained, it is saved locally, and the 3D scene is used as the base map, that is, other concerns The data in the three-dimensional scene is fused, and the three-dimensional scene is used as the starting point for basic analysis.

Furthermore, the scene overlay and analysis system divides the three-dimensional data of the three-dimensional scene into blocks, and when the on-site three-dimensional scene is updated, the scene overlay and analysis system receives the three-dimensional update data package of the corresponding block, and the three-dimensional update data package should include The pointed block is used for updated 3D data. The scene overlay and analysis system replaces the 3D data of the corresponding block with the 3D data in the 3D update data package to ensure the timeliness of the 3D scene.

S202: The video real-time calculation system performs real-time calculation on the received video stream to obtain a video frame.

Specifically, further, the video real-time solution system includes a video frame extraction module and a hardware decoder, where:

The video frame extraction module uses FFMPEG library to extract frame data from the video stream. The FFMPEG library is a set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams, which can meet the requirements of extracting frame data in this embodiment.

The hardware decoder is used to calculate the frame data to obtain the video frame. In this embodiment, the hardware decoder is an independent video decoding module built in the NVIDIA graphics card.

S203: The multi-source data acquisition and processing analysis system receives and stores multi-source data, performs basic analysis and processing, and converts the multi-source data into a three-dimensional representation format.

Specifically, the multi-source data acquisition and processing analysis system includes a multi-source data acquisition system and a multi-source data analysis system, among which:

Multi-source data acquisition system, used to receive and store multi-source data returned by multi-source sensors.

The selection of multi-source sensors is based on the target, equipment and actual conditions of the scene, and is installed in the corresponding position. Install a multi-source data access switch on site to access and aggregate the monitoring data of multi-source sensors, and transmit it to the multi-source data access switch set on the side of the multi-source data acquisition system via wired or wireless means. Similar to the communication method of the data return system, the multi-source data access switch sends the received multi-source data to the multi-source data acquisition system, and the multi-source data acquisition system sends the received multi-source data to the multi-source data analysis system .

Specifically, a multi-source data analysis system is used to perform basic analysis and processing on multi-source data and convert the multi-source data into a three-dimensional representation format.

Exemplarily, after the multi-source data acquisition system outputs multi-source data reflecting the condition of the monitored equipment, the multi-source data analysis system receives multi-source data according to demand and performs basic analysis and processing of the multi-source data, such as AD conversion and threshold analysis , Trend analysis, early warning analysis, numerical range, working status, etc., its three-dimensional representation format should be understood as the format corresponding to the target expression in the virtual three-dimensional rendering system, and its target expression can be the real-time value and real-time status of the monitored data , Data table, color and other forms of one or more combinations.

S204: The virtual 3D rendering system uses the 3D scene as the base map, maps and merges the video frame in the 3D scene, matches and merges the position of the multi-source data in the 3D scene, and interacts the virtual and real The middleware performs mapping in the three-dimensional scene, and renders and interacts with the merged three-dimensional scene.

S205: The virtual three-dimensional rendering system generates an interactive instruction in response to the interactive operation on the virtual-real interactive middleware and sends it to the virtual-real interactive middleware.

Specifically, when multi-source data is matched and fused in a three-dimensional scene, the multi-source is determined according to the position correspondence between the multi-source sensor and the virtual-real interaction middleware, that is, according to the location information or device identification number carried in the multi-source data The data corresponds to the position in the 3D scene, and the multi-source data is mapped in the 3D scene according to the target expression form, so that the rendering position of the multi-source data in the 3D scene corresponds to the position of the virtual and real interactive middleware, thereby completing the multi-source data Position matching and fusion in the 3D scene.

Further, the virtual 3D rendering system also responds to changes in multi-source data to change the rendering state of the virtual and real interactive middleware in the 3D scene. Exemplarily, the color or performance state of the virtual and real interactive middleware in the three-dimensional scene can be correspondingly changed according to the value range or working status of the multi-source data of the corresponding device, such as distinguishing different value ranges with different colors. The working status is expressed in the form of switch status.

S206: The virtual-real interaction middleware sends out the interaction instruction issued by the virtual 3D rendering system.

Specifically, the virtual-real interactive middleware is used to realize the transmission of interactive instructions between the virtual 3D rendering system and the physical control system. The physical control system receives the interactive instructions from the virtual-real interactive middleware, and controls the field devices in response to the interactive instructions. .

Further, the virtual-real interaction middleware includes an instruction receiving module and an instruction transmission module. The instruction receiving module is used to receive the interactive instruction issued by the virtual 3D rendering system; the instruction transmission module is used to transmit the interactive instruction to the entity control directed by the interactive instruction. system.

As mentioned above, the received video stream is solved in real time by the video real-time solving system, the video frame obtained from the result of the calculation, and the multi-source data collection and processing analysis system processes the received multi-source data and converts it into a three-dimensional representation Format, and then map, merge and visualize the video frame, multi-source data and 3D scene through the virtual 3D rendering system. Interactive operations can be performed on the rendered 3D scene. The interactive commands generated by the interaction are sent through the virtual and real interaction middleware. In the control module of the field device, the field device can execute corresponding actions in response to interactive instructions, thereby realizing the control of the field device.

On the basis of the foregoing embodiment, FIG. 5 is a schematic structural diagram of a computer device provided by an embodiment of this application. 5, the computer equipment provided by this embodiment includes: a display screen 24, an input device 25, a memory 22, a communication module 23, and one or more processors 21; the communication module 23 is used to communicate with the outside world; The display screen 24 is used to display virtual and real interactive interfaces; the input device 25 is used to receive interactive operations; the memory 22 is used to store one or more programs; when the one or more programs are The one or more processors 21 execute, so that the one or more processors 21 implement the digital twin method and system functions as provided in the embodiments of the present application.

As a computer-readable storage medium, the memory 22 can be used to store software programs, computer-executable programs, and modules, such as the digital twin method and system functions described in any embodiment of the present application. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some examples, the memory 22 may further include a memory remotely provided with respect to the processor, and these remote memories may be connected to the device through a network. Examples of the aforementioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

Further, the computer device further includes a communication module 23, which is used to establish wired and/or wireless connections with other devices and perform data transmission.

The processor 21 executes various functional applications and data processing of the device by running software programs, instructions, and modules stored in the memory 22, that is, realizes the aforementioned digital twin method and system functions.

The digital twin system and computer equipment provided above can be used to execute the digital twin method provided in the above embodiments, and have corresponding functions and beneficial effects.

The embodiment of the present application also provides a storage medium containing computer-executable instructions, when the computer-executable instructions are executed by a computer processor, they are used to execute the digital twin method provided in the embodiments of the present application to implement the embodiments of the present application. The functions of the digital twin system provided.

Storage medium-any of various types of storage devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc. ; Non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements. The storage medium may further include other types of memory or a combination thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system connected to the first computer system through a network (such as the Internet). The second computer system may provide program instructions to the first computer for execution. The term "storage media" may include two or more storage media that may reside in different locations (for example, in different computer systems connected through a network). The storage medium may store program instructions executable by one or more processors 21 (for example, embodied as a computer program).

Of course, the storage medium containing computer-executable instructions provided by the embodiments of the present application is not limited to the digital twin method described above, and can also execute the digital twin methods provided in any embodiment of the present application. Related operations to realize the functions of the digital twin system provided by any embodiment of the present application.

The digital twin system and computer equipment provided in the above embodiments can execute the digital twin method provided in any embodiment of this application. For technical details not described in detail in the above embodiments, please refer to the digital twin provided in any embodiment of this application. System and method.

The foregoing are only the preferred embodiments of the present application and the technical principles used. The application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not depart from the protection scope of the application. Therefore, although the application has been described in more detail through the above embodiments, the application is not limited to the above embodiments, and may also include more other equivalent embodiments without departing from the concept of the application. The scope of is determined by the scope of the claims.

Claims

A digital twin system, characterized in that it includes a scene superimposition and analysis system, a video real-time solution system, a multi-source data acquisition and processing analysis system, a virtual three-dimensional rendering system, and virtual-real interaction middleware, including:

The scene overlay and analysis system saves the three-dimensional scene of the scene and uses the three-dimensional scene as the base map;

Video real-time solution system, which performs real-time solution on the received video stream to obtain video frames;

Multi-source data acquisition and processing analysis system, receiving and storing multi-source data, and performing basic analysis and processing, and converting multi-source data into a three-dimensional representation format;

The virtual three-dimensional rendering system uses the three-dimensional scene as a base map, maps and merges the video frames in the three-dimensional scene, and performs position matching and fusion of the multi-source data in the three-dimensional scene, and combines the virtual and real The interactive middleware performs mapping in the three-dimensional scene, renders and interacts with the fused three-dimensional scene, generates an interactive instruction in response to the interactive operation of the virtual-real interaction middleware, and sends it to the virtual-real interaction middleware;

The virtual and real interaction middleware is used to send out the interaction instructions issued by the virtual 3D rendering system.
The digital twin system of claim 1, wherein the video stream is generated by a multi-channel image acquisition system collecting images at multiple locations on the scene, and the video stream generated by the multi-channel image acquisition system flows through multiple The road image real-time return control system for return.
The digital twin system according to claim 2, wherein the system further comprises a data synchronization system for data synchronization of the video stream returned by the multi-channel image real-time return control system, and the data synchronization is specifically Time synchronization makes the returned video streams of the same batch located in the same time slice space.
The digital twin system of claim 1, wherein the video real-time resolution system comprises a video frame extraction module and a hardware decoder, wherein:

The video frame extraction module uses FFMPEG library to extract frame data from the video stream;

The hardware decoder is used to calculate the frame data to obtain the video frame.
The digital twin system of claim 1, wherein the multi-source data collection and processing analysis system comprises a multi-source data collection system and a multi-source data analysis system, wherein:

Multi-source data acquisition system for receiving and storing multi-source data returned by multi-source sensors;

Multi-source data analysis system, used for basic analysis and processing of multi-source data, and convert multi-source data into three-dimensional representation format.
The digital twin system according to claim 1, wherein the virtual-real interaction middleware comprises an instruction receiving module and an instruction transmission module, wherein:

The instruction receiving module is used to receive interactive instructions issued by the virtual 3D rendering system;

The instruction transmission module is used to transmit the interactive instruction to the entity control system pointed to by the interactive instruction.
The digital twin system according to claim 1, characterized in that, according to the position correspondence between the multi-source sensors used to return the multi-source data and the virtual-real interaction middleware, the rendering position of the multi-source data in the three-dimensional scene Corresponds to the position of the virtual and real interactive middleware.
The digital twin system according to claim 7, wherein the virtual three-dimensional rendering system also responds to changes in multi-source data to change the rendering state of the virtual-real interaction middleware in the three-dimensional scene.
A digital twin method, characterized in that it includes:

The scene overlay and analysis system saves the on-site three-dimensional scene, and uses the three-dimensional scene as a base map;

The video real-time solution system performs real-time solution on the received video stream to obtain the video frame;

The multi-source data acquisition and processing analysis system accepts and stores multi-source data, performs basic analysis and processing, and converts the multi-source data into a three-dimensional representation format;

The virtual three-dimensional rendering system uses the three-dimensional scene as the base map, maps and fuses the video frames in the three-dimensional scene, matches and fuses the multi-source data in the three-dimensional scene, and integrates the virtual and real interaction middleware Performing mapping in the three-dimensional scene, and rendering and interacting with the merged three-dimensional scene;

The virtual three-dimensional rendering system generates and sends interactive instructions to the virtual-real interactive middleware in response to the interactive operation of the virtual-real interactive middleware;

The virtual-real interaction middleware sends out the interaction instructions issued by the virtual 3D rendering system.
A computer device, characterized by comprising: a display screen, an input device, a memory, and one or more processors;

The display screen is used to display virtual and real interactive interfaces;

The input device is used to receive interactive operations;

The memory is used to store one or more programs;

When the one or more programs are executed by the one or more processors, the one or more processors implement the digital twin method according to claim 9.