KR102630904B1 - Roi tracking and optimization technology in multi projection system for building xr environment - Google Patents

Abstract

The image processing method for removing interference with the user in the projection image, which is performed by the control module according to the present invention, is (a) photographing the front through a camera, recognizing the area where the user is located, and determining the distance between the user and the camera. calculating a distance value; (b) extracting the user's silhouette from the distance value and the image captured through the camera; (c) generating a masking image based on the distance value and the silhouette; and (d) overlapping the masking image with the image to be projected and projecting the image through a projector. Includes.

Description

ROI tracking and optimization technology in a multi-projection system for building an XR environment {ROI TRACKING AND OPTIMIZATION TECHNOLOGY IN MULTI PROJECTION SYSTEM FOR BUILDING XR ENVIRONMENT}

The present invention relates to ROI tracking and optimization technology in a multi-projection system for building an XR environment. More specifically, when a person is located in front of a screen on which an image is being projected, the person is tracked and recognized to prevent interference with the person. It's about removal technology.

This invention was developed by Seoul Business Agency as a project management agency, the 2021 XR Industry-Academic Research Council as the research project name, research on expression and exhibition techniques of art based on Metaverse XR technology as the research project name, and Bizwave and 5 other organizations (consortium) as the project implementation agency. ) and the research period is from May 1, 2021 to November 31, 2021.

The number of stages using video in the background of cultural and artistic performances such as musicals, plays, concerts, and exhibitions through the application of various realistic technologies such as VR, AR, and MR is increasing. However, when a performance or exhibition is held in front of an image projected from a projector to a screen, a phenomenon occurs in which the actor acting or the audience watching is obscured by the image. Therefore, it is difficult to recognize actors hidden in the video, and there is a disadvantage in that it becomes difficult to distinguish between visitors when taking pictures in front of exhibition works. Accordingly, when a person is positioned in front of an image projected on a screen, technology is required to recognize the person and project the image separately.

Existing realistic technologies, such as VR, AR, and MR, were technologies that used single-person devices, so their scope of application was limited, and they were limited to the extent to which individuals could view recorded videos or images using devices. Additionally, there are technical limitations such as low resolution, narrow FOV (Field of View), uncomfortable fit, and dizziness.

In addition, existing immersive technologies were limited to passive functions in which one person participates in a given action in a set manner within a set range using a single-person device within a pre-recorded video.

The present invention is intended to solve the problems of the prior art described above, and its purpose is to establish an XR environment and provide technology to remove interference after recognizing actors and audiences hidden in images projected from a projector to a screen.

As a technical means for achieving the above-described technical problem, an image processing method for removing interference with the user in a projection image, which is performed by a control module according to an embodiment of the present invention, includes (a) viewing the front through a camera; Taking pictures, recognizing the area where the user is located, and calculating the distance between the user and the camera; (b) extracting the user's silhouette from the distance value and the image captured through the camera; (c) generating a masking image based on the distance value and the silhouette; and (d) overlapping the masking image with the image to be projected and projecting the image through a projector.

The step (a) defines a plane at the position where the object is closest and furthest from the camera in the image, and converts the camera image and the window image that capture each plane between the distances to the subject. It is calculated by calculating the matrix.

Step (b) includes (b-1) recognizing an object corresponding to a user; (b-2) defining ROI space around the object; (b-3) recognizing the joint in the ROI space; and (b-4) recognizing the joint, performing binarization conversion into a grayscale image, and dividing the wall onto which the image is projected and the user's area based on a specific threshold.

The ROI space is an area required to extract the silhouette from the image captured by the camera.

The joint information extracted according to the user's movement and posture is corrected based on continuous frame information.

Before step (c), the image containing the silhouette is input to a gesture recognition learning model to receive an output of gesture recognition information, and the output gesture recognition information is input to a Tensor Flow Model to perform a gesture. It includes the steps of outputting type information and compensating the silhouette by combining the output gesture type information.

In step (c), the silhouette is extracted and corrected based on the distance value to generate a masking image according to the outline of the silhouette.

In step (d), the masking image is generated by applying a transformation matrix appropriate for the depth based on the distance value information of the part where the user is located, and the masking image cut out according to the outline of the silhouette from the image is added to the existing masking image. The video is transmitted to the screen by overlaying it on the video.

The image processing method to remove interference with the user in the projection image, which is performed by the control module, has a stored program to perform the image processing method to remove the interference with the user in the projection image, which is performed by the control module. Memory; and a processor for executing the program, wherein the processor performs the method according to execution of the program. The method includes (a) photographing the front through a camera, recognizing the area where the user is located, and calculating a distance value between the user and the camera; (b) extracting the user's silhouette from the distance value and the image captured through the camera; (c) generating a masking image based on the distance value and the silhouette; and (d) overlapping the masking image with the image to be projected and projecting the image through a projector.

The present invention is an ROI tracking and optimization technology in a multi-projection system for building an XR environment. It tracks and recognizes people located in front of a screen projected through a projector, removes interference, and then reprojects the image to provide high-quality performances and exhibitions. make it possible

Figure 1 is a conceptual diagram of ROI tracking and optimization technology in a multi-projection system for building an XR environment according to an embodiment of the present invention.
Figure 2 is a configuration diagram of ROI tracking and optimization technology in a multi-projection system for building an XR environment according to an embodiment of the present invention.
Figure 3 is a flowchart of ROI tracking and optimization technology in a multi-projection system for building an XR environment according to an embodiment of the present invention.
Figure 4 is an example diagram illustrating ROI tracking of ROI tracking and optimization technology in a multi-projection system for building an XR environment according to an embodiment of the present invention.
Figure 5 is an example diagram for explaining ROI tracking and optimization technology in a multi-projection system for building an XR environment according to an embodiment of the present invention.
Figure 6 is an example diagram illustrating ROI tracking and optimization technology for multiple people in a multi-projection system for building an XR environment according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware. Meanwhile, '~ part' is not limited to software or hardware, and '~ part' may be configured to reside in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

The “user terminal” mentioned below may be implemented as a computer or portable terminal that can connect to a server or other terminal through a network. Here, the computer is, for example, a laptop equipped with a web browser, a desktop, a laptop, a VR HMD (e.g., HTC VIVE, Oculus Rift, GearVR, DayDream, PSVR, etc.), etc. may include. Here, VR HMD is for PC (e.g. HTC VIVE, Oculus Rift, FOVE, Deepon, etc.), mobile (e.g. GearVR, DayDream, Storm Magic, Google Cardboard, etc.), and console (PSVR). Includes independently implemented Stand Alone models (e.g. Deepon, PICO, etc.). Portable terminals are, for example, wireless communication devices that ensure portability and mobility, including smart phones, tablet PCs, and wearable devices, as well as Bluetooth (BLE, Bluetooth Low Energy), NFC, RFID, and ultrasonic devices. , may include various devices equipped with communication modules such as infrared, WiFi, and LiFi. In addition, “network” refers to a connection structure that allows information exchange between nodes such as terminals and servers, including a local area network (LAN), a wide area network (WAN), and the Internet. (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, and ultrasound. This includes, but is not limited to, communication, Visible Light Communication (VLC), LiFi, etc.

It may be comprised of a memory in which a program (or application) is stored and a processor that executes the above program. Here, the processor can perform various functions depending on the execution of programs stored in memory, and each function can be represented by detailed components included in the processor.

First, Figure 1 is a conceptual diagram to explain an image processing method for removing interference with a user in a projection image, which is performed by the control module 200.

The camera 100 can track the movement and position of a person in front of the screen on which the projector 300 image is projected. Videos can be not only moving images, but also images such as photographs. The control module 200 can extract pose data from the person's movement and position. After masking the extracted pose data, the image is projected on the screen through the projector 300.

Hereinafter, Figure 2 is a configuration diagram of ROI tracking and optimization technology in a multi-projection system for building an XR environment according to an embodiment of the present invention.

ROI tracking and optimization technology in a multi-projection system for building an XR environment according to an embodiment of the present invention consists of a camera 100, a control module 200, and a projector 300. The camera 100 can capture the screen on which the image of the projector 300 is projected and track the movement of the person in front of it. The captured image is transmitted to the control module 200 to extract the person's pose data. After masking the extracted pose data, the image is projected through the projector 300.

Hereinafter, Figure 3 is a flowchart of ROI tracking and optimization technology in a multi-projection system for building an XR environment according to an embodiment of the present invention.

First, the user in front of the image projected from the projection system is photographed (S110).

For example, in a musical, performance, or exhibition, when a user (actor or visitor) is positioned in front of a video or image projected on the background, the camera 100 photographs the user and tracks the actor or visitor within the frame.

Extract the distance value from the captured image (S120).

For example, in images captured by users (actors or spectators) in front of projected images and images, the user defines the planes at the closest and farthest positions from the camera 100, and the camera (100) captured each plane. ) Calculate the transformation matrix between the image and the window image to extract the distance value.

The user silhouette is extracted based on the distance value (S130).

For example, in a musical, performance, or exhibition, a user (actor or audience member) in front of a video or image projected on the background can recognize the user from the distance value extracted from the captured video and track the user's movement in the part where silhouette extraction is required. Define ROI space. After recognizing the object's joints in the defined ROI space, it is binarized and converted into a grayscale image, and based on a specific threshold, the wall on which the image is projected and the user's area are separated and tracked. Joint information that is extracted unstable depending on the user's movement and posture can be corrected based on continuous frame information. Correcting the irregularly shaped user silhouette area may reduce the accuracy of the user silhouette information due to the mixing of the target human body data to be extracted from the captured image information and information extracted from surrounding objects with similar distance values, so only the target data is selected. Post-processing work is required. Extract silhouette and joint structure from separately tracked images. The process of defining the ROI space and extracting the silhouette and joint (skeleton) structure is explained with reference to FIG. 4 below. In addition, specifying a specific threshold and dividing the wall on which the image is projected and the user's area into near and far areas and tracking them separately will be explained with reference to FIG. 5.

Referring to FIG. 4 below, the process of extracting the silhouette and joint (skeleton) structure through the distance value in the image captured by the user (actor or visitor) in front of the video and image projected on the background in a musical, performance, or exhibition can be seen. You can. Based on the user's pose, joint (skeleton) points are set and connected with lines to extract the joint (skeleton) structure.

Referring to FIG. 5 below, a user (actor or visitor) in front of a video or image projected on the background in a musical, performance, or exhibition defines an ROI space to track the user's movement in the captured video, recognizes the joints of the object, and then , when binarizing and converting to a grayscale image and applying a specific threshold, it can be confirmed that the shape of the object is removed within the frame.

Silhouette correction is performed by inputting it into the learning model (S140).

The image containing the extracted silhouette is input into a learning model (gesture recognition model) to receive output of gesture information. The output gesture information is input into the TensorFlow model to derive the type of gesture, and the silhouette is corrected by combining the derived information. Extracting silhouettes when there are multiple people in a captured image is explained with reference to FIG. 6 below.

Referring to FIG. 6, objects of multiple people are tracked in a captured image and silhouettes are extracted by defining an ROI space. At this time, a silhouette is extracted from the ROI space at a distance value similar to the human body data. After recognizing the object's joints in the extracted silhouette, the image is binarized and converted to a grayscale image, and when a specific threshold is applied, the shapes of multiple people are removed.

In an additional embodiment of the present invention, when tracking and removing objects of multiple people, only the objects that are specifically selected can be removed.

Create a masking image considering the distance value and silhouette. (S150)

The screen on which the projector 300 image is projected is photographed, the distance value is extracted by tracking the movement of the user in front, and the silhouette is extracted and corrected based on the distance value to create a masking image according to the outline of the silhouette.

A masking image is overlaid on the video to be transmitted and transmitted. (S160)

A masking image cut from the existing video according to the outline of the silhouette is overlaid on the existing video and the video is transmitted to the screen. Since the masking image is located in the part where the user is located in the transmitted video, it is possible to identify the user located in front of the screen where the video is transmitted.

In an additional embodiment of the present invention, when a prop used in a performance or exhibition is placed in front of a video or image projected on the background in a musical, performance, or exhibition, the silhouette information of the prop is extracted to create a masking image and added to the video to be transmitted. It can be transmitted by overlaying a masking image.

In an additional embodiment of the present invention, a masking image is created on an object (person or object) located in front of the screen where the image is transmitted from the projector, and when the masking image is overlaid on the image to be transmitted and transmitted, the masking image is removed and the image is transmitted. By having the object obscured, the appearance and disappearance of the object can be arbitrarily controlled.

One embodiment of the present invention may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may include all computer storage media. Computer storage media includes both volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data.

Although the methods and systems of the present invention have been described with respect to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: Camera 200: Control module
300: Projector

Claims (9)

In an image processing method for removing interference with a user in a projection image, which is performed by a control module,
(a) capturing the front through a camera, recognizing the area where the user is located, and calculating the distance between the user and the camera;
(b) extracting the user's silhouette from the distance value and the image captured through the camera;
(c) generating a masking image based on the distance value and the silhouette; and
(d) overlapping the masking image with the image to be projected and projecting the image through a projector;
Including,
Step (b) above is
(b-1) recognizing an object corresponding to a user;
(b-2) defining ROI space around the object;
(b-3) recognizing a joint in the ROI space; and
(b-4) comprising the step of recognizing the joint and then binarizing and converting it into a grayscale image, dividing the wall on which the image is projected and the user's area based on a specific threshold,
If there are multiple recognized objects, select and remove one or more objects,
If the object is an object, extract silhouette information of the object to generate the masking image,
Control exposure of the object by creating or removing the masking image based on the object,
The ROI space is
This is an area necessary to extract the silhouette from the image captured by the camera,
Correcting the joint information extracted according to the user's movement and posture based on continuous frame information,
Before step (c) above
The image containing the silhouette is input to a gesture recognition learning model to receive output of gesture recognition information, and the output gesture recognition information is input to a Tensor Flow Model to output gesture type information. Including the step of compensating the silhouette by synthesizing the gesture type information,
An image processing method to remove user interference in a projection image, performed by a control module. According to claim 1,
Step (a) above is
In the image, the user defines the planes at the closest and farthest positions from the camera, and calculates the conversion matrix between the camera image and the window image taken for each plane between the distances to the user. To do,
An image processing method to remove user interference in a projection image, performed by a control module. delete delete delete delete According to claim 1,
Step (c) above is
Extracting and correcting the silhouette based on the distance value to generate a masking image according to the outline of the silhouette,
An image processing method to remove user interference in a projection image, performed by a control module. According to claim 1,
Step (d) above is
The masking image is generated by applying a transformation matrix appropriate for the depth based on the distance value information of the part where the user is located,
The masking image cut from the video according to the outline of the silhouette is overlaid on the existing video and the video is transmitted to the screen.
An image processing method to remove user interference in a projection image, performed by a control module. In an image processing method to remove interference with a user in a projection image, performed by a control module
a memory storing a program for performing an image processing method to remove user interference in a projection image, which is performed by a control module; and
Includes a processor for executing the program,
The processor performs the method according to execution of the program,
The above method is
(a) capturing the front through a camera, recognizing the area where the user is located, and calculating the distance between the user and the camera;
(b) extracting the user's silhouette from the distance value and the image captured through the camera;
(c) generating a masking image based on the distance value and the silhouette; and
(d) overlapping the masking image with the image to be projected and projecting the image through a projector;
Including,
Step (b) above is
(b-1) recognizing an object corresponding to a user;
(b-2) defining ROI space around the object;
(b-3) recognizing a joint in the ROI space; and
(b-4) comprising the step of recognizing the joint and then binarizing and converting it into a grayscale image, dividing the wall on which the image is projected and the user's area based on a specific threshold,
If there are multiple recognized objects, select and remove one or more objects,
If the object is an object, extract silhouette information of the object to generate the masking image,
Control exposure of the object by creating or removing the masking image based on the object,
The ROI space is
This is an area necessary to extract the silhouette from the image captured by the camera,
Correcting the joint information extracted according to the user's movement and posture based on continuous frame information,
Before step (c) above
The image containing the silhouette is input to a gesture recognition learning model to receive output of gesture recognition information, and the output gesture recognition information is input to a Tensor Flow Model to output gesture type information. Including the step of compensating the silhouette by synthesizing the gesture type information,
An image processing device for removing user interference in a projection image, performed by a control module.

Images