KR102021851B1 - Method for processing interaction between object and user of virtual reality environment - Google Patents

Abstract

The present invention relates to a method of processing interaction between a user and an object in a virtual reality environment, and controls a real-time user motion in a virtual reality environment using a head mounted device (HMD) and a hand controller. A method for processing interaction between a user and an object in a virtual reality environment through an image control terminal including an algorithm, the method comprising: a hand controller and the user to implement a user's hand read through user motion capture through a virtual hand in a virtual reality environment; Performing synchronization between the virtual hands; Surrounding environment detection step of automatically processing and activating metadata for representing motion types of a plurality of virtual objects and a virtual hand in the virtual reality environment; Check the characteristics of the virtual object including the type, size, weight, and position value of each virtual object based on the metadata, and process the motion type of the virtual hand for each characteristic of the virtual object to interact with the virtual object in real time. An interaction processing step for acting; And detecting a finger motion by subdividing the virtual hand into units of measure when a contact between the virtual hand and the virtual object is detected, and then controlling a finger by units of measure according to the detected finger motion to execute motion of the virtual hand. Motion control step. Therefore, the present invention, according to the present invention is not only the whole script control of the knuckles of each finger joint, but also independently control each finger joint by the knuckles to perform the finger motion not only excellent realism and precise omime expression can be expressed It is possible to implement the technology with commercially available HMD without additional equipment, and thus it is efficient because there is no need for a separate process for operating image information processing and eye display, and it can be applied to more than one HMD device. And flexibility can be improved.

Description

{METHOD FOR PROCESSING INTERACTION BETWEEN OBJECT AND USER OF VIRTUAL REALITY ENVIRONMENT}

The present invention relates to a method of processing an interaction between a user and an object in a virtual reality environment. More specifically, the script is controlled by the node unit of each finger joint rather than the entire hand, and the finger joint is independently controlled by the node unit. The present invention relates to a method of processing interaction between a user and an object in a virtual reality environment that enables excellent motion and precise motion expression by executing motion.

Recently, various games with excellent graphic effects have been released and are attracting much attention from gamers. These games are evaluated for their high realism with excellent content including realistic graphics and sound effects. However, due to the realistic limitation of controlling the game with only the keyboard and mouse, there is a disadvantage that the immersion is low when participating in the game. In order to improve the immersion of the game, a virtual reality game in which a user manipulates a character by taking a real action has been released.

Virtual reality refers to a cyberspace environment that is similar to reality but not real. Currently, virtual reality technology is being developed to restrict immersive virtual reality by utilizing existing interface devices such as HMD (HEAD MOUNT DISPLAY). That is, the present invention seeks to use the existing interface devices such as a display (monitor, TV, HMD, etc.), a speaker, a keyboard, and a mouse. The elements required to implement virtual reality are three-dimensional spatiality, real-time interaction and immersion. In order to implement each element, a number of input / output device development technologies that stimulate the five senses such as computer graphic technology, network communication technology, and HMD have been developed.

In particular, the input device for virtual reality, unlike a keyboard and mouse, which are conventionally used physical interfaces, are devices that can directly recognize a user's physical activity. Motion capture is an input device that adopts motion capture technology, one of 3D motion techniques. Among motion capture input devices, motion capture cameras were originally capable of normal motion recognition by attaching various types of markers to the whole body. However, due to the development of technology, devices capable of motion recognition without markers have been developed. Currently, motion capture cameras are commercialized in various motion capture devices such as Kinect and OptiTrack.

However, most of the games in the virtual reality environment still use only simple gestures and gestures, and the user's hands are detected in real time using the head, hands, legs, or the like with a sensored glove. It also recognizes movement. In this case, regardless of the movement of the finger, it is detected by the unit of the hand in real time to animate the movement of the hand or the complex movement as a blending effect between the animation to reflect in the virtual reality environment, so the shape of the finger or the movement of the finger There is a problem that the precise and natural expression of the expression is impossible.

In addition, when implementing a virtual reality environment using the HMD and a controller linked with the HMD, the image information processing and the eye to reflect the image of the user motion or gaze processing in real time to the image of the virtual reality displayed through the HMD The user needs to use a specific controller for each HMD due to the need for a separate processing process for display operation and compatibility between the HMD and the controller. Therefore, a user who wants to enjoy a variety of virtual reality can select a scenario and equipment for implementing the virtual reality. There is a problem of narrowing the width.

Korean Patent Registration No. 10-1779569 "Virtual Reality Device Interworking Input Device" Korean Patent Publication No. 10-2017-0120299 "Realistic Content Service System Using Lip Motion"

The present invention uses the trigger button of the hand controller to control scripts in the unit of each finger joint, not the entire hand, so that the motion of the hand corresponding to each virtual object in the virtual reality environment is realized on the screen. Provides a method of handling user-object interaction in a virtual reality environment that enables superior motion and precise ombudsmanship by executing finger motion by controlling each unit independently.

Among the embodiments, a method of processing interaction between a user and an object in a virtual reality environment may include a virtual reality control that controls real-time user motion in a virtual reality environment using a head mounted device (HMD) and a hand controller. A method for processing interaction between a user and an object in a virtual reality environment through an image control terminal including an algorithm, the method comprising: a hand controller and the user to implement a user's hand read through user motion capture through a virtual hand in a virtual reality environment; Performing synchronization between the virtual hands; Surrounding environment detection step of automatically processing and activating metadata for representing motion types of a plurality of virtual objects and a virtual hand in the virtual reality environment; Check the characteristics of the virtual object including the type, size, weight, and position value of each virtual object based on the metadata, and process the motion type of the virtual hand for each characteristic of the virtual object to interact with the virtual object in real time. An interaction processing step for acting; And detecting a finger motion by subdividing the virtual hand into units of measure when a contact between the virtual hand and the virtual object is detected, and then controlling a finger by units of measure according to the detected finger motion to execute motion of the virtual hand. It characterized in that it comprises a motion control step.

In this case, the hand controller is characterized in that it comprises a trigger button for the user hand and the virtual hand to work together.

The interaction processing step may include identifying a type of the virtual object based on the metadata; When the type of the virtual object is a tool, confirming a preset position value of the tool based on the virtual hand so that a tool motion of the virtual hand proceeds; When the type of the virtual object is a general object rather than a tool, the size and weight of the object are checked. When the size and weight of the object are greater than or equal to a preset size and weight, the snap / grab motion is performed. Allowing it to proceed; And picking motion when the size and weight of the object is less than a predetermined size and weight when the type of the virtual object is a general object instead of a tool.

The motion control step may include a motion type execution command for processing a finger motion of the virtual hand corresponding to each virtual object in units of knuckles of the finger joint when the contact point at which the virtual object is in contact with the virtual hand is detected. Detecting a finger motion; And a finger motion control step of performing a finger motion to move each finger joint in units of nodes so that the corresponding virtual object and the virtual hand interact with each other after the contact point in accordance with the motion type execution command.

The finger motion detection may be performed by dividing the finger joint of the virtual hand into nodes of the distal joint, the auxiliary joint, and the main joint, and detecting the distal joint, the auxiliary joint, and the main joint at the contact point of contact with the virtual object. Detecting a finger motion.

The finger motion control may include dividing a finger of the virtual hand into nodes of the distal joint, the auxiliary joint, and the main joint, and controlling the primary finger motion or the motion of the virtual object. The script control in the control order.

In the virtual reality environment of the present invention, a method of processing interaction between a user and an object may be performed by script control of each finger joint instead of the entire hand and independently of each finger joint to execute finger motion. And precise grammatical expression is possible.

In addition, the present invention can be implemented with a commercially available HMD without the addition of additional equipment, which is efficient because it does not require a separate process for operating image information processing and eye display, it is applied to more than one HMD device As a result, interoperability, extensibility and flexibility can be improved.

1 is a diagram illustrating a system for processing interaction between a user and an object in a virtual reality environment according to an embodiment of the present invention.
2 is a flowchart illustrating a method for processing interaction between a user and an object in a virtual reality environment according to an embodiment of the present invention.
3 is a diagram illustrating a motion implementation state of a virtual hand using a hand controller according to an embodiment of the present invention.
4 is a flowchart illustrating the operation of the interaction processing algorithm of FIG.
FIG. 5 is a diagram illustrating a finger motion implementation state of a virtual object and a virtual hand according to the operation of the interaction algorithm of FIG. 4.
FIG. 6 is a diagram illustrating a finger motion detection process of the motion control algorithm of FIG. 2.
FIG. 7 is a diagram illustrating a finger motion control process of the motion control algorithm of FIG. 2.
8 is a diagram illustrating an operation of a motion control algorithm in which the finger motion detection process and the finger motion control process of FIGS. 6 and 7 are organically linked.

Configurations shown in the embodiments and drawings described in the present invention are merely preferred embodiments of the present invention, and do not represent all of the technical idea of the present invention, the scope of the invention is the embodiments and drawings described in the text It should not be construed as limited by That is, since the embodiments may be variously modified and may have various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical idea. In addition, the objects or effects presented in the present invention does not mean that a specific embodiment should include all or only such effects, the scope of the present invention should not be understood as being limited thereby.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. The terms defined in the commonly used dictionary should be interpreted to correspond with the meanings in the context of the related art, and should not be interpreted as having an ideal or excessively formal meaning not explicitly defined in the present invention.

1 is a diagram illustrating a system for processing interaction between a user and an object in a virtual reality environment according to an embodiment of the present invention.

Referring to FIG. 1, an interaction processing system between a user and an object in a virtual reality environment includes a head mounted device (HMD) 100, a hand controller 200, and an image control terminal 300. do.

The HMD 100 includes, but is not limited to, a camera 110, a display 120, a motion detector 130, an image processor 140, a communicator 150, and a controller 160. It may further include components such as an LED.

The camera 110 is attached to the front to capture the external environment and the user motion, the motion detection unit 120 detects the user's movement, the image processing unit 130 for the external environment that is captured through the camera 110 The captured image is displayed on the display 120 in real time. In some cases, the motion detection unit 120 is embedded in the HMD 100 or disposed at a position spaced apart from the HMD 100 by a predetermined distance or more to detect the motion detection data of the user's movement through mutual communication with the HMD 100. Can be processed.

The communicator 150 wirelessly communicates with the hand controller 200, and the controller 160 displays the virtual reality image reflecting the user's movement on the HMD 100 according to a user's manipulation command transmitted through the communicator 150. do.

The hand controller 200 may include, but is not limited to, a communication unit 210, an operation unit 220, and a control unit 230, and may further include components such as a power supply device, a memory, an LED, a vibration motor, and the like.

The communication unit 210 transmits a user's operation command while communicating with the HMD 100, the operation unit 220 inputs an operation command for a user's movement using a plurality of buttons or sensors, and the control unit 230 controls the communication unit ( By controlling the operation of the 210 and the operation unit 220 to transmit the user's operation command to the HMD 100 to check the operating state of each component to be displayed on the LED. In this case, the manipulation unit 220 may include a trigger button 221 for allowing the motion of the virtual hand corresponding to the virtual object to be implemented in the virtual reality environment.

The image control terminal 300 is a terminal including a communication module 310, an interface module 320, a control module 330, and a storage module 340, and may include a desktop PC, a notebook computer, a tablet PC, a laptop PC, and a smart device. It may be any one of a video device capable of communication and image processing such as, and may be linked to an external device.

The communication module 310 communicates with not only the HMD 100 and the hand controller 200 but also with an external device in various manners including short range wireless communication such as Bluetooth, Wi-Fi, and mobile communication.

The interface module 320 is responsible for interfacing with the HMD 100, the hand controller 200, and an external device.

 The control module 330 extracts the real object image of the real space through the captured image of the HMD 100, determines a virtual image to be displayed at a specific position in the surrounding environment of the captured real object, and controls the hand controller 200. When the user motion for the virtual object is determined, the virtual image including the virtual object and the determined user motion is provided to the HMD 100.

 In this case, the control module 330 uses the hand controller 200 to synchronize the virtual hand in the virtual image with the actual user's hand, a metadata processing algorithm for processing basic information about the surrounding environment, and a user motion. Virtual reality control algorithms that allow user motions to be reflected in the virtual reality environment, including interaction processing algorithms that allow virtual hands to interact with virtual objects, and motion control algorithms that detect the finger motions of the user and then control the finger motions of the virtual hands. And each algorithm is organically linked to control user motion. In this case, the virtual reality control algorithm may be provided in the form of an application through a service provider that provides a virtual reality service, and may be periodically updated through a network.

The storage unit 340 stores scenario images, a virtual reality control algorithm, and virtual reality related device information related to the virtual reality environment.

2 is a flowchart illustrating a method for processing interaction between a user and an object in a virtual reality environment according to an embodiment of the present invention, and FIG. 3 illustrates a motion of a virtual hand using a hand controller according to an embodiment of the present invention. It is a figure explaining a state.

Referring to FIGS. 2 and 3, a method of processing interaction between a user and an object in a virtual reality environment may be displayed on the captured real space when a captured image of the real space photographed by the HMD 100 is received. The virtual object is determined, and the virtual image is output according to a preset scenario based on the type and location of the virtual object.

In this case, the virtual reality control algorithm provides the HMD 100 with metadata regarding the type of the virtual object, location information for each virtual object, motion type of the virtual hand for each virtual object, and the like.

The virtual reality control algorithm performs a synchronization algorithm to interact with the virtual hand and the virtual object in the virtual image to synchronize the hand controller 200 with the virtual hand using the trigger button 221 of the hand controller 200 (S10). )

That is, as shown in FIG. 3, when the image of the virtual hand 10 is output in the virtual reality environment, the user clicks the trigger button 221 of the hand controller 200 and the trigger button 221. When the trigger signal is transmitted by the synchronization with the virtual hand 300. In this manner, only a click of the trigger button 221 of the hand controller 200 may implement various motion types such as catching, bending, and pinching a tool or an object in the virtual reality environment.

The virtual reality control algorithm automatically processes metadata by performing a metadata processing algorithm for expressing the types of virtual objects and motion types of various virtual hands in the virtual reality environment (S20). The virtual object is a user in the virtual reality environment. Refers to all objects requiring motion control of the virtual hand when contacting through the hand controller 200.

The virtual reality control algorithm detects the surrounding environment through the captured image of the HMD 100 to detect virtual objects including general objects such as hammers, cameras, pencils, tools / writing tools, bottles, cups, hands, keys, and coins. Automatically process the object metadata for each object and automatically process the hand metadata of the virtual hand including virtual object-specific motion types (tool, snap, grab, pick, etc.). In this case, the object metadata includes a type, a size, a weight of a virtual object, and a position value determined based on the virtual hand.

The interaction processing algorithm checks the characteristics of the virtual object, including the type, size, weight, and position value of each virtual object, based on metadata, and performs interaction processing on the motion type of the virtual hand by the characteristics of the virtual object. (S30)

 The motion control algorithm detects the contact between the virtual hand and the virtual object and detects the finger motion by subdividing the virtual hand into the knuckles of the finger joints, that is, the main joint, the secondary joint, and the distal joint, rather than the entire hand. According to the motion, the finger motion of the virtual hand is script-controlled by each node.

In this case, the interaction processing algorithm S3 and the motion control algorithm S4 are classified for convenience of description, and the interaction processing algorithm and the motion control algorithm operate in an organic manner.

That is, if the motion control algorithm detects a finger motion when the virtual hand touches the virtual object, the interaction processing algorithm determines the motion type of the virtual hand according to the type, size, weight, and position value of the virtual object to which the virtual hand is contacted. , snap, grab, pick, etc.) and then the motion control algorithm outputs the execution command for the finger motion according to the motion type of the virtual hand.

4 is a flowchart illustrating an operation of the interaction processing algorithm of FIG. 2, and FIG. 5 is a diagram illustrating a finger motion implementation state of a virtual object and a virtual hand according to the operation of the interaction algorithm of FIG. 4.

4 and 5, when the virtual object contact of the virtual hand is detected through the motion control algorithm, the interaction processing algorithm determines whether the type of the virtual object is a tool or a general object based on the object metadata. (S31)

When the type of the virtual object is a writing tool or a tool tool, the interaction processing algorithm checks a predetermined position value of the corresponding tool based on the virtual hand so that the tool motion of the virtual hand proceeds (S2, S33, S34)

The hand controller 200 allows the user to directly feel the vibration when the user touches the virtual object through the vibration function, and allows the user's finger motion for the virtual object to be reflected in the virtual hand in real time. That is, the hand controller 200 outputs a vibration effect when the user grabs the pliers in the virtual reality environment, and the interaction processing algorithm and the motion control algorithm execute the finger motion of the virtual hand such as tightening the screw with the pliers.

The interaction processing algorithm checks the size and weight of the virtual object if it is a normal object, not a tool, and if the size and weight of the object is greater than or equal to the preset size and weight, that is, a cup or a smartphone. Allows snap / grab motion to proceed (S35, S36, S37).

However, the interaction processing algorithm allows the pick motion to proceed when the size and weight of the object is less than or equal to the preset size and weight, that is, a key or a coin (S38, S39).

As described above, the interaction processing algorithm checks the preset position value and the characteristic of the virtual object and implements the finger motion through the motion control algorithm for the virtual object.

6 is a diagram illustrating a finger motion detection process of the motion control algorithm of FIG. 2, and FIG. 7 is a diagram illustrating a finger motion control process of the motion control algorithm of FIG. 2.

As shown in FIG. 6, when the contact point of contact between the virtual object and the virtual hand is detected, the motion control algorithm detects the distal joint 1, the auxiliary joint 2, and the main joint (1) contacting each virtual object at the contact point. Finger motion is detected by the detection sequence in 3).

As shown in FIG. 7, the motion control algorithm is used to implement a natural finger motion when the virtual hand executes a basic finger motion or a tool motion, snap / grab motion, or pick motion on a virtual object after a contact point. Finger motion is controlled to move the virtual hand in the control order of the elbow, secondary joint and distal joint.

8 is a diagram illustrating an operation of a motion control algorithm in which the finger motion detection process and the finger motion control process of FIGS. 6 and 7 are organically linked.

Referring to FIG. 8, the motion control algorithm performs motion control using a basic finger motion before holding the pliers, a finger motion in a pliers grabbing attempt, a finger motion in a pliers grabbing state, a finger motion after releasing the pliers, and a basic finger motion again. Perform.

At this time, a finger motion detection process for detecting contact between the pliers and the virtual hand is performed between the finger motion in the object catch attempt state, the finger motion in the pinch catch state, and the finger motion after placing the pliers. That is, in the finger motion detection process, a motion type execution command is output so that the finger motion corresponding to the virtual object can be processed at the contact point of contact between the virtual object and the virtual hand. In the finger control process, according to the motion type execution command, the script motion for the object is scripted by the node of the finger rather than the entire hand.

Therefore, the virtual hand can express various finger motions by independently moving the main joint, the secondary joint, and the distal joint, which are nodes of each finger, and can perform fast and precise motion control to express natural movement.

Although described above with reference to a preferred embodiment of the present invention, those skilled in the art will be variously modified and changed within the scope of the invention without departing from the spirit and scope of the invention described in the claims below I can understand that you can.

100: HMD 200: hand controller
300: image control terminal 310: communication module
320: interface module 330: control module
340: storage module

Claims (6)

Interaction between the user and the object in the virtual reality environment through an image control terminal including a virtual reality control algorithm for controlling real-time user motion in the virtual reality environment using a head mounted device (HMD) and a hand-controller. In the processing method,
A photographed image obtaining step of obtaining a photographed image of the real space photographed by the HMD;
A virtual image output step of determining a virtual object to be displayed on the photographed real space and outputting a virtual image according to a preset scenario based on a type and a position of the virtual object;
Providing metadata to the HMD, the metadata including a position value determined based on the type, size, weight, and virtual hand of the virtual object;
When a trigger signal generated by a user operating a trigger button provided in the hand controller is received, the hand controller and the virtual device are synchronized by a synchronization algorithm such that the user's hand, which is read through user motion capture, is implemented through a virtual hand in a virtual reality environment. A synchronization performing step of performing synchronization between hands;
Surrounding environment detection step of automatically processing and activating metadata for representing motion types of a plurality of virtual objects and a virtual hand in the virtual reality environment;
Check the characteristics of the virtual object including the type, size, weight, and position value of each virtual object based on the metadata, and process the motion type of the virtual hand for each characteristic of the virtual object to interact with the virtual object in real time. An interaction processing step for acting;
When a contact point at which the virtual object is in contact with the virtual hand is detected, a finger motion detection outputting a motion type execution command for processing a finger motion of the virtual hand corresponding to each virtual object at each node of a finger joint at the contact point is performed. step; And
And a finger motion control step of performing a finger motion to move each finger joint in units of nodes so that the corresponding virtual object and the virtual hand interact with each other after the contact point in accordance with the motion type execution command. How to handle interactions between users and objects in the environment.
delete The method of claim 1,
The interaction processing step,
Identifying a type of the virtual object based on the metadata;
When the type of the virtual object is a tool, confirming a preset position value of the tool based on the virtual hand so that a tool motion of the virtual hand proceeds;
When the type of the virtual object is a general object rather than a tool, the size and weight of the object are checked. When the size and weight of the object are greater than or equal to a preset size and weight, the snap / grab motion is performed. Allowing it to proceed; And
And if the type of the virtual object is a general object rather than a tool, pick motion when the size and weight of the object is less than or equal to a preset size and weight. How to handle interactions between users and objects in.
delete The method of claim 1,
The finger motion detection step,
The finger joint of the virtual hand is divided into nodes of the distal joint, the auxiliary joint, and the main joint, and finger motion is detected in the order of detection of the distal joint, the auxiliary joint, and the main joint at the point of contact with the virtual object. A method of handling interaction between user and object in virtual reality environment.
The method of claim 1,
The finger motion control step,
The finger of the virtual hand is divided into nodes of the distal joint, the secondary joint, and the main joint, and script control is performed in the control order of the main joint, the secondary joint, and the distal joint when controlling the basic finger motion or motion corresponding to the virtual object. A method of handling interaction between a user and an object in a virtual reality environment.

Images