WO2018086295A1

WO2018086295A1 - Application interface display method and apparatus

Info

Publication number: WO2018086295A1
Application number: PCT/CN2017/078027
Authority: WO
Inventors: 曹海恒; 谭利文; 孙伟; 陈心
Original assignee: 华为技术有限公司
Priority date: 2016-11-08
Filing date: 2017-03-24
Publication date: 2018-05-17
Also published as: CN108604385A

Abstract

Disclosed are an application interface display method and apparatus. The method in the embodiments of present application is used for displaying an interface of a 2D application program on a VR device by an application interface display apparatus. The method comprises: acquiring an interface to be displayed, wherein the interface to be displayed is an interface of a 2D application program; performing dimension conversion processing on the interface to be displayed to obtain a first left eye image and a first right eye image corresponding to the interface to be displayed, wherein the first left eye image and the first right eye image are used for presenting the interface to be displayed having a three-dimension visual effect; acquiring a first head posture of a user; adjusting the first left eye image to obtain a second left eye image, and adjusting the first right eye image to obtain a second right eye image according to the first head posture; and displaying the second left eye image in a left eye view field of the VR device, and displaying the second right eye image in a right eye view field of the VR device.

Description

Application interface display method and device

This application claims priority to Chinese Patent Application No. 201610980760.2, entitled "A Method and Apparatus for Low-Delay Stereoscopic Display of Two-Dimensional Images", filed on November 8, 2016, the entire contents of which are incorporated herein by reference. This is incorporated herein by reference.

Technical field

The present application relates to the field of computer applications, and in particular, to an application interface display method and apparatus.

Background technique

Virtual Reality (VR) technology is a computer simulation system that can create and experience virtual worlds. It uses computer to generate a simulation environment. It is a multi-source information fusion, interactive 3D dynamic vision and entity. System simulation of behavior immerses users in the environment. With the demand for quality of life, the development of virtual reality display technology has become the focus of social attention.

The virtual reality device requires that the left-eye image and the right-eye image are respectively rendered to generate a stereoscopic effect, and most of the existing application interfaces are two-dimensional (2D), which cannot meet the requirements of the virtual reality device. This makes a large number of applications impossible to use in virtual reality systems.

The prior art writes a virtual reality scene into a frame buffer of an Android system by using an Open Graphics Library (OpenGL) function in a left-right split screen manner, and uses the Android system to read the content in the frame buffer to perform rendering. Displaying the reality scene on the left and right screens of the virtual device, forming a virtual screen in the virtual reality scene, and finally directly drawing the acquired texture of the two-dimensional application interface to be displayed on the virtual screen in the virtual reality scene of the left and right screens, thereby The two-dimensional application interface simultaneously renders the left and right eye images, and has a three-dimensional effect.

However, rendering a two-dimensional application interface into an image with a three-dimensional visual effect takes a long time, and the rendered result is lagging behind, and cannot be attached to the user's field of view, which easily causes the user to have a visual misalignment, thereby causing dizziness and experience. difference.

Summary of the invention

The embodiment of the present application provides an application interface display method, which can avoid the image position and the user's visual field misalignment caused by the change of the user's head posture during the process of rendering the two-dimensional application interface into an image having a three-dimensional visual effect. Dizziness and enhance the user experience.

In view of this, the first aspect of the present application provides an application interface display method, which is used by an application interface display device to display an interface of a two-dimensional application on a VR device, and the method includes:

The application interface display device obtains an interface to be displayed. The interface to be displayed is an interface of the two-dimensional application. After the application interface display device obtains the interface to be displayed, the interface to be displayed is subjected to dimension conversion processing to obtain a first left eye corresponding to the interface to be displayed. An image and a first right eye image, the first left eye image and the second right eye image are used to present a to-be-displayed interface having a three-dimensional visual effect, and then the application interface display device acquires a current head posture of the user, that is, the first head And the second left eye image and the second right eye image are obtained by adjusting the first left eye image and the first right eye image according to the first head posture, and finally displaying the second left in the left eye field of view of the VR device. The eye image displays the second right eye image in the right eye field of view of the VR device.

It should be noted that the dimension conversion process refers to converting an interface of a two-dimensional application into an interface having a three-dimensional visual effect. The left eye image refers to an image generated for the user's left eye field of view, and the right eye image refers to an image generated for the user's right eye field of view. The VR device is divided into a left-eye view area and a right-eye view area according to the left and right eyes of the user, and the left-eye view area is a screen area or an optical lens group in the VR device aligned with the user's left-eye view, and the right-eye view area is in the VR device. A screen area or optical lens set that is aligned with the user's right eye field of view.

After performing the dimension conversion processing on the display interface to obtain the image corresponding to the left screen and the right screen, the current head posture of the user is obtained, and the image corresponding to the left screen and the right screen is adjusted according to the head posture, and then the adjustment is performed. The subsequent images are displayed on the left and right screens, respectively. That is to say, after the dimension conversion of the display interface is performed to obtain an image having a three-dimensional visual effect, the converted result is adjusted according to the latest head posture of the user, so that the position of the finally displayed image is more closely matched. The user's field of vision avoids the dizziness caused by the image position and the user's visual field misalignment caused by the user's head posture change during rendering of the two-dimensional application interface into an image with a three-dimensional visual effect, thereby improving the user experience.

With reference to the first aspect of the present application, in the first implementation manner of the first aspect of the present application, the process of performing the dimension conversion processing by the application interface display device may specifically include:

The application interface display device performs binocular rendering on the display interface, obtains a third left eye image and a third right eye image of the interface to be displayed, and performs barrel distortion processing on the third left eye image and the third right eye image to obtain a to-be-distorted process. The first left eye image of the interface and the first right eye image of the interface to be displayed are displayed.

It should be noted that, after optical processing, the magnification away from the optical axis region is lower than that near the optical axis, and the convex scene shown in the figure appears in the image plane is called barrel distortion, and the barrel distortion in the present application. The process is used to counteract the distortion produced by the optical lens in the VR device.

After obtaining an image with a three-dimensional visual effect by binocular rendering, the embodiment of the present application also performs barrel distortion processing on the image to offset the distortion generated by the optical lens in the VR device, thereby improving image quality and enhancing user experience.

With reference to the first implementation manner of the first aspect of the present application, in a second implementation manner of the first aspect of the application, the application interface display device performs binocular rendering on the display interface to obtain a third left eye image and a third right eye image. The process specifically includes:

After acquiring the interface to be displayed, the application interface display device acquires the current head posture of the user, that is, the second head posture, and then determines the first region, that is, the second region, according to the second head posture, and draws the to-be-displayed region in the first region. The third left eye image is obtained by the interface, and the third right eye image is obtained by drawing the interface to be displayed in the second area, wherein the first area is an area for displaying the interface to be displayed in the left eye image of the preset three-dimensional scene, and the second area is The area in the right eye image of the preset three-dimensional scene for displaying the interface to be displayed.

The embodiment of the present application performs binocular rendering by drawing the interface to be displayed to the preset three-dimensional scene, so that the user can browse the interface to be displayed in the preset three-dimensional scene, thereby improving the flexibility of the solution and further enhancing the user experience.

With reference to the first aspect of the present application, the first or second implementation manner of the first aspect, in the third implementation manner of the first aspect of the application, the application interface display device adjusts the first left eye according to the first head posture The process of obtaining the second left eye image and the second right eye image by the image and the first right eye image specifically includes:

The application interface display device performs asynchronous time warping on the first left eye image according to the first partial posture to obtain the second left eye. The image is subjected to asynchronous time warping of the first right eye image to obtain a second right eye image. It should be noted that asynchronous time warping is a technique of image correction. When using a virtual reality device, the head motion has been too fast, and the delay of rendering the scene, that is, the head has already passed, but the image has not been rendered. Or rendering the image of the previous frame, the asynchronous time warping solves this delay problem by distorting the image before being sent to the display device.

The embodiment of the present application provides a specific manner of adjusting an image, which improves the achievability of the solution.

With reference to the first aspect of the present application, any one of the first to third implementation manners of the first aspect, in the fourth implementation manner of the first aspect of the present application,

The application interface display device obtains the interface to be displayed, and the application interface display device obtains the interface to be displayed from the mobile terminal;

Correspondingly, the application interface display device displays the second left eye image in the left eye view area of the VR device, and displays the second right eye image in the right eye view area of the VR device. The left eye image and the second right eye image are sent to the mobile terminal, the screen of the mobile terminal includes a third area and a fourth area, the third area corresponds to a left eye view area of the VR device, and the fourth area corresponds to a right eye view of the VR device In the area, after the mobile terminal receives the second left eye image and the second right eye image sent by the application interface display device, the mobile terminal displays the second left eye image in the third area of the screen, and displays the second image in the fourth area of the screen. Right eye image.

The embodiment of the present application provides a specific manner for obtaining an interface to be displayed and displaying an interface to be displayed, thereby improving the achievability of the solution.

With reference to the fourth implementation manner of the first aspect of the embodiment of the present application, in a fifth implementation manner of the first aspect of the embodiment, the mobile terminal includes a SurfaceFlinger module, where the SurfaceFlinger is a module responsible for display synthesis in the Android system. The position in the final composite image of each layer can be calculated, and then the final display buffer is generated and displayed on a particular display device.

The application interface display device may obtain the to-be-displayed interface from the mobile terminal by: the application interface display device acquires the to-be-displayed interface from the SurfaceFlinger module;

Correspondingly, the application interface display device can send the second left eye image and the second right eye image to the mobile terminal by: the application interface display device sending the second left eye image and the second right eye image to the SurfaceFlinger module, So that the SurfaceFlinger module displays the second left eye image in the third area of the screen of the mobile terminal, and displays the second right eye image in the fourth area of the screen.

The application interface display device in the embodiment of the present application may be a device that is independent of the Android system, that is, the application interface display method in the embodiment of the present application may not depend on the Android system, and may reduce the computing load of the mobile terminal, and is used in the method. When the algorithm needs to be updated, the update can be performed independently of the Android system. When the internal architecture in the Android system is updated, the algorithm used in the method does not need to be modified accordingly, and the flexibility and versatility are higher.

The second aspect of the present application provides an application interface display device for displaying an interface of a two-dimensional application on a VR device, the device comprising:

a first acquiring module, configured to acquire an interface to be displayed, where the interface to be displayed is an interface of a 2D application;

a processing module, configured to perform dimension conversion processing on the interface to be displayed acquired by the first acquiring module, to obtain a first left eye image and a first right eye image corresponding to the interface to be displayed, and the first left eye image and the first right eye image Presenting an interface to be displayed having a three-dimensional visual effect;

a second acquiring module, configured to acquire a first head posture of the user;

The adjusting module is configured to adjust the first left eye image to obtain the second left eye image according to the first head posture acquired by the second acquiring module, and adjust the first right eye image to obtain the second right eye image;

And a display module, configured to display a second left eye image in a left eye view area of the VR device, and display a second right eye image in a right eye view area of the VR device.

With reference to the second aspect of the present application, in a first implementation manner of the second aspect of the present application, the processing module specifically includes:

a rendering unit, configured to perform binocular rendering on the display interface, and obtain a third left eye image and a third right eye image of the interface to be displayed;

The processing unit is configured to perform barrel distortion processing on the third left eye image and the third right eye image to obtain a first left eye image of the interface to be displayed and a first right eye image of the interface to be displayed.

The processing unit of the embodiment of the present application can perform distortion processing on the image to offset the distortion generated by the optical lens in the VR device, improve image quality, and enhance user experience.

With reference to the second aspect of the present application, in a first implementation manner of the second aspect of the present application, the rendering unit may specifically include:

a first acquiring subunit, configured to acquire a second head posture of the user;

Determining a sub-unit, configured to respectively determine a first area and a second area according to the second head posture, where the first area is an area for displaying an interface to be displayed in a left eye image of a preset three-dimensional scene, and the second area is a pre- The area of the right eye image of the three-dimensional scene for displaying the interface to be displayed;

The drawing subunit is configured to draw the interface to be displayed in the first area to obtain the third left eye image, and draw the interface to be displayed in the second area to obtain the third right eye image.

The rendering unit of the embodiment of the present application draws the interface to be displayed into the preset three-dimensional scene by using the drawing sub-unit, so that the user can browse the interface to be displayed in the preset three-dimensional scene, thereby improving the flexibility of the solution and further enhancing the user experience.

With reference to the second aspect of the present application, the first or second implementation manner of the second aspect, in the third implementation manner of the second aspect of the application, the adjustment module may specifically include:

The time warping unit is configured to perform asynchronous time warping on the first left eye image according to the first head posture to obtain a second left eye image, and perform asynchronous time warping on the first right eye image to obtain a second right eye image.

The embodiment of the present application provides a specific manner for adjusting an image of an adjustment module, which improves the achievability of the solution.

With reference to the second aspect of the present application, the first or second implementation manner of the second aspect, in the third implementation manner of the second aspect of the application, the first obtaining module may specifically include:

An obtaining unit, configured to acquire the to-be-displayed interface from the mobile terminal;

Correspondingly, the display module may specifically include:

a sending unit, configured to send the second left eye image and the second right eye image to the mobile terminal, so that the mobile terminal displays the second left eye image in the third area of the screen, and displays the second right eye image in the fourth area of the screen The screen of the mobile terminal includes a third area and a fourth area, the third area corresponds to a left eye view area of the VR device, and the fourth area corresponds to a right eye view area of the VR device.

The embodiment of the present application provides a specific manner for an acquiring unit to obtain an interface to be displayed and a display module to display an interface to be displayed, thereby improving the achievability of the solution.

With reference to the third implementation manner of the second aspect of the present application, in a fourth implementation manner of the second aspect of the present application, the mobile terminal includes a SurfaceFlinger module, and the acquiring unit may specifically include:

a second obtaining subunit, configured to obtain an interface to be displayed from the SurfaceFlinger module;

Correspondingly, the sending unit may specifically include:

a sending subunit, configured to send the second left eye image and the second right eye image to the SurfaceFlinger module, so that the SurfaceFlinger module end displays the second left eye image in the third area of the screen of the mobile terminal, in the fourth area of the screen The second right eye image is displayed.

The application interface display device in the embodiment of the present application may be a device independent of the Android system, that is, the method in the embodiment of the present application may be executed independently of the Android system, and the computing load of the mobile terminal may be reduced, and the algorithm used in the method may be used. When an update is required, the update can be performed independently of the Android system. When the internal architecture in the Android system is updated, the algorithm used in the method does not need to be modified accordingly, and the flexibility and versatility are higher.

A third aspect of the present application provides a computer readable storage medium having stored therein instructions that, when run on a computer, cause the computer to perform the first aspect, the first aspect of the first aspect A method as claimed in any one of the fifth implementations.

A fourth aspect of the present application provides a computer program product comprising instructions which, when executed on a computer, cause the computer to perform the first aspect described above, any one of the first to fifth implementations of the first aspect The method described in the manner.

As can be seen from the above technical solutions, the embodiments of the present application have the following advantages:

DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the present application.

1 is a flow chart of an embodiment of an application interface display method in an embodiment of the present application;

2 is a flow chart of another embodiment of an application interface display method in an embodiment of the present application;

3 is a schematic diagram of a third left eye image and a fourth left eye image in the embodiment of the present application;

4 is a schematic diagram of an embodiment of an application interface display system in an embodiment of the present application;

FIG. 5 is a flowchart of another embodiment of an application interface display method in an embodiment of the present application;

6 is a schematic diagram of an embodiment of an application interface display device in an embodiment of the present application;

7 is a schematic diagram of another embodiment of an application interface display device in an embodiment of the present application;

8 is a schematic diagram of another embodiment of an application interface display device in an embodiment of the present application;

FIG. 9 is a schematic diagram of another embodiment of an application interface display device in an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application are clearly and completely described in the following with reference to the drawings in the embodiments of the present application. It is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments.

The terms "first", "second", "third", "fourth", etc. (if present) in the specification and claims of the present application and the above figures are used to distinguish similar objects, and are not necessarily used for Describe a specific order or order. It is to be understood that the data so used may be interchanged where appropriate, such that the embodiments of the present application described herein can be implemented, for example, in a sequence other than those illustrated or described herein. In addition, the terms "comprises" and "comprises" and "the" and "the" are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to Those steps or units may include other steps or units not explicitly listed or inherent to such processes, methods, products or devices.

The embodiment of the present application provides an application interface display method and device for avoiding dislocation of an image position and a user's visual field due to a change in a user's head posture during rendering of a two-dimensional application interface into an image having a three-dimensional visual effect. The resulting dizziness enhances the user experience.

In order to facilitate the understanding of the embodiments of the present application, the technical background of the embodiments of the present application is briefly introduced:

VR devices refer to hardware products related to the field of virtual reality technology and are hardware devices used in virtual reality solutions. At present, the hardware devices commonly used in virtual reality can be roughly divided into four types: modeling devices, three-dimensional visual display devices, sound devices, and interactive devices. The VR device in the embodiment of the present application refers to a three-dimensional visual display device, such as a three-dimensional display system, a large projection system (such as CAVE), a head-mounted display device, and the like.

The VR head-mounted display device, abbreviated as VR head display, is a kind of use of a head-mounted display device to close the user's visual and auditory sense to the outside world, and guide the user to create a feeling in the virtual environment. The VR device in the embodiment of the present application includes a left-eye view area for displaying a left-eye image in a left eye of the user, and a right-eye view for displaying a right-eye image in the right eye of the user. After the user's left and right eyes respectively display the left eye image and the right eye image with differences, the user can create a stereoscopic effect in the mind. VR head display can be subdivided into three categories: external connector display, integrated machine head display, mobile terminal display. Among them, the external connector display and the integrated machine head have an independent screen, and the external connector displays the left eye image and the right eye image on the self-contained screen through the data input by the external device, so that the user is immersed in the virtual environment, The headphone display allows users to immerse themselves in a virtual environment without any input/output devices. The mobile terminal display, also called the VR glasses box, needs to put the mobile terminal into the VR glasses box, and displays the left eye image and the right eye image on the screen of the mobile terminal, and the user obtains the left side of the mobile terminal through the VR glasses box. The eye image and the right eye image produce a sense of three-dimensionality and immersion in the mind.

The application interface display method in the embodiment of the present application is used for the interface of the application interface display device to display the 2D application on the VR device. Specifically, for the external connector display, the application interface display device may be the external connector display, or may be an input device capable of being connected to the external connector, such as a personal computer (PC), a mobile phone, etc.; The application interface display device may be the integrated head display, or may be a component for rendering an image in the integrated head display; for the mobile terminal display, the application interface display device may be the mobile terminal display, or A mobile terminal capable of being placed in the mobile terminal for displaying a left eye image and a right eye image. The application interface display device may also be other devices capable of communicating with the above three types of headphones, or input devices, or mobile terminals, such as a cloud server.

The following describes the application interface display method in the embodiment of the present application. Referring to FIG. 1 , an embodiment of the application interface display method in the embodiment of the present application includes:

101. The application interface display device obtains an interface to be displayed;

The application interface display device obtains an interface to be displayed. The interface to be displayed is an interface that needs to be displayed on a screen of the display device. The interface to be displayed may be an interface of any two-dimensional application, or may be a plurality of two-dimensional applications. The interface synthesized by the interface is not limited herein. It should be understood that a two-dimensional application refers to an application developed based on two-dimensional display.

102. The application interface display device performs dimension conversion processing on the display interface, and obtains a first left eye image and a first right eye image of the interface to be displayed;

After the application interface display device obtains the interface to be displayed, performing dimension conversion processing on the interface to be displayed, and obtaining a first left eye image and a first right eye image of the interface to be displayed, the first left eye image and the first right eye image Used to render an interface to be displayed with a three-dimensional visual effect.

It should be understood that, in the embodiment of the present application, the dimension conversion process refers to converting an interface of a two-dimensional application into an interface having a three-dimensional visual effect. It should also be understood that the left eye image in the embodiment of the present application refers to an image generated for the user's left eye field of view, and the right eye image refers to an image generated for the user's right eye field of view.

103. The application interface display device acquires a first head posture of the user;

In the process of the application interface display device performing the above 101 and 102, the posture of the user's head may change. When the application interface display device performs the step 102 and obtains the first left eye image and the first right eye image, Get the user's latest head gesture, the first head gesture. It should be understood that the head posture may specifically include a user's head yaw direction, a yaw angle of the user's head, or a motion mode of the user's head, and may also include other posture information, which is not limited herein.

104. The application interface display device adjusts the first left eye image according to the first head posture to obtain a second left eye image, and adjusts the first right eye image to obtain a second right eye image.

After acquiring the first head posture, the application interface display device adjusts the first left eye image according to the first head posture to obtain a second left eye image, and adjusts the first right eye image to obtain a second right eye image.

105. The application interface display device displays the second left eye image in the left eye view area of the VR device, and displays the second right eye image in the right eye view area of the VR device.

It should be understood that, in the embodiment of the present application, the VR device is divided into a left-eye view area and a right-eye view area according to the left and right eye views of the user. Specifically, if the VR device has an independent screen, the left-eye view area is the left of the user on the screen. The area seen by the eye, the application interface display device displays the second left eye image in the area, and the right eye view area is the user on the screen The area seen by the right eye, the application interface display device displays the second right eye image in the area, and the second left eye image and the second right eye image are displayed in the left and right eyes of the user through the corresponding optical lens group; if the VR device Without a separate screen, the left eye field of view is the optical lens group on which the user's left eye is aligned on the VR device, and the application interface display device displays the second left eye image in the area of the external screen where the optical lens group is aligned. The right eye field of view is an optical lens group on which the user's right eye is aligned on the VR device, and the application interface display device displays the second right eye image on the external screen in the area where the optical lens group is aligned, and the second left eye image And the second right eye image is finally displayed in the left and right eyes of the user through the optical path deformation. In this way, the left eye field of view and the right eye field of view of the VR device respectively display the second left eye image and the second right eye image in the left and right eyes of the user, and the user can synthesize a stereoscopic image in the brain. A three-dimensional effect to be displayed interface. After performing the dimension conversion process on the display interface to obtain the image corresponding to the left and right eyes, the current head posture of the user is acquired, and the image corresponding to the left and right eyes is adjusted according to the head posture, and the adjusted image is respectively displayed in the image. The left and right eye view areas of the VR device. That is to say, after the dimension conversion of the display interface is performed to obtain an image having a three-dimensional visual effect, the converted result is adjusted according to the latest head posture of the user, so that the position of the finally displayed image is more closely matched. The user's field of vision avoids the dizziness caused by the image position and the user's visual field misalignment caused by the user's head posture change during rendering of the two-dimensional application interface into an image with a three-dimensional visual effect, thereby improving the user experience.

Based on the corresponding embodiment of FIG. 1 , the application interface display device can convert the interface of the two-dimensional application into an interface with a three-dimensional visual effect in various manners, and one of the following uses the application interface as an example in the embodiment of the present application. The display method is described in detail. Referring to FIG. 2, another embodiment of the application interface display method in the embodiment of the present application includes:

201. The application interface display device obtains an interface to be displayed.

The application interface display device obtains an interface to be displayed. The interface to be displayed is an interface that needs to be displayed on a screen of the display device. The interface to be displayed may be an interface of any two-dimensional application, or may be a plurality of two-dimensional applications. The interface synthesized by the interface is not limited herein. It should be understood that a two-dimensional application refers to an application developed based on two-dimensional display. It should also be understood that SurfaceFlinger is the module responsible for display synthesis in Android system. It can receive windows and layers as input, and calculate the final synthesis of each layer according to the parameters such as depth, transparency, size and position of each layer. The position in the image is then generated into the final display buffer (Buffer) and displayed on a specific display device.

202. The application interface display device performs binocular rendering on the display interface, and obtains a third left eye image and a third right eye image of the interface to be displayed;

It should be understood that the left eye and the right eye of the user can independently view the object, and there is a certain distance between the left and right eyes, so for the same target, the image in the left eye of the user is different from the image in the right eye of the user. The difference produced by observing one target from two points with a certain distance is called parallax. The user's brain can fuse the left-eye image and the right-eye image with parallax to produce a stereoscopic visual effect, so that the user can see the stereoscopic object.

Based on the above principle, after the application interface display device acquires the interface to be displayed, the left eye image and the right eye image of the interface to be displayed are drawn for the left eye and the right eye of the user, that is, the Stereoscopic Rendering of the display interface is obtained. Display the left eye image and the right eye image of the interface. For the convenience of description, the left eye image obtained by binocular rendering is here. The right eye image is referred to as a third left eye image and a third right eye image. 3 is an example of a third left eye image and a third right eye image. The third left eye image and the third right eye image are acquired by the VR device, and the user brain can fuse the two images to generate a stereo image. The visual effect allows the user to see the three-dimensional interface to be displayed.

Specifically, the application interface display device may draw a third left eye image and a third right eye image of the interface to be displayed for the left eye and the right eye of the user by acquiring the second head posture of the user according to the second head The first posture and the second region are respectively determined, the third left eye image is obtained by drawing the interface to be displayed in the first region, and the third right eye image is obtained by drawing the interface to be displayed in the second region, wherein the first region is An area for displaying the interface to be displayed in the left eye image of the preset three-dimensional scene, and the second area is an area for displaying the interface to be displayed in the right eye image of the preset three-dimensional scene.

In the embodiment of the present application, the user or the system may preset one or more three-dimensional scenes, and draw a left-eye image and a right-eye image of the three-dimensional scene for the left and right eyes of the user, and the user points the left eye to the VR device. The left eye field of view and the right eye are aligned with the right eye field of view of the VR device to obtain the left eye image and the right eye image, and the brain synthesis can generate a stereoscopic feeling and a immersive feeling, so that the user is placed in the preset In a 3D scene. The preset three-dimensional scenes include a display area for displaying an interface to be displayed. The corresponding area in the left eye image of the three-dimensional scene is the first area, and the corresponding area in the right eye image of the three-dimensional scene. The area is the second area. After the application interface display device draws the to-be-displayed interface in the first area, that is, the second area, when the user is in the preset three-dimensional scene, the display area sees the to-be-displayed interface. Specifically, the preset three-dimensional scene may be a movie theater, a shopping mall, a classroom, etc., and is not enumerated here. The corresponding display area may be a screen in a movie theater, an advertisement screen in a shopping mall, a blackboard in a classroom, and the like.

It should be understood that the VR technology is to immerse the user in the simulation environment, so the three-dimensional scene that the user sees using the VR device simulates the reality situation. When the user's head rotates, the three-dimensional scene seen by the user also rotates, and the scene thereof The elements in the room will change. Taking the classroom scene as an example, the user's initial field of view is set at the center of the classroom. At this time, the user can see the table and chairs in front, the podium and the entire blackboard, and when the user's head is lifted up, The user can only see the upper half of the blackboard and ceiling. Therefore, as the user's head moves, the position of the display area in the user's field of view changes, even within the user's field of view. Therefore, the application interface display device acquires the current head posture of the user, that is, the second head posture, and then determines the position of the display area in the user's field of view according to the second head posture, that is, determines the first position information of the first area. And the second location information of the second area, where the location information may be the coordinate information corresponding to the vertices of the area in the screen, or may be other information that can determine the location, which is not limited herein. Then, the application interface display device may draw the interface to be displayed into the first area according to the first location information to obtain a third left eye image, and draw the interface to be displayed into the second area according to the second location information to obtain a third right eye. image.

It should be noted that, in the embodiment of the present application, the second head posture refers to a head posture acquired when the application interface display device performs binocular rendering on the display interface by the above manner, and the first header in the following step 204 The attitude refers to the head posture acquired before the image adjustment is performed after the barrel distortion processing. The first head pose and the second head pose are application display interface devices that acquire the user's head pose at different times, the first head pose for binocular rendering and the second head pose for image adjustment. The first head posture or the second head posture in the embodiment of the present application is determined by a sensor, and may be a sensor in the VR device, a sensor in the application interface display device, or other external devices. The sensor is not limited herein.

It should also be noted that the application interface display device can also obtain the third left eye of the interface to be displayed by other means. The image and the third right eye image are not limited herein.

203. The application interface display device performs barrel distortion processing on the third left eye image and the third right eye image to obtain a first left eye image and a first right eye image of the interface to be displayed.

Since the VR device includes a plurality of sets of optical lenses, when the user obtains an image through the optical lens, the edge of the image may be distorted to different degrees. In the embodiment of the present application, the application interface display device draws a third left image for the left and right eyes of the user. After the three right eye images, the third left eye image is subjected to barrel distortion processing to obtain a first left eye image, and the third right eye image is subjected to barrel distortion processing to obtain a first right eye image, thereby generating an optical lens. The distortion is offset.

Specifically, the application interface display device may use a shader to perform barrel distortion on each of the fifth image and the sixth image to the first left eye image and the first right eye image through a set of preset parameters. The preset parameters are set for lens parameters in the VR device, such as thickness, refractive index, pitch, and the like. The application interface display device can also perform the barrel distortion processing in other manners, which is not limited herein.

204. The application interface display device acquires a first head posture of the user.

In the process of the application interface display device performing the foregoing 201 to 203, the posture of the user's head may change, and after the application interface display device performs step 203 to obtain the first left eye image and the first right eye image, the application The interface display device acquires the user's latest head posture, that is, the first head posture. It should be understood that the head posture may specifically include a user's head deflection direction, a deflection angle of the user's head, or a motion mode of the user's head. The motion mode may specifically be a left-right swing, a swing up and down, or the like, which is not limited herein. The gesture may also include other gesture information, which is not limited herein.

205. The application interface display device adjusts the first left eye image to obtain a second left eye image according to the first head posture, and adjusts the first right eye image to obtain a second right eye image.

After acquiring the first head posture, the application interface display device adjusts the first left eye image according to the first head posture to obtain a second left eye image, and adjusts the first right eye image to obtain a second right eye image. Specifically, the application interface display device may calculate a transformation matrix according to the first head posture, transform the first left eye image according to the transformation matrix to obtain a second left eye image, and transform the first right eye image to obtain a second right image. The eye image, that is, the first left eye image is asynchronously time warped to obtain a second left eye image, and the first right eye image is asynchronously time warped to obtain a second right eye image. Specifically, the application interface display device may perform an asynchronous time warping operation on the texture data of the first left eye image and the first right eye image through a set of preset parameters by using a shader to obtain a second left eye image and a second image. Right eye image. The second left eye image and the second right eye image may be obtained by performing asynchronous time warping in other manners, which is not limited herein.

It should be understood that Asynchronous Time Warp (ATW) is a technique of image correction. When using a virtual reality device, the delay of the scene rendering due to the head movement is too fast, that is, the head has been turned, but the image It has not been rendered, or the image of the previous frame is rendered. The asynchronous time warping solves this delay problem by distorting the image before being sent to the display device. Specifically, the asynchronous time warping refers to an operation of stretching and shifting an image, for example, when the acquired first head posture is rotated to the left, the application interface display device according to the first head a posture, stretching and translating the first left eye image and the first right eye image to the left to obtain a second left eye image and a second right eye image, for example, when the acquired first head posture is downward rotation And the application interface display device stretches and translates the first left eye image downward according to the first head posture to obtain the second left eye image and the second right eye. image. The manner of adjustment is different based on the acquired first head posture information, and will not be enumerated here.

206. The application interface display device displays the second left eye image on the left eye view area of the VR device, and displays the second right eye image on the right eye view area of the VR device.

It should be understood that, in the embodiment of the present application, the VR device is divided into a left-eye view area and a right-eye view area according to the left and right eye views of the user. Specifically, if the VR device has an independent screen, the left-eye view area is the left of the user on the screen. The area seen by the eye, the application interface display device displays the second left eye image in the area, the right eye view area is the area seen by the user's right eye on the screen, and the application interface display device displays the second right in the area The eye image, the second left eye image and the second right eye image are displayed in the left and right eyes of the user through the corresponding optical lens group; if the VR device does not have a separate screen, the left eye visual field is aligned on the left eye of the VR device The optical lens group, the application interface display device displays the second left eye image in the area of the external screen where the optical lens group is aligned, and the right eye field of view is the optical lens group on which the user's right eye is aligned on the VR device, the application The interface display device displays the second right eye image on the area of the external screen where the optical lens group is aligned, and the second left eye image and the second right eye image are finally deformed by the optical path. Displayed in the left and right eyes of the user. In this way, the left eye field of view and the right eye field of view of the VR device respectively display the second left eye image and the second right eye image in the left and right eyes of the user, and the user can synthesize a stereoscopic image in the brain. A three-dimensional effect to be displayed interface.

After performing the dimension conversion process on the display interface to obtain the image corresponding to the left and right eyes, the current head posture of the user is acquired, and the image corresponding to the left and right eyes is adjusted according to the head posture, and the adjusted image is respectively displayed in the image. The left and right eye view areas of the VR device. That is to say, after the dimension conversion of the display interface is performed to obtain an image having a three-dimensional visual effect, the converted result is adjusted according to the latest head posture of the user, so that the position of the finally displayed image is more closely matched. The user's field of vision avoids the dizziness caused by the image position and the user's visual field misalignment caused by the user's head posture change during rendering of the two-dimensional application interface into an image with a three-dimensional visual effect, thereby improving the user experience.

Secondly, the embodiment of the present application provides binocular rendering of the display interface, and after rendering the two-dimensional application interface into an image having a three-dimensional visual effect, the image is also subjected to barrel distortion to eliminate optical lens generation in the VR device. Distortion improves image quality and enhances user experience.

Again, the embodiment of the present application provides a variety of manners for image adjustment according to the head posture, which improves the flexibility of the solution.

In order to facilitate the understanding of the embodiments of the present application, an application scenario to which the embodiments of the present application are applied is briefly introduced. Referring to FIG. 4, a schematic diagram of a system component structure applied to the application interface display method and apparatus provided by the embodiments of the present application is provided. The system can include a mobile terminal display 401 and a mobile terminal 402. The mobile terminal 402 includes a screen, and the screen includes a third area and a fourth area. The user first needs to place the mobile terminal in the mobile terminal display 401, and the third area and the mobile end are used. The left eye field of view of the head display 401 is aligned, the fourth area is aligned with the right eye field of view of the moving head display 401, and then the moving end display 401 is worn, and the left eye is aligned with the moving end display 401. In the left-eye view area, the right eye is aligned with the right-eye view area of the moving end display 401.

The left-eye view area and the right-eye view area of the mobile terminal respectively include at least one set of optical lenses for optically processing the image displayed by the mobile terminal 402, and displaying the processed image on the user's retina. It creates a sense of three-dimensionality and immersion in the minds of users. The mobile terminal display 401 may also include a pass for tracking the posture of the user's head. Sensor, CPU for processing data, etc.

Based on the scenario corresponding to FIG. 4, another embodiment of the application interface display method in the embodiment of the present application includes:

501. The application interface display device obtains an interface to be displayed from the mobile terminal;

In the embodiment of the present application, after the user wears the mobile terminal display, when the user needs to display the interface of the two-dimensional application, the mobile terminal determines an interface of the two-dimensional application that needs to be displayed according to the user operation, and the application interface display device moves from the mobile device. The interface to be displayed is obtained at the terminal.

Specifically, in the embodiment of the present application, the mobile terminal may include a SurfaceFlinger module. SurfaceFlinger is a module responsible for display synthesis in Android. It can receive windows and layers as input, and calculate the position in the final composite image of each layer according to the depth, transparency, size, position and other parameters of each layer. , then generate the final display buffer (Buffer), and then display it to a specific display device. The mobile terminal can generate the to-be-displayed interface by using a SurfaceFlinger module, and the application interface display device acquires the to-be-displayed interface from the SurfaceFlinger module.

Optionally, in the embodiment of the present application, the application interface display device may be the mobile terminal, and the mobile terminal synthesizes the to-be-displayed interface through the SurfaceFlinger module, and then transmits the to-be-displayed interface to another independent Android through the cross-process communication interface. In the process of the system, the following steps 502 to 503 are performed in the process.

Optionally, the application interface display device in the embodiment of the present application may also be other user equipments, such as PCs, that are independent of the Android system. The user equipment can establish a connection with the mobile terminal by using a data line, a wireless network, a Bluetooth, or other means. After the mobile terminal synthesizes the to-be-displayed interface through the SurfaceFlinger module, the user equipment obtains the to-be-displayed interface from the mobile terminal through the connection. And perform the following steps 502 to 503.

Optionally, the application interface display device in the embodiment of the present application may further be a cloud server independent of the Android system, the mobile terminal communicates with the cloud server through the wireless network, and transmits the interface to be displayed synthesized by the SurfaceFlinger module to the cloud server. The cloud server receives the interface to be displayed, and performs the following steps 502 to 503.

502. The application interface display device performs a dimension conversion process on the display interface, and obtains a first left eye image and a first right eye image corresponding to the interface to be displayed.

After the application interface display device obtains the interface to be displayed, the first left eye image and the first right eye image may be obtained by performing dimension conversion processing on the display interface in the manner described in steps 202 to 203 in the corresponding embodiment of FIG. 2 above. The first left eye image and the first right eye image of the interface to be displayed may be obtained by other methods, which are not limited herein.

503. The application interface display device acquires a first head posture of the user.

In the embodiment of the present application, the sensor in the mobile terminal display can track the head posture of the user in real time, and after the application interface display device obtains the first left eye image and the second right eye image, the application interface display device moves from the moving head. The sensor in the display acquires the current head posture of the user, that is, the first head posture.

504. The application interface display device adjusts the first left eye image to obtain a second left eye image according to the first head posture, and adjusts the first right eye image to obtain a second right eye image.

After the application interface display device obtains the first head posture, the first left eye image and the first right eye image may be adjusted to obtain the second left eye image and the second manner as described in step 505 of the corresponding embodiment of FIG. 2 . The first left eye image and the first right eye image may be adjusted by other methods, which are not limited herein.

505. The application interface display device sends the second left eye image and the second right eye image to the mobile terminal.

After the application interface display device obtains the second left eye image and the second right eye image, the second left eye image and the second right eye image are sent to the mobile terminal, so that the mobile terminal displays the second left eye in the third area of the screen. The image displays the second right eye image in a fourth area of the screen. Then, the left eye of the user acquires the second left eye image in the third area by moving the headed left eye field of view, and the right eye acquires the second right eye image in the fourth area by moving the headed right eye field of view. In the brain, the second left eye image and the second right eye image can be combined into a stereoscopic image to present a three-dimensional effect to be displayed interface.

Optionally, in the embodiment of the present application, the application interface display device may be the mobile terminal, and when the mobile terminal performs the

steps

502 and 503 in another process independent of the Android system, the second left eye image and the second image are obtained. After the right eye image, the human left eye image and the second right eye image are sent to the SurfaceFlinger module through the cross-process communication interface, the display buffer is generated by the SurfaceFlinger module, and the second left eye image is displayed on the screen and Second right eye image.

Optionally, the application interface display device in the embodiment of the present application may be another user device or a cloud server that is independent of the Android system. When the user device or the cloud server performs

steps

502 and 503, the second left eye image and the second image are obtained. After the second right eye image, the second left eye image and the second right eye image may be sent to the SurfaceFlinger module through a wireless network or other manner, the display buffer is produced by the SurfaceFlinger module, and the second left eye image is displayed on the screen. And a second right eye image.

Secondly, the application interface display method in the embodiment of the present application may be run in another process independent of the Android system in the mobile terminal, or may be run in a user device or a cloud server independent of the Android system. That is, the application interface display method in the embodiment of the present application does not depend on the Android system, and can reduce the computational burden of the mobile terminal. In addition, when the algorithm used in the method needs to be updated, the update can be performed independently of the Android system, when the Android system When the internal architecture in the update is updated, the algorithm used in the method does not need to be modified accordingly, and the flexibility and versatility are higher.

In another embodiment of the application interface display method provided by the embodiment of the present application, the mobile terminal may be an Android-based mobile phone, which includes a SurfaceFlinger process, and another independent. The 3DConverter process for Android.

In the embodiment of the present application, when the user clicks the icon of the two-dimensional application in the Android mobile phone, the mobile phone starts the process corresponding to the two-dimensional application (Process100), and SurfaceFlinger creates a layer for the Process 100. At the same time, a graphic buffer (GraphicBuffer) corresponding to the Surface is created. For convenience of description, the graphic buffer corresponding to the Surface is referred to as a first image buffer (gb100). SurfaceFlinger passes the data in the first graphics buffer to Process100 through the Binder mechanism, and Process100 maps the data in gb100 to the process space. Then Process100 performs the drawing operation through the OpenGL function according to the drawing logic of the application. The drawing result is written into the process space, and the SurfaceFlinger drawing is notified through the Binder mechanism.

Driven by the timer signal, SurfaceFlinger detects whether the data in gb100 is updated every fixed period. If there is an update, it marks gb100. The content of the mark is mainly SurfaceFlinger's synthesis strategy for gb100. For example, SurfaceFlinger is through the graphics processor ( Graphics Processing Unit (GPU) or Hardware Compose (HWC) handles gb100. Here, processing refers to the synthesis of graphics buffers of multiple applications and sends them to the framebuffer display. SurfaceFlinger traverses the data in gb100 to be displayed, and calls the data into the graphics buffer (gb200) corresponding to the framebufferSurface through the call of glDrawArray function.

After the mobile phone starts the 3DConverter process, the 3DConverter obtains the data in the gb200 from the SurfaceFlinger through the cross-process communication interface (Interfacer100), that is, the texture data of the interface to be displayed, and then updates the texture data to the first texture block (P200_texture100), and then the first A texture block is used as input to the OpenGL function to render the display interface once and store the result of one rendering into the second texture block (P200_texture200). The specific process of rendering is as follows:

After updating the data in the gb200 to the first texture block, the 3DConverter determines that the preset three-dimensional scene is a cinema scene, and the scene includes a screen (display area), and the 3DConverter acquires the current head of the user through the sensor in the VR device. Gesture (first head pose), then model data of the cinema scene (including the vertices, geometry, color, etc. of the cinema model), texture data of the interface to be displayed stored in the first texture block, and the acquired head pose, etc. As the input, the process calls glDrawArray twice, calculates the position of the screen in the virtual scene, obtains the coordinates of the four vertices, and then draws the interface to be displayed to the three-dimensional according to the coordinates of the vertex and the texture data of the interface to be displayed. In the scene, a third left eye image and a third right eye image corresponding to the three-dimensional scene (including the interface to be displayed) are obtained. Then, in the shader of the OpenGL, the third left eye image and the third right eye image are barrel-distorted to the first left eye image and the first right eye image through a set of preset parameters, and the first left is The eye image and the first right eye image storage are stored in texture form into the second texture block.

After the 3DConverter stores the first left eye image and the first right eye image to the second texture block, the second texture block is used as an input of the OpenGL function, and the secondary rendering result is stored in the first texture block. . The specific process of secondary rendering is as follows:

The 3DConverter acquires the current head posture (first head posture) of the user through the sensor in the VR device, and then calculates a transformation matrix according to the head posture, and uses the transformation matrix to transform the image stored in the second texture block, And drawing the transformed image. Specifically, when drawing with OpenGL, the OpenGL shader performs an asynchronous time warping operation on the texture data in the second texture block by another set of preset parameters to obtain the second left eye image and the first Two right eye images, and the second left eye image and the second right eye image are stored in a texture form into the first texture block.

It should be understood that, in a normal case, the texture block is an input of OpenGL drawing, and the frame buffer is an output of the drawing, but the embodiment of the present application outputs the drawing result to the texture block, specifically, the first texture block is associated with the first frame buffer. (p200_faramebuffer100) at the color mount point, the second texture block is associated with the color mount point of the second pin buffer (p200_faramebuffer200), and the first render buffer can be used to store the first render result to the second. In the texture block, the result of the secondary rendering can be stored in the first texture block by calling the second frame buffer.

Finally, 3DConverter notifies SurfaceFlinger of the end of rendering through the cross-process communication interface (Interfacer200), and sends the texture data in the first texture block to SurfaceFlinger. Based on the texture data, SurfaceFlinger displays the second left-eye image in the third area of the screen of the mobile phone. At the same time, the second right eye image is displayed in the fourth area of the screen. Then, through the VR device, the user points the left eye to the left screen of the mobile phone, the right eye to the right screen of the mobile phone, and feels in the preset movie theater scene, and sees the to-be-displayed on the screen of the movie theater. interface.

The application interface display device in the embodiment of the present application is described above. The following describes the application interface display device in the embodiment of the present application. It should be understood that the application interface display device in the embodiment of the present application is used to display a 2D application on the VR device. The interface of the program, the application interface display device may be the VR device, or may be a communication device capable of connecting with the VR device, such as a PC, a mobile terminal, a cloud server, etc., or may be a component in the VR device or the communication device. Specifically, it is not limited here.

The following describes the application interface display device in the embodiment of the present application from the perspective of a function module. Referring to FIG. 6 , an embodiment of the application interface display device in the embodiment of the present application includes:

The first obtaining module 601 is configured to obtain an interface to be displayed, where the interface to be displayed is an interface of a 2D application;

The processing module 602 is configured to perform dimension conversion processing on the interface to be displayed acquired by the first obtaining module 601, and obtain a first left eye image and a first right eye image corresponding to the interface to be displayed, and the first left eye image and the first right eye. The image is used to present an interface to be displayed having a three-dimensional visual effect;

a second acquiring module 603, configured to acquire a first head posture of the user;

The adjusting module 604 is configured to adjust the first left eye image to obtain the second left eye image according to the first head posture acquired by the second acquiring module, and adjust the first right eye image to obtain the second right eye image;

The display module 605 is configured to display a second left eye image in a left eye view area of the VR device, and display a second right eye image in a right eye view area of the VR device.

After the processing module 602 performs the dimension conversion processing on the display interface to obtain the image corresponding to the left and right eyes, the second obtaining module 603 acquires the current head posture of the user, and the adjusting module 604 adjusts the image corresponding to the left and right eyes according to the head posture. The display module then displays the adjusted images on the left and right eye view areas of the VR device. That is to say, after the dimension conversion of the display interface is performed to obtain an image having a three-dimensional visual effect, the converted result is adjusted according to the latest head posture of the user, so that the position of the finally displayed image is more closely matched. The user's field of vision avoids the dizziness caused by the image position and the user's visual field misalignment caused by the user's head posture change during rendering of the two-dimensional application interface into an image with a three-dimensional visual effect, thereby improving the user experience.

Based on the embodiment corresponding to FIG. 6 , the processing module can convert the interface of the two-dimensional application into an interface having a three-dimensional visual effect in a plurality of manners, and one of the following is an example of the application interface display device in the embodiment of the present application. For a detailed description, please refer to FIG. 7. Another embodiment of the application interface display device in the embodiment of the present application includes:

The first obtaining module 701 is configured to obtain an interface to be displayed, where the interface to be displayed is an interface of a 2D application.

The processing module 702 is configured to perform dimension conversion processing on the interface to be displayed acquired by the first obtaining module 701, and obtain a first left eye image and a first right eye image corresponding to the interface to be displayed, a first left eye image and a first right eye. The image is used to present an interface to be displayed having a three-dimensional visual effect;

a second acquiring module 703, configured to acquire a first head posture of the user;

The adjusting module 704 is configured to adjust the first left eye image according to the first head posture acquired by the second acquiring module a second left eye image, and adjusting the first right eye image to obtain a second right eye image;

The display module 705 is configured to display a second left eye image in a left eye view area of the VR device, and display a second right eye image in a right eye view area of the VR device;

In the embodiment of the present application, the processing module 702 includes:

a rendering unit 7021, configured to perform binocular rendering on the display interface, and obtain a third left eye image and a third right eye image of the interface to be displayed;

The processing unit 7022 is configured to perform barrel distortion processing on the third left eye image and the third right eye image to obtain a first left eye image of the interface to be displayed and a first right eye image of the interface to be displayed.

Optionally, in the embodiment of the present application, the rendering unit 7021 may include:

a first obtaining subunit 70211, configured to acquire a second head posture of the user;

The determining subunit 70212 is configured to respectively determine the first area and the second area according to the second head posture, where the first area is an area for displaying an interface to be displayed in a left eye image of the preset three-dimensional scene, and the second area is An area in the right eye image of the preset three-dimensional scene for displaying the interface to be displayed;

The drawing sub-unit 70213 is configured to draw an interface to be displayed in the first area to obtain a third left-eye image, and draw an interface to be displayed in the second area to obtain a third right-eye image.

Optionally, in the embodiment of the present application, the adjusting module 704 may include:

The time warping unit 7041 is configured to perform asynchronous time warping on the first left eye image according to the first head posture to obtain a second left eye image, and perform asynchronous time warping on the first right eye image to obtain a second right eye image.

After the processing module 702 performs the dimension conversion processing on the display interface to obtain the image corresponding to the left and right eyes, the second obtaining module 703 acquires the current head posture of the user, and the adjusting module 704 adjusts the image corresponding to the left and right eyes according to the head posture. The display module then displays the adjusted images on the left and right eye view areas of the VR device. That is to say, after the dimension conversion of the display interface is performed to obtain an image having a three-dimensional visual effect, the converted result is adjusted according to the latest head posture of the user, so that the position of the finally displayed image is more closely matched. The user's field of vision avoids the dizziness caused by the image position and the user's visual field misalignment caused by the user's head posture change during rendering of the two-dimensional application interface into an image with a three-dimensional visual effect, thereby improving the user experience.

The embodiment of the present application provides a method for performing image adjustment according to the head posture, which improves the achievability of the solution.

For an embodiment of the present application, referring to FIG. 4, based on the system scenario corresponding to FIG. 4, another embodiment of the application interface display device in the embodiment of the present application includes:

The first obtaining module 801 is configured to obtain an interface to be displayed, where the interface to be displayed is an interface of a 2D application;

The processing module 802 is configured to perform dimension conversion processing on the interface to be displayed acquired by the first obtaining module 801, and obtain a first left eye image and a first right eye image corresponding to the interface to be displayed, a first left eye image and a first right eye. The image is used to present an interface to be displayed having a three-dimensional visual effect;

a second acquiring module 803, configured to acquire a first head posture of the user;

The adjusting module 804 is configured to adjust the first left eye image to obtain a second left eye image according to the first head posture acquired by the second acquiring module, and adjust the first right eye image to obtain a second right eye image;

The display module 805 is configured to display a second left eye image in a left eye view area of the VR device, and display a second right eye image in a right eye view area of the VR device;

The first obtaining module 801 includes:

The obtaining unit 8011 is configured to acquire the to-be-displayed interface from the mobile terminal;

Correspondingly, the display module 805 includes:

The sending unit 8051 is configured to send the second left eye image and the second right eye image to the mobile terminal, so that the mobile terminal displays the second left eye image in the third area of the screen, and displays the second right eye in the fourth area of the screen. The image, the screen of the mobile terminal includes a third area and a fourth area, the third area corresponds to a left eye view area of the VR device, and the fourth area corresponds to a right eye view area of the VR device;

Optionally, in the embodiment of the present application, the obtaining unit 8011 may include:

a second obtaining subunit 80111, configured to obtain an interface to be displayed from the SurfaceFlinger module;

Correspondingly, the sending unit 8051 can include:

The sending subunit 80511 is configured to send the second left eye image and the second right eye image to the SurfaceFlinger module, so that the SurfaceFlinger module end displays the second left eye image in the third area of the screen of the mobile terminal, in the fourth screen The area displays the second right eye image;

It should be understood that, in the embodiment of the present application, the application interface display device may be a mobile terminal as shown in FIG. 4, and may be other user devices independent of the Android system, such as a PC, and may be a cloud server independent of the Android system. It can also be other devices, which are not limited herein.

After the processing module 802 performs the dimension conversion processing on the display interface to obtain the image corresponding to the left and right eyes, the second obtaining module 803 obtains the current head posture of the user, and the adjusting module 804 adjusts the image corresponding to the left and right eyes according to the head posture. The display module then displays the adjusted images on the left and right eye view areas of the VR device. That is to say, after the dimension conversion of the display interface is performed to obtain an image having a three-dimensional visual effect, the converted result is adjusted according to the latest head posture of the user, so that the position of the finally displayed image is more closely matched. The user's field of vision avoids the dizziness caused by the image position and the user's visual field misalignment caused by the user's head posture change during rendering of the two-dimensional application interface into an image with a three-dimensional visual effect, thereby improving the user experience.

Secondly, the application interface display device in the embodiment of the present application may be a user device or a cloud server that is independent of the Android system, that is, the application interface display method in the embodiment of the present application does not depend on the Android system, and the computing burden of the mobile terminal may be reduced. In addition, when the algorithm used in the method needs to be updated, the update can be performed independently of the Android system. When the internal architecture in the Android system is updated, the algorithm used in the method does not need to be modified accordingly, and the flexibility and More versatile.

The application interface display device in the embodiment of the present application is introduced from the perspective of the function module. The application interface display device in the embodiment of the present application is introduced from the perspective of the physical hardware. Please refer to FIG. 9 , which is an application of the embodiment of the present application. A schematic structural view of the interface display device 90. The application interface display device 90 can include an input device 910, an output device 920, a processor 930, and a memory 940.

Memory 940 can include read only memory and random access memory and provides instructions and data to processor 930. A portion of the memory 940 may also include a Non-Volatile Random Access Memory (NVRAM).

Memory 940 stores the following elements, executable modules or data structures, or subsets thereof, or their extended sets:

Operation instructions: include various operation instructions for implementing various operations.

Operating system: Includes a variety of system programs for implementing various basic services and handling hardware-based tasks.

In the embodiment of the present application, the application interface display device or the VR device includes at least one display, and the processor 930 in the application interface display device is specifically configured to:

Obtaining an interface to be displayed, where the interface to be displayed is an interface of a 2D application;

Performing a dimension conversion process on the display interface, and obtaining a first left eye image and a first right eye image corresponding to the interface to be displayed, where the first left eye image and the first right eye image are used to display a to-be-displayed interface having a three-dimensional visual effect;

Obtaining a first head gesture of the user;

Adjusting the first left eye image to obtain a second left eye image according to the first head posture, and adjusting the first right eye image to obtain a second right eye image;

The control display displays a second left eye image in the left eye view area of the VR device and a second right eye image in the right eye view area of the VR device.

The processor 930 controls the operation of the application interface display device 90, which may also be referred to as a Central Processing Unit (CPU). Memory 940 can include read only memory and random access memory and provides instructions and data to processor 930. A portion of the memory 940 can also include an NVRAM. In a specific application, the components of the application interface display device 90 are coupled together by a bus system 950. The bus system 950 may include a power bus, a control bus, a status signal bus, and the like in addition to the data bus. However, for clarity of description, various buses are labeled as bus system 950 in the figure.

The method disclosed in the foregoing embodiment of the present application may be applied to the processor 930 or implemented by the processor 930. Processor 930 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the foregoing method may be completed by an integrated logic circuit of hardware in the processor 930 or an instruction in a form of software. The processor 930 may be a general-purpose processor, a digital signal processing (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or Other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general purpose processor may be a microprocessor or the processor or any conventional processor or the like. The steps of the method disclosed in the embodiments of the present application may be directly implemented by the hardware decoding processor, or may be performed by a combination of hardware and software modules in the decoding processor. The software module can be located in a conventional storage medium such as random access memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, registers, and the like. The storage medium is located in memory 940, and processor 930 reads the information in memory 940 and, in conjunction with its hardware, performs the steps of the above method. In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product.

The computer program product includes one or more computer instructions. Loading and executing the computer on a computer The program or function described in the embodiment of the present application is generated in whole or in part when the program is instructed. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be from a website site, computer, server or data center Transfer to another website site, computer, server, or data center by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL), or wireless (eg, infrared, wireless, microwave, etc.). The computer readable storage medium can be any available media that can be stored by a computer or a data storage device such as a server, data center, or the like that includes one or more available media. The usable medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (such as a Solid State Disk (SSD)) or the like.

A person skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the system, the device and the unit described above can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

In the several embodiments provided by the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application, in essence or the contribution to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium. A number of instructions are included to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, and the like, which can store program codes. .

The above embodiments are only used to explain the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still The technical solutions described in the embodiments are modified, or the equivalents of the technical features are replaced by the equivalents. The modifications and substitutions of the embodiments do not depart from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

An application interface display method for an interface of an application interface display device for displaying a two-dimensional 2D application on a virtual reality VR device, comprising:

Obtaining an interface to be displayed, where the interface to be displayed is an interface of a 2D application;

Performing a dimension conversion process on the interface to be displayed to obtain a first left eye image and the first right eye image corresponding to the interface to be displayed, where the first left eye image and the first right eye image are used Presenting the interface to be displayed having a three-dimensional visual effect;

Obtaining a first head gesture of the user;

Adjusting the first left eye image to obtain a second left eye image according to the first head posture, and adjusting the first right eye image to obtain a second right eye image;

Displaying the second left eye image in a left eye view area of the VR device, and displaying the second right eye image in a right eye view area of the VR device.
The method according to claim 1, wherein the performing dimension conversion processing on the interface to be displayed comprises:

Performing binocular rendering on the interface to be displayed to obtain a third left eye image and a third right eye image of the interface to be displayed;

Performing barrel distortion processing on the third left eye image and the third right eye image to obtain a first left eye image of the interface to be displayed and a first right eye image of the interface to be displayed.
The method according to claim 2, wherein the binocular rendering of the interface to be displayed, the obtaining of the third left eye image and the third right eye image of the interface to be displayed include:

Obtaining a second head gesture of the user;

Determining, according to the second head posture, a first area and a second area, where the first area is an area for displaying the interface to be displayed in a left eye image of a preset three-dimensional scene, the second area An area for displaying the interface to be displayed in the right eye image of the preset three-dimensional scene;

Draw the to-be-displayed interface in the first area to obtain the third left-eye image, and draw the to-be-displayed interface in the second area to obtain the third right-eye image.
The method according to claim 1, wherein the adjusting the first left eye image to obtain a second left eye image according to the first head posture, and adjusting the first right eye image to obtain the first The second right eye image includes:

Performing asynchronous time warping on the first left eye image according to the first head posture to obtain a second left eye image, and performing asynchronous time warping on the first right eye image to obtain a second right eye image.
The method according to any one of claims 1 to 4, wherein the obtaining the interface to be displayed comprises:

Obtaining the to-be-displayed interface from the mobile terminal;

Displaying the second left eye image in a left eye view area of the VR device, and displaying the second right eye image in a right eye view area of the VR device includes:

Transmitting the second left eye image and the second right eye image to the mobile terminal, so that the mobile terminal displays the second left eye image in a third area of the screen, and displays the fourth area of the screen The second right eye image, the screen of the mobile terminal includes a third area and a fourth area, where the third area corresponds to a left eye view area of the VR device The fourth area corresponds to a right eye view area of the VR device.
The method of claim 5, wherein the mobile terminal comprises a SurfaceFlinger module, and the obtaining the interface to be displayed from the mobile terminal comprises:

Obtaining the interface to be displayed from the SurfaceFlinger module.

The sending the second left eye image and the second right eye image to the mobile terminal includes:

Sending the second left eye image and the second right eye image to the SurfaceFlinger module, so that the SurfaceFlinger module end displays the second left eye image in a third area of the screen of the mobile terminal, The second right eye image is displayed in a fourth area of the screen.
An application interface display device for displaying an interface of a two-dimensional 2D application on a virtual reality VR device, comprising:

a first acquiring module, configured to acquire an interface to be displayed, where the interface to be displayed is an interface of a 2D application;

a processing module, configured to perform a dimension conversion process on the to-be-displayed interface acquired by the first acquiring module, to obtain a first left-eye image and the first right-eye image corresponding to the to-be-displayed interface, where the first The left eye image and the first right eye image are used to present the to-be-displayed interface having a three-dimensional visual effect;

a second acquiring module, configured to acquire a first head posture of the user;

And an adjusting module, configured to adjust the first left eye image to obtain a second left eye image according to the first head posture acquired by the second acquiring module, and adjust the first right eye image to obtain a second right image Eye image

a display module, configured to display the second left eye image in a left eye view area of the VR device, and display the second right eye image in a right eye view area of the VR device.
The device according to claim 7, wherein the processing module comprises:

a rendering unit, configured to perform binocular rendering on the interface to be displayed, to obtain a third left eye image and a third right eye image of the interface to be displayed;

a processing unit, configured to perform barrel distortion processing on the third left eye image and the third right eye image to obtain a first left eye image of the interface to be displayed and a first right eye of the interface to be displayed image.
The apparatus according to claim 8, wherein the rendering unit comprises:

a first acquiring subunit, configured to acquire a second head posture of the user;

a determining subunit, configured to respectively determine a first area and a second area according to the second head posture, where the first area is an area for displaying the interface to be displayed in a left eye image of a preset three-dimensional scene The second area is an area in the right eye image of the preset three-dimensional scene for displaying the interface to be displayed;

And a drawing subunit, configured to draw the to-be-displayed interface in the first area to obtain the third left-eye image, and draw the to-be-displayed interface in the second area to obtain the third right-eye image.
The apparatus according to claim 7, wherein the adjustment module comprises:

a time warping unit, configured to perform asynchronous time warping on the first left eye image according to the first head posture to obtain a second left eye image, and perform asynchronous time warping on the first right eye image to obtain a second right eye image.
The apparatus according to any one of claims 7 to 10, wherein the first acquisition module comprises:

An obtaining unit, configured to acquire the to-be-displayed interface from the mobile terminal;

The display module includes:

a sending unit, configured to send the second left eye image and the second right eye image to the mobile terminal, so that the mobile terminal displays the second left eye image in a third area of the screen, on the screen The fourth area displays the second right eye image, the screen of the mobile terminal includes a third area and a fourth area, where the third area corresponds to a left eye view area of the VR device, and the fourth area corresponds to The right eye field of view of the VR device.
The device according to claim 11, wherein the mobile terminal comprises a SurfaceFlinger module, and the obtaining unit comprises:

And a second obtaining subunit, configured to obtain the interface to be displayed from the SurfaceFlinger module.

The sending unit includes:

a sending subunit, configured to send the second left eye image and the second right eye image to the SurfaceFlinger module, so that the SurfaceFlinger module end displays the third area of the screen of the mobile terminal The second left eye image displays the second right eye image in a fourth area of the screen.
An application interface display device, configured to display an interface of a two-dimensional 2D application on a virtual reality VR device, wherein the application interface display device or the VR device includes at least one display; the application interface The display device comprises: an input device, an output device, a processor and a memory;

The memory is used to store a program;

The processor is configured to execute the program in the memory, and specifically includes the following steps:

Obtaining an interface to be displayed, where the interface to be displayed is an interface of a 2D application;

Performing a dimension conversion process on the interface to be displayed to obtain a first left eye image and the first right eye image corresponding to the interface to be displayed, where the first left eye image and the first right eye image are used Presenting the interface to be displayed having a three-dimensional visual effect;

Obtaining a first head gesture of the user;

Adjusting the first left eye image to obtain a second left eye image according to the first head posture, and adjusting the first right eye image to obtain a second right eye image;

Controlling the display to display the second left eye image in a left eye view area of the VR device, and displaying the second right eye image in a right eye view area of the VR device.
A computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 6.
A computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 6.