JP2018036720A - Virtual space observation system, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual space observation system which takes a burden applied to a body and safety into consideration, while providing an intuitive observation experience in observation in a virtual space.SOLUTION: The virtual space observation system includes: a display; a motion sensor capable of detecting the attitude of the display; and an information processing part for displaying an image in a predetermined viewing angle frame in a predetermined observation direction in the virtual space according to the attitude of the display detected by the motion sensor, on the display. The information processing part further includes an observation direction control part for changing an observation direction in the virtual space according to an observation direction control signal generated regardless of the attitude change of the display on the basis of a predetermined operation by the user of the virtual space observation system.SELECTED DRAWING: Figure 20

Description

  The present invention relates to a virtual space observation system that observes a virtual space image, for example.

  In recent years, a system in which a predetermined image is presented to both eyes of a user using a head-mounted display (HMD) or the like, thereby causing the user to perceive a virtual space or an object has been spreading. In such a virtual space observation system using a head-mounted display, a head tracking function is often implemented in order to realize an intuitive observation experience similar to that in a real space. According to this head tracking function, by changing the display posture corresponding to the user's head posture, an image within a predetermined angle frame in a direction corresponding to the display posture in the virtual space is presented on the display. .

  As an example, Patent Document 1 discloses a technique for displaying an image that allows a user to perceive a virtual selection item called an information belt around a user on a see-through head mounted display including a motion sensor. In this example, when the user is not walking and moves the head and observes the information belt, the display coordinates in the virtual space correspond to the head direction in a fixed state. A predetermined image is drawn.

Japanese Patent Laid-Open No. 2006-82411

  However, the virtual space observation system having the head tracking function has the following inconveniences because the intuitive observation experience similar to that in the real space is realized. That is, when a user who stands up or sits on a chair and observes the virtual space tries to observe his / her back side in the virtual space, the head corresponding to the display must also face the back side, You have to twist your neck and upper body to the back. At this time, when using an immersive head-mounted display that not only places a heavy burden on the neck and upper body, but also shields the surrounding field of view, twist the neck and upper body to the back without checking the surroundings. Therefore, it is not preferable in terms of safety because of the possibility of contact with people and objects. Such a problem can occur not only when observing the back direction but also when observing other directions such as vertically upward.

  In addition, in the virtual space observation system using the above-described head mounted display, by holding a smartphone capable of detecting the display posture in a virtual space having a coordinate system fixed in the space, Even in the virtual space observation system for observing the state of a certain direction, the same problem arises because, in order to observe a predetermined direction, the smartphone must be held in that direction.

  The present invention has been made in view of the above-described technical background, and its purpose is to provide an intuitive observation experience for observation in a virtual space, while taking into consideration the burden on the body and safety aspects. It is to provide a virtual space observation system made.

  Still other objects and operational effects of the present invention will be easily understood by those skilled in the art by referring to the following description of the specification.

  The above technical problem is solved by the following configuration.

  That is, the virtual space observation system according to the present invention includes a display, a motion sensor capable of detecting the orientation of the display, and a virtual space on the display corresponding to the orientation of the display detected by the motion sensor. An information processing unit that displays an image within a predetermined field angle frame in a predetermined observation direction within the virtual space observation system, wherein the information processing unit is further predetermined by a user of the virtual space observation system And an observation direction control unit that changes the observation direction in the virtual space in accordance with an observation direction control signal generated regardless of the change in the posture of the display based on the above operation.

  According to such a configuration, it is possible to arbitrarily change the observation direction in the virtual space as needed without changing the display posture while providing an intuitive observation experience by the head tracking process.

  In the above-described invention, the observation direction control unit further continuously changes the observation direction in the virtual space in accordance with a continuous observation direction control signal, whereby the observation direction is displayed on the display. The images within a predetermined field angle frame corresponding to may be continuously displayed.

  According to such a configuration, the process in which the observation direction is changed can be confirmed through continuous images. Therefore, it is possible to intuitively understand which direction the observation direction is changed in the virtual space, and it is also possible to observe the state up to the target observation direction.

  In the above-described invention, the continuous display of the image within the predetermined field angle frame may be performed along a predetermined circular orbit centered on the observation point in the virtual space.

  According to such a configuration, if the observation direction is rotated, it is possible to return to the original direction once after making a round, and therefore, the possibility of losing sight of the own observation direction in the virtual space is reduced.

  In the above-described invention, the information processing unit may further include an observation point control unit that moves the observation point in the virtual space to a nearest observation direction by a predetermined distance in accordance with a predetermined observation point movement signal.

  According to such a configuration, since the observation point itself can be freely moved in the virtual space, the virtual space can be freely searched.

  In the above-described invention, the display, the motion sensor, and the information processing unit are included in a single information processing apparatus, and the virtual space observation system further includes the display arranged in front of the eyes. The information processing apparatus can be incorporated, has a head mounting tool that can be mounted on the user's head, and a predetermined operator, and can be connected to the information processing apparatus by a wired or wireless connection, And a body-mounted operation device.

  According to such a configuration, a head-mounted display can be realized using the display, the motion sensor, and the information processing unit provided in the information processing device incorporated in the head-mounted device.

  In the above-described invention, the observation direction control signal may be generated by operating the operation element of the operation device.

  According to such a configuration, the observation direction control signal can be reliably generated by the operation element even when the field of view is limited or completely absent by wearing the head-mounted tool.

  In the above-described invention, the observation direction control signal continuously changes the observation direction in the virtual space while an input operation is performed using the operation element of the operation device. An image within a predetermined field angle frame corresponding to the viewing direction may be continuously displayed on the display.

  According to such a configuration, since the observation direction can be changed only while input is performed by the operation element, there is little possibility of losing sight of the observation direction of the observation direction change operation. This can be realized with a simple operation.

  In the above-described invention, the operating device further includes a motion sensor for an operating device that detects the attitude and motion of the operating device, and the observation direction control signal is generated according to the attitude or motion of the operating device. Also good.

  According to such a configuration, the observation direction in the virtual space can be controlled by changing the posture or movement of the controller device without changing the head posture.

  In the above-described invention, the observation direction control unit starts from an instruction area in an angle-of-view frame when a predetermined direction is instructed by the operating device, and the start point when the instruction area moves from the start point With respect to the movement of the pointing region, the observation direction may be changed symmetrically with respect to the point.

  According to such a configuration, it is possible to change the observation direction in the virtual space as if dragging, starting from a predetermined area of the image in the angle frame.

  In the above-described invention, the observation direction control unit starts from an instruction area in an angle-of-view frame when a predetermined direction is instructed by the operating device, and the start point when the instruction area moves from the start point The observation direction may be changed so as to coincide with the relative positional relationship between the display area and the indication area.

  According to such a configuration, the observation direction can be changed intuitively using the operation device.

  In the above-described invention, the virtual space observation system further includes a camera that captures a real world image, and the information processing unit displays the real world image on the display and superimposes the real world image on the display. Then, instead of the image in the predetermined field angle frame, a predetermined image in a predetermined observation direction in the virtual space is displayed corresponding to the attitude of the display detected by the motion sensor. Also good.

  According to such a configuration, it is possible to arbitrarily change the observation direction in the virtual space without changing the display posture, and thereby it is possible to change only the image related to the virtual space to be superimposed.

  In the above-described invention, the virtual space observation system further includes a microphone for voice input, and the information processing unit further generates the observation direction control signal based on the microphone input, thereby You may provide the audio | voice input process part which changes the said observation direction in virtual space.

  According to such a configuration, there is no need to have any operation device or the like for the operation of changing the observation direction, both hands are free, and the field of view is limited or completely absent by wearing the head wearing tool. In addition, the observation direction in the virtual space can be changed appropriately.

  In the above-described invention, the virtual space observation system further includes a camera for detecting a line of sight, and the observation direction control unit further includes a gaze area in the image within the predetermined field angle frame based on a camera input. And a line-of-sight detection processing unit that changes the observation direction so that the area becomes the center of the image when it is detected that the same area is watched for a predetermined time or more.

  According to such a configuration, there is no need to have any operation device or the like for the operation of changing the observation direction, both hands are free, and the field of view is limited or completely absent by wearing the head wearing tool. In addition, the observation direction in the virtual space can be changed appropriately.

  In the above-described invention, the motion sensor may include at least a 3-axis acceleration sensor, a 3-axis angular velocity sensor, and a 3-axis geomagnetic sensor.

  According to such a configuration, the orientation of the display can be detected with high accuracy from the detection values of the sensors. In particular, it is also possible to calculate the attitude in the absolute coordinate system using the detected values of the triaxial acceleration sensor and the triaxial geomagnetism.

  In the above-described invention, the motion sensor for the operation device may include at least a 3-axis acceleration sensor, a 3-axis angular velocity sensor, and a 3-axis geomagnetic sensor.

  According to such a configuration, the orientation of the display can be detected with high accuracy from the detection values of the sensors. In particular, it is also possible to calculate the attitude in the absolute coordinate system using the detected values of the triaxial acceleration sensor and the triaxial geomagnetism.

  The present invention can also be considered as a virtual space observation method. That is, the virtual space observation method is a virtual space observation method for observing a virtual space recognized through an image displayed on a display, and includes a motion detection step for detecting an attitude of the display, An information processing step of displaying an image in a predetermined field angle frame in a predetermined observation direction in a virtual space corresponding to the posture of the display detected by the motion detection step, and the information processing step further includes And an observation direction control step of changing an observation direction in the virtual space in accordance with an observation direction control signal generated regardless of a change in posture of the display based on a predetermined operation by the user.

  According to such a configuration, it is possible to provide a method capable of arbitrarily changing the observation direction in the virtual space without changing the display posture.

  Furthermore, the present invention can also be considered as a virtual space observation computer program. That is, the computer can be thought of as a virtual space observation computer program that causes a computer to function as an information processing unit in the virtual space observation system described above.

  According to such a configuration, it is possible to provide a computer program that can arbitrarily change the observation direction in the virtual space without changing the display posture.

  ADVANTAGE OF THE INVENTION According to this invention, the virtual space observation system which can change the observation direction in virtual space arbitrarily, without changing a display attitude | position can be provided. Accordingly, it is possible to provide a virtual space observation system in which consideration for the burden on the body and safety is provided while providing an intuitive observation experience in observation in the virtual space.

FIG. 1 is an explanatory diagram showing a use state of the system. FIG. 2 is a perspective view showing an appearance of the head-mounted device. FIG. 3 is an explanatory diagram of a procedure for incorporating the smartphone into the head-mounted device. FIG. 4 is an explanatory diagram illustrating a usage state of the head-mounted device. FIG. 5 is a block diagram showing an electrical hardware configuration of the smartphone. FIG. 6 is a perspective view showing the appearance of the controller. FIG. 7 is a block diagram showing an electrical hardware configuration of the controller. FIG. 8 is a flowchart regarding the preparation procedure of the game. FIG. 9 is a flowchart regarding mode selection. FIG. 10 is a screen view of the initial state of the game. FIG. 11 is an explanatory diagram illustrating a display example of a parallax image. FIG. 12 is a conceptual diagram when a global image is used. FIG. 13 is a flowchart regarding the head tracking process. FIG. 14 is an explanatory diagram relating to the coordinate system. FIG. 15 is a conceptual diagram relating to coordinate system matching. FIG. 16 is an explanatory diagram for explaining the state of head tracking. FIG. 17 is an interrupt processing flowchart regarding the moving operation in the virtual space. FIG. 18 is an explanatory diagram regarding the forward movement in the virtual space. FIG. 19 is an interrupt processing flowchart (part 1) relating to the observation direction changing operation in the virtual space. FIG. 20 is an explanatory diagram (first embodiment) regarding a change mode of an observation direction. FIG. 21 is an explanatory diagram of the movement trajectory of the picture frame. FIG. 22 is an interrupt processing flowchart (part 2) relating to the observation direction changing operation in the virtual space. FIG. 23 is an explanatory diagram (second embodiment) regarding a change mode of an observation direction. FIG. 24 is an explanatory diagram (third embodiment) regarding a change mode of an observation direction. FIG. 25 is an explanatory diagram (fourth embodiment) relating to a manner of changing the observation direction. FIG. 26 is a conceptual diagram when a panoramic image is used. FIG. 27 is a conceptual diagram when hemispherical images are used. FIG. 28 is a conceptual diagram when main information processing is performed by an external device. FIG. 29 is an explanatory diagram relating to a modified example in the case of the non-immersive type.

  Hereinafter, a preferred embodiment of a virtual space observation system according to the present invention will be described in detail with reference to the accompanying drawings.

[1. First Embodiment]
[1.1 Overall system configuration]
FIG. 1 shows an external view relating to a use state of an example of a virtual space observation system 1 according to the present invention. As is apparent from the figure, in this example, the virtual space observation system 1 includes a controller 10 that is a hand-held operating device held by the hand of the user (or operator) 40, and the head of the user 40. The head mounting device 20 is mounted on the head 41 via the head circumference belt 21, and the smartphone 50 is an information processing device incorporated in the head mounting device 20. In addition, in the same figure, since the smart phone 50 is built in the inside of the head mounting tool 20, it is not illustrated.

[1.2 Structure of head-mounted device]
The configuration of the head mounting tool 20 will be described with reference to FIGS. As shown in FIGS. 2 and 3, the head-mounted device 20 has a main body 23 having an opening 24 on the front surface and a rear surface applied to both eyes 43 of the user 40. The main body 23 is combined with a head belt 21 applied to the periphery of the head of the user 40 and a head belt 22 applied to the top of the head. A lid plate 25 that opens outward is hinged to the front opening 24 so as to be openable and closable. The lid plate 25 closes the front opening 24, and both are fixed by lid plate fasteners 27 and 28. The On the inner surface side of the cover plate 25, the smartphone 50 is fixed with an appropriate holder 26 with the display screen 51 facing the inner surface side.

  3A and 3B show details of the procedure for incorporating the smartphone 50 into the head-mounted device 20. First, as shown in FIG. 3A, the smartphone 50 has the center of the four holders 26 arranged on the inner surface side of the lid plate 25 in a state where the lid plate 25 on the front surface of the head-mounted device 20 is opened. It is arranged and incorporated in the part. FIG. 3B shows the head mounted device 20 after the smartphone 50 is incorporated. As is clear from the figure, the smartphone 50 is detachably held on each side by a holder 26 on the inner surface side of the lid plate 25. In this state, by engaging the male lid plate fastener 28 and the female lid plate fastener 27 and closing the lid plate 25, the smartphone with the display screen 51 facing the inner surface side of the main body 23. 50 can be fixed.

  When the user 40 wears the head wearing tool 20 in which the smartphone 50 is incorporated in the head 41, the optical held by the partition plate 29 is placed in front of the eyes 43 of the operator 40 as shown in FIG. 4. The display screen 51 of the smartphone 50 faces through the system 30. Therefore, the head mounting tool 20 functions as a so-called head mounted display (HMD).

[1.3 Configuration of smartphone]
As shown in FIG. 3 or FIG. 6, the smartphone 50 has a display 51 at the center of one surface, operation buttons, speakers, and an in-camera above and below, and an out-camera on the other surface. . As will be described later with reference to FIG. 5, the smartphone 50 incorporates a triaxial acceleration sensor 506, a triaxial angular velocity sensor 507, and a triaxial geomagnetic sensor 508 that constitute a motion sensor. Therefore, it is possible to freely detect changes including the position, posture, and / or movement of the smartphone 10 by these sensors.

  A block diagram showing an example of the electrical hardware configuration of the smartphone is shown in FIG. As shown in the figure, the electric circuit of the smartphone 50 includes a control unit 501, a storage unit 502, a display unit 51, an input operation unit (touch panel) 504, an input operation unit (button) 505, and a triaxial acceleration sensor 506, 3. Axial angular velocity sensor 507, triaxial geomagnetic sensor 508, GPS unit 509, illuminance sensor 510, timer 511, battery 512, vibration unit 513, communication unit 514, audio output unit 515, audio input unit 516, speaker unit 517, camera unit 518 And a connector 519.

  The control unit 501 includes a CPU (including a SoC (System-on-chip), MCU (Micro Control Unit), or FPGA (Field-Programmable Gate Array)) that executes programs for controlling various operations of the smartphone. Prepare.

  The storage unit 502 includes a ROM, a RAM, a storage, and the like, and stores a program for realizing general functions of the smartphone 50, a game program described later, and various other data.

  The display 51 includes a display device such as a liquid crystal display, an organic EL display, or an inorganic EL display, and displays characters, figures, symbols, and the like. The display 51 also functions as a display unit of a head mounted display (HMD) in connection with the present invention. That is, an image constituting a video game or the like is displayed on the display screen of the display 51. This image is preferably subjected to 3D processing by a known method. As an example of a technique for performing 3D display, a technique is adopted in which a horizontally long display screen of the display 51 is divided into left and right parts and two images each having a parallax are displayed.

  The communication unit 514 performs wireless communication with an external device via a LAN or the Internet. Examples of wireless communication standards include 2G, 3G, 4G, IEEE 802.11, Bluetooth (registered trademark), and the like. In particular, in connection with the present invention, the smartphone 50 performs short-range communication with the operation unit 10 via Bluetooth.

  The speaker unit 517 outputs music or the like from the smartphone, and the speaker unit 517 is also used to generate a game sound in connection with the present invention.

  The triaxial acceleration sensor 506 detects the direction and magnitude of acceleration acting on the smartphone casing, and the triaxial angular velocity sensor 507 detects the angle and angular velocity of the smartphone casing, and the triaxial geomagnetic sensor 508. Detects the direction of geomagnetism. These three sensors 506, 507, and 508 constitute a motion sensor in the present invention. Then, the orientation of the head 41 of the user 40 is detected based on the outputs of the three sensors 506, 507, and 508. Details of the sensors 506, 507, and 508 will be described later.

  The camera unit 518 is connected to, for example, an in camera or an out camera provided on the front surface or the back surface of the housing of the smartphone 50, and is used when capturing a still image or a moving image. As will be described later, the camera unit 518 captures an image of the real world using an out camera provided on the back side of the display 51 or is provided on the same surface as the display 51 in connection with the present invention. It can also be used to acquire a line-of-sight image with the in-camera provided.

  The voice input unit 516 is for inputting voice (such as a microphone) when making a call or the like using the smartphone 50. As will be described later in connection with the present invention, the voice input via the voice input unit 516 may be analyzed to perform various operations on the smartphone or application program.

  In addition, the vibration part 513 is a vibration part used by various application programs etc., for example, can be comprised with the vibration mechanism using an electromagnetic coil, various other vibration motors, etc. The connector 519 is a terminal used for connection with other devices. The terminal may be a general-purpose terminal such as USB or HDMI (registered trademark). The audio output unit 515 outputs audio when making a call or the like using the smartphone. The input operation unit 504 is, for example, a touch-type input operation unit, and performs various inputs by detecting contact with a finger, pen, stylus pen, or the like. A GPS (Grand Positioning System) unit 509 detects the position of the smartphone. The illuminance sensor 510 detects illuminance. The illuminance indicates light intensity, brightness, or luminance. The timer 511 measures time. Note that the timer may be provided integrally with the CPU or the like. The battery 512 functions as a power source for the smartphone by charging.

[1.4 Controller configuration]
FIG. 6 shows an external perspective view of the controller 10 which is an operation device. The controller 10 is formed so as to have a recess at the lower front portion in the longitudinal direction, and is provided behind the recess so as to be wrapped in at least the little finger, the ring finger and the base of the thumb when gripped by the user 40, and conform to the hand. , And a projecting portion 12 connected to the grip portion 11 and provided on the front side of the recess. Provided on the upper surface of the grip portion 11 are a first operating element 14 and a second operating element 15 that are operated with the thumb when gripped. On the front side of the projecting portion, there are provided a third operating element 16 and a fourth operating element 17 that are operated with an index finger or a middle finger when grasped vertically. A direction indicating operation element 13 is provided on the uppermost surface of the controller 10 corresponding to the back side of the depression, and can be freely specified. A switchable operation element 18 is provided on the side surface of the controller 10. Each of these operators is a momentary type operator that continuously generates an operation signal while being pressed.

  As will be described later with reference to FIG. 7, the controller 10 incorporates a three-axis acceleration sensor 1003, a three-axis angular velocity sensor 1004, and a three-axis geomagnetic sensor 1005 that constitute a motion sensor for the operation device. . Therefore, it is possible to freely detect changes including the position, posture, and / or movement of the controller 10 by these sensors.

  A block diagram showing the electrical hardware configuration of the controller 10 is shown in FIG. As shown in the figure, an electric circuit inside the operation element is constituted by a microprocessor, an ASIC, or the like, and a control unit 1001 and. A communication unit 1002 for realizing short-range communication such as Bluetooth, three sensors constituting a motion sensor for an operation device, that is, a three-axis acceleration sensor 1003, a three-axis angular velocity sensor 1004, a three-axis magnetic sensor 1005, and a battery 1006, a storage unit 1007, a first switch 14 a that turns on and off in conjunction with the first operator 14, a second switch 15 a that turns on and off in conjunction with the second operator 15, and a third operation A third switch 16 a that is turned on / off in conjunction with the child 16 and a fourth switch 17 a that is turned on / off in conjunction with the fourth operator 17 are configured.

  As is well known to those skilled in the art, the three-axis acceleration sensor 1003 is a sensor that can measure the acceleration in the three directions of the X axis, the Y axis, and the Z axis with a single device. It can be detected and can also measure gravity (static acceleration). Many 3-axis accelerometers apply microfabrication technology such as semiconductor manufacturing technology and laser processing technology to integrate micromechanical elements on a silicon substrate. “Micro Electro Mechanical Systems, MEMS” , Mems, Micromachine) ”. The MEMS 3-axis acceleration sensor is called “low g” type, which can measure in the range of ± several g and can follow the acceleration fluctuation from 0 Hz to several hundred Hz. In this case, 0 Hz is a state in which only gravitational acceleration is applied to the sensor, and the direction to the ground can be measured from the sum of the X-axis, Y-axis, and Z-axis acceleration vectors at this time. . There are three types of MEMS triaxial acceleration sensors: semiconductor piezoresistive triaxial acceleration sensors, capacitive triaxial acceleration sensors, and thermal detection (gas temperature distribution type) triaxial acceleration sensors. The measurement method is different. The semiconductor piezoresistive triaxial acceleration sensor measures the acceleration by detecting the distortion of the diaphragm generated when the acceleration acts on the weight. Capacitance type 3-axis acceleration sensor measures the change in capacitance for measuring acceleration, and heat detection type (gas temperature distribution type) 3-axis acceleration sensor measures the acceleration using the movement of gas heated by the heater To do.

  As is well known to those skilled in the art, the triaxial angular velocity sensor 1004 is a kind of inertial sensor that realizes measurement of rotational angular velocity with three orthogonal axes (X axis, Y axis, Z axis), It is also called a gyro sensor. Angular velocity sensors measure rotational movements that are not responsive to acceleration sensors. The gyro sensor can be classified by a method for detecting rotation. At present, vibration gyro sensors using IC type micro electro mechanical system (MEMS) technology are most commonly installed in consumer devices. As the name suggests, an inertial sensor using MEMS technology constitutes a sensor with a technology that combines an element that makes mechanical movement and an electronic circuit that processes the signal, and detects the movement. Among vibration-type gyro sensors, there are a capacitive type using silicon and a piezo type using crystal and other piezoelectric materials. Types other than the vibration type gyro sensor include a geomagnetic type, an optical type, and a mechanical type. The three axes are generally defined as three axes of up / down, left / right, and front / rear, and the up / down axis is often called a “yaw axis”, the left / right axis is called a “pitch (pitching) axis”, and the front / rear axis is often called a “roll axis”. All vibratory gyro sensors detect rotation using Coriolis force (turning force).

  As is well known to those skilled in the art, the three-axis geomagnetic sensor 1005 is a kind of magnetic sensor that detects the direction of geomagnetism and has three orthogonal axes (X axis, Y axis, Z axis). ) Refers to a sensor that is calculated with the value of. That is, this sensor has a third geomagnetic sensor that detects the vertical geomagnetism in addition to the first and second magnetic sensors in the front-rear direction and the left-right direction. For this reason, for example, even when the electronic compass is tilted at an angle with the operating element, if it is determined how many times the electronic compass is tilted, the horizontal magnetism can be calculated by subtracting the tilt and the correct orientation can be displayed. There are mainly three types of magnetic sensor elements that are used in geomagnetic sensor ICs incorporated in triaxial geomagnetic sensors. They are MR (magneto-resistive) elements, MI (magneto-impedance) elements, and Hall elements. The MR element uses, for example, the MR effect of a soft metal magnetic material such as permalloy (NiFe) whose resistance value changes with the intensity change of the external magnetic field, and the MI element uses the MI effect of an amorphous wire whose impedance changes when the external magnetic field changes. . A pulse current is allowed to flow through the wire. The Hall element measures the change in the external magnetic field by detecting the potential difference caused by the Hall effect of the semiconductor. In addition to such magnetic sensor elements, the geomagnetic sensor IC generally incorporates a drive circuit, an amplifier circuit, and the like.

[1.5 Operation]
Next, the operation of the virtual space observation system according to the present embodiment will be described.

[1.5.1 Preparation procedure]
First, a series of preparation procedures when using the virtual space observation system 1 according to the present embodiment will be described with reference to FIGS. 8 and 9. The user 40 who uses the virtual space observation system 1 according to the present embodiment first performs connection setting for short-range wireless communication between the smartphone 50 and the controller 10 (S101). Accordingly, the smartphone 50 and the controller 10 can communicate with each other. For example, various information such as operation signals from the controller 10 and detection values of the motion sensors 1003, 1004, and 1005 provided in the controller are transmitted to the smartphone 50. Can be transmitted. Any known standard can be adopted as the communication standard, but Bluetooth (registered trademark) is adopted in this embodiment.

  After the connection setting for short-range wireless communication is made, the touch panel 504 of the smartphone 50 is operated to select an application program for displaying a virtual space image from the list of application programs (S102). By this selection operation, the application program is activated, and a virtual space observation game can be started as will be described later. This application program is downloaded in advance from the application program distribution server and stored in the storage unit 502.

  The smartphone 50 in a state where the application program is activated is incorporated into the head-mounted device 20 as shown in FIG. 3 (S103), and the head-mounted device 20 is mounted on the head of the user 40 (S104). ). Through these steps, the display 51 of the smartphone 50 is arranged in front of the user 40, and the virtual space can be presented to the user.

  Thereafter, in this example, a predetermined mode is selected after the application program is started. FIG. 9 shows a flowchart relating to mode selection. In this example, three modes of “game start”, “game end”, and “game setting” are prepared, and the user 40 corresponds to the posture and position of the controller 10 or the operation of the operator. One of these three options can be selected using the cursor displayed on the screen. When the user 40 selects “GAME START”, a virtual space observation game described later starts (S202). When the user 40 selects “End Game”, the execution of the application program is ended (S203). When the user 40 selects “game setting”, various game setting processes can be performed (S204).

[1.5.2 Virtual space observation game operation]
Next, a series of operations of the virtual space observation game will be described.

[1.5.2.1 Initial screen of game]
The initial screen of the game will be described with reference to FIGS. When the game starts (S203), an initial screen shown in FIG. As is clear from the figure, an image corresponding to the field of view of the user 40 in the virtual space is obtained from a background image 52 corresponding to an angle-of-view frame in the observation direction in the virtual space, and an aiming 53 arranged on the extension of the controller 10. It is configured. Such an image perceiving the virtual space allows the user 40 to experience as if searching for outer space.

  Note that the initial screen shown in FIG. 10 is to allow the user 40 to perceive a sense of depth, and actually, on the display 51 of the smartphone 50, an L (left) image having parallax shown in FIG. An R (right) image is displayed on the left and right. The background image 52 is a part of a virtual space simulating outer space. As shown in FIG. 12, the background image 52 is actually arranged around the user 40 so that the image is fixed in a spherical shape. Has been.

[1.5.2.2 Head tracking process]
The head tracking process will be described with reference to FIGS. When the game starts (S203), the head tracking process is started simultaneously. FIG. 13 shows a flowchart relating to the head tracking process. When the process starts, the control unit 501 acquires the detection values of the motion sensors 506, 507, and 508 provided in the smartphone 50 (S301). After acquiring each detection value, the control unit 501 determines the position coordinates of the smartphone 50 (that is, the position of the head 41 of the user 40) P (p, q, r) and the observation direction (gaze direction) V from the position. (Θ1, φ1) is calculated (S302).

  In parallel with calculating the position coordinate P and the observation direction V of the smartphone 50, the control unit 1001 of the controller 10 also acquires the detection values of the motion sensors 1003, 1004, and 1005 provided in the controller 10 ( S303). After acquiring each detection value, the control unit 1001 calculates the position coordinates Q (u, v, w) of the controller 10 and the instruction direction U (θ2, φ2) from the position (S304). After this calculation, the calculation result is transmitted from the controller 10 to the smartphone 50.

  Here, a coordinate system in the virtual space will be described with reference to FIG. The point P (p, q, r) is a position coordinate of the smartphone 50 in the virtual space, and an image is observed in a spherical shape with the point P as the center. The observation direction V from the point P is the angle φ1 between the observation direction V and the XY plane (or X′-Y ′ plane), and the observation direction V is the XY plane (or X′-Y ′ plane). The image is defined using an angle θ1 formed by a straight line projected onto the X axis and the X axis (or X ′ axis), and an image within a predetermined field angle frame in the observation direction V is observed. Similarly, the controller 10 is defined as the position coordinate Q (u, v, w) of the controller 10 and its designated direction U (θ2, φ2). The position coordinates P of the smarton 50 and the position coordinates of the controller Q may be linked.

  Returning to FIG. 13, when the current position P of the smartphone 50, the current position Q of the controller 10, the observation direction V, and the instruction direction U are calculated, next, they are finally displayed on the display 51 by a predetermined calculation process. An image drawing area specifying process is performed (S305). This drawing area is an area corresponding to a field angle frame on a plane which is on the extended line in the observation direction V and is located at a predetermined distance from the observation center.

  Further, after this drawing area is determined, whether or not the extension line in the pointing direction U and the drawing area in the angle of view frame have an intersection, and if so, the coordinates on the drawing area are calculated. In performing this calculation, the coordinate systems of the smartphone 50 and the controller 10 are matched with each other. FIG. 15 shows a conceptual diagram of matching between coordinate systems. As is clear from FIG. 15, the coordinate system (FIG. 15 (a)) of the head-mounted device 20 incorporating the smartphone 50 therein and the coordinate system (FIG. 15 (b)) of the controller 10 are the same as those in FIG. ) So that the coordinate axes are shared. By such coordinate system matching, various calculations can be performed on a common coordinate system.

  When the drawing area specifying process (S305) ends, a background image drawing process and a display process on the display 51 are performed (S306). Further, if the pointing direction U of the controller 10 and the drawing area have an intersection by the above-described processing (YES in S307), the aim is drawn and displayed at the intersection coordinates on the background image with the aim being in the drawing area. (S308). Thereafter, the sensor detection value of each motion sensor is acquired again, and the above-described processing is repeated. On the other hand, if the pointing direction U of the controller 10 and the drawing area do not have an intersection (NO in S307), the sensor detection value of each motion sensor is again set without drawing the aim on the background image without aiming in the drawing area. And the above-described processing is repeated. In other words, according to the head tracking process according to the present embodiment, a predetermined picture size is displayed on the display 51 in accordance with the observation direction (or line-of-sight direction) of the smartphone 50 in the virtual space and the instruction direction of the controller 10. The background image and the aiming image within the frame are always displayed.

  The relationship between the head movement and the display screen when the head tracking process is performed will be specifically described with reference to FIG. The left side of the figure shows the user 40 holding the controller 10 with the right hand and the head wearing tool 20 attached to the head 41 as viewed from above, and the right side of the figure shows the head. An image within a predetermined field angle frame that is observed through the display 51 corresponding to the orientation of the unit 41 and the orientation of the controller 10 is shown.

  As shown in FIG. 16A, when the head 41 faces “front” and the operation device 10 is also “front” and points to the center of the screen, as shown on the right side of FIG. The aiming 53 is drawn at the center of the background image (S307 YES, S308).

  On the other hand, as shown in FIG. 16B, when the head 41 is directed to the “right side” and the operating device 10 is instructing “forward”, as shown in the right side of FIG. Since there is no angle-of-view frame corresponding to the direction of the head 41 above, the aim 53 is not drawn (NO in S307), and only the background image within the predetermined picture frame arranged on the right side is drawn (S306).

  That is, when a background image within a predetermined field angle frame is drawn according to the orientation of the head 41 of the user 40 and an intersection with an extension line in the direction indicated by the controller 10 exists on the image, the intersection coordinates The aiming image 53 is drawn at a position corresponding to.

[1.5.2.3 Translation processing in virtual space]
Next, the translational movement operation in the virtual space will be described with reference to FIGS. 17 and 18.

  FIG. 17 shows an interrupt processing flowchart for the translational movement operation of the observation point in the virtual space. In this example, when the third operating element 16 provided in the protruding portion 12 of the controller 10 is operated in the state in which the head tracking process is being performed, an interrupt process for the head tracking process is performed. Starts.

  When the interrupt process is started, first, the control unit 501 of the smartphone 50 is based on the detected values of the motion sensors 506, 507, and 508 on the virtual space of the smartphone 50 (or the head 41 of the user 40). The current position P and the current observation direction (gaze direction) V are acquired (S401). Next, based on the current position P and the current observation direction V, a translational movement process in the virtual space is performed (S402). Specifically, the observation position is moved from the current observation position P in the current observation direction V by a distance of ΔL, and the observation position is moved from P to P ′. Thereafter, the background image within the field angle frame existing in the observation direction V from the new observation position P ′ is displayed on the display 51 (S403). At this time, the control unit 501 also acquires information related to the controller 10, and when the angle of view frame exists on the extension line in the direction indicated by the controller 10, the sight 53 is drawn and displayed at the coordinates corresponding to the intersection. (S403). In this way, after displaying a predetermined observation image from a new observation position (or observation center), the interruption process ends (interrupt return).

  The movement operation of the observation point in the virtual space will be described in detail with reference to FIG. The left side of the figure shows the user 40 holding the controller 10 with the right hand and the head wearing tool 20 attached to the head 41 as viewed from above, and the right side of the figure shows the head. An image within a predetermined field angle frame that is observed through the display 51 corresponding to the orientation of the unit 41 and the orientation of the controller 10 is shown.

  FIG. 18A shows a state immediately before the interruption process is performed, the head 41 of the user 40 is “front”, and the pointing direction of the controller 10 is also “front”. Further, at this time, a background image shown on the right side of FIG.

  FIG. 18B shows a state after the third operator 16 is operated in the state of FIG. If the observation position is moved by ΔL to the “forward” direction, which is the observation direction, by operating the third operation element 16 (S402), an image observed from a new observation position that is separated by ΔL with this movement. Is newly drawn (S403). Specifically, as is clear from the screen diagram on the right side of the drawing, the image is advanced forward by ΔL, so that a screen diagram that slightly enlarges the front image is depicted and displayed.

  According to such a configuration, since the observation point itself can be freely moved in the virtual space, the virtual space can be freely searched.

[1.5.2.4 Observation direction changing process in virtual space]
Next, the process of changing the observation direction in the virtual space will be described with reference to FIGS. 19 and 20.

  FIG. 19 is a flowchart of an interrupt process related to an observation direction changing operation in the virtual space. In this example, when the push button type second operation element 15 provided in the grip portion 11 of the controller 10 is operated in a state where the head tracking process is being performed, the head tracking process is performed. Interrupt processing starts.

  When the interrupt process starts, the control unit 501 of the smartphone 50 determines the current position P in the virtual space of the smartphone 50 (or the head 41 of the user 40) based on the detection values of the motion sensors 506, 507, and 508. And the present observation direction (namely, line-of-sight direction) V is acquired (S501). After acquiring the current position P and the current observation direction V, the control unit 501 adds a predetermined angle Δθ1 to the observation direction V to obtain a new observation direction V ′ (S502). As a result, the angle formed by the straight line projected from the observation direction V onto the XY plane (or X′-Y ′ plane) and the X axis increases, and when viewed from vertically above, the Z axis (or Z ′ The observation direction (or line-of-sight direction) rotates horizontally and continuously around the axis).

  After acquiring this new observation direction V ′, the background image within the angle-of-view frame corresponding to the observation direction V ′ and, if present, the aiming image are drawn and displayed on the display 51 (S503). ). After this display process, when it is detected that the input operation is still continued in the push button type second operator 15 (YES in S504), the process of adding the predetermined angle Δθ is repeated again. That is, as long as the input operation is continued in the push button type second operation element 15, the predetermined angle Δθ is cumulatively added. In other words, the rotation around the Z axis in the observation direction is continued. It becomes.

  On the other hand, if no input operation is detected by the push button type second operation element 15 after the display process (NO in S504), the rotation in the observation direction is stopped, and the total based on the cumulative addition of Δθ so far is stopped. A rotation amount ΔΘ is calculated. After correcting the reference of the indication direction of the controller 10 so as to correspond to the new observation direction using this total rotation amount ΔΘ (S505), the interruption process is ended (interrupt return). That is, the user 40 can associate the latest observation direction in the virtual space with the front direction of the user.

  A mode in which the observation direction is changed will be specifically described with reference to FIG. The left side of the figure shows the user 40 holding the controller 10 with the right hand and the head wearing tool 20 attached to the head 41 as viewed from above. The center of the figure shows the orientation of the head 41. In addition, an image within a predetermined field angle frame observed through the display 51 corresponding to the orientation of the controller 10 is shown on the right side of the figure, and a conceptual diagram showing the observation direction on the horizontal plane in the virtual space with arrows. Has been.

  As shown in FIG. 20A, in the initial state, the head 41 of the user 40 is directed to “front” and the controller 10 is directed to “front”. A background image and a sight 53 within the direction angle frame are displayed. The observation direction in the virtual space at this time is 0 degree, that is, coincides with the direction of the actual head 41.

  Next, when the user continues the input operation of the push button type second operation element 15 provided in the controller 50 (S504 YES), the physical direction of the head 41 of the user 40 and the physical direction of the controller 10 In spite of no change, only the observation direction in the virtual space is changed (S502, S503). FIG. 20B shows a state in which the 90-degree observation direction changes in the process of rotating the observation direction in the virtual space. At this time, a deviation of 90 degrees from the actual direction of the head 41 occurs.

  FIG. 20C shows a state in which the user 40 continues the input operation via the push button type second operation element 15 and cancels the input operation when the observation direction changes just 180 degrees (NO in S504). Is shown. Also at this time, although the head 41 of the user 40 is directed to “front” and the controller 10 is directed to “front”, the observation direction in the virtual space is shifted by 180 degrees from the actual head 41 direction. .

  That is, according to such a configuration, the observation direction in the virtual space can be arbitrarily changed without twisting the neck or upper body backward. Therefore, it is possible to arbitrarily change the observation direction in the virtual space as needed without changing the display posture while using the head tracking process as a basis.

  Further, if the above-described translational movement process is performed immediately after the observation direction in the virtual space is rotated by 180 degrees, the observation position can be advanced 180 degrees behind the observation direction of the head 41 in the virtual space. That is, according to the above configuration, the user 40 can freely search the virtual space.

  Furthermore, according to such a configuration, the process of changing the observation direction can be confirmed through continuous images. Therefore, it is possible to intuitively understand which direction the observation direction is changed in the virtual space, and it is also possible to observe the state up to the target observation direction.

  In addition, according to such a configuration, when the observation direction is rotated, it goes around once and returns to the original direction again. Therefore, there is less possibility of losing sight of its own observation direction in the virtual space.

  Moreover, according to such a structure, an observation direction can be rotated along a horizontal surface. Therefore, the risk of screen sickness is reduced.

  Furthermore, according to such a configuration, it is possible to change the observation direction only while an input is performed by the operator. Therefore, the observation direction changing operation can be realized by an intuitive and simple operation.

  In the example of the present embodiment, the observation direction in the virtual space is changed to 180 degrees behind, but the target angle is not limited to this. For example, the input operation of the operator is canceled halfway By doing (that is, releasing the push button), the observation direction can be arbitrarily set.

[1.6 Modification]
In the above-described embodiment, the movement of the view angle frame corresponding to the observation direction in the virtual space is performed linearly along a circular orbit on the horizontal plane. However, the present invention is not limited to such a configuration.

  Therefore, for example, as shown in FIG. 21 (a), even if it is a circular orbit, it is not a circular orbit on the horizontal plane, but another orbit, for example, an orbit in which the angle-of-view frame moves in the diagonally upper right direction and makes one round Alternatively, it may be a trajectory in which the angle-of-view frame moves almost vertically upward and goes around as it is.

  In addition, as shown in FIG. 21B, the angle of view frame may be moved along another trajectory, for example, a curved trajectory, instead of a linear trajectory.

[2. Second Embodiment]
With reference to FIGS. 22 and 23, a second embodiment in which the observation direction changing process is performed according to the posture and movement of the controller 10 will be described. The virtual space observation system according to the present embodiment also includes the smartphone 50, the head-mounted device 20, and the controller 10, and the hardware configuration of each device is the same as that described in the first embodiment, except for the following points. Since they are substantially the same, a description of the hardware configuration is omitted here.

  FIG. 22 shows a flowchart of an interrupt process relating to the observation direction changing operation in the virtual space. In this example, when the push button type second operation element 15 provided in the grip portion 11 of the controller 10 is operated in a state where the head tracking process is being performed, the head tracking process is performed. Interrupt processing starts.

  When the interrupt process starts, the control unit 501 of the smartphone 50 determines the current position P in the virtual space of the smartphone 50 (or the head 41 of the user 40) based on the detection values of the motion sensors 506, 507, and 508. Then, the current observation direction (that is, the line-of-sight direction) V is acquired (S601).

  Next, the controller 10 determines the current position Q (u, v, w) in the virtual space of the controller 10 and the current indication direction U of the controller 10 based on the detection values of the motion sensors 1003, 1004, and 1005. (Θ2, φ2) is acquired and transmitted to the smartphone 50 (S602).

  After that, the control unit 501 calculates the intersection coordinates of the extension line in the instruction direction of the controller 10 and the image area in the angle of view frame corresponding to the observation direction V in the virtual space, and stores the coordinates as the start point (S603). . Thereafter, when the controller 10 is moved, the intersection coordinates between the extension line of the indication direction of the controller 10 and the image area in the angle-of-view frame corresponding to the observation direction V in the virtual space are calculated as the indication point, and the above-described start point and indication Corresponding to the relative positional relationship with the point, the view angle frame, that is, the observation direction V is moved symmetrically with respect to the start point (S604). A predetermined background image is displayed according to the movement of the angle of view frame, and an aim is displayed if it exists (S605).

  After this display process, if it is detected that the input operation is still continued in the push button type second operation element 15 (YES in S606), the process from the calculation process of the indicated point is repeated again. That is, by changing the indication point of the controller 10 in a state where the input operation is continued in the push button type second operation element 15, point symmetry is performed as if so-called dragging is performed with respect to the above-described start point. In addition, the view angle frame or the observation direction can be changed.

  On the other hand, after the display process, when no input operation is detected in the push button type second operator 15 (NO in S606), the reference direction reference of the controller 10 is corrected so as to correspond to the new observation direction ( S607), interrupt processing is terminated (interrupt return).

  A mode in which the observation direction is changed will be specifically described with reference to FIG. The left side of the figure shows the user 40 holding the controller 10 with the right hand and the head mounting device 20 on the head 41 as viewed from above. The right side of the figure shows the orientation of the head 41. In addition, an image within a predetermined field angle frame observed through the display 51 corresponding to the orientation of the controller 10 is shown.

  As shown in FIG. 23A, in the initial state, the head 41 of the user 40 is directed to “front”, and the controller 10 is directed to “front”. A background image and a sight 53 within the direction angle frame are displayed.

  Next, as shown in FIG. 23 (b), the user 40 confirms the image and uses the aim 53 drawn in the extension of the instruction direction of the controller 10 to use the region as the starting point, that is, in this example. In this case, a slightly lower area of the planet with the upper right wheel is indicated, and in this state, the input operation of the push button type second operation element 15 provided in the controller 50 is performed (pressing the push button). Thereby, the interrupt process according to the present embodiment is started and the area is stored as a start point (S603). Thereafter, by moving the aim 53 according to the indication area of the controller 10 while maintaining the input state of the second operation element 15, the relationship between the start point and the indication area is point-symmetric with respect to the angle of view frame, That is, the observation direction moves (S604, S605, S606 YES).

  FIG. 23C illustrates a state in which the controller 10 is returned to the state of FIG. 23A while maintaining the input state of the second operation element 15. In this state, when the user cancels the input state of the second operator 15 (NO in S606), the above-described interrupt process ends. According to such a series of operations, the controller 10 can be used to arrange a desired area as a starting point in the front direction of the virtual space. In particular, if the aim is moved with the center of the screen as the end point, the desired observation direction can be arranged at the front center of the user 40.

  That is, according to such a configuration, it is possible to change the observation direction in the virtual space as if dragging, starting from a predetermined area of the image in the angle of view frame.

[3. Third Embodiment]
Next, a virtual space observation system according to the third embodiment will be described. The basic configuration of the virtual space observation system according to the third embodiment is substantially the same as that of the virtual space observation system according to the second embodiment. That is, the observation direction in the virtual space can be changed by a change in the posture or movement of the controller 10 regardless of the posture of the head 41. However, unlike the second embodiment, in the third embodiment, when the designated area is moved with the predetermined area in the predetermined angle frame as the starting point, the predetermined angle frame or observation is centered on the starting point. The direction does not move point-symmetrically, but moves so as to coincide with the relative positional relationship between the starting point and the pointing area.

  With reference to FIG. 24, the aspect of changing the observation direction according to the third embodiment will be specifically described. The left side of the figure shows the user 40 holding the controller 10 with the right hand and the head mounting device 20 on the head 41 as viewed from above. The right side of the figure shows the orientation of the head 41. In addition, an image within a predetermined field angle frame observed through the display 51 corresponding to the orientation of the controller 10 is shown.

  As shown in FIG. 24A, in the initial state, the head 41 of the user 40 is directed to “front” and the controller 10 is directed to “front”. A background image and a sight 53 within the direction angle frame are displayed. In this state, when the input operation of the push button type second operation element 15 provided in the controller 50 is performed (when the push button is pressed), the interrupt processing according to the present embodiment is started and the aim 53 is The pointing area corresponding to the position is stored as a starting point (S603). In this example, the starting point is the center of the screen.

  Next, as shown in FIG. 24B, the user 40 moves the controller 10 to the right side while keeping the input state to the second operation element 15 (while pressing the push button). Move 90 degrees. At this time, on the display 51, images within a predetermined field angle frame are continuously displayed in the process from the front image to the right side image, and finally shown in the right side of FIG. The image when facing the right side is displayed. In the state where the controller 10 faces to the right side, the input state of the second operation element 15 is canceled (the push button is released), and the observation direction changing operation is completed.

  Thereafter, after the user 40 returns the controller 10 to the front, a predetermined calibration operation regarding the aiming 53 is performed, and when the controller 10 is set to the front center, the aiming 53 is set to be centered again. As shown in FIG. 24C, only the observation direction in the virtual space can be directed to the right side without actually turning the neck to the right side.

  According to the above configuration, the observation direction can be changed by an intuitive operation using the controller 10.

[4. Fourth Embodiment]
Next, a virtual space observation system according to the fourth embodiment will be described. In the virtual space observation system according to the first to third embodiments, a completely virtual space image is displayed on the display 51 using the immersive head wearing tool 20. In the present embodiment, an example of the development of the present invention to a so-called AR (Augmented Reality) technique in which a real world image is displayed on a display and a specific object image is displayed thereon is shown. The smartphone 50 in the present embodiment can capture an image from the out-camera into the control unit 51 via the camera unit 518 and display the image on the display 51. The observation direction changing method in the virtual space is similar to that of the virtual space observation system according to the second embodiment.

  With reference to FIG. 25, an aspect of changing the observation direction will be described. The left side of the figure shows the user 40 holding the controller 10 with the right hand and the head wearing tool 20 attached to the head 41 as viewed from above. The center of the figure shows the orientation of the head 41. In addition, an image within a predetermined field angle frame observed through the display 51 corresponding to the orientation of the controller 10 is shown on the right side of the figure, and a conceptual diagram showing the observation direction on the horizontal plane in the virtual space with arrows. Has been.

  As is apparent from FIG. 25A, in the initial state, when the user 40 wearing the head wearing device 20 incorporating the smartphone 50 having the head tracking function on the head 41 observes the right side, the right side The object image having the shape of a monster is superimposed and displayed on the actual road image existing on the road.

  As shown in FIG. 25 (b), the user 40 who wants to change the observation direction in the virtual space to the right side, after directing the head 41 forward, extends the right hand holding the controller 10 to the side, An input operation of the push button type second operation element 15 is performed (push button is pressed). Thereby, the interrupt process according to the present embodiment is started and the area is stored as the start point. Thereafter, by moving the aim 53 according to the indication area of the controller 10 while maintaining the input state of the second operator 15, the angle of view is symmetric with respect to the relationship between the start point and the indication area. The frame, that is, the observation direction rotates 90 degrees to the right.

  FIG. 25C shows a state in which the position and posture of the controller 10 are returned to the state of FIG. 25A while maintaining the input state of the second operator 15. In this state, when the user cancels the input state of the second operation element 15, the above-described interrupt process ends. As is clear from the figure, the object image originally present on the right side of the user 40 is displayed superimposed on the real world image in front of the user 40.

  That is, according to such a configuration, it is possible to arbitrarily change the observation direction in the virtual space without twisting the neck and upper body, thereby changing only the object image relating to the virtual space to be superimposed. .

[5. Others]
[5.1 Modification using voice input]
In the above-described embodiment, the operation signal of the controller 10 or the attitude and / or movement of the controller 10 is used for controlling the observation direction in the virtual space without changing the display attitude. However, the present invention is limited to such a configuration. Not. Therefore, for example, the observation direction in the virtual space may be controlled using sound.

  In this modification, the controller 10 itself is not used, or the controller 10 is not used for controlling the observation direction in the virtual space. Instead, the control unit 501 of the smartphone 50 is provided with a voice input processing unit that analyzes a voice input via the voice input unit 516 using a known voice analysis algorithm.

  When the voice input processing unit detects that the voice “rotate” is input via the voice input unit 516, the voice input processing unit horizontally rotates the observation direction in the virtual space to a predetermined direction, for example, the right direction. Further, when it is detected that the voice of “stop” is input, the observation direction that is rotating in the virtual space is stopped. That is, the user 40 can freely change the observation direction using the words “rotation” and “stop”. Note that the words to be detected are not limited to these words, and various words can be used for the command.

  According to such a configuration, there is no need to have any operation device or the like for the operation of changing the observation direction, both hands are free, and the field of view is limited or completely absent by wearing the head wearing tool. In addition, the observation direction in the virtual space can be changed appropriately.

[5.2 Modification using eye-gaze input]
In the above-described embodiment, the operation signal of the controller 10 or the attitude and / or movement of the controller 10 is used for controlling the observation direction in the virtual space without changing the display attitude. However, the present invention is limited to such a configuration. Not. Therefore, for example, the observation direction in the virtual space may be controlled using the line of sight.

  In this modification, the controller 10 itself is not used, or the controller 10 is not used for controlling the observation direction in the virtual space. Instead, the image captured by the in-camera provided on the same surface as the display 51 is taken into the control unit 501 of the smartphone 50 into the control unit 501 via the camera unit 518. In the present modification, the control unit 501 is provided with a line-of-sight detection processing unit that detects the line of sight of the user 40 using a known line-of-sight detection algorithm based on the in-camera image and estimates which area on the screen is being viewed. It has been.

  When the line of sight is placed in a predetermined area in the image for a predetermined time or longer, the line-of-sight detection processing unit changes the line-of-sight direction so that the area of interest is the center of the observation direction. . That is, the observation direction in the virtual space can be changed only by the line of sight without changing the head posture or the like.

  According to such a configuration, there is no need to have any operation device or the like for the operation of changing the observation direction, both hands are free, and the field of view is limited or completely absent by wearing the head wearing tool. In addition, the observation direction in the virtual space can be changed appropriately.

[5.3 Other Modifications]
In the above-described embodiment, an all-spherical virtual space is assumed, but the present invention is not limited to such a configuration. Therefore, for example, a virtual space such as a panoramic image shown in FIG. 26 may be adopted, or a hemispherical virtual space shown in FIG. 27 may be adopted.

  Moreover, in the above-mentioned embodiment, although the control part 501 shall be contained in the smart phone 50 grade | etc., This invention is not limited to such a structure. Therefore, for example, as shown in FIG. 28, an information processing unit 70 is provided as an external device, and processing with a relatively high calculation cost or all calculation processing is performed by the external information processing unit 70. It is good.

  Furthermore, in the above-described embodiment, a so-called immersive head mounted display in which the field of view other than the display 51 is blocked is employed, but the present invention is not limited to such a configuration. Therefore, for example, in a virtual space having a coordinate system fixed in space, a virtual space observation system that observes the state of the holding direction by holding the smartphone 50 or the like that can detect the orientation of the display 51 is adopted. Also good.

  FIG. 29 is an explanatory diagram of a non-immersive virtual space observation system for observing a virtual space having a coordinate system fixed in space by holding the smartphone 50 over it. As shown in FIG. 29A, for example, the user 40 sitting on a chair or the like can check the surrounding virtual space image via the display 51 of the smartphone 50 held in his hand. Since the smartphone 50 has a built-in motion sensor, the observation direction in the virtual space can be changed by changing the posture of the smartphone. In such a state, in order for the user 40 to observe the left and right virtual spaces, as shown in FIG. 29B, the smartphone with the neck and upper body twisted to the left and right and the hand held in the right and left directions must be directed respectively. In this virtual space observation system as well, the burden on the body accompanying the change of the observation direction in the virtual space is large. Therefore, in this example, as shown in FIG. 29C, software for changing the observation direction in the virtual space without changing the display posture in the lower right area of the background image 52 of the display 51. A button is provided. This software button includes a forward operation button 54 that advances the observation position (or observation center) in the observation direction, a right rotation button 56 that rotates the horizontal plane in the right direction, and a left rotation button that rotates the horizontal plane in the left direction. 58, an upper rotation button 55 that rotates vertically upward, and a lower rotation button 57 that rotates vertically downward. Any of the software buttons continuously input signals while pressed. This is a momentary expression that is generated.

  According to such a configuration, the observation direction in the virtual space can be arbitrarily changed by operating the software button without directing the display in an arbitrary direction.

  In the above-described embodiments, the virtual space observation system has been described, but it is needless to say that it can be considered as a device constituting the system, a method or program as a processing process thereof. In particular, in the above-described embodiment, the general-purpose smartphone 50 is incorporated into the head-mounted device 20, but the present invention is not limited to such a configuration, and is a head-mounted display dedicated to a virtual space observation system. It may be configured.

  In the above description, the virtual space observation system has been described while exemplifying a game. However, the present invention is not limited to a game application. Therefore, it goes without saying that the present invention can be applied to a virtual space observation system for all uses.

  According to the present invention, a virtual space observation system is provided. Thereby, for example, a video game for observing or searching in the virtual space can be provided.

DESCRIPTION OF SYMBOLS 1 Virtual space observation system 10 Controller 11 Grip part 12 Projection part 13 Direction instruction | indication operation element 14 1st operation element 15 2nd operation element 16 3rd operation element 17 4th operation element 18 Switchable operation element 1001 Control Part 1002 Communication unit 1003 Three-axis acceleration sensor 1004 Three-axis angular velocity sensor 1005 Three-axis geomagnetic sensor 1006 Battery 20 Head mounting tool 21 Head belt 22 Head belt 23 Main body 24 Opening 25 Cover plate 26 Holder 27 Female side cover plate fastening Tool 28 Male lid plate fastener 29 Partition plate 30 Optical system 40 User 41 Head 43 Eye 50 Smartphone (or information processing device)
51 Display 52 Background Image 52L Left Eye Image 52R Right Eye Image 53 Aim 501 Control Unit 502 Storage Unit 504 Input Operation Unit (Touch Panel)
505 Input operation unit (button)
506 Triaxial acceleration sensor 507 Triaxial angular velocity sensor 508 Triaxial geomagnetic sensor 509 GPS unit 510 Illuminance sensor 511 Timer 512 Battery 513 Vibration unit 514 Communication unit 515 Audio output unit 516 Audio input unit 517 Speaker unit 518 Camera unit 519 Connector 54 Advance operation Button 55 Up rotation button 56 Right rotation button 57 Down rotation button 58 Left rotation button 70 Information processing section

Claims (17)

Display,
A motion sensor capable of detecting the orientation of the display;
An information processing unit that displays an image in a predetermined field angle frame in a predetermined observation direction in a virtual space on the display in correspondence with the posture of the display detected by the motion sensor;
A virtual space observation system comprising:
The information processing unit further includes:
An observation direction control unit configured to change an observation direction in the virtual space in accordance with an observation direction control signal generated regardless of a change in posture of the display based on a predetermined operation by a user of the virtual space observation system; , Virtual space observation system. The observation direction control unit further includes:
In response to a continuous observation direction control signal, the observation direction in the virtual space is continuously changed, thereby continuously displaying an image in a predetermined field angle frame corresponding to the observation direction on the display. The virtual space observation system according to claim 1, displayed on the screen.   The virtual space observation system according to claim 2, wherein the continuous display of the image within the predetermined field angle frame is performed along a predetermined circular orbit centered on the observation point in the virtual space. The information processing unit further includes:
The virtual space observation system according to claim 3, further comprising an observation point control unit that moves the observation point in the virtual space by a predetermined distance in a nearest observation direction in accordance with a predetermined observation point movement signal. The display, the motion sensor, and the information processing unit are included in a single information processing device,
The virtual space observation system further includes:
A head-mounted device that can be incorporated into the information processing apparatus so that the display is placed in front of the eye and can be mounted on the user's head;
The virtual space observation system according to claim 1, further comprising: a hand-held or body-mounted operation device that has a predetermined operation element and can be connected to the information processing device by wired or wireless communication.   The virtual space observation system according to claim 5, wherein the observation direction control signal is generated by operating the operation element of the operation device. The observation direction control signal is
While the input operation is performed using the operation element of the operation device, the observation direction in the virtual space is continuously changed, and thereby a predetermined image corresponding to the observation direction is displayed on the display. The virtual space observation system according to claim 6, wherein images in the corner frame are continuously displayed. The operating device further includes a motion sensor for the operating device that detects the attitude and movement of the operating device,
The virtual space observation system according to claim 5, wherein the observation direction control signal is generated according to a posture or a movement of the operation device. The observation direction control unit is
The movement of the pointing area is centered on the starting point when the pointing area is moved from the starting point when the pointing area is moved from the starting point when the pointing direction is designated by the operating device. The virtual space observation system according to claim 8, wherein the observation direction is changed symmetrically. The observation direction control unit is
When the pointing area in the angle of view frame when a predetermined direction is instructed by the operating device is the starting point, and the pointing area is moved from the starting point, the relative positional relationship between the starting point and the pointing area coincides The virtual space observation system according to claim 8, wherein the observation direction is changed so as to. The virtual space observation system further includes:
Including a camera that captures real-world images,
The information processing unit displays the real world image on the display, and superimposes the real world image on the display, replacing the image within the predetermined field angle frame, and detecting the motion sensor. The virtual space observation system according to claim 1, wherein a predetermined image in a predetermined observation direction in the virtual space is displayed corresponding to the posture of the display. The virtual space observation system further includes:
Has a microphone for voice input,
The information processing unit further includes:
The virtual space observation system according to claim 1, further comprising: an audio input processing unit that generates the observation direction control signal based on a microphone input and thereby changes the observation direction in the virtual space. The virtual space observation system further includes:
Equipped with a camera for gaze detection,
The observation direction control unit further includes:
Based on a camera input, a gaze area in the image within the predetermined angle-of-view frame is detected, and when it is detected that the same area has been gaze for a predetermined time or more, the area becomes the center of the image. The virtual space observation system according to claim 1, comprising a line-of-sight detection processing unit that changes an observation direction.   The virtual space observation system according to claim 1, wherein the motion sensor includes at least a triaxial acceleration sensor, a triaxial angular velocity sensor, and a triaxial geomagnetic sensor.   The virtual space observation system according to claim 5, wherein the motion sensor for the operating device includes at least a 3-axis acceleration sensor, a 3-axis angular velocity sensor, and a 3-axis geomagnetic sensor. A virtual space observation method for observing a virtual space recognized through an image displayed on a display,
A motion detection step of detecting the orientation of the display;
An information processing step of displaying an image in a predetermined field angle frame in a predetermined observation direction in a virtual space on the display in correspondence with the posture of the display detected by the motion detection step;
The information processing step further includes:
A virtual space observation method comprising: an observation direction control step of changing an observation direction in the virtual space in accordance with an observation direction control signal generated regardless of a change in posture of the display based on a predetermined operation by a user.   A computer program for causing a computer to function as an information processing unit in the virtual space observation system according to claim 1.

Images