JP7530206B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program for generating a virtual viewpoint image.

A technology that generates a virtual viewpoint image according to a specified virtual viewpoint using multiple viewpoint images obtained by synchronously capturing images from multiple cameras installed in different positions is attracting attention. With a virtual viewpoint image, for example, a specific scene (such as a goal shot) in a soccer or basketball game can be viewed from various angles. Therefore, the virtual viewpoint image can give the user a higher sense of realism than a conventional captured image.

There is a known technology that sequentially generates virtual viewpoint images corresponding to the position, orientation, and time of the virtual viewpoint set on an arbitrary path while moving the virtual viewpoint on the path. Hereinafter, the path along which the virtual viewpoint moves is referred to as a flow line. Also, a set of the position, orientation, and time of the virtual viewpoint set on the flow line is referred to as flow line information. One method for generating such virtual viewpoint flow line information is the key frame method. In the key frame method, the position and orientation of the virtual viewpoint with respect to a number of basic frames (base frames) are specified on the flow line along which the virtual viewpoint moves. The position and orientation on the flow line of the virtual viewpoint with respect to intermediate frames located between the base frames are automatically generated by interpolation. Patent Document 1 discloses a technology that generates a motion path of a camera in a three-dimensional virtual space by interpolating intermediate frames with a spline function.

JP 2007-25979 A

However, in the key frame method disclosed in Patent Document 1 and other documents, the position and orientation of the virtual viewpoint on the movement line for the intermediate frames are interpolated based on the position and orientation of the virtual viewpoint on the movement line set in the base frame, and the position of the object while the virtual viewpoint is moving along the movement line is not taken into consideration. As a result, there is a possibility that the generated virtual viewpoint image will not include a noteworthy object. In order to ensure that the desired object is included in the virtual viewpoint image, the base frame must be set more precisely, which results in increased effort involved in generating the movement line information for the virtual viewpoint.

The present invention aims to provide a technology that makes it easy to set a virtual viewpoint for generating a virtual viewpoint image that includes a specified object.

An information processing device according to an aspect of the present invention has the following arrangement:
An information processing device that generates a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint for generating a virtual viewpoint image from a plurality of images obtained by photographing a predetermined space with a plurality of photographing devices,
an acquisition means for acquiring a first position and a second position of the virtual viewpoint, and a first time and a second time corresponding to the first position and the second position and indicating a time of shooting of the predetermined space ;
a generating means for generating a flow line along which the virtual viewpoint moves by connecting the first position and the second position;
a setting means for setting a plurality of positions of the virtual viewpoint between the first position and the second position of the traffic line, and setting a photographing time of the predetermined space corresponding to each of the plurality of positions based on the first time and the second time;
a determination means for determining a line of sight direction from the virtual viewpoint at each of the plurality of positions based on a position of a specific object in the predetermined space at the shooting time set at each of the plurality of positions,
the acquiring means acquires first flow line information including a plurality of positions including the first position of the virtual viewpoint, a line of sight from the virtual viewpoint at each of the plurality of positions, and a photographing time, and second flow line information including a plurality of positions including the second position of the virtual viewpoint, and a line of sight from the virtual viewpoint at each of the plurality of positions, and a photographing time,
The generating means generates the flow line connecting the first position and the second position as a flow line of third flow line information along which the virtual viewpoint moves.

The present invention makes it possible to easily set a virtual viewpoint for generating a virtual viewpoint image that includes a specified object.

FIG. 1 is a diagram illustrating an example of an image processing system. FIG. 1 is a diagram showing an example of the configuration of an image processing system. FIG. 2 is a diagram showing an example of the hardware configuration of each device. FIG. 2 is a diagram showing an example of the functional configuration of an image generating apparatus. FIG. 2 is a diagram showing an example of the functional configuration of an information processing device. 11A to 11C are diagrams showing an example of a virtual viewpoint operation. FIG. 4 is a diagram showing an example of the configuration of virtual viewpoint parameters. FIG. 4 is a diagram showing an example of metadata configuration. 4 is a flowchart showing the flow of a series of processes in an information processing device. FIG. 13 is a diagram illustrating a group of virtual viewpoint parameters. FIG. 13 is a diagram showing an example of a connecting flow line generated by a flow line generating unit. FIG. 11 is an explanatory diagram relating to the determination of an object to be followed. FIG. 13 is a diagram showing an example of a virtual viewpoint moving on a connecting flow line. FIG. 13 is a diagram showing an example of a state in which three flow lines are connected. FIG. 13 is a diagram showing an example of a condition for automatically joining two flow lines.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

(System Configuration)
1 is a diagram for explaining an example of an image processing system 100 according to an embodiment. The image processing system 100 has a plurality of imaging devices (hereinafter, cameras 104) installed in a stadium 101. The stadium 101 includes spectator seats 102 and a field 103 where a sport or the like is actually played. Each of the plurality of cameras 104 captures at least a portion of the field 103, which is the subject of the image capture. The plurality of cameras 104 are also arranged so that the angles of view of at least two of the cameras overlap.

2 is a diagram showing an example of a schematic device configuration of the image processing system 100. In one example, the image processing system 100 includes a plurality of cameras 104 installed in the stadium 101, an image generating device 201, an information processing device 202, and a user terminal 203. The plurality of cameras 104 are connected to each other via, for example, a transmission cable, and are also connected to the image generating device 201. The plurality of cameras 104 transmit images acquired by each of them by photographing the field 103 to the image generating device 201 via the transmission cable. In the example of FIG. 1, the plurality of cameras 104 are arranged so that all or part of the range of the stadium 101, such as a soccer field, is photographed. The plurality of cameras 104 may be configured to photograph still images or may be configured to photograph moving images. The plurality of cameras 104 may also be configured to photograph both still images and moving images. In this embodiment, unless otherwise specified, the term "image" includes both still images and moving images.

The image generating device 201 aggregates and accumulates a plurality of images taken by a plurality of cameras 104, and generates a virtual viewpoint image corresponding to a specified virtual viewpoint. Hereinafter, an image taken by the camera 104 may be referred to as a taken image. In addition, a plurality of virtual viewpoint images generated for a plurality of virtual viewpoints according to the position, posture, and time specified by a group of virtual viewpoint parameters are referred to as a group of virtual viewpoint images. In addition, in this specification, a group of virtual viewpoint parameters including a set of the position, posture, and time of the virtual viewpoint set along the path (flow line) of the virtual viewpoint is referred to as flow line information. The image generating device 201 generates a group of virtual viewpoint images based on the flow line information received from the information processing device 202, and transmits the generated group of virtual viewpoint images to the information processing device 202. In addition, the image generating device 201 transmits metadata to the information processing device 202. The metadata includes, for example, position information indicating the position of a player playing on the field 103. Details of the metadata will be described later. The image generating device 201 has a database function for storing multiple captured images and a group of generated virtual viewpoint images, and an image processing function for generating a group of virtual viewpoint images. The multiple cameras 104 in the stadium 101 and the image generating device 201 are connected, for example, by a wired or wireless communication network line, or a cable line such as SDI (Serial Digital Interface). The image generating device 201 receives captured images from the multiple cameras 104 through this line, and stores the received captured images in a database.

The information processing device 202 is a device that edits the virtual viewpoint (position, posture, etc.). The information processing device 202 stores the virtual viewpoint parameter group received from the user terminal 203 in the storage unit 502 (FIG. 5). The information processing device 202 transmits flow line information to the image generating device 201 and receives a virtual viewpoint image group corresponding to the transmitted flow line information. The storage unit 502 stores a plurality of virtual viewpoint parameter groups corresponding to a plurality of flow line information. The information processing device 202 acquires two pieces of flow line information (virtual viewpoint parameter groups) stored in the storage unit 502 and combines the flow line information. More specifically, the information processing device 202 generates a flow line (hereinafter, sometimes referred to as a connecting flow line) that connects two flow lines indicated by two pieces of flow line information, and generates the position and posture information of the virtual viewpoint moving on the connecting flow line to generate the connecting flow line information. At that time, the information processing device 202 generates posture information of the virtual viewpoint moving on the connecting flow line so as to follow a predetermined object. The information processing device 202 combines the two pieces of movement line information using the connection movement line information to generate a new piece of movement line information. That is, the information processing device 202 connects three pieces of movement line information to generate new movement line information (a group of virtual viewpoint parameters). The information processing device 202 is, for example, a personal computer. The information processing device 202 may be incorporated into the image generating device 201 or the user terminal 203. That is, the information processing device 202 may be realized as a device integrated with at least one of the image generating device 201 and the user terminal 203.

The user terminal 203 is an information processing device owned by a user who uses the image processing system 100. The user terminal 203 can accept user operation instructions related to the position and posture of the virtual viewpoint. FIG. 6 is an example of virtual viewpoint operation. For example, the user can move the position of the virtual viewpoint 602 so that it is diagonally forward to the right of the player 601, and direct the line of sight of the virtual viewpoint 602 toward the player 601. The movement line along which the virtual viewpoint 602 moves is called the movement line 603. The user terminal 203 transmits virtual viewpoint parameters indicating the contents of the operation instructions to the information processing device 202 each time a change occurs in the position and posture of the virtual viewpoint. In addition, the user terminal 203 can also transmit a group of virtual viewpoint parameters indicating the contents of the series of operation instructions to the information processing device 202 when a series of operations for specifying the position and posture of the virtual viewpoint as shown in FIG. 6 is completed.

Figure 7 shows an example of the configuration of virtual viewpoint parameters that make up the movement line information. The virtual viewpoint parameters include time information, position information, and attitude information. The time information is composed of HH (hours): MM (minutes): SS (seconds). FF (frames). The position information is obtained by indicating three-dimensional orthogonal coordinates of a coordinate system in which three coordinate axes (x-axis, y-axis, z-axis) intersect at the origin so as to be perpendicular to each other. The origin can be any position in the shooting space. For example, the origin can be the center of the center circle of a soccer field. The attitude information can be indicated by the angles relative to the three axes of pan (horizontal direction), tilt (vertical direction), and roll (direction in which the camera rotates). The virtual viewpoint parameter group is data in which virtual viewpoint parameters are compiled in chronological order.

The user terminal 203 receives a virtual viewpoint image generated based on the virtual viewpoint parameters from the information processing device 202 and displays it on a display device. The display device may be a display device built into the user terminal 203 or a display device of an external device. The user terminal 203 is, for example, a personal computer or a mobile terminal such as a smartphone or tablet. The user terminal 203 has an interface for accepting user operations such as at least one of a mouse, a keyboard, a six-axis controller, and a touch panel.

The image generating device 201, the information processing device 202, and the user terminal 203 are configured to be able to exchange information with each other via a network such as the Internet. Note that communication between the devices may be performed by either wireless communication or wired communication, or a combination of these.

(Hardware configuration of the device)
Next, an example of the hardware configuration of the image generating device 201, the information processing device 202, and the user terminal 203 will be described with reference to Fig. 3. In this embodiment, each device has a hardware configuration shown in the block diagram of Fig. 3, and includes a controller unit 300, an operation unit 309, and a display device 310. Note that the image generating device 201, the information processing device 202, and the user terminal 203 do not need to have the same hardware configuration, and they do not need to have all of the configurations described below.

The controller unit 300 includes a CPU 301, a ROM 302, a RAM 303, a HDD 304, an operation unit I/F 305, a display unit I/F 306, and a communication I/F 307. These hardware blocks are connected so as to be able to communicate with each other, for example, via a system bus 308. Note that CPU is an acronym for Central Processing Unit, HDD is an acronym for Hard Disk Drive, and I/F is an interface.

The CPU 301 controls the operation of the ROM 302, the RAM 303, the HDD 304, the operation unit I/F 305, the display unit I/F 306, and the communication I/F 307 via the system bus 308. The CPU 301 starts the OS (operating system) by a boot program stored in the ROM 302. The CPU 301 executes an application program stored in the HDD 304, for example, on this OS. The CPU 301 executes the application program to realize various processes of each device. The ROM 302 is a read-only semiconductor memory, and stores various programs and parameters executed by the CPU 301. The RAM 303 is a semiconductor memory that can be read and written at any time, and provides, for example, an area for storing temporary information and a working area for the CPU 301. The HDD 304 is a large-capacity storage device such as a hard disk, and stores application programs, image data, etc. The CPU 301 may be replaced by one or more arbitrary processors (e.g., an ASIC (application specific integrated circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), etc.).

The operation unit I/F 305 is an interface that connects the operation unit 309 to the controller unit 300. The operation unit I/F 305 transfers information on user operations received by the operation unit 309 to the CPU 301. The operation unit 309 is configured to include devices that can receive user operations, such as a mouse, a keyboard, and a touch panel. The display unit I/F 306 is an interface that connects the display device 310 to the controller unit 300. For example, the CPU 301 outputs image data to be displayed to the display device 310 via the display unit I/F 306. The display device 310 is configured to include a display such as a liquid crystal display. The communication unit I/F 307 is an interface for performing communication such as Ethernet (registered trademark). The communication unit I/F 307 has a connector for receiving a connection of a transmission cable. The communication unit I/F 307 may be a wireless communication interface, in which case the communication unit I/F 307 has, for example, a baseband circuit, an RF circuit, and an antenna. The controller unit 300 inputs and outputs information with external devices via the communication unit I/F 307.

Note that the display device 310 in each device may be an external device. That is, the device may also perform display control to display an image on the external display device 310 connected via a cable or network. In this case, the device executes display control to output display data to the display device 310. Note that the configuration in FIG. 3 is an example, and some of the configurations may be omitted or configurations not shown may be added, and further, the configurations shown in the figures may be combined. For example, the image generating device 201 may not have a display device 310.

(Functional configuration of image generating device 201)
Fig. 4 shows an example of a functional configuration of the image generating device 201. The functional configuration shown in Fig. 4 is realized by a CPU 301 of the image generating device 201 reading various programs recorded in a ROM 302 or the like and executing control of each unit. Note that a part or all of the functional configuration shown in Fig. 4 may be realized by dedicated hardware (e.g., an ASIC or an FPGA).

In one example, the image generating device 201 has a control unit 401, a storage unit 402, an image input unit 403, an image storage unit 404, an image generating unit 405, a metadata generating unit 406, and a data transmitting/receiving unit 407. These functional units are interconnected by an internal bus 408, and can transmit and receive data to and from each other under the control of the control unit 401. The control unit 401 controls the operation of the entire image generating device 201, including each of the functional units described below. The storage unit 402 includes a ROM 302, a RAM 303, and a HDD 304, and stores and reads out various programs, various data such as captured images, etc.

The image input unit 403 acquires images (photographed images) taken by multiple cameras 104 installed in the stadium 101 at a predetermined frame rate via the communication unit I/F 307. The image input unit 403 acquires the photographed images from the cameras 104, for example, by a wired or wireless communication module, or an image transmission module such as SDI. The image storage unit 404 stores the photographed images acquired by the image input unit 403, a group of virtual viewpoint images generated based on the photographed images, and metadata indicating the position of the object, for example, in the HDD 304. The group of virtual viewpoint images is generated by the image generation unit 405 described later, and the metadata is generated by the metadata generation unit 406 described later. The image storage unit 404 can be realized, for example, by a magnetic disk, an optical disk, a semiconductor memory, or the like. The image storage unit 404 may be realized by a device (RAM 303, HDD 304) built into the image generating device 201, or may be realized by an external device physically separated from the image generating device 201. The captured images and virtual viewpoint images stored in the image storage unit 404 are stored in an image format, for example, MXF (Material exchange format). These captured images and virtual viewpoint images are compressed in, for example, MPEG2 format. However, the image format and data compression method are not limited to these, and any image format and data compression method may be used.

The image generating unit 405 generates a group of virtual viewpoint images corresponding to a plurality of virtual viewpoints from a plurality of captured images stored in the image storage unit 404. The group of virtual viewpoint images can be generated, for example, using image-based rendering (hereinafter, IBR). IBR is a rendering method that generates virtual viewpoint images from images captured from a plurality of actual viewpoints without performing modeling (a process of creating the shape of an object using geometric figures). However, the method of generating the group of virtual viewpoint images is not limited to IBR. For example, the group of virtual viewpoint images can be generated using model-based rendering (hereinafter, MBR). MBR is a method of generating virtual viewpoint images using a three-dimensional model that is generated based on a plurality of captured images obtained by capturing an object from a plurality of directions. In MBR, the appearance of the scene from a virtual viewpoint is generated as an image by using the three-dimensional shape (model) of the target scene obtained by three-dimensional shape reconstruction methods such as volume intersection and Multi-View-Stereo (MVS).

The virtual viewpoint image group generated by the image generation unit 405 includes virtual viewpoint images of various virtual viewpoint positions and line of sight directions, and is compressed and encoded in the spatial direction and the temporal direction as one image stream. However, the form of the virtual viewpoint image group is not limited to such an image stream, and each may be composed of a plurality of independent images. Also, the virtual viewpoint image group does not have to be compressed and encoded. Also, in this embodiment, the image generation device 201 generates the virtual viewpoint image group, but this is not limited to this. For example, the image generation device 201 may generate information for generating a virtual viewpoint image, such as information indicating a three-dimensional model such as three-dimensional shape data, or an image mapped to a three-dimensional model indicated by the information. That is, instead of generating a rendered virtual viewpoint image, the image generation unit 405 may generate information necessary for rendering the virtual viewpoint image in the information processing device 202 or the user terminal 203.

The metadata generating unit 406 analyzes the captured images stored in the image storage unit 404 and acquires the position information of the object. The metadata generating unit 406 acquires the position information of a specific object using, for example, a technique such as visual hull. The metadata generating unit 406 acquires the position information of the object at all times from the start of shooting to the end of shooting of the captured images stored in the image storage unit 404. The metadata generating unit 406 generates metadata including the acquired position information for the object. FIG. 8 shows an example of the configuration of metadata generated by the metadata generating unit 406. The metadata includes, for example, time information and position information of each object. The time information is composed of HH (hours): MM (minutes): SS (seconds). FF (frames). The position information indicates the position of an object (for example, a ball, a player, a referee) using three-dimensional orthogonal coordinates. The object name is composed of an arbitrary name and an object ID. The object ID is an identification symbol composed of alphabets and numbers and assigned to each object so that each object can be distinguished. Note that the metadata is not limited to that obtained by the analysis of the captured image by the metadata generating unit 406, but may be that registered in advance in the image generating device 201 or the information processing device 202.

The data transmission/reception unit 407 outputs the group of virtual viewpoint images stored in the image storage unit 404 to the information processing device 202 at a predetermined frame rate. In addition, the data transmission/reception unit 407 outputs metadata stored in the image storage unit 404 to the information processing device 202 before outputting the group of virtual viewpoint images. In addition, the data transmission/reception unit 407 receives movement line information (group of virtual viewpoint parameters) from the information processing device 202.

(Functional configuration of information processing device 202)
Fig. 5 is a block diagram showing an example of a functional configuration of the information processing device 202. The functional configuration shown in Fig. 5 is realized by the CPU 301 of the information processing device 202 reading various programs recorded in the ROM 302 or the like and executing control of each unit. Note that some or all of the functional units shown in Fig. 5 may be realized by dedicated hardware (e.g., ASIC or FPGA).

In one example, the information processing device 202 has a control unit 501, a memory unit 502, a data transmission/reception unit 503, an image memory unit 504, a movement line generation unit 505, a posture generation unit 506, a movement line connection unit 507, an image acquisition unit 508, and an input/output unit 509. These functional units are connected to each other by an internal bus 510, and can transmit and receive data to each other under the control of the control unit 501. The control unit 501 controls the operation of the entire information processing device 202, including each of the functional units described below. The memory unit 502 includes a ROM 302, a RAM 303, and a HDD 304, and stores and reads out various programs, various data such as captured images, etc.

The data transmission/reception unit 503 receives a group of virtual viewpoint images and metadata from the image generation device 201. The data transmission/reception unit 503 also transmits movement line information (group of virtual viewpoint parameters) to the image generation device 201. The image storage unit 504 stores the group of virtual viewpoint images and metadata received by the data transmission/reception unit 503, the movement line information (group of virtual viewpoint parameters) acquired by the movement line connection unit 507 and the input/output unit 509, and the like. The image storage unit 504 is realized, for example, by a magnetic disk, an optical disk, a semiconductor memory, or the like. The image storage unit 504 may be realized by a device (ROM 302, RAM 303, HDD 304) built into the information processing device 202, or may be realized by an external device physically separated from the information processing device 202.

The flow line generating unit 505, the posture generating unit 506, and the flow line connecting unit 507 generate connecting flow line information for connecting two pieces of flow line information stored in the image storage unit 504. For example, the flow line generating unit 505 determines a point on one of the two flow lines to be connected (hereinafter, the first flow line) as the start point of the connecting flow line, and determines a point on the other flow line (hereinafter, the second flow line) as the end point of the connecting flow line. In this embodiment, the end point of the first flow line and the start point of the second flow line are used. The flow line generating unit 505 generates a linear connecting flow line so that the virtual viewpoint moves from the start point to the end point of the determined connecting flow line, and interpolates the position information of the virtual viewpoint moving on the connecting flow line in units of time. Note that in this embodiment, an example is shown in which a straight line is used to interpolate between the start point and the end point, but this is not limited to this. For example, a curve expressed by a spline function or the like may be used as the connecting flow line. In addition, which of the two flow lines to be combined will be the first flow line and which will be the second flow line may be automatically determined by the flow line generation unit 505, or may be explicitly specified by the user via the user terminal 203. For example, the flow line generation unit 505 determines, of the two flow lines to be combined, the one having the earlier time information (start time) of the virtual viewpoint parameter located at the top of the virtual viewpoint parameter group, as the first flow line.

Furthermore, the flow line generating unit 505 determines the playback speed when displaying the virtual viewpoint image on the connecting flow line. The flow line generating unit 505 adjusts the playback speed according to the distance or time difference (time difference) between the start point and the end point of the connecting flow line. For example, when the distance between the start point and the end point of the connecting flow line is greater than a predetermined value, the flow line generating unit 505 sets a playback speed faster than the normal playback speed. As a result, fast-forward playback is performed while the virtual viewpoint moves along the connecting flow line. Also, the playback speed may be set faster as the distance between the start point and the end point of the connecting flow line is greater. Also, for example, the flow line generating unit 505 may set a playback speed faster than the normal playback speed when the time difference between the start point and the end point of the connecting flow line is greater than a predetermined value. Also, the playback speed may be set faster as the time difference between the start point and the end point of the connecting flow line is greater. Note that fast-forward playback at a predetermined speed may be performed unconditionally, regardless of the distance or time difference between the start point and the end point of the connecting flow line.

The posture generation unit 506 generates posture information of a virtual viewpoint moving on the line of the connecting flow line generated by the flow line generation unit 505. The posture generation unit 506 acquires a virtual viewpoint parameter located at the start point or end point of one of the two flow lines that the flow line generation unit 505 has selected as the target of joining. The posture generation unit 506 provides the acquired virtual viewpoint parameter to the image acquisition unit 508, thereby causing the image acquisition unit 508 to acquire a virtual viewpoint image. The posture generation unit 506 detects an object located at the center or near the center of the virtual viewpoint image acquired by the image acquisition unit 508, and calculates the three-dimensional orthogonal coordinates of the detected object. The posture generation unit 506 uses the time information of the virtual viewpoint parameter and the calculated three-dimensional orthogonal coordinates to collate with the metadata stored in the image storage unit 504, and identifies the object ID of the detected object. The posture generation unit 506 extracts the position information of the object at each time from the metadata based on the time information of the virtual viewpoint moving on the flow line of the connecting flow line and the identified object ID. The orientation generation unit 506 determines the orientation of the virtual viewpoint based on the position information of the virtual viewpoint moving on the flow line of the connecting flow line and the position information of the object extracted from the metadata. In other words, the orientation generation unit 506 generates the orientation information of the virtual starting point so that the straight line connecting the position of the virtual viewpoint and the position of the object coincides with the direction of the virtual viewpoint.

The movement line combination unit 507 combines two pieces of movement line information to be combined, acquired by the movement line generation unit 505 from the image storage unit 504, with the connected movement line information (connected movement line to which the position and posture information of the virtual viewpoint have been added) generated by the movement line generation unit 505 and the posture generation unit 506. The movement line combination unit 507 stores the movement line information (a group of virtual viewpoint parameters) obtained by the combination in the image storage unit 504. At that time, the movement line combination unit 507 applies the playback speed determined by the movement line generation unit 505.

The image acquisition unit 508 selects a virtual viewpoint image from the group of virtual viewpoint images stored in the image storage unit 504 based on the virtual viewpoint parameters provided by the posture generation unit 506 or the virtual viewpoint parameters acquired from the input/output unit 509. The image acquisition unit 508 can also acquire a group of virtual viewpoint parameters (traffic line information) from the image storage unit 504 and select a virtual viewpoint image from the group of virtual viewpoint images based on the virtual viewpoint parameters extracted in order from the top of the group of virtual viewpoint parameters. That is, the image acquisition unit 508 can sequentially acquire virtual viewpoint images from the image storage unit 504 according to the traffic line information and provide the acquired virtual viewpoint images to the user terminal 203 via the input/output unit 509. The input/output unit 509 outputs the virtual viewpoint image acquired by the image acquisition unit 508 to the user terminal 203. The input/output unit 509 also inputs operation instructions, virtual viewpoint parameters, and virtual viewpoint parameter groups from the user terminal 203.

(Operation of information processing device 202)
9 is a flowchart showing a series of processing steps in the information processing device 202. This flowchart shows a process of generating connected trajectory information that connects two pieces of trajectory information and combining them to generate one piece of trajectory information. This flowchart is executed, for example, when a request to combine two pieces of trajectory information that have already been created is received from the user terminal 203.

In S901, the movement line generating unit 505 acquires two pieces of movement line information stored in the image storage unit 504. FIG. 10 illustrates a group of virtual viewpoint parameters included in the movement line information. In FIG. 10, 1001 and 1002 are players, and show how the players move in the direction of the dashed arrows. Also, 1011 and 1012 are virtual viewpoints, and show how the virtual viewpoints move in the direction of the solid arrows. Also, 1021 and 1022 show the movement lines of the virtual viewpoints 1011 and 1012, respectively. The virtual viewpoint 1011 moves diagonally in front of the player 1001 in accordance with the movement of the player 1001, and its posture is adjusted so that the player 1001 can always be photographed. The virtual viewpoint 1012 gradually approaches the player 1001 from a distance, and finally moves so as to rotate around the player 1001, and its posture is adjusted so that the player 1001 can always be photographed. For ease of explanation, the players and virtual viewpoints are thinned out in the illustration, but in reality they exist in each frame. Also, the shooting times of the movement lines 1021 and 1022 are shown in FIG. 15(a). The shooting time of the movement line 1021 is the movement line A in FIG. 15(a), and the shooting time of the movement line 1022 is the movement line B in FIG. 15(a). There is an overlapping period between the movement lines A and B, with the shooting of the movement line A starting earlier than that of the movement line B, and the shooting ending time of the movement line B being later.

Returning to FIG. 9, in S902, the movement line generation unit 505 acquires the movement lines contained in the two pieces of movement line information acquired in S901, and generates a connecting movement line that connects these two movement lines. FIG. 11 is an example of a connecting movement line generated by the movement line generation unit 505. In FIG. 11, movement lines 1021 and 1022 are the movement lines shown in FIG. 10. 1101 indicates the end point of movement line 1021, and 1102 indicates the start point of movement line 1022. Also, the connecting movement line 1110 is a movement line generated from the end point 1101 to the start point 1102. Of the two acquired movement lines, the movement line generation unit 505 determines movement line 1021 as the forward movement line, and movement line 1022 as the backward movement line. The flow line generation unit 505, for example, compares the time information of the virtual viewpoint parameter located at the top of the virtual viewpoint parameter group, and determines the one with the earlier shooting start time as the forward flow line. The flow line generation unit 505 determines the end point 1101 of the forward flow line 1021 as the start point of the connecting flow line 1110, and determines the start point 1102 of the rear flow line 1022 as the end point of the connecting flow line 1110. The flow line generation unit 505 generates a flow line from the end point 1101 to the start point 1102 as the connecting flow line 1110.

In this embodiment, the flow line generating unit 505 sets the connecting flow line 1110 as a straight line connecting the end point 1101 and the start point 1102, and interpolates the position information of the virtual viewpoint moving on the connecting flow line 1110 in units of time. Of course, the connecting flow line 1110 may be a curve represented by a predetermined function such as a spline function. Also, since the time of the start point 1102 is earlier than the time of the end point 1101, the time information of the virtual viewpoint moving on the connecting flow line 1110 goes back in time. Therefore, the playback of the virtual viewpoint image by the connecting flow line 1110 is reverse playback. Also, the posture information of the virtual viewpoint at this time is empty and is set in S905. The interpolation method may be a spline function. Also, the flow line selected as the forward flow line from the two acquired flow lines may be selected by the user from the user terminal 203.

In S903, the flow line generating unit 505 determines the playback speed when displaying the virtual viewpoint image of the connecting flow line 1110 generated in S902. The flow line generating unit 505 sets the playback direction of the connecting flow line 1110 generated in S902 to reverse (reverse playback) and determines the playback speed to fast forward. As described above, the playback speed may be adjusted according to the distance or time difference between the start point and the end point of the connecting flow line. Also, the fast forward speed may be increased as the distance or time difference increases. Also, the playback speed may be set by the user from the user terminal 203. Also, fast forward may be set for reverse playback, and normal playback may be set for forward playback. Once the playback speed is determined, the position of the virtual viewpoint on the connecting flow line 1110 is determined. For example, the playback time for the connecting flow line 1110 is determined from the time difference between the start point and the end point and the playback speed, and the position of the number of frames determined based on the playback time is arranged at equal distance intervals on the connecting flow line 1110, thereby determining the position of the virtual viewpoint.

Next, in S904, the posture generation unit 506 determines an object to be followed by the virtual viewpoint moving on the connecting movement line 1110 generated in S902. FIG. 12 is a diagram for explaining a method for determining an object to be followed. In FIG. 12(a), movement lines 1021 and 1022 and players 1001 and 1002 are as described in FIG. 10. 1201 indicates the virtual viewpoint located at the end point of movement line 1021. Also, 1211 indicates the shooting range of virtual viewpoint 1201. Note that the diagonal lines of players 1001 and 1002 indicate the locations where the players are playing at the time when virtual viewpoint 1201 is located at the end point of movement line 1021. FIG. 12(b) is a virtual viewpoint image acquired for virtual viewpoint 1201.

The posture generation unit 506 acquires the virtual viewpoint parameters of the virtual viewpoint 1201 located at the end point of the forward movement line 1021 determined in S902. The posture generation unit 506 sends the acquired virtual viewpoint parameters to the image acquisition unit 508 and acquires a virtual viewpoint image from the image acquisition unit 508. At this time, the virtual viewpoint image acquired from the image acquisition unit 508 is shown in FIG. 12(b). The posture generation unit 506 detects an object located at the center or near the center of the virtual viewpoint image and calculates the three-dimensional orthogonal coordinates of the detected object. The posture generation unit 506 uses the time information of the virtual viewpoint parameters of the virtual viewpoint 1201 and the calculated three-dimensional orthogonal coordinates to match them with the metadata stored in the image storage unit 504 and identify the object ID. In the example of FIG. 12(a) and (b), the position of the player 1001 is detected from the virtual viewpoint image of the virtual viewpoint 1201, and the object ID of the player 1001 is identified by matching it with the metadata. Note that, although the object to be followed is detected from the virtual viewpoint image at the end point of the flow line 1021, this is not limited to the above. For example, the object to be followed may be detected from the virtual viewpoint image at the start point of the flow line 1022. Also, the object to be followed may be specified from the user terminal 203.

In S905, the posture generation unit 506 generates the posture of each position of the virtual viewpoint moving on the line of the connecting flow line 1110 generated in S902. FIG. 13 is an example of a virtual viewpoint moving on the line of the connecting flow line 1110. In FIG. 13, 1301 indicates a virtual viewpoint moving on the line of the connecting flow line 1110. The posture of the virtual viewpoint 1301 is adjusted so that the player 1001 can always be photographed while moving in the direction of the arrow along the solid arrow. Note that the time information of the virtual viewpoint 1301 moving on the line of the connecting flow line 1110 goes back in time so that the player 1001 also moves backward in the direction of the arrow along the dashed arrow. The posture generation unit 506 acquires the position information of the player 1001 from the metadata based on the time information of each position of the virtual viewpoint 1301 moving on the line of the connecting flow line 1110 and the object ID determined in S904. The posture generation unit 506 generates posture information of the virtual viewpoint by adjusting the posture of the virtual viewpoint at each position so that the line of sight of the virtual viewpoint 1301 faces the position of the player 1001 extracted from the metadata.

In S906, the flow line combination unit 507 combines the two pieces of flow line information acquired in S901 with the connecting flow line information generated in S902 to S905. FIG. 14 shows an example of a state in which these three pieces of flow line information are combined. The flow line combination unit 507 connects in the following order: the forward flow line 1021 determined in S902, the connecting flow line 1110 to which posture information has been added in S905, and the rearward flow line 1022 determined in S902. The flow line combination unit 507 stores the flow line information (virtual viewpoint parameter group) obtained by the combination in the image storage unit 504.

As described above, according to this embodiment, when two pieces of already created flow line information are combined, a connecting flow line that connects the two flow lines is generated, and the line of sight of the virtual viewpoint that moves on the line of the connecting flow line can always be directed to the moving object. Therefore, even if the user does not perform an operation related to the position or posture of the virtual viewpoint, flow line information that connects two flow lines can be generated so that the object to be focused on does not leave the range of the virtual viewpoint image. For example, in a soccer game, consider moving the virtual viewpoint to follow a player who has performed a series of actions from dribbling to shooting as an object of interest, and then moving the virtual viewpoint to another position while rewinding time (while playing backwards). In this case, if the flow line of the virtual viewpoint for reverse playback is generated by simply interpolating the position and posture on the flow line set corresponding to the basic frame, the position of the player who is the object of interest is not taken into consideration. Therefore, there is a case where the player who is the object of interest moves out of the field of view of the virtual viewpoint during reverse playback. In contrast, according to the above-mentioned embodiment, the flow line is interpolated taking into account the position of the object of interest, so that the risk of the player who is the object of interest moving out of the field of view of the virtual viewpoint can be reduced.

In the above embodiment, an example of generating a connecting flow line that connects two flow lines has been described, but the present invention is not limited to this. For example, by specifying two virtual viewpoints for which a time, position, and orientation are set, the present invention can be applied to a configuration in which flow line information connecting these virtual viewpoints is automatically generated. That is, in the key frame method, the above-mentioned method of generating flow line information can be applied to determine the position and orientation of a virtual viewpoint corresponding to an intermediate frame between two basic frames. This makes it possible to determine the virtual viewpoint of the intermediate frame so that a specified object does not fall out of the virtual viewpoint image.

Other Embodiments
In the above embodiment, when a request to combine two pieces of already created flow line information is received from the user terminal 203, the process of the flowchart shown in Fig. 9 is executed. However, the present invention is not limited to this. For example, the control unit 501 may control the process of the flowchart shown in Fig. 9 to be executed every time new flow line information (a group of virtual viewpoint parameters) is added to the image storage unit 504.

In this case, the control unit 501 may analyze the newly added flow line information and the existing flow line information stored in the image storage unit 504, and determine whether or not to combine these pieces of flow line information. For example, as shown in FIG. 15(a), if the ranges of the shooting times of the two flow lines at least partially overlap, and a common object is photographed by the two pieces of flow line information as shown in FIG. 15(b), the control unit 501 determines to combine the two pieces of flow line information. Note that the determination conditions for executing the combination are not limited to this. In this way, two pieces of flow line information that are highly related to each other can be automatically combined, even if the user does not issue an instruction to combine the two flow lines. Note that the execution of the combination of the flow line information may be set by the user from the user terminal 203.

In addition, while the above embodiment illustrates the case of photographing a soccer match, the subject of the photograph is not necessarily limited to this. For example, this embodiment can also be applied to photographing other sports matches such as rugby, tennis, ice skating, and basketball, as well as live performances and concerts.

As described above, according to each embodiment, when the virtual viewpoint of an intermediate frame is interpolated using the key frame method, the attitude of the virtual viewpoint is determined so as to track the object that is the subject of the virtual viewpoint's shooting, so that the object does not move out of the shooting range of the virtual viewpoint.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present invention. Therefore, in order to publicize the scope of the present invention, the following claims are appended.

202: Information processing device, 501: Control unit, 502: Storage unit, 503: Data transmission/reception unit, 504: Image storage unit, 505: Movement line generation unit, 506: Posture generation unit, 507: Movement line connection unit, 508: Image acquisition unit, 509: Input/output unit, 510: Internal bus

Claims (15)

An information processing device that generates a position of a virtual viewpoint and a line of sight direction from the virtual viewpoint for generating a virtual viewpoint image from a plurality of images obtained by photographing a predetermined space with a plurality of photographing devices,
an acquisition means for acquiring a first position and a second position of the virtual viewpoint, and a first time and a second time corresponding to the first position and the second position and indicating a time of shooting of the predetermined space ;
a generating means for generating a flow line along which the virtual viewpoint moves by connecting the first position and the second position;
a setting means for setting a plurality of positions of the virtual viewpoint between the first position and the second position of the traffic line, and setting a photographing time of the predetermined space corresponding to each of the plurality of positions based on the first time and the second time;
a determination means for determining a line of sight direction from the virtual viewpoint at each of the plurality of positions based on a position of a specific object in the predetermined space at the shooting time set at each of the plurality of positions,
the acquiring means acquires first flow line information including a plurality of positions including the first position of the virtual viewpoint, a line of sight from the virtual viewpoint at each of the plurality of positions, and a photographing time, and second flow line information including a plurality of positions including the second position of the virtual viewpoint, and a line of sight from the virtual viewpoint at each of the plurality of positions, and a photographing time,
The information processing device, characterized in that the generation means generates the flow line connecting the first position and the second position as a flow line of third flow line information along which the virtual viewpoint moves. The information processing device according to claim 1, further comprising a combining means for combining the first movement line information, the second movement line information, and the third movement line information to generate combined movement line information. The information processing device according to claim 1 or 2, further comprising a selection means for selecting the specific object based on a virtual viewpoint image obtained from the first flow line information or the second flow line information. The information processing device according to claim 3, characterized in that the selection means selects, as the specific object, an object located closest to the center in a virtual viewpoint image obtained from a virtual viewpoint at the first position or the second position. The information processing device according to any one of claims 1 to 4, characterized in that the generating means generates a flow line whose starting point is the end point of the virtual viewpoint represented by the first flow line information and whose ending point is the start point of the virtual viewpoint represented by the second flow line information. The information processing device according to any one of claims 1 to 5, characterized in that the generating means generates the movement line by connecting the first position and the second position with a straight line. The information processing device according to any one of claims 1 to 5, characterized in that the generating means generates the movement line by connecting the first position and the second position with a curve represented by a predetermined function. The information processing device according to any one of claims 1 to 7, characterized in that the setting means sets a playback speed based on a distance between the first position and the second position or a time difference between the first time and the second time , and sets the position of the virtual viewpoint and the shooting time on the traffic line based on the playback speed. The information processing device according to claim 8, characterized in that the setting means sets a playback speed faster than the normal playback speed when the distance is greater than a predetermined value or when the time difference is greater than a predetermined value. The information processing device according to claim 8 or 9, characterized in that the setting means sets a faster playback speed the greater the distance or the greater the time difference. The information processing device according to any one of claims 1 to 5, further comprising a determination means for determining whether or not to generate the third movement line information based on the first movement line information and the second movement line information. The information processing device according to claim 11, characterized in that the determination means determines to generate the third traffic line information when the range of shooting times included in the first traffic line information and the range of shooting times included in the second traffic line information at least partially overlap. The information processing device according to claim 11 or 12, characterized in that the determination means determines to generate the third flow line information when a virtual viewpoint image generated based on the first flow line information and a virtual viewpoint image generated based on the second flow line information include a common object. 1. An information processing method for generating a position of a virtual viewpoint and a line of sight direction from a virtual viewpoint for generating a virtual viewpoint image from a plurality of images obtained by photographing a predetermined space with a plurality of photographing devices, comprising:
an acquisition step of acquiring a first position and a second position of the virtual viewpoint, and a first time and a second time corresponding to the first position and the second position and indicating a shooting time of the predetermined space ;
a generating step of connecting the first position and the second position to generate a flow line along which the virtual viewpoint moves;
a setting step of setting a plurality of positions of the virtual viewpoint between the first position and the second position of the traffic line, and setting a shooting time of the predetermined space corresponding to each of the plurality of positions based on the first time and the second time;
determining a line of sight direction from the virtual viewpoint at each of the plurality of positions based on a position of a specific object in the predetermined space at the shooting time set at each of the plurality of positions;
The acquiring step acquires first flow line information including a plurality of positions including the first position of the virtual viewpoint, a line of sight from the virtual viewpoint at each of the plurality of positions, and a photographing time, and second flow line information including a plurality of positions including the second position of the virtual viewpoint, and a line of sight from the virtual viewpoint at each of the plurality of positions, and a photographing time,
An information processing method characterized in that the generation step generates the flow line connecting the first position and the second position as a flow line of third flow line information along which the virtual viewpoint moves. A program for causing a computer to function as each of the means of an information processing device according to any one of claims 1 to 13.

Images