WO2020189341A1

WO2020189341A1 - Image display system, image distribution method, and program

Info

Publication number: WO2020189341A1
Application number: PCT/JP2020/009701
Authority: WO
Inventors: Osamu Ogawara; Yohei Fujita; Takayuki Hara
Original assignee: Ricoh Company, Ltd.
Priority date: 2019-03-18
Filing date: 2020-03-06
Publication date: 2020-09-24
Also published as: JP2020154457A

Abstract

An image display system in which a terminal device receives a content from an image distribution apparatus 10 for distributing an image, wherein the image distribution apparatus includes a display pattern generation unit that generates a display pattern in which one or more of the content is displayed from different viewpoints, a constraint unit that constrains the viewpoints, and a distribution unit that distributes the display pattern and the content, or a plurality of display images generated based on the display pattern to the terminal device.

Description

IMAGE DISPLAY SYSTEM, IMAGE DISTRIBUTION METHOD, AND PROGRAM

The present invention relates to an image display system, image distribution method, and program.

A distribution type display advertisement, in which an advertiser displays an advertisement within an advertisement space provided in a Web pages is known.
The advertiser can have a product or service advertisement displayed in advertisement spaces provided by various Web sites. A viewer interested in the product or service displayed in the advertisement space click (or tap) the advertisement space, allowing the advertiser to attract viewers to his or her Web site.

While the advertiser wants to increase the frequency with which the viewers click on the advertisements, the number of advertisements posted on a single Web page tends to increase, and the advertiser are devising ways to attract viewers' attention (see, for example, PTL 1). PTL 1 discloses a technology for increasing the effect (click rate) of advertising by distributing video advertising. People tend to change gaze to a moving object. This change of gaze can increase advertising attention and improve advertising effectiveness.

In addition, there are interactive contents, such as 3DCG (Three-Dimensional Computer Graphics) images and wide-angle images, that can be viewed differently by the viewer. There are also known attempts to increase attention and improve the effectiveness of advertisement by having such content appear on Web pages.

Japanese Laid-Open Patent Application No. 2008-192115. C.H. Lee, A. Varshney, D.W. Jacobs, "Mesh saliency," ACM transactions on graphics (TOG)、 Vol. 24, No. 3, 2005

However, there is a problem that the content that can change the viewpoint such as 3DCG image or wide angle image does not necessarily display an image from an appropriate viewpoint. In addition, there is a display method in which the terminal device that receives the content displays the content on the advertisement space while continuously changing the viewpoint of the content. With this display method, a part of the content is displayed one after another in an advertisement space, allowing a still image to be perceived like a video image. It is known that advertisement is more effective than a normal still image advertisement because the image within the media is automatically changed.

However, such a display method of displaying a content may automatically display the content viewed from a viewpoint not often required by the viewer, which possibly impair the effectiveness of advertisement. For example, the viewer may look down from above or up directly below the content, thereby upsetting a viewer's sense of earth.

An object of the present invention is to provide an image display system capable of displaying a content from an appropriate viewpoint in consideration of the above object.

In view of the above-described object, the present invention provides an image display system in which a terminal device receives a content from an image distribution apparatus for distributing an image, wherein the image distribution apparatus includes a display pattern generation unit that generates a display pattern in which one or more of the content is displayed from different viewpoints, a constraint unit that constrains the viewpoints, and a distribution unit that distributes the display pattern and the content, or a plurality of display images generated based on the display pattern to the terminal device, and wherein the terminal device includes an acquisition unit that acquires a plurality of display images generated based on the display pattern and the content or the display pattern, and a display unit that displays the content acquired by the acquisition unit by changing the viewpoint of the display pattern or the plurality of display images generated based on the display pattern.

The present invention can provide an image display system capable of displaying the content from an appropriate viewpoint.

An Example of a diagram explaining a full spherical image as an example of a wide angle image. An Example of a diagram explaining a full spherical image as an example of a wide angle image. An Example of a diagram explaining a full spherical image as an example of a wide angle image. A schematic view illustrating an outline of an image display system. A diagram illustrating a transition prohibition area in which a transition is prohibited when a terminal device displays an animation display full spherical image or a 3DCG image. A diagram illustrating a transition prohibition area in which a transition is prohibited when a terminal device displays an animation display full spherical image or a 3DCG image. A diagram illustrating a transition prohibition area in which a transition is prohibited when a terminal device displays an animation display full spherical image or a 3DCG image. A diagram illustrating a transition prohibition area in which a transition is prohibited when a terminal device displays an animation display full spherical image or a 3DCG image. A diagrammatic representation of an example of an image display system. An example of a hardware configuration diagram of a computer system. An example of a hardware configuration diagram of a terminal device. An example of using a full spherical camera. An outline of the process from an image captured by a full spherical camera to the creation of a full spherical image. An outline of the process from an image captured by a full spherical camera to the creation of a full spherical image. An outline of the process from an image captured by a full spherical camera to the creation of a full spherical image. A diagram illustrating an outline of the process from an image captured by a full spherical camera to the creation of a full spherical image. A diagram illustrating an outline of the process from an image captured by a full spherical camera to the creation of a full spherical image. An example of a diagram explaining a viewer's eye gaze direction. An example of a relationship between predetermined area information and an image of a predetermined area. A schematic representation of a 3D model and a projection transformation. A schematic representation of a 3D model and a projection transformation. An example of a 3D model with a different viewpoint. An example of a 3D model with a different viewpoint. An example of a 3D model with a different viewpoint. An example of a 3D model with a different viewpoint. An example of a 3D model with a different viewpoint. An example of a 3D model with a different viewpoint. An example of information used by a Web page. An example of information used by a Web page. An example of information used by a Web page. A functional block diagram illustrating a function of a terminal device, an advertiser Web server, a partner site Web server, an image distribution apparatus, an SSP, and a DSP in a block shape. An example of a functional block diagram illustrating functions of a content reception unit and the image registration terminal in a block shape. An example of a content registration screen displayed on a display of the image registration terminal. An example of a content registration screen displayed on a display of the image registration terminal. An example of a flowchart illustrating a procedure in which a point-of-interest determination unit determines a point-of-interest. A partial image of an example of a full spherical image. A partial image of an example of a full spherical image. A flowchart illustrating an example of an overall flow in which a display pattern generation unit generates a display pattern. An example of a diagram illustrating a parameter of the position of a virtual camera in an eye gaze direction of full spherical image or a 3DCG image. An example of a diagram illustrating a parameter of the position of a virtual camera in an eye gaze direction of full spherical image or a 3DCG image. An example of a diagram illustrating a parameter of the position of a virtual camera in an eye gaze direction of full spherical image or a 3DCG image. An example of a simplified parameter of the position of a virtual camera in an eye gaze direction of full spherical image or a 3DCG image. An example of a simplified parameter of the position of a virtual camera in an eye gaze direction of full spherical image or a 3DCG image. An example of a virtual camera in the eye gaze direction of full spherical image or the 3DCG image when a horizontal orbit is employed. An example of a virtual camera in the eye gaze direction of full spherical image or the 3DCG image when a horizontal orbit is employed. An example of a virtual camera position in the eye gaze direction of full spherical image or the 3DCG image when a large circle orbit is employed. An example of a virtual camera position in the eye gaze direction of full spherical image or the 3DCG image when a large circle orbit is employed. An example of a virtual camera position in the 3DCG image using an arbitrary axis. An example of a virtual camera position in the 3DCG image using a helical orbit. A flowchart illustrating a procedure performed in step S103 of FIG. 20 when a display pattern is generated using an element orbit. A diagram illustrating an accumulated value of an interest on the element orbit. A flowchart illustrating an example of a procedure for determining two or more element orbits. An example of a sequence diagram illustrating a procedure for delivering a full spherical image or 3DCG image in which an image display system is displayed as an advertisement. A flowchart illustrating an operation of the terminal device that receives the advertisement data. A flowchart illustrating an operation of the terminal device that receives the advertisement data. A diagram illustrating a form in which the advertisement data is distributed. A diagram illustrating a form in which the advertisement data is distributed. A diagram illustrating a form in which the advertisement data is distributed.

Hereinafter, as an example of a method for practicing the present invention, an image display system and an image distribution method performed by the image display system will be described.
<Summary of image display system of this embodiment>

FIGS. 1A to 1C are examples of wide angle image, which are a full spherical image 6. FIG. 1A illustrates the full spherical image 6 represented by a three-dimensional sphere CS. FIG. 1B illustrates an image represented by an equirectangular projection (hereinafter referred to as an "equirectangular projection image"). The equirectangular projection image 6 generated by the full spherical camera has a three-dimensional structure in which the equirectangular projection image as illustrated in FIG. 1B is aligned to form a solid sphere CS.
The virtual camera IC corresponds to a viewer's eye position. In FIG. 1A, the eye position is at the center of the full spherical image 6. The viewer may rotate the three axes centered on the X-axis, Y-axis, and Z-axis through the virtual camera IC to display any area of the full spherical image 6 as a predetermined area image Q on a display of the terminal device. By continuously changing the parameters such as the position and size of the predetermined area image (A1 to A3 in FIG. 1B), the image can be displayed as an animation.

FIG. 1C is a diagram illustrating a 3DCG image and a display method thereof. The 3DCG (Three-Dimensional Computer Graphics) image is an image (projected) of a 3D model 95 comprising a three-dimensional point group, a 2D mesh, or a texture, etc., imaged with a virtual camera 330. The terminal device sets the position of the virtual camera 330 and the direction of illumination, and projects the 3D model 95 onto the virtual camera 330 to generate a projected image. Parameters such as the position and angle of view of the virtual camera 330 can be continuously changed to display an animation.

FIG. 2 is a diagram illustrating an outline of the image display system 100 according to the present embodiment. In this embodiment, the image display system 100 using the full spherical image 6 for the advertisement distributed by the third party will be described. A third-party distribution is a system in which advertisements are distributed to media from servers owned by advertisement agencies, not from servers owned by media companies. The third-party distribution server controls the frequency of distributing an advertisement and measures the effectiveness of the advertisement. In FIG. 2, the image distribution apparatus 10 corresponds to a third-party distribution server. In (S1), when the terminal device 30 opens a Web page having an advertisement space 7 (a display frame for displaying an advertisement) by the browser 8 or the like, the DSP 20 (Demand-Side Platform) who wishes to display the advertisement on the terminal device 30 is notified to the terminal device 30. The terminal device 30 requests the advertisement for this DSP 20. In (S2), the DSP 20 requests advertisements to the image distribution apparatus 10. In (S3), the image distribution apparatus 10 stores the advertisement request and generates access information 2 for access by the terminal device 30 to the own machine and transmits the information to the DSP 20. In (S4), the DSP 20 transmits the received access information 2 to the terminal device 30. In (S5), the terminal device 30 requests advertisement data from the image distribution apparatus 10 based on the access information 2. In (S6), the image distribution apparatus 10 transmits the advertisement data (at least one of the full spherical image 6 or the 3DCG image, which is referred to as the content) requested by the access information 2 to the terminal device 30. In (S7), the terminal device 30 receives the content and displays the browser 8 or the like operating in the terminal device 30 as the predetermined area image Q according to the display pattern of the content (a plurality of regions-of-interest A1 to A3 in the full spherical image 6 in FIG. 2). For example, in the full spherical image 6 of FIG. 2, three regions-of-interest A1 to A3 and each of the regions-of-interest A1 to A3 are maintained to have a field angle, and a plurality of displayed images are generated in the order of rotation of region-of-interest A1 (field angle 1) → region-of-interest A2 (field angle 2) → region-of-interest A3 (field angle 3), even if the viewer does not operate the image.

FIGS. 3C and 3D illustrate a transition prohibition area 81 in which the terminal device 30 performs the animation display of the full spherical image 6 or the 3DCG image. FIG. 3A illustrates the point-of-interest P of the 3DCG image, and FIG. 3B illustrates the point-of-interest P of the full spherical image 6. FIGS. 3A and 3B illustrate high points-of-interest P determined using a method such as a prominence detection and object detection. In this way, the image distribution apparatus 10 generates an interest map and generates a display pattern with a viewpoint on the orbit through the region having a high interest.

On the other hand, the display pattern with a viewpoint to the orbit through the region having a high interest may adopt a viewpoint that the viewer does not designate much, which may damage the advertisement effectiveness. Smoothness can also be lost by transiting different viewpoints in a complex orbit.

Therefore, as illustrated in FIGS. 3C and 3D, the person in charge sets the transition prohibition area 81. FIG. 3C illustrates the transition prohibition area 81 of the 3DCG image and FIG. 3D illustrates the transition prohibition area 81 of the full spherical image 6. The transition prohibition area 81 of the 3DCG image means the position at which the virtual camera 330 cannot (or is difficult to take) and the transition prohibition area 81 of the full spherical image 6 means the coordinates (the eye gaze direction) of the full spherical image 6 which the virtual camera IC cannot take. The transition prohibition area 81 is the area in which transition is prohibited. In addition, the coordinates (the eye gaze direction) of the full spherical image 6, which is not completely prohibited and is difficult to be captured by the virtual camera IC, may be used. In this case, the transition is prohibited or restricted in the transition prohibition area 81.

In FIGS. 3C and 3D, shaded portions of the zenith and bottom are the position of the virtual camera 330 or the prohibited area 81 of the coordinates of the full spherical image 6. The terminal device 30 according to this embodiment generates a display pattern so that the terminal device 30 does not enter the transition prohibition area 81.

This restriction of the viewpoint prevents an unnatural transition of the viewpoint. For example, it can be prevented from moving to a viewpoint where the content is viewed upward from above or upward from below. In addition, by constraining the viewpoint after setting the element orbit described later, it is possible to prevent the transition of different viewing directions in a complex orbit. Therefore, the content can be displayed from an appropriate viewpoint, making it easier to enhance the advertising effect.
<Terminology>

A viewpoint is information about which direction to project when the same content is displayed in a planar image.

The plurality of display images generated based on the display pattern is a plurality of images in which the slightly different still images are consecutively arranged, and may be referred to as a movie, an animation, or the like.
<Example of system configuration>

FIG. 4 is an example of a schematic configuration diagram of the image display system 100. The image display system 100 includes the terminal device 30, an image distribution apparatus 10, a DSP 20, an SSP (Supply Side Platform) 50, a partner site Web server 60, and an advertiser Web server 70 capable of communicating through the network N.

The network N is constructed by a LAN constructed in a facility in which the terminal device 30 is installed, a network of providers connecting the LAN to the Internet, and a line provided by the line provider. If the network N has multiple LANs, the network N is called a WAN or the Internet. The network N may be constructed of either wired or wireless, and may be combined with wired and wireless. Further, when the terminal device 30 is directly connected to the public network, the terminal device 30 can be connected to the provider network without using the LAN.

The terminal device 30 is an information processing device that operates as a client terminal according to this embodiment. In the terminal device 30, the browser software or application software having equivalent functions is operated, and a Web page requested by the terminal device 30 is received from the partner site Web server 60 and displayed on the display.

The terminal device 30 is, for example, a PC (Personal Computer), a tablet device, a smartphone, a PDA (Personal Digital Assistant), a game machine, a navigation terminal, a wearable PC, or the like, provided that the Web page can be displayed.
For example, if the printer has a function or display for displaying Web pages, the printer may be used as the terminal device 30. Digital signage may also display Web pages.
Digital signage refers to a system, display device, or information displayed that transmits information using an electronic display device, such as a display, at a location where a person passes through or may exist, such as an outdoor, store, public space, and transportation facility. In this embodiment, a Web application is included in the Web page. A Web application is a software or mechanism used on a browser that operates by coordinating a program in a programming language (e.g., JavaScript., "JavaScript" is a registered trademark.) and a program on a Web server. According to this embodiment, in addition to the Web application, an application that depends on the OS called a native application can be operated in the same manner.

The terminal device 30 may be connected to the network N via an access point of a wired LAN or a wireless LAN access point, or it may be connected to the network N by a switched circuit type communication such as 3G, 4G, or LTE (Long Term Evolution).

The partner site Web server 60 is a server (an ordinary information processing apparatus) that provides information and functions to a client computer (the terminal device 30 according to this embodiment) used by a viewer through a network. The Web page provided by the partner site Web server 60 has an advertisement space 7. The partner site Web server 60 requests the SSP 50 to display the advertisement on the advertisement space 7. This allows the advertisement space 7 to be mapped to the advertisement tag issued by the SSP 50.

The advertiser Web server 70 is a server (an ordinary information processing apparatus) that provides information and functions to a client computer (the terminal device 30 according to this embodiment) used by a viewer through a network. The advertiser asks DSP20 to purchase the advertisement space 7. The image distribution apparatus 10 that holds the advertisement data is registered in the DSP 20. The advertiser also places the advertisement data into the image distribution apparatus 10. The advertisement data contains URLs (linked URLs) of websites (landing pages) for viewers such as advertiser Web server 70 as well as the full spherical image 6 or the 3DCG image.

The advertiser Web server 70 and the partner site Web server 60 communicate with the terminal device 30 using, for example, HTTP or HTTPs communication protocols. In response to a request from the terminal device 30, the advertiser Web server 70 and the partner site Web server 60 transmit screen information to the terminal device 30. Screen information is a program written in HTML, script language, and CSS (cascading style sheet) where the structure of a Web page is predominantly specified by HTML, the script language defines the behavior of a Web page, and CSS identifies the style of a Web page. In this embodiment, it is the script language that reflects the operation of the viewer's Web page on the full spherical image 6. Specifically, the programming language is known as JavaScript ("JavaScript" is the registered trademark) or ECMAScript.

The advertiser Web server 70 and the partner site Web server 60 both record cookies in the browser software of the terminal device 30. The advertiser Web server 70 records both its own cookies and the cookies of the DSP 20, and the partner site Web server 60 records both its own cookies and the cookies of the SSP 50.

The SSP 50 is a mechanism for maximizing profits by selling the advertisement space 7 through a partner site Web server 60 that lists the advertisement space 7. The figure illustrates a network of one or more information processing apparatus although it is described as a single device. The partner site Web server 60 requests the SSP 50 to sell the advertisement space 7. Specifically, the advertisement tag issued by the SSP 50 is described in the advertisement space 7, and when the terminal device 30 displays the Web page of the partner site Web server 60, the advertisement tag requests the SSP 50 to distribute the advertisement (advertisement request). The SSP 50 receives a bid for the advertisement space 7 from the DSP 20 and transmits access information 1 for accessing the accepted DSP 20 to the terminal device 30.

The DSP20 is a mechanism to efficiently and effectively distribute advertisements to advertisers, such as purchasing and distributing ads for advertisement space. The figure illustrates a network of at least one information processing apparatus, although it is described as a single device. That is, the DSP 20 acquires cookies (referred to as SSP cookies to distinguish between SSP cookies) from the SSP 50 and determines the attributes of the viewer based on the information corresponding to the DSP cookies managed by the DSP cookies and the SSP cookies. Then, bid is done on the SSP 50 at a price determined based on the advertisement distribution setup based on the viewer attributes, budget, etc.

The DSP 20 whose bid has been accepted is required to be advertised from the terminal device 30 by the access information 1. The DSP 20 notifies the image distribution apparatus 10 of an advertisement request and acquires the access information 2 for requesting the advertisement data from the image distribution apparatus 10 by the terminal device 30. The advertisement data includes a display program for displaying at least one of the full spherical image 6 or the 3DCG image and an operation history program for acquiring an operation history.

The DSP 20 transmits the access information 2 to the terminal device 30. The DSP20 charges advertisers in accordance with a contract. Between the image distribution apparatus 10 and the advertiser are charged in accordance with a contract. However, depending on the contract, the advertisement will be charged only if it is displayed.

The image distribution apparatus 10 is at least one information processing apparatus which provides the advertisement data including advertising material (the full spherical image 6 or the 3DCG image) suitable for the viewer attribute to the advertisement space 7 purchased by the DSP 20. The image distribution apparatus 10 holds submission materials such as banners, text, and advertisement images. Advertisement data may be merely banners (text, photographs, or pictorial images) or may include scripting in addition to the images. This embodiment includes a display program for rotating the full spherical image 6 in a display order or for reflecting an operation on at least one of the viewer's full spherical image 6 or the 3DCG image in the appearance of the full spherical image 6 , and an operation history program for recording an operation history on at least one of the full spherical image 6 or the 3DCG image. Both are written in the script language.

When the terminal device 30 requests the advertisement data from the image distribution apparatus 10 based on the access information 2, the image distribution apparatus 10 transmits the advertisement data to the terminal device 30. The advertisement data includes full spherical image 6 which automatically rotates and enlarges or shrinks at least one of the full spherical image 6 or the 3DCG image so as to follow the point-of-interest of at least one of the full spherical image 6 or the 3DCG image. The operation history program transmits the operation history for the full spherical image 6 together with the cookie (referred to as the image cookie for distinguishing) of the image distribution apparatus 10 and the image ID to the image distribution apparatus 10. The image cookie is written to the browser software from the image distribution apparatus 10. The operation history is, for example, information about which viewing angle is displayed. Because this operation history is used to determine the point-of-interest, if the operation history is not used to determine the point-of-interest, the operation history may not be transmitted.

The operation history program transmits a fact that it is clicked together with the pattern ID of the display pattern to be described later to the image distribution apparatus 10. The image distribution apparatus 10 defines a plurality of display patterns for one universal image 6. By monitoring the click rate of each display pattern, only the display pattern having a high click rate can be distributed.
<Example hardware configuration>

Next, the hardware configuration of the advertiser Web server 70, the partner site Web server 60, the image distribution apparatus 10, the SSP 50, and the DSP 20 in the image display system according to the present embodiment will be described.

The advertiser Web server 70, the partner site Web server 60, the image distribution apparatus 10, the SSP 50, and the DSP 20 are implemented, for example, by a computer system having a hardware configuration illustrated in FIG. 5. FIG. 5 is an example of a hardware configuration diagram of a computer system 200 according to this embodiment.

The computer system 200 illustrated in FIG. 5 includes an input device 201, a display device 202, an external I/F 203, a RAM (Random Access Memory) 204, a ROM (Read Only Memory) 205, a CPU (Central Processing Unit) 206, a communication I/F 207, and an HDD (Hard Disk Drive) 208, each of which is connected to each other by a bus B.

The input device 201 includes a keyboard, a mouse, a touch panel, or the like, and is used by an operator to input each operation signal. Display 202 includes a display or the like to display the results of processing by computer system 200.

The communication I/F 207 is an interface that connects the computer system 200 to an internal network, the Internet, or the like. This allows the computer system 200 to perform data communication via communication I/F 207.

The HDD 208 is a non-volatile memory device that stores a program and data. Examples of the programs and data to be stored include the operating system (OS), which is the basic software for controlling the entire computer system 200, and application software that provides various functions on the OS. The HDD 208 manages the stored program 208p and data using a predetermined file system and/or DB (database).

The external I/F 203 is an interface with an external apparatus. An external apparatus is a recording medium 203a or the like. This allows the computer system 200 to read and/or write the recording medium 203a via the external I/F 203. The recording medium 203a includes a flexible disk, a CD (Compact Disk), a DVD (Digital Versatile Disk), an SD Memory Card (SD Memory Card), and a Universal Serial Bus memory (USB Memory).

The ROM 205 is a non-volatile semiconductor memory (memory device) that can hold programs and data even when the power is turned off. The ROM 205 stores programs and data such as the BIOS (Basic Input/Output System), OS settings, and network settings that are executed when the computer system 200 is started. The RAM 204 is a volatile semiconductor memory (memory device) that temporarily holds programs and data.

The CPU 206 is an arithmetic device which substantializes a control and function of the entire computer system 200 by reading out a program and data from a memory device such as the ROM 205 and the HDD 208 to the RAM 204 and executing processing.

Each server may be compatible with cloud computing, or it may be a so-called single-unit information processing apparatus. The cloud computing is a form of use in which resources on a network are used without being aware of specific hardware resources.
<<Terminal device>>

FIG. 6 is an example of a hardware configuration diagram of a terminal device 30. The terminal device 30 illustrated in FIG. 6 is assumed to be a tablet device or a smartphone. The terminal device 30 includes a CPU 601, a ROM 602, a RAM 603, an EEPROM 604, a CMOS sensor 605, an acceleration and orientation sensor 606, and a media drive 608.

CPU 601 controls the overall operation of terminal device 30. ROM 602 stores the basic input/output program. RAM 603 is used as the work area of CPU 601. The EEPROM 604 reads or writes data according to the control of the CPU 601. The CMOS sensor 605 captures the subject according to the control of the CPU 601 and obtains image data. The acceleration and orientation sensor 606 is an electromagnetic compass, gyrocompass, acceleration sensor, or the like that detects geomagnetic field.

The media drive 608 controls reading or writing (memory) of data to a media 607, such as a flash memory. The media drive 608 is configured to detachably mount the media 607 for storing data read from or newly written from already recorded data.

The program 604p executed by the CPU 601 is stored in the EEPROM 604. The program 604p is an application software, an OS, or the like for executing various processing in the embodiment. The program 604p may be distributed as stored in media 607 or from a program distribution server.

The CMOS sensor 605 is a charge-coupled device that converts light into an electric charge to electronically convert an image of the subject. The CMOS sensor 605 may be, for example, a CCD (Charge Coupled Device) sensor if the subject can be imaged. The CMOS sensor 605 can read bar codes or two-dimensional bar codes.

The terminal device 30 further includes an RF tag reader/writer 622, an antenna I/F 623 and a vibration actuator 624. RF tag reader/writer 622 communicates according to standards such as, for example, NFC (Near Field Communication).

A vibration actuator 624 is a motor that vibrates the terminal device 30. For example, vibration occurs when an incoming call or mail is received so as to inform the viewer the issue.

The terminal device 30 further includes a voice input unit 609, a voice output unit 610, an antenna 611, a communication unit 612, a wireless LAN communication unit 613, a short range wireless communication antenna 614, a short range wireless communication unit 615, a display 616, a touch panel 617, and a bus line 619.

The voice input unit 609 converts the voice into a voice signal. The voice output unit 610 converts the speech signal to speech. The communication unit 612 communicates with the nearest base station apparatus by a radio communication signal using the antenna 611. The wireless LAN communication unit 613 performs wireless LAN communication conforming to the IEEE 802.11 standard.

The short-range wireless communication unit 615 is a communication device that uses the antenna 614 for short-range wireless communication and complies with, for example, the communication standard of Bluetooth ("Bluetooth" is a registered trademark) or Bluetooth Low Energy ("Bluetooth Low Energy" is a registered trademark).

A display 616 is a liquid crystal or organic EL for displaying an image of a subject, various icons, or the like. A touch panel 617 is mounted on the display 616 and includes a pressure-sensitive or electrostatic panel that detects a touch position on the display 616 by touching it with a finger, a touch pen or the like. A bus line 619 is an address bus, data bus, or the like for electrically connecting the above portions.

The terminal device 30 also includes a dedicated battery 618, which may be driven by either battery 618 or a commercial power supply. The voice input unit 609 includes a microphone for inputting voice. An audio output 610 includes a speaker that outputs audio.
<Full spherical image>

The full spherical image 6 captured by the full spherical camera 9 will be described with reference to FIGS. 7 to 11. FIG. 7 is a diagram illustrating the use of the full spherical camera 9. The full spherical camera 9 is an imaging device in which a user holds an image of a subject around the user's hand, as illustrated in FIG. 7. The full spherical camera 9 has a structure in which the rear surfaces of the two image elements are opposed to each other. Each camera acquires two hemispherical images by imaging a subject around the user.

Next, an outline of the process from an image captured by the full spherical camera 9 to the creation of the full spherical image 6 will be described with reference to FIGS. 8A to 9B. FIG. 8A illustrates a hemispherical image (front side) captured by the full spherical camera 9, FIG. 8B illustrates a hemispherical image (rear side) captured by the full spherical camera 9, and FIG. 8C illustrates an image (equirectangular projection) represented by the equidistant cylindrical diagram. FIG. 9A is a conceptual diagram illustrating the sphere covered by an equirectangular projection image, and FIG. 9B is a diagram illustrating the full spherical image 6.

As illustrated in FIG. 8A, the image obtained by the full spherical camera 9 is a hemispherical image (front) curved by the fish-eye lens. As illustrated in FIG. 8B, the image obtained by the full spherical camera 9 is a hemispherical image (rear) curved by the fish-eye lens. The hemispherical image (front) and the 180-degree inverted hemispherical image (rear) are then synthesized by the full spherical camera 9 and the equirectangular projection image is generated as illustrated in FIG. 8C. This is full spherical image.

Then, the OpenGL ES (Open Graphics Library for Embedded Systems) is utilized so that the equirectangular projection image is attached onto the spherical surface as illustrated in FIG. 9A, and thefull spherical image 6 as illustrated in FIG. 9B is displayed. Thus, the full spherical image 6 is represented as an image in which the equirectangular projection image is oriented toward the center of the sphere. OpenGL ES is a graphical library used to visualize data in 2D (2-Dimensions) and 3D (3-Dimensions). The full spherical image 6 may be a still image or a video image.

Advertisement data distributed by the image distribution apparatus 10 includes the full spherical image 6. The full spherical image 6 is curved because it is an image that is pasted over to cover a spherical surface. Therefore, when viewed from a human eye, it has a feeling of discomfort. Therefore, the terminal device 30 displays a predefined area T of a part of the full spherical image 6 as a plane image with a low curvature so as not to cause a discomfort to humans. The predetermined region T is indicated by the coordinates (X, Y, Z) in the three-dimensional virtual space. On the other hand, because the display 616 is a two-dimensional plane, the terminal device 30 cannot be displayed while the predetermined area T is still present. Thus, the terminal device 30 obtains a predetermined area T through a perspective projection transformation that projects a three-dimensional object to a two-dimensional plane using a 3D computer graphic technique.

FIG. 10 is an example of a diagram illustrating a viewer's eye gaze direction. Because the full spherical image 6 has three-dimensional coordinates, the eye gaze direction is specified by information specifying the coordinates of the sphere, such as the three-dimensional coordinates or latitude and longitude. In this embodiment, the central point CP of the predetermined region T is in the eye gaze direction. The center point CP of the predetermined region T is specified by the spherical coordinate system (θ, φ). θ is longitude and φ is latitude.

The viewer can change the viewing direction by operation, but assuming that the virtual camera IC does not move in parallel, the virtual camera IC can rotate rigidly in three ways: roll (rotation about the Z axis), yaw (rotation about the Y axis), and pitch (rotation about the X axis). Any of these three rotations cause a change in the direction of vision. For example, the yaw angle changes when the viewer rotates the full spherical image 6 horizontally, the pitch angle changes when the viewer rotates it vertically, and the roll angle changes when the viewer rotates the full spherical image 6 about the center of the display 616. In this embodiment, the operation of the Web page of the viewer is reflected in the line-of-sight direction (roll angle, yaw angle, pitch angle, etc.). How to be reflected shall be pre-stated in the display program. The predefined region T can also be enlarged or reduced.

The viewer can change the eye gaze direction (rotate the image data) by flicking up, down, left, and right against the advertisement space 7.

A relationship between the predetermined area information and the image of the predetermined area T will be described with reference to FIG. 11. FIG. 11 is a diagram illustrating an example of the relationship between the predetermined area information and the image of the predetermined area T. As illustrated in FIG. 11, rH represents Horizontal Radian (longitude θ), rV represents Vertical Radian (latitude φ), and α represents the field angle (angle). That is, the attitude of the virtual camera IC is changed so that the viewpoint of the virtual camera IC represented by the imaging direction (θ, φ) is the center point CP of the predetermined area T that is the imaging area of the virtual camera IC. The predetermined area image Q is an image of the predetermined area T in the full spherical image 6. The f is the distance from the virtual camera IC to the center point CP. The L is the distance between any vertex and center point CP in the given region T (2L is diagonal). In FIG. 11, the trigonometric function represented by the following equation (A) is generally true:

Lf=tan(α/2) ・・・(A)
<For viewing projected images using 3D model>

3DCG images can be viewed differently (viewpoint is changed) and an interactive advertisement can be substantialized.
<For 3DCG>

3DCG is a method of creating images with a sense of depth (three-dimensional feeling) by converting virtual three-dimensional objects in three-dimensional space into two-dimensional information. The 3DCG is created by modeling, world transformation, view transformation, projection transformation, lighting, and other processes. Modeling and world transformation are performed on dedicated software, and view transformation, projection transformation, and lighting are performed on the display program.

Modeling refers to the author defining the shape of an object in a local coordinate system, for example, creating a shape in a polygon. The world transformation is the transformation of an object's coordinates (translation and rotation) into a world coordinate system that represents the three-dimensional space to be drawn (placement of the object in the world coordinate system). The view transformation is equivalent to defining the viewer's viewpoint in the world coordinate system. It sets the camera's coordinates, camera's viewpoint, and camera's upward direction, and sets the object in a specific direction. Perspectives can be determined by the viewer's operations or can be changed automatically. A projection transformation is a projection of an object in three-dimensional space onto a two-dimensional plane. Lighting is a process that gives light and darkness according to the position of the light source.

FIGS. 12A and 12B are a schematic representation of the 3D model and the projection conversion. In FIG. 12A, a virtual camera 330 is arranged in the 3D model 95 with the optical axis oriented. The 3D model 95 of FIG. 12A is a car, but it is only an example. The virtual camera 330 can vary in angle in a horizontal and vertical direction 360 degrees about the 3D model 95. The viewer can view the 3D Model 95 in any direction. Alternatively, the virtual camera 330 may rotate the 3D model 95 on a turntable 96 while remaining fixed.

FIG. 12B schematically illustrates a projection transformation of the 3D model into a virtual camera 330. A 3D model 95 formed by polygons or three-dimensional point groups is projected onto a plane. An image projected onto a plane is called a projection image. Changing the position of the virtual camera 330 allows 3D model 95 to be projected depending on the position of the (view transform) camera. Because the position of the virtual camera 330 can be changed to any 360 degrees around the 3D model 95, the viewer can view the 3D model 95 from various angles within a limited advertisement space 7.

FIGS. 13A to 13F is an example of a 3D model 95 displayed from different perspectives. The projection images of the 3D model 95 captured from various angles as illustrated in FIGS. 13A to 13C are illustrated in FIGS. 13D to 13F. These different projected images are displayed one after another in the ad frame 7 according to the display pattern. In addition, the viewer may rotate the 3D model 95 displayed on the advertisement space 7 by his/her own operation and view one 3D model 95 from various angles in one advertisement space 7.
<Information used by Web pages>

FIG. 14A schematically illustrate information used by the Web page provided by the partner site Web server 60. FIG. 14A is a web page provided by the partner site Web server 60. A Web page provided by the partner site Web server 60 has one or more advertisement space 7. The browser of the terminal device 30 holds the SSP cookie and the DSP cookie beforehand. The URL of the SSP 50 and the advertisement space ID are associated with the advertisement space 7. The browser of the terminal device 30 (An advertisement space detection unit 36, which will be described later) transmits the SSP cookie and the request contents to the SSP 50. In addition, the access information 1 including the URL of the DSP 20 that accepted the advertising time limit 7 and the successful bid ID are transmitted from the SSP 50 to the terminal device 30.

FIG. 14B is a diagram illustrating an operation based on access information 1. The terminal device 30 transmits the bid acceptance ID and DSP cookie to DSP 20 based on the URL of DSP 20. The DSP 20 identifies the advertisement request with the bid acceptance ID and acquires the URL of the image distribution apparatus 10 and the access information 2 including the advertisement opportunity ID from the image distribution apparatus 10. The terminal device 30 acquires the access information 2 and the advertisement opportunity ID from the DSP 20.

FIG. 14C is a diagram illustrating an operation based on the access information 2. The terminal device 30 transmits an advertisement opportunity ID to the image distribution apparatus 10 based on the URL of the image distribution apparatus 10. The image distribution apparatus 10 generates the advertisement data based on the advertisement request identified by the advertisement opportunity ID and transmits the data together with the image cookie, the ID of the advertisement space, and the display pattern to the terminal device 30. In this way, the full spherical image 6 is displayed in the ad frame 7 of the Web page.
<Function>

FIG. 15 is an example of a functional block diagram illustrating functions of the terminal device 30, an advertiser Web server 70, a partner site Web server 60, an image distribution apparatus 10, an SSP 50, and a DSP 20 in a block shape.
<<Advertiser Web server, partner site Web server>>

The advertiser Web server 70 and the partner site Web server 60 include a Web page providing unit 71. Each of these functions of the advertiser Web server 70 and partner site Web server 60 is a function or means substantialized by CPU 206 executing program 208p deployed from HDD 208 to RAM 204.

For example, the Web page providing unit 71 performs general HTTP communication and constructs a Web page in response to a Web page request from the terminal device 30 and transmits the Web page to the terminal device 30. If necessary for processing, request processing from the application server, place the processing result on the Web page, and send it.

The Web page of the advertiser Web server 70 includes the advertiser's cookie and the DSP cookie. The advertiser Web server 70 asks the DSP 20 to deliver the advertisement. This makes it possible to advertise in return. The Web page of the partner site Web server 60 includes the partner site cookie and the SSP cookie. This is because the partner site Web server 60 requests the SSP 50 to sell the advertisement space 7.
<<SSP>>

The SSP 50 includes an advertisement request unit 51, an advertisement request acquisition unit 52, an access information transmission unit 53, and a bid acceptance unit 54. Each of these functions of the SSP 50 is a function or means substantialized by CPU 206 executing a program 208p deployed from the HDD 208 to the RAM 204.

The advertisement request acquisition unit 52 acquires the advertisement request requested together with the SSP cookie by executing the advertisement tag corresponding to the advertisement space 7 of the terminal device 30. The advertisement request acquisition unit 52 sends an advertisement request including an SSP cookie to the advertisement request unit 51.

The advertisement request unit 51 sends an advertisement request to the DSP 20. Advertisement requests may include SSP cookies, domain of the partner site Web server 60, media limit ID, 7-size media limits, advertising formats, browser types, OS types, and the like.

The bid acceptance unit 54 receives a bid from DSP 20 and essentially sells the 7 ad space to DSP 20 with the highest bid amount (no bid depending on the advertiser's products or services). The bid acceptance unit 54 generates the bid acceptance ID and notifies DSP 20.

The access information transmission unit 53 generates the access information 1 for requesting an advertisement against the accepted DSP 20 and transmits the information together with the successful bid ID to the terminal device 30. Access information 1 includes the URL (IP address) of DSP 20.
<<DSP>>

The DSP 20 includ unit 23, an advertisement request reception unit 24, and an image requesting unit 25. Each of these functions of the DSP 20 is a function or means substantialized by CPU 206 executing the program 208p deployed from the HDD 208 to the RAM 204.

A cookie information DB 291 and a distribution setting DB 292 are constructed in the memory unit 29. The memory unit 29 is substantialized by the HDD 208 or the RAM 204 illustrated in FIG. 5.

Table 1 schematically illustrates the information stored in cookie information DB 291. The cookie information DB 291 maps the DSP cookie to the SSP cookie and registers the visitor's visit domain. The mapping of DSP cookies to SSP cookies is possible by a technique called cookie sinks. A visit domain can also be obtained by visitors visiting a Web site where DSP 20 has attached a tag (behavioral monitoring tag). Thus, the DSP 20 can identify the DSP cookie from the SSP cookie and determine which Web sites the visitor is interested in.

Table 2 schematically illustrates the information stored in the distribution setting DB 292. The distribution setting DB 292 registers the advertiser ID, the advertisement target attribute (the attribute of the preferred target), and the non-advertisement target attribute (the attribute of the unwanted target). The advertiser ID is an identifying information to identify or identify the advertiser from which the wide-angle image is provided. Accordingly, the DSP 20 or the image distribution apparatus 10 can determine whether to transmit a wide angle image by referring to the attributes of a preferred subject or the attributes of an undesirable subject set for a wide angle image provider.

The ID stands for Identification and means identifier or identification information. An ID is a name, code, character string, numeric value, or one or more of these combinations used to uniquely distinguish a particular object from multiple objects. The same applies to the following IDs. Advertising Attributes are those attributes of the viewer who the advertiser wishes to deliver the advertisement, and Non-Advertising Attributes are those of the viewer who the advertiser does not wish to deliver the advertisement. The DSP 20 quantifies the attributes of the viewer by comparing the attributes of the viewer determined from cookie information DB 291, etc. and the information of the distribution setting DB 292, and determines the bid amount. In determining the bid amount, various information such as time zone, day of the week, and area of the viewer may be considered.
(DSP Functions)

The request reception unit 21 receives an advertisement request from the SSP 50. The request information provides basic information such as the media limit ID of the terminal device 30.

The bid determination unit 22 evaluates the advertisement request received by the request reception unit 21 by referring to the cookie information DB 291 and the distribution setting DB 292 and determines the bid amount (sometimes no bid). As described above, the bid amount is determined by identifying the DSP cookie from the SSP cookie and how much attribute information stored in association with the DSP cookie matches the advertiser's request stored in the distribution setting DB 292.

A bid unit 23 bids on SSP 50 in the determined bid amount. If the bid is accepted, the winning bidder shall obtain the bid acceptance ID and match it with the advertisement request.

The advertisement request reception unit 24 acquires the advertisement request (the bid acceptance ID and the DSP cookie) from the terminal device 30 based on the access information 1 . The advertisement request reception unit 24 identifies the advertisement request by the bid acceptance ID. The DSP cookie may not be present and is used when the SSP cookie and the DSP cookie are not associated.

The image requesting unit 25 requests the image distribution apparatus 10 to the access information 2 along with the DSP cookie, the advertiser ID, and the request contents. The advertiser is identified by the advertiser ID. The media space ID in the request contents identifies the advertisement space 7. Further, the attributes of the viewer may be transmitted to the image distribution apparatus 10. Accordingly, the image distribution apparatus 10 can distribute the full spherical image 6 suitable for the viewer.
<<Terminal device>>

The terminal device 30 includes a Web page acquisition unit 31, a Web page analysis unit 32, a Web page display unit 33, an operation reception unit 34, and a script execution unit 35. Each of these functions of the terminal device 30 is a function or means substantialized by CPU 601 executing the program 604p expanded from EEPROM 604 to RAM 603.

The operation reception unit 34 receives various operations on the terminal device 30. Specifically, the operation of the browser executed by the terminal device 30 is received. The operation reception unit 34 receives the operation for the Web page. The image operation reception unit 43 accepts the operation of the image data based on the script.

The Web page acquisition unit 31 acquires the Web page from the advertiser Web server 70 and the partner site web server 60 in communication with the advertiser Web server 70 and the partner site web server 60 by operation of the viewer or by operation of a script.

The Web page analysis unit 32 reads the HTML included in the screen information sequentially from the top to the bottom and analyzes the structure of the text or the image data included in the HTML. It also detects the association of HTML with the text or image data described in CSS and maps the style of the text or image data contained in HTML. The Web page analysis unit 32 detects the script tag from the HTML and extracts the script written in the script language. The Web page analysis unit 32 transmits HTML and CSS to the Web page display unit 33 and transmits scripts to the script execution unit 35.

The Web page display unit 33 displays the Web page on the display 616 in the order in which the analysis is completed from the beginning of the HTML. It also updates the Web page according to the operation of the Web page.

The script execution unit 35 executes the script extracted by the Web page analysis unit 32. The specific content of the script varies depending on the Web page. According to this embodiment, as a script, the advertisement tag associated with the advertisement space 7, the access information 1 acquired from the SSP 50, the access information 2 acquired from the DSP 20, and the advertisement data distributed from the image distribution apparatus 10 are detected. The script execution unit 35 is substantialized by executing the program 604p by the CPU 601 illustrated in FIG. 6. Further, each function or means of the script execution unit 35 is substantialized by executing the script by the CPU 601.

The advertisement space detection unit 36 is a function in which the terminal device 30 executes the advertisement tag associated with the advertisement space 7 included in the screen information acquired by the Web page acquisition unit 31. The advertisement space detection unit 36 transmits an advertisement request to the SSP 50 together with the SSP cookie based on the URL corresponding to the advertisement space 7.

The access information acquisition unit 37 acquires the access information 1 from the SSP 50. Access information 1 is written in a script. The advertisement request unit 38 accesses the DSP 20 based on the URL included in the access information 1 and requests an advertisement together with the bid acceptance ID and the DSP cookie. The advertisement request unit 38 acquires the access information 2 and the advertisement opportunity ID in response to this request. The access information 2 is also written in script.

The advertisement acquisition unit 39 specifies the advertisement opportunity ID based on the URL of the image distribution apparatus 10 included in the access information 2 and requests the advertisement data to the image distribution apparatus 10. Because the image distribution apparatus 10 generates the advertisement data, the advertisement acquisition unit 39 acquires the advertisement data. The dvertisement data of this embodiment include content (full spherical image 6 and/or 3DCG images), display patterns, display programs and operation history programs. The operation history program causes the operation history to be transmitted to the image distribution apparatus 10. The display program rotates the full spherical image 6 or the 3DCG image in the display pattern and reflects the manipulation of the full spherical image 6 or the 3DCG image in the appearance of the full spherical image 6 or the 3DCG image. The operation history program and the display program are also described in a script language and are executed by the script execution unit 35.

The advertisement display unit 40 displays the advertisement data acquired from the image distribution apparatus 10. The advertisement display unit 40 is implemented mainly in a display program, and displays a predetermined area image generated from the full spherical image 6 or a projected image generated from the 3DCG image in the advertisement space 7, and rotates the image automatically.

The display image generation unit 44 generates an image (a predetermined area image or a projected image) to be displayed on the advertisement space 7 by the display pattern acquired from the image distribution apparatus 10 from the content. That is, the full spherical image 6 is rotated or the position of the virtual camera 330 in the 3DCG image is changed one after another to generate a plurality of displayed images (animations) that are displayed on the advertisement space 7 from one content. The advertisement display unit 40 displays a plurality of display images generated by the display image generation unit 44.

When the advertisement space 7 includes the coordinates of a pointing device (a finger, a bar-like member, or a dedicated pen, and a touch panel when the display 616 is a touch panel, a mouse pointer when a mouse is used, or a mouse pointer when a trackball is used), the image manipulation reception unit 43 receives the operation on the image data in preference to the operation on the Web page. Thus, if the viewer flicks or sweeps on the advertisement space 7, the image data can be changed rather than the Web page.

When the advertisement data is acquired from the image distribution apparatus 10, the terminal device 30 acquires the image cookie of the image distribution apparatus 10 and stores the cookie in the storage unit 49. When the advertisement acquisition unit 39 transmits the image cookie to the image distribution apparatus 10, the image distribution apparatus 10 can correspond the DSP cookie to the image cookie.

The operation history recording unit 41 is implemented mainly by the operation history program and records the operation information for the full spherical image 6 or the 3DCG as the operation history. Details of the operation history will be described in the image distribution apparatus 10. The operation history transmitting unit 42 is implemented mainly by the operation history program and transmits the operation history for the full spherical image 6 or the 3DCG displayed on the advertisement space 7 to the image distribution apparatus 10 together with the image cookie and the image ID (identifying the full spherical image 6 ). When the image is clicked, this issue is further transmitted to the image distribution apparatus 10 together with the display pattern.
<<Image distribution apparatus>>

The image distribution apparatus 10 includes an image information response unit 11, an advertisement distribution unit 12, a point-of -interest determination unit 13, an operation history acquisition unit 14, a content reception unit 15, a display pattern generation unit 16, and a constraint unit 17. Each of these functions of the image distribution apparatus 10 is a function or means substantialized by the CPU 206 executing the program 208p deployed from the HDD 208 to the RAM 204.

The image distribution apparatus 10 includes the memory unit 19 substantialized by the HDD 208 or the RAM 204 illustrated in FIG. 5. A distribution history DB 191, an operation history DB 192, a cookie correspondence DB 193, an advertisement image DB 194, a pattern DB 195, and a display program 196 are constructed in the memory unit 19.

Table 3 schematically illustrates the information stored in the operation history DB 192. The operation history DB 192 records history of operations by each viewer for the full spherical image 6. In the operation history DB 192, items of the image ID, browsing time, image cookie, and viewing image angles 1 to 3 are recorded. The image ID is information for identifying the full spherical image 6. The image cookie provides information to determine the identity of the terminal device 30 or the viewer of the terminal device 30 operating the terminal device 30 (an example of device identification information). The viewing field angles 1 to 3 are field angles when viewing the full spherical image 6 by the viewer. A viewing time, information of whether the image is enlarged, an enlarged field angle, and information of whether the image is reduced are stored in each viewing field angle 1 to 3. Thus, for each image, the field angle that the viewer is thought to be interested in is recorded. The viewing field angles 1 to 3 are field angles where the viewer stays for a predetermined period of time (for example, one second) or longer without rotating the full spherical image 6. The operation history recording unit 41 of the terminal device 30 records the three image angles corresponding to the longest three stationary times. One or at least four field angles may be used. Similarly, the operation history recording unit 41 records whether the viewer has enlarged or reduced at the viewing field angles of 1 to 3. The image distribution apparatus 10 may use such operation history as a point to interest P. In addition, image cookies associated with image IDs allow for retargeting advertisement. The field angle at the time of clicking may be recorded as the operation history.

Table 4 schematically illustrates the information stored in the advertisement image DB 194. The advertisement image DB194 contains information about the full spherical image 6 or 3DCG to be advertised. The advertisement image DB 194 is registered in association with the advertiser ID, image ID, priority, and target attributes. Because the DSP 20 determines the advertiser, among the images that the advertiser wishes to distribute, the image to be distributed is determined based on at least one of the degree of agreement or priority between the viewer attribute and the target attribute in Table 4. At distribution, the initial position of the full spherical image 6 is specified (when the display pattern is also distributed, the display pattern takes precedence). The image ID is associated with at least one of the full spherical image 6 or 3DCG image. In the case of the retargeting advertisement, an image associated with an image cookie is distributed by the operation history DB 192.

Table 5 schematically illustrates the information stored in the cookie correspondence DB 193. The cookie correspondence DB 193 is associated with a DSP cookie and an image cookie. The DSP cookie is notified by the DSP 20 and the image cookie is notified by the terminal device 30. When the two are associated, the image distribution apparatus 10 can determine the viewer using only the DSP cookie.

Table 6 schematically illustrates the information registered in the pattern DB 195. In the pattern DB195, a display pattern for displaying the point-of-interest P of the full spherical image 6 or the 3DCG image is registered. A pattern ID is associated with the image ID, and a display pattern is associated with the pattern ID. The display pattern is one or more (preferably a plurality of numbers) parameters set in time series. One parameter refers to the location (i.e., the viewer's viewpoint) of the virtual camera 330 in the central point CP or 3DCG image of the full spherical image 6. Each parameter is determined not to include a transition prohibition area 81 constrained by the constraint section 17. Details of the parameters are described later. The number of clicks is the number of clicks the viewer has made for each of these display pattern. That is, the image distribution apparatus 10 displays the same full spherical image 6 or 3DCG image 6 or 3DCG image 3 in the same display pattern on a plurality of terminal devices 30 and records that the terminal device 30 clicked. In this way, the distribution of low-clicking patterns is gradually avoided. It is preferable to record the number of clicks according to the attributes (sex, age, family composition, etc.) of the viewer. This allows the viewer attributes to be displayed in a display pattern with more clicks.

Table 7 schematically illustrates the information registered in the distribution history DB191. In the distribution history DB 191, the image IDS and display patterns of the full spherical image 6 or 3DCG image distributed corresponding to the image cookie are registered. When the image cookie is registered in the terminal device 30 and the image distribution apparatus 10 transmits the advertisement data, the image distribution apparatus 10 can acquire the image cookie from the terminal device 30. The image distribution apparatus 10 may avoid distributing the same full spherical image 6 or 3DCG image to the same viewer or distributing the same full spherical image 6 or 3DCG image in the same display pattern. Conversely, the retargeting distribution is possible.
(Functions of Image Distribution Apparatus)

The image information response unit 11 gives the advertisement opportunity ID to the DSP cookie, the advertiser ID, and the request content (mainly the advertisement space ID) obtained from the DSP 20 and returns the same to the DSP 20 together with the access information 2. The advertisement opportunity ID identifies the distribution opportunity of the advertisement for each of the advertisement space 7 of the terminal device 30.

The advertisement distribution unit 12 determines the full spherical image 6 and the display pattern to be distributed when the advertisement data is requested together with the advertisement opportunity ID from the terminal device 30. First, the full spherical image 6 or the 3DCG image associated with the advertiser ID obtained from the DSP 20 is determined from the advertisement image DB 194. Preferably, the attributes of the viewer are considered. The viewer attributes may be notified from the DSP 20, or the image distribution apparatus 10 may be stored in association with the image cookie. Next, the advertisement distribution unit 12 determines the display pattern of the full spherical image 6 with reference to the pattern DB 195. For example, the pattern having the highest number of clicks is determined, or the pattern having the highest number of clicks is arbitrarily determined from the pattern having the highest number of clicks. It is preferable that the viewer attributes be taken into account when determining the pattern.

The operation history acquisition unit 14 acquires the operation history together with the image cookie and the image ID from the terminal device 30 and sets the operation history DB 192. In addition, there are two types of operation histories, one for determining the point-of-interest and the other for updating the number of clicks (clicked).

The operation history recording unit 41 of the terminal device 30 may be provided by the image distribution apparatus 10. Because the operation history recording unit 41 requires the operation contents in a time series, it may be difficult for the image distribution apparatus 10 to acquire the content depending on the communication band or the like. On the other hand, the operation history recording unit 41 of the terminal device 30 can transmit only the final operation history to the image distribution apparatus 10, thereby reducing the communication load.

The point-of-interest determination unit 13 determines the point-of-interest P of the full spherical image 6. Various methods for determining the point-of-interest P, such as estimating the point-of-interest by image processing, etc., are considered. The details will be described later. The point-of-interest P determined by the point-of -interest determination unit 13 is used for the display pattern.

The content reception unit 15 receives the content for distributing by the image distribution apparatus 10 (that is, accepts the input of the advertisement). The content reception unit 15 accepts the setting of the transition prohibition area 81 by the person in charge. A method of receiving an image by the content reception unit 15 will be described with reference to FIG. 16.

The display pattern generation unit 16 generates a display pattern that passes through a point P determined by the point-of-interest determination unit 13 under a constraint by the constraint unit 17. Details are given below.
<Accept content>

FIG. 16 is an example of a functional block diagram illustrating functions of the content reception unit 15 and the image registration terminal 90 in a block shape. The content reception unit 15 includes a communication unit 151, a screen information creation unit 152, and an image storage unit 153.

The communication unit 151 communicates with the image registration terminal 90 using,, for example, a communication protocol of HTTP or HTTPs. In the present embodiment, the screen information of the content registration screen for registering the content in response to a request from the image registration terminal 90 is transmitted to the image registration terminal 90. The contents are received from the image registration terminal 90.

The screen information creation unit 152 creates screen information of the content registration screen illustrated in FIG. 16. The image storage unit 153 stores the content received by the communication unit 151 in the advertisement image DB 194.

The image registration terminal 90 is, for example, a terminal operated by a person in charge of an advertiser or an agent (hereinafter, simply referred to as the "person in charge"), and is used to register at least one of the full spherical image 6 or the 3DCG image illustrated in FIGS. 3A to 3D as content in the image distribution apparatus 10. The image registration terminal 90 includes a communication unit 91, a display control unit 92, and an operation reception unit 93. The hardware configuration of the image registration terminal 90 may be the same as that of FIG. 6.

The communication unit 91 receives the screen information of the content registration screen from the content reception unit 15 and transmits the content input by the person in charge to the content reception unit 15.

The display control unit 92 displays the screen information of the content registration screen received by the communication unit 91 on the display of the image registration terminal 90. The operation reception unit 93 receives the operation by the person in charge of the content registration screen (full spherical image 6 and 3DCG, the transition prohibition area 81, and the like).

FIGS. 17A and 17B illustrates an example of the content registration screen 220 displayed on the display of the image registration terminal 90. As illustrated in FIG. 17A, the content registration screen 220 includes a work area 221, a file selection field 222, a setting field 223, a cancel button 224, and a new creation button 225.

The file selection field 222 is a field for a person in charge to select and set the file of the full spherical image 6 or 3DCG image. The full spherical image 6 or 3DCG image selected in the file selection field 222 is displayed in the work area 221.

The full spherical image 6 displayed in the work area 221 is displayed, for example, as a equirectangular projection image. The person can set the transition prohibition area 81 with a pointing device such as a mouse or finger. The transition prohibition area 81 of the full spherical image 6 is set to a range in which the central point CP in the eye gaze direction does not take or is difficult to take. In FIG. 17A, a region in which the absolute value of the latitude exceeds the predetermined value is set as the transition prohibition area 81. The person can set any region to be the transition prohibition area 81.

FIG. 17B illustrates an example of the content registration screen 220 in which the 3DCG image is displayed in the work area 221. The 3DCG image is displayed in the work area 221. The person in charge sets the transition prohibition area 81 with a pointing device such as a mouse or finger. The transition prohibition area 81 of the 3DCG image is set to a range of locations on the sphere where the virtual camera 330 does not take or is difficult to take. In FIG. 17B, a region in which the absolute value of latitude exceeds the predetermined value is set as the transition prohibition area 81. The person can set any region to be the transition prohibition area 81.

In this embodiment, for convenience, the optical axis of the virtual camera 330 is oriented toward the center of the sphere, but the direction of the optical axis can be set to any direction.

In the setting field 223 with numerical values, the transition prohibition area 81 can be set with numerical values described below instead of the transition prohibition area 81 visually. For example, "|zt|<a" specifies the range of the axial direction numerically, or sets other parameters such as the speed of parameter change, field angle, etc. for the transition orbit, which will be described later. In addition to the numeric value, the user may select the area where transition is prohibited using a mouse, stylus, or the like in the work area.
<Determination of point-of-interest>

Next, we will explain how to determine the point-of-interest in several ways. <How to determine the point-of-interest from the feature quantity>>
FIG. 18 is an example of a flowchart illustrating a procedure in which the point-of-interest determination unit 13 determines the point-of-interest. In Step S11, the point-of-interest determination unit 13 defines a polyhedron having a center common to the unit sphere and performs a perspective projection transformation with the normal direction of each surface as the eye gaze direction to obtain a partial image. FIG. 19A illustrates an example where the octahedron is defined as the projection plane of the full spherical image, and FIG. 19B illustrates an example where the dodecahedron is defined as the projection plane of the full spherical image.

In step S12, the point-of-interest determination unit 13 extracts a predetermined feature amount from each partial image obtained in step S11. Because the input image is divided by the above-described method and the feature quantity is calculated from the less distorted partial image, it is possible to robustly process the wide angle image exceeding 180 degrees. As the feature quantity, the color, edge, conspicuity, object position/label, and the like may be used.

In Step S13, the point-of-interest determination unit 13 calculates the importance for each position (for each pixel) of the input image from the feature amount extracted from each partial image using a predetermined regression model. Although the details are omitted, the degree of importance and characterization by location are expressed by functions using the regression. Use training data in advance to determine the degree of importance g, which is the output (teacher data), by an appropriate method, and calculate the relationship between the feature quantity and the degree of importance g.

Subsequent to Step S14, the likelihood distribution of the point-of-interest is calculated based on the distribution of the degree of importance calculated in Step S13 under the design concept that the point-of-interest determination unit 13 has the user's point-of-interest in the direction of high importance. For example, after defining the region R passing through the viewpoint on the unit image plane, the sum value obtained by adding the importance of each position in the region R can be calculated as the likelihood of the region-of-interest.

In step S15, the point-of-interest determination unit 13 calculates the point-of-interest based on the region-of-interest likelihood distribution. In this embodiment, for example, the position corresponding to the imaging direction corresponding to the maximum likelihood value of the region-of-interest is calculated as the point-of-interest.

In this way, several points-of-interest can be calculated. The point-of-interest determination unit 13 specifies a predetermined number of points-of-interest or a point-of-interest exceeding the threshold value among the upper N points-of-interest. The point-of-interest is transmitted to the display pattern generation unit 16.
<Determination of a person as the point-of-interest>

The point-of-interests can also be determined in light of viewer preferences. The subject of interest is considered to be a person. Face recognition is known as a method for detecting the person. Deep learning, various other machine learning, and pattern matching may be employed for facial recognition. The point-of-interest determination unit 13 detects the center of the face as the point-of-interest. If there are a large number of faces, the number of faces can be reduced to a fixed number, regarded as a single face with a close distance.
<Focus on smiles>

A smile on the face of a person may be considered as a subject of interest. As a method of detecting the smile, in addition to the face recognition described above, a known method of pattern matching using a pre-retained pattern of smiles can be used. If there are many faces, only smiling faces may be the point-of-interest. In addition to the smiling face, facial expressions such as sadness, anger, or embarrassment may also be determined as the point-of-interest.
<Determination of a child as the point-of-interest>

If a child is present in an adult, a person may focus on the child. Alternatively, when an adult is present in a large number of children, people may focus on the adult. A face position can be used as a method for detecting the child or adult. If the face identified by the face recognition above is lower than the position of the other face, it is presumed to be the child, and if the face identified by the face recognition is higher than the position of the other face, it is presumed to be the adult.
<Determination an arbitrary subject as the point-of-interest>

The advertiser may be aware of the point-of-interest that the advertiser wants viewers to see. Accordingly, the point-of-interest determination unit 13 detects the subject designated by the advertiser and makes it the point-of-interest. For example, the products and services of the advertiser, such as automobiles, furniture, and food, are detected using the pattern recognition to make it the point-of-interest.
<<Use of the operation history>>

If the full spherical image or 3DCG image is clicked, the clicked eye gaze direction (coordinate) in the full spherical image or the position of the virtual camera 330 may be the point-of-interest.
<Generation of display pattern>

Hereinafter, a method for generating a display pattern in which the transition prohibition area 81 is considered will be described with reference to FIGS. 20 to 28.

FIG. 20 is a flowchart illustrating an example of the overall flow in which the display pattern generation unit 16 generates the display pattern.

In step S101, the display pattern generation unit 16 acquires the full spherical image or 3DCG image from the advertisement image DB 194.

In step S102, as described in FIGS. 18 and 19, the point-of-interest determination unit 13 determines the point-of-interest (the area displayed in the advertisement space including the point-of-interest is referred to as the region-of-interest). Meanwhile, a method such as NPL 1 may be used for determining the point-of-interest. This allows attention to each area of the full spherical image or 3DCG image (the minimum unit is each pixel).

In step S103, the display pattern generation unit 16 generates the display pattern while the constraint unit 17 imposes constraints on the eye gaze direction of the full spherical image or the parameters of the position of the virtual camera 330 in the 3DCG image.

FIGS. 21A to 21C are examples of explaining parameters of the eye gaze direction of the full spherical image or the position of the virtual camera 330 in the 3DCG image. The parameters of the eye gaze direction of the full spherical image or the position of the virtual camera 330 in the 3DCG image are camera parameters arranged in time series. The camera parameters are the parameters (a position and attitude, a direction of optical axis, a focal length (field angle), etc.) of the camera captured when the full spherical image 66 and the 3DCG image are virtually placed in a three-dimensional space.

FIG. 21A illustrates a situation in which the virtual camera 330 moves on the sphere 340 between times t=1 and t=T. The vertical direction of the sphere 340 is the Z direction. In FIGS. 21A, and 21B, the sphere 340 is assumed to be the full spherical image, with a 3D model 95 positioned at the center. In this case, the terminal device 30 may display the full spherical image 6 as the background of the 3D model 95.

FIG. 21B illustrates the camera parameters in the situation illustrated in FIG. 21A. The camera parameters are as follows:
Camera projection center (xt, yt, zt);
Optical axis direction (ut, vt, wt);
Optical axis rotation θt; and
Horizontal angle φt.

Because it is taken from t=1 to t=T, the parameters of the position of the virtual camera 330 in the line-of-sight or 3DCG images of the full spherical image are in chronological order as illustrated in FIG. 21C. One line (camera parameter) in FIG. 21c is one parameter in Pattern DB195 in Table 6.

Further, as illustrated in FIG. 22, the number of parameters may be reduced by setting the center of the sphere 340, which places the full spherical image 6 , to pass through the optical axis of the virtual camera 330. The camera parameters of FIGS. 22a and 22b are as follows:
Imaging direction (xt, yt, zt);
Camera center displacement dt;
Optical axis rotation θt; and
Horizontal angle φt.

Because the optical axis of the virtual camera 330 passes through the center of the sphere 340, the position of the virtual camera 330 is determined from the imaging direction and the camera center displacement dt. Therefore, parameters can be reduced by not requiring an optical axis.

The constraint 17 imposes constraints on the camera parameters defined in this manner. For example, for the transition prohibition area 81 of FIGS. 17A and 17B, established for the appropriate threshold a ∈ [0 1] is:
|zt|<a (t=1,2, ...,T).

The constraint can be expressed as follows. That is, it constrains the possible values for a given axis. This constraint means that the absolute value of z at the position of the virtual camera 330 in the line-of-sight or 3DCG image of the full spherical image is less than a, which corresponds to the transition prohibition area 81 of FIGS. 17A and 17B. The parameters of the position of the virtual camera 330 are limited within the constrained coordinates of the constraint and the point-of-interest is set in time series.

The display pattern generation unit 16 generates the display pattern under such constraints. For example, (1) constrained generation, (2) relaxation of constraints, and (3) re-parameterization. The method in (1) means to limit the search range to within the constraints when searching for the above point-of-interest. The method of (2) generates a display pattern without restrictions on the constraint of |zt|<a (t=1, 2,..., T), for example, after reducing the viewpoint of the region outside the constraint (in the extreme case, 0). When searching for the above-described points-of-interest, it is sufficient to multiply the viewpoint of the search range by a value (for example, 0) that is less than 1. By extracting a predetermined number of restricted points as described above and randomly setting the order, a display pattern is obtained. That is, the parameters of the position of the virtual camera 330 are set in a time order with priority given to the points-of-interest having a high degree of interest after the degree of interest of the range of coordinates is reduced.

The method of (3) is a method of re-parameterizing parameters using an element orbit, as described in detail in FIGS. 23A to 28.

Returning to FIG. 20, in Step S104, the display parameters generated by the display pattern generation unit 16 above are stored in the pattern DB 195. Although the present embodiment describes that the display pattern and the content are distributed to the terminal device 30, a plurality of display images may be generated from the display pattern and the content and distributed to the terminal device 30. The processing load of the terminal device 30 can be reduced if the communication cost or the communication bandwidth is not hindered. Because a plurality of displayed images is reproduced as a movie, a plurality of displayed images may be referred to as a movie or animation. In this case, the image distribution apparatus 10 includes a display image generation unit 44.
<Constraint method using element orbit>

Next, a constraint due to (3) re-parameterization will be described. In FIGS. 20 to 22B, the display pattern is generated while the constraint unit 17 imposes the constraints on the parameters of the eye gaze direction of the full spherical image or the position of the virtual camera 330 in the 3DCG image. On the other hand, there is a method of generating the display pattern using the element orbit.

For example, for convenience of illustration, for the appropriate variable w₁, w₂, w₃ ∈ R, if the position of the virtual camera 330 in the eye gaze direction of the full spherical image or in the 3DCG image is limited to a large circular orbit (the center of the circular orbit coincides with the center of the sphere), the following constraints are set:

Equation 1

w₁.x_t+w₂.y_t+w₃.z_t=0 and x_t ²+y_t ²+z_t ²-1=0 (t=1,2,..., T)...(1)

The left hand side represents the dot product of (w₁ ,w₂ ,w₃) and (x_t ,y_t ,z_t), and the dot on the sphere (x_t ,y_t ,z_t) is above the great circle because the dot is orthogonal. Various large circles can be expressed by setting the variables w₁, w₂, and w₃. The right-hand side of this equation represents a sphere having a diameter of 1.

Generally, the constraint can be expressed using a parameter θ, using N functions g_n:(n=1, 2,..., N) and M functions h_m:(m=1, 2,..., M), in Equation 2 having a following form:

Equation 2

g₁(Θ) < 0 ∧...∧ g_N(Θ) < 0 ∧ h₁(Θ)= 0 ∧...∧ h_M(Θ)=0...(2)

Here, the parameter space is a real number R.
<Example of the element orbit>

FIGS. 23A to 24B illustrate examples of the element orbit. FIG. 23A illustrates a horizontal orbit at the position of the virtual camera 330 and FIG. 23B illustrates the horizontal orbit in the eye gaze direction of the full spherical image 6.

Equation 3

w₁.x_t+w₂.y_t+w₃.z_t-u=0 and x_t ²+y_t ²+z_t ²-1=0 (t=1,2,..., T)...(3)

In Equation 3, when the z-axis is vertical and w₁=w₂=0, the horizontal orbits illustrated in FIGS. 23A and 23B are obtained from the relationship zt=u/w₃.

As illustrated in FIGS. 24A and 24B, an orbit around any axis can be set. FIG. 24A illustrates the large circle orbit 97 of the virtual camera 330 and FIG. 24B illustrates the large circle orbit 98 in the eye gaze direction of the full spherical image 6. In Equation 3, where u = 0, the orbit becomes the large circle illustrated in FIG. 24A and 24B. Thus, Equation 3 is a generalization of horizontal and large circular orbits.

Also, as illustrated in FIG. 25A, the horizontal rotation orbit can be set for an arbitrary axis 99. Also, as illustrated in FIG. 25B, the helical orbit may be used as the element orbit. In FIG. 25B, the position of the virtual camera 330 or the eye gaze direction of the full spherical image 6 is moved spirally over the sphere 340. Such helical orbits can be represented by Equation 4.

Equation 4

w₁.x_t+w₂.y_t+w₃.z_t-(u+vt) = 0 and x_t ²+y_t ²+z_t ²-1=0 (t=1,2,...,T)...(4)

The constraint incorporates the time component vt, as illustrated in Equation 4. If v=0, it is equivalent to Equation 3. Here, v is a parameter for the speed of axial movement.
<<Procedure for generating display pattern using element orbit>>

Such element orbits can reduce inconveniences such as the loss of smoothness by shifting different perspectives to complex orbits.

When the display pattern is generated by reflecting the constraint on the element orbit, the processing content of step S103 of FIG. 20 is changed. FIG. 26 is a flowchart illustrating a procedure performed in step S103 of FIG. 20 when a display pattern is generated using element orbits.

In step S201, the display pattern generation unit 16 generates concrete values of the parameters (w₁, w₂, w₃, u, v) of the element orbit. For example, it is randomly generated based on a uniform distribution in a particular region of the element orbit's parameters (intended to be limited to a certain range, which may be determined from the processing power of the computer, etc.).

In step S202, the display pattern generation unit 16 calculates the accumulated value of the interest on the element orbit generated by parameters randomly generated.

FIG. 27 is a diagram illustrating the accumulated value of the interest on the element orbit. As the element orbit selection method, it is possible to select the one in which the accumulated value of attention on the element orbit takes the extreme value. The horizontal orbit can represent the element orbit with the elevation angle as a parameter. FIG. 27 is an example of calculating the accumulated interest on the element orbit corresponding to each elevation angle. In this example, there are two extreme values, and the element orbit of the elevation angle taking the extreme value is selected.

Alternatively, the elevation angle may be clustered with accumulated attention regarded as a frequency. In the case of a large circle orbit, the direction vector of the rotation axis can be used as a parameter to make the same explanation as described above.

Returning to FIG. 26, in step S203, the display pattern generation unit 16 determines the adoption of parameters from the value of the accumulated interest. If the parameter is greater than the accumulated interest previously searched, the accumulated interest is taken and recorded.

In step S204, when the termination condition (for example, repeating the specified number of times) is satisfied, the process ends, or otherwise returns to step S201. This gives a single element orbit. The display pattern generation unit 16 generates the display pattern having a continuous viewpoint on the element orbit.

FIG. 26 is a method for detecting the parameter of the element orbit by random search, but there is an alternative method for dividing the parameter space of the element orbit onto the grid and searching the entire grid point. In this case, the grid points are sequentially selected in step S201. The termination condition is that all grid points have been selected in step S204.

Alternatively, meta-heuristic search techniques such as annealing, genetic algorithms, and taboo searching may be used. In this case, a parameter is generated according to each meta-heuristic algorithm in step S201 and a parameter is adopted according to each meta-heuristic algorithm in step S203. When using the annealing method, step S201 randomly generates a parameter near the parameter of the previous iteration, and step S203 adopts the new parameter when it has a higher accumulated interest than the previous repeating parameter, even when it is low, with a constant probability.

Incidentally, when FIG. 26 is executed in the element orbits of Equations 2 to 4, the degree of freedom of the speed of the parameter change on the element orbit is left. This is set manually by a person in charge. In addition, the element orbits of Equations 2 to 4 are simply determined by the position of the virtual camera 330 or the eye gaze direction of the full spherical image 6, and the degree of freedom remains in other parameters such as the field angle. This can be set manually or generated in a manner similar to that used to determine the point-of-interest.
<<How to obtain multiple element orbits>>

Because FIG. 26 is an example of selecting parameters of one element orbit, when more than one is desired, the above method is performed multiple times with different random number species to use a method of obtaining multiple quasi-optimal solutions.

Another approach is to use clustering techniques. A processing flow of the clustering approach is illustrated in FIG. 28, and the details of each step are described below. FIG. 28 is a flowchart illustrating a procedure for determining two or more element orbits.

In step S301, the constraint unit 17 appropriately discretizes the parameter space of the element orbit (limiting the value taken by each parameter to a predetermined value such as 1, 2, etc.) to obtain the accumulated value of the interest for each parameter.

In step S302, the accumulated interest calculated by the display pattern generation unit 16 is regarded as a probability of the parameter of the element orbit, and the parameter of the element orbit is divided into a specific number of clusters. Common clustering techniques can be used, such as a k-means and an EM algorithms/variance Bayesian method for a mixed distribution. Incidentally, in the above-described method of determining the point-of-interest, the "interest" is regarded as the probability and the "camera parameters" are clustered. However, in FIG. 28, the "accumulated interest" is regarded as the probability and the "parameter of the element orbit" is clustered, and there is a difference.

In step S303, the display pattern generation unit 16 selects a representative value (an average value, a median value, a mode, or the like) of each cluster. This allows the number of clusters to yield the parameters of the element orbits.

In step S304, the display pattern generation unit 16 determines the transition sequence of the obtained plurality of element orbits. One way is to sort the element orbit's parameter space with the appropriate ordering (e.g., the Euclidean distance to the origin) to determine the transition order. Alternatively, the order can be determined by randomly selecting one element orbit and then recursively selecting the element orbit closest to that parameter.

In step S305, the display pattern generation unit 16 generates a transition orbit between the ordered element orbits. Simply joining the element orbits results in discontinuity at the junction, and thus interpolation between the element orbits is performed. This may be linear interpolation or spline interpolation. Specifically, in the case of linear interpolation, the animation parameter θ´i(t) can be generated by interpolating the ith and i+1 element orbits by naming the animation parameter θ(t) at time t corresponding to the ith element orbit. T is the total number of frames.

If the time lengths of θi and θi+1 are different, either copy or resize the elements so that the time lengths are the same. For example, θi(1)...θi(10) has a time length of 10, θi+1(1)...θi+1(20) has a time length of 20, so the time lengths of the two are different. In this case, each element of the former animation parameter is copied to a total length of 20, or the elements of the latter animation are deleted one at a time to resize.

The display parameters interpolated in this way can be aligned with the pre-interpolation display parameters (θ1, θ´1, θ2, θ´2, θ3,...) to obtain a continuously varying animation.

Thus, it is possible to generate a display pattern based on multiple element orbits. In addition, a plurality of element orbits may be obtained by the search method illustrated in FIG. 26, and the display pattern may be generated by performing the steps from Step S304 onwards.
<Operation Procedure>

FIG. 29 is an example of a sequence diagram illustrating a procedure for delivering the full spherical image 6 or the 3DCG image displayed as the advertisement by the image display system 100.

S1: The viewer first operates the terminal device 30 to connect with the partner site Web server 60. The operation reception unit 34 of the terminal device 30 receives the operation, and the Web page acquisition unit 31 acquires the Web page.

S2: The Web page analysis unit 32 of the terminal device 30 analyzes the Web page and extracts HTML, CSS, and scripts. The Web page display unit 33 displays the Web page based on HTML and CSS.

S3: The advertisement space detection unit 36 of the terminal device 30 transmits an advertisement request to the SSP 50 by executing an advertisement tag (script) associated with the advertisement space 7. The advertisement requests may include, for example, SSP cookies, domains of the partner site Web server 60, an advertisement space ID, a size of the advertisement space 7, an advertising format, a browser type, an OS type and the like.

S4: The advertisement request acquisition unit 52 of the SSP 50 acquires the advertisement request, and the advertisement request unit 51 accepts the advertisement request. The advertisement request unit 51 detects the SSP cookie and notifies the DSP 20 of the advertisement request and sells the advertisement space 7.

S5: The request reception unit 21 of the DSP 20 sends an advertisement request to the bid determination unit 22. The bid determination unit 22 of the DSP 20 specifies the DSP cookie from the SSP cookie with reference to cookie information DB 291.

S6: The bid determination unit 22 determines attributes from the DSP cookie. The visit domain of the cookie information DB 291 may be referenced, or the viewer attribute information identified by the DSP cookie may be referenced, as appropriate.

S7: The bid determination unit 22 determines whether to transmit a wide angle image to the viewer based on the attributes of a preferred target or an undesirable target set for a source of a wide angle image or a 3DCG image referring to the distribution setting DB 292. Here, the distribution is determined to at least one advertiser.

S8: The bid determination unit 22 determines the bid amount in accordance with an advertiser's budget and the degree of agreement of attributes. The bid acceptance unit 54 of the SSP 50 determines the DSP 20, which, in principle, offers the highest bid amount from among bids from multiple DSPs 20, as the accepted bidder.

S10: Here, the DSP20 is accepted in this case. The SSP 50 notifies DSP 20 of the bid acceptance ID.

S11: The access information transmission unit 53 of the SSP 50 transmits the access information 1 to the terminal device 30 together with the bid acceptance ID.

S12: The advertisement request unit 38 of the terminal device 30 transmits the advertisement request to the DSP 20 together with the bid acceptance ID by executing the access information 1.

S13: The advertisement request reception unit 24 of the DSP 20 identifies the advertisement request by the bid acceptance ID. The image requesting unit 25 requests the image distribution apparatus 10 with the advertiser ID, the DSP cookie, and the advertisement request.

S14: The image information response unit 11 of the image distribution apparatus 10 generates an advertisement opportunity ID for identifying an advertisement request upon receipt of an image request.

S15: The image information response unit 11 transmits the access information 2 associated with the advertisement opportunity ID to the DSP 20.

S16: The image requesting unit 25 of the DSP 20 receives the access information 2 and transmits the advertisement request reception unit 24 to the terminal device 30.

S17: The advertisement acquisition unit 39 of the terminal device 30 transmits an advertisement request (advertisement opportunity ID) based on the URL of the image distribution apparatus 10 included in the access information 2. If the terminal device 30 holds the image cookie, the image cookie can be transmitted.

S18: The advertisement distribution unit 12 of the image distribution apparatus 10 specifies the image request by the advertisement opportunity ID.

S19: The advertisement distribution unit 12 determines an attribute from the DSP cookie or determines an attribute from the image cookie.

S20: The advertisement distribution unit 12 identifies the full spherical image 6 based on the advertiser ID and the attributes. That is, referring to the advertisement image DB 194, the full spherical image 6 or the 3DCG image to be transmitted based on the attributes or priorities of a target of distribution set for the full spherical image 6 or the 3DCG image is determined. The display pattern is higher than the threshold number of clicks, but is selected randomly while the number of distributions of the full spherical image 6 or the 3DCG image is small.

S21: The advertisement distribution unit 12 transmits the advertisement data including the full spherical image 6 or the 3DCG image (which may be both), the initial position, the display pattern, the image ID, the image cookie, and the advertisement space ID to the terminal device 30. The full spherical image 6 or the 3DCG image are associated with the URL of the advertiser Web server 70. In addition, when the image distribution apparatus 10 creates a plurality of display images (animations), a plurality of display images are distributed instead of the full spherical image 6 or the 3DCG image.

S22: The advertisement acquisition unit 39 of the terminal device 30 acquires the advertisement data, and the advertisement display unit 40 displays the full spherical image 6 or the 3DCG image on the advertisement space 7. When the viewer presses (clicks or taps) the advertisement space 7, the operation reception unit 34 receives it, and the Web page acquisition unit 31 starts communication with the advertiser Web server 70 based on the URL of the advertiser Web server 70.
<Operation of terminal device>

FIG. 30A and 30B are examples of a flowchart illustrating an operation of the terminal device 30 that receives the advertisement data. FIG. 30A is an operation procedure in which the terminal device 30 generates a plurality of display images, and FIG. 30B is an operation procedure in which the image distribution apparatus 10 generates a plurality of display images.

In step S401, the advertisement acquisition unit 39 of the terminal device 30 receives the full spherical imageor the 3DCG image (both may be) and the display pattern.

In step S402, the display image generating unit 44 acquires the time series parameters included in the display pattern in sequence.

In step S403, the display image generation unit 44 generates a predetermined area image or a projected image in accordance with parameters, and the advertisement display unit 40 displays a predetermined area image or a projected image in the advertisement space 7.

In Step S404, the display image generating unit 44 determines whether or not playback is completed. Judgment of whether playback has been completed refers to the determination of whether or not the display pattern has been repeated a fixed number of times. Processing returns to step S402 until the reproduction ends, and when the reproduction ends, the advertisement display unit 40 stops the predetermined area image or the projected image and ends the processing.

In Step S501, when the image distribution apparatus 10 generates a plurality of display images, the advertisement acquisition unit 39 of the terminal device 30 receives a plurality of display images generated from a whole globe image or a 3DCG image and a display pattern.

In step S502, the advertisement display unit 40 reproduces a plurality of display images as a movie (or animation). When playing to the end, playback is terminated.
<Supplement to distribution format of advertisement data>

In the present embodiment, the advertisement was distributed by a third party distribution mechanism, but the advertisement may be displayed on a Web page displayed by the terminal device 30. FIGS. 31A and 31B illustrate some forms of advertisement data distribution.

In FIG. 31A, the Web page is transmitted with the advertisement data included in the Web page distributed by the partner site Web server 60. In this case, the advertiser has preliminarily submitted the image data to the partner site Web server 60.

In FIG. 31B, an advertisement distribution server 80 distributes the advertisement data. The partner site Web server 60 registers the Web page on which the advertisement is to be placed in the advertisement distribution server 80. Accordingly, because the advertisement distribution server 80 outputs an ad tag, the partner site Web server 60 pastes the advertisement to the Web page on which the advertisement is to be placed. The advertiser also places the advertisement data into the advertisement distribution server 80.

When the viewer accesses the partner site Web server 60, the advertisement distribution server 80 is called by the ad tag. The advertisement distribution server 80 determines which advertisements are distributed from each advertiser and transmits the advertisement data to the terminal device 30.

FIG. 31C is an example of distribution of advertisement data according to the embodiment described in this embodiment. As described in this embodiment, the image distribution apparatus 10 may deliver advertisement data or the DSP 20 may distribute the advertisement data. Advertisers can submit ad data to DSP 20. In the configuration illustrated in FIG. 31C, the DSP 20 or the image distribution apparatus 10 serves as the advertisement distribution server 80.

Further, although the image data displayed in the advertisement space 7 has been described as a preferred example of the image data, if the image data is distributed to the terminal device 30 and varies with the operation of the viewer (the image data has interactivity), the present embodiment is not limited to advertising, but can be suitably applied.
<Summary of Embdiments>

As described above, by constraining the viewpoint in this manner, the image display system according to this embodiment can prevent an unnatural transition of the viewpoint. For example, it can be prevented from moving to a viewpoint where the content is viewed upward from above or upward from below. In addition, by constraining the viewpoint after setting the element orbit, it is possible to prevent the transition of different viewing directions in a complex orbit. Therefore, the content can be displayed from an appropriate viewpoint, making it easier to enhance the advertising effect.
<Other Applications>

While the preferred embodiment of the present invention has been described with reference to examples, various modifications and substitutions may be made thereto without departing from the spirit and scope of the invention.

For example, the position of the virtual camera is not limited to a sphere, but may be an ellipse, a rectangular body, or the like.

Further, in the present embodiment, an example in which the full spherical image 6 is displayed in the advertisement space 7 has been described. However, the whole-heavenly image 6 may be a wide angle image. That is, it is sufficient that the image data has an angle of field that cannot be inserted into the advertisement space 7 and the range that can be seen by the operation of the viewer is changed. In addition, the image displayed on the Web page need not be an advertising image, but only image data that can be viewed.

Further, the application of the terminal device 30 for displaying the full spherical image 6 is not limited to the browser software, and the present embodiment can be applied when the arbitrary application software displays the full spherical image 6 or the 3DCG image on the screen.

According to this embodiment, the terminal device 30 accesses the DSP 20 and then accesses the image distribution apparatus 10 to acquire the advertisement data. However, the DSP 20 may acquire the advertisement data directly from the image distribution apparatus 10 and transmit it to the terminal device 30. The terminal device 30 can obtain the advertisement data only by accessing the DSP 20, thus reducing the time until the advertising is displayed. The DSP 20 and the image distribution apparatus 10 may be integrally configured, and the DSP 20 may distribute advertisement data (an all-orbit image) to the terminal device 30.

Further, the method for distributing the advertisement in which the SSP 50, the DSP 20, and the image distribution apparatus 10 cooperating with each other described in the present embodiment is merely an example, and it is not intended to restrict the process until the terminal device 30 displays the advertisement. For example, as a simple process, the partner site Web server 60 may store and deliver advertisement data from the advertiser to the terminal device 30. In this case, the SSP 50, the DSP 20, and the image distribution apparatus 10 may not be present.

Further, the full spherical image 6 is not limited to the case where the two images are created in a manner in which the two images are held together. For example, overlapping portions of the image data captured by the image data may be created while moving in the image direction little by little. In other words, the method of creating the full spherical image 6 is not limited.

Further, the configuration example such as FIGS. 13A to 13C illustrated in the above-described embodiment is divided according to the main functions for easier understanding of the processing of the image display system 100. However, the present invention is not limited by the method of dividing or the name of each treatment unit. The image display system 100 may be divided into more processing units depending on the processing content. Alternatively, one processing unit can be divided to include more processing.

The functions of the embodiments described above may be implemented by one or more processing circuits. As used herein, a "processing circuit" includes a processor programmed to perform each function by software, such as a processor implemented in electronic circuits, an ASIC (Application Specific Integrated Circuit) designed to perform each function as described above, a digital signal processor (DSP), a field programmable gate array (FPGA), or a conventional circuit module.

7 Advertisement space
10 Image distribution apparatus
20 DSP
30 Terminal device
40 Advertisement display unit
50 SSP
60 Partner site Web server
70 Advertiser Web server
100 Image display system

The present application is based on Japanese Priority Patent Application No. 2019-050320 filed on March 18, 2019 with the Japan Patent Office, the entire contents of which are hereby incorporated by reference.

Claims

　　An image display system in which a terminal device receives a content from an image distribution apparatus for distributing an image,
　　wherein the image distribution apparatus includes
　　　　a display pattern generation unit that generates a display pattern in which one or more of the content is displayed from different viewpoints,
　　　　a constraint unit that constrains the viewpoints, and
　　　　a distribution unit that distributes the display pattern and the content, or a plurality of display images generated based on the display pattern to the terminal device, and
　　wherein the terminal device includes
　　　　an acquisition unit that acquires a plurality of display images generated based on the display pattern and the content or the display pattern, and
　　　　a display unit that displays the content acquired by the acquisition unit by changing the viewpoint of the display pattern or the plurality of display images generated based on the display pattern.
　　The image display system according to claim 1,
　　wherein the constraint unit prohibits or limits a transition of the viewpoint to a specific region in coordinates representing the viewpoint for viewing the content.
　　The image display system according to claim 1 or 2,
　　wherein the constraint unit constrains a value applicable for a predetermined axis of coordinates representing the viewpoint from a position of a virtual camera representing the viewpoint or an eye gaze direction with respect to the content.
　　The image display system according to claim 3, further comprising:
　　a point-of-interest determination unit that determines a point-of-interest of the content,
　　wherein the display pattern generation unit generates the display pattern by setting the point-of-interest in a time series by limiting the point-of-interest within a range of the coordinates constrained by the constraint unit from among a plurality of points-of-interest determined by the point-of-interest determination unit.
　　The image display system according to claim 3, further comprising:
　　a point-of-interest determination unit that determines a point-of-interest of the content,
　　wherein the display pattern generation unit generates the display pattern by reducing a range of the coordinates out of the constraint by the constraint unit from among a plurality of points-of-interest determined by the point-of-interest determination unit and prioritizing the point-of-interests having a greater interest in a time series.
　　The image display system according to claim 1 or 2, wherein the display pattern generation unit generates the display pattern from a viewpoint of an element orbit based on the degree of viewpoint of the point to which the element orbit rotating around the content passes.
　　The image display system according to claim 6, further comprising:
　　a point-of-interest determination unit that determines a point-of-interest of the content,
　　wherein the display pattern generation unit accumulates the interest of the point-of-interest determined by the point-of-interest determination unit through which the element orbit passes and determines the element orbit having a highest interest and generates the display pattern in which the element orbit is the viewpoint.
　　The image display system according to claim 6,
　　wherein the display pattern generation unit discretizes a parameter space of the element orbit to obtain the accumulated value of the interest for each parameter, regards the accumulated value of the interest as a probability, performs clustering, determines a representative value of each cluster in the parameter of the element orbit so as to generate the display pattern using the determined element orbit as the viewpoint.
　　The image display system according to claim 8,
　　wherein the display pattern generation unit generates the display pattern interpolated between parameters of the element orbit.
　　An image distribution method performed by an image display system in which a terminal device receives a content from an image distribution apparatus that distributes an image, the image distribution method comprising:
　　generating, by a display pattern generation unit of the image distribution apparatus, a display pattern in which at least one content is displayed from different viewpoints;
　　constraining, by a constraint unit of the image distribution apparatus, the viewpoint;
　　distributing, by a distribution unit of the image distribution apparatus, a plurality of display images generated based on the display pattern and the content or the display pattern to the terminal device;
　　acquiring, by an acquisition unit of the terminal device, a plurality of display images generated based on the display pattern and the content or the display pattern; and
　　displaying, by the display unit of the terminal device, the content acquired by the acquisition unit by changing the viewpoint of the display pattern or the plurality of display images generated based on the display pattern.
　　A program causing an image distribution apparatus for distributing an image to a terminal device, the program comprising:
　　a display pattern generation unit that generates a display pattern in which at least one content is displayed from different viewpoints;
　　a constraint unit that constrains the viewpoint; and
　　a distribution unit that distributes a plurality of display images generated based on the display pattern and the content or the display pattern to the terminal device.