JP7393361B2 - Digital supplementary association and search for visual search - Google Patents

Description

REFERENCES TO RELATED APPLICATIONS This application is a continuation of U.S. Nonprovisional Patent Application No. 16/014,520, filed on June 21, 2018, entitled “DIGITAL SUPPLEMENT ASSOCIATION AND RETRIEVAL FOR VISUAL SEARCH,” and has priority thereto. , the disclosure of which is incorporated herein by reference in its entirety.

Background Mobile computing devices, such as smartphones, often include a camera. These cameras may be used to capture images of entities in the environment surrounding the computing device. Various types of content or experiences related to these entities may be available to the user via the mobile computing device.

Overview This disclosure describes systems and methods for digital complementary association and retrieval for visual search. For example, the systems and techniques described herein may be used to provide digital supplements, such as augmented reality (AR) content or experiences, in response to visual searches. Visual searches may be based on images or entities identified within images, for example. Digital supplementation may include, for example, providing information or functionality associated with an image.

One aspect is a computer-implemented method that includes receiving data specifying a digital supplement, the data identifying a digital supplement and a supplement anchor for associating the digital supplement with image content. The method also includes generating a data structure instance that specifies the digital supplement and the supplement anchor. The method further includes, after generating the data structure instance, enabling triggering of digital supplementation with the image based on storing at least the data structure instance in a database that includes a plurality of other data structure instances. Each of the other data structure instances specifies a digital supplement and one or more supplement anchors.

Another aspect is a computing device that includes at least one processor and memory that stores instructions. The instructions, when executed by the at least one processor, cause the computing device to receive data specifying a digital supplement, the data including a digital supplement and a supplement for associating the digital supplement with image content. An anchor and context information are received. The instructions also cause the computing device to generate a data structure instance that specifies the digital supplement, the supplement anchor, and the context information. The instructions further cause the computing device to trigger digital supplementation with an image based at least on: generating the data structure instance and then storing the data structure instance in a database that includes a plurality of other data structure instances. Activate. Each of the other data structure instances specifies a digital supplement and one or more supplement anchors.

Yet another aspect is a computer-implemented method that includes receiving an image content query from a computing device and identifying a supplemental anchor based on the image content query. The method also includes generating an ordered list of digital supplements based on the identified supplement anchors and transmitting the ordered list to the client computing device.

The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

1 is a block diagram illustrating a system according to an example embodiment. FIG. 2 is a third-person view of an exemplary physical space in which an embodiment of the client computing device of FIG. 1 is accessing a digital supplement; FIG. FIG. 3 is an illustration of an example method for enabling a trigger for digital supplementation, according to embodiments described herein. FIG. 3 is an illustration of an example method for enabling a trigger for digital supplementation, according to embodiments described herein. FIG. 2 is an illustration of an example method for searching and presenting digital supplements in accordance with embodiments described herein. FIG. 2 is an illustration of an example method for identifying and presenting digital supplements based on images, according to embodiments described herein. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for performing image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 to perform various image content searches within a store; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 to perform various image content searches within a store; FIG. 2 is a schematic diagram of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 to perform various image content searches within a store; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 during various image content searches; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 during various image content searches; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 during various image content searches; FIG. 1 is a schematic illustration of an example of a computing device and a mobile computing device that may be used to implement the techniques described herein.

DETAILED DESCRIPTION The present disclosure describes technical improvements that simplify the identification and presentation of digital supplements based on image content. Certain embodiments of the techniques described herein generate an index of digital supplements related to particular types of image content and index those digital supplements in response to an image content query received from a client computing device. Provide supplements. This index allows users to access relevant digital supplements provided by network-accessible resources (eg, web pages) located around the world. This may provide a functional data structure that allows more efficient retrieval of information.

For example, a client computing device, such as a smartphone, may obtain images of supplementary anchors, such as entities. The client computing device may then send an image content query based on the image to the server computing device to retrieve digital supplements associated with the identified supplement anchors. In some embodiments, the supplemental anchor is based on the physical environment surrounding the client computing device, and the digital supplement is virtual content that may supplement the user's experience in the physical environment.

The image content query may include images or data determined from the image (eg, indicators of identified supplemental anchors). An example of data determined from an image is text extracted from the image using, for example, optical character recognition. Other examples of data extracted from images include barcodes, QR codes, etc. on the image, an identifier or description of the entity, the product, or the type of entity identified in the image.

An entity, product, or entity type may be identified within an image using a neural network system, such as a convolutional neural network system, for example. An identifier or description of an entity, product, or entity type may include metadata or a reference to a record in a database that is associated with the entity, product, or entity type. Non-limiting examples of entities include buildings, works of technology, products, books, posters, photographs, catalogs, signs, documents (eg, business cards, receipts, coupons, catalogs), people, and body parts.

Various types of digital supplements associated with supplemental anchors may be available. Digital supplements may be provided by network accessible resources, such as web pages available on the Internet. A method is needed to locate and provide these digital supplements in response to image content queries. Certain embodiments generate and maintain an index of digital supplements associated with entities for use in responding to image content queries. The index may be configured, for example, by crawling network-accessible resources to determine whether the network-accessible resources include or provide digital supplements and determining supplementary anchors associated with those digital supplements. , may be added.

For example, a network-accessible resource may include metadata that identifies a supplement anchor (eg, text, code, entity, or type of entity) with which a digital supplement is associated. Metadata may be included by a network-accessible resource in response to a Hypertext Transfer Protocol (HTTP) request. Metadata may be provided in various formats, such as extensible markup language (XML), JavaScript Object Notation (JSON), or other formats.

Metadata for a digital supplement may include one or more of the following: type indicator, anchor indicator, name, description, snippet of content (i.e., an excerpt or preview of a portion of the content), associated images, digital supplement and an identifier of the application associated with the digital supplement. The metadata may also include information about the digital supplement's publisher. For example, the metadata may include one or more of a publisher name, a publisher description, and an image or icon associated with the publisher. In some embodiments, the metadata includes contextual information associated with providing digital supplements. For example, metadata may also include conditions associated with providing a digital supplement or accessing a digital capture (eg, geographic conditions, required applications).

The identified digital supplements may be added to an index stored in memory. In at least some embodiments, the associated supplementary anchor for the digital supplement is used as a key to the index. Digital supplements may also be associated with various scores. For example, a digital supplement can determine how many other links that refer to the digital supplement or network-accessible resources associated with the digital supplement are found (e.g., while crawling network-accessible resources); These links may be associated with a prestige score based on the prestige of the network-accessible resource providing the link. As another example, a digital supplement may be associated with one or more relevance scores that correspond to the relevance of the digital supplement (or associated network-accessible resource) to a particular anchor. Relevance scores may also be associated with keywords or subjects. The relevance score is one of the following: the content of a digital supplement, the content of a network-accessible resource, the content of a site that links to a network-accessible resource, and the content (e.g., text) of a link to a network-accessible resource. or more than one.

FIG. 1 is a block diagram illustrating a system 100 according to an example embodiment. System 100 can associate digital supplements with entities or entity types and can retrieve digital supplements in response to visual searches. Visual search is a search based on image content. For example, a visual search may be performed based on an image content query. Image content queries are queries based on images or other image content. For example, an image content query may include images. In some embodiments, the image content query may include text or data based on the image. For example, text or data may be generated by recognizing one or more entities within an image. Some image content queries do not include images (eg, an image content query may include only data or text generated from images). In some embodiments, system 100 includes client computing device 102, search server 152, and digital supplementation server 172. Also shown is a network 190 with which client computing device 102, search server 152, and digital supplementary server 172 may communicate.

Client computing device 102 may include a processor assembly 104, a communications module 106, a sensor system 110, and memory 120. Sensor system 110 may include various sensors, such as a camera assembly 112, an inertial motion unit (IMU) 114, and a global positioning system (GPS) receiver 116. Embodiments of sensor system 110 may also include, for example, light sensors, audio sensors, image sensors, distance and/or proximity sensors, contact sensors such as capacitive sensors, timers, and/or other sensors; Other sensors may be included including different combinations. In some embodiments, client computing device 102 is a mobile device (eg, a smartphone).

Camera assembly 112 captures images or video of the physical space around client computing device 102 . Camera assembly 112 may include one or more cameras. Camera assembly 112 may also include an infrared camera. Images captured with camera assembly 112 may be used to supplement anchors and identify them to form image content queries.

In some embodiments, the images captured by camera assembly 112 also capture images of a client computing device within a physical space, such as an interior space, based on a representation of the physical space received from memory 120 or an external computing device. 102 can be used to determine the position and orientation of 102. In some embodiments, a representation of a physical space may include visual features of the physical space (eg, features extracted from an image of the real space). The representation may also be associated with a position that can be used by a visual positioning system to determine a position and/or location within a physical space based on one or more images of the physical space. May include decision data. The representation may also include a three-dimensional model of at least some structure in physical space. In some embodiments, the representation does not include a three-dimensional model of the physical space.

IMU 114 may detect motion, motion, and/or acceleration of the client computing device. IMU 114 may include a variety of different types of sensors, such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The orientation of client computing device 102 may be detected and tracked based on data provided by IMU 114 or GPS receiver 116.

GPS receiver 116 may receive signals transmitted by GPS satellites. The signal includes the time and location of the satellite. Based on receiving signals from a number (eg, at least four) of satellites, GPS receiver 116 may determine the global location of client computing device 102.

Memory 120 may include applications 122, other applications 140, and device positioning system 142. Other applications 140 include any other applications installed or otherwise available for execution on client computing device 102. In some embodiments, application 122 may cause one of the other applications 140 to provide digital supplements. In some embodiments, some digital supplements may be available only if other applications 140 include the particular applications associated with or required to provide the digital supplements. .

Device positioning system 142 determines the location of client computing device 102 . Device positioning system 142 may use sensor system 110 to determine the location and orientation of client computing device 102 globally or within physical space. In some embodiments, device positioning system 142 determines the location of client computing device 102 based on, for example, cellular triangulation.

In some embodiments, client computing device 102 compares images captured by camera assembly 112 (or features extracted from those images) to known placements of features within a representation of physical space, and The system may include a visual positioning system that determines a six-degree-of-freedom pose (eg, position and orientation) of client computing device 102 in virtual space.

Application 122 may include a supplemental anchor identification engine 124, a digital supplementary search engine 126, a digital supplementary presentation engine 128, and a user interface engine 130. Certain embodiments of application 122 may include fewer, additional, or other components.

Supplemental anchor identification engine 124 identifies supplemental anchors based on images captured with camera assembly 112, for example. In some embodiments, supplemental anchor identification engine 124 analyzes images to identify text. That text can then be used to identify the anchor. For example, text may be mapped to nodes in a knowledge graph. For example, the text may be recognized as the name of an entity, such as a person, place, product, building, artwork, movie, or other type of entity. In some embodiments, the text may be recognized as phrases commonly associated with or describing a particular entity. For example, the text may then be recognized as an anchor associated with that particular entity.

In some embodiments, supplemental anchor identification engine 124 identifies one or more codes within the image, such as a barcode, QR code, or another type of code. The codes may then be mapped to supplementary anchors.

Supplemental anchor identification engine 124 may include a machine learning module that can recognize at least some types of entities within an image. For example, a machine learning module may include a neural network system. A neural network is a computational model used for machine learning, consisting of nodes organized into layers with weighted connections. Training a neural network uses multiple training examples, each example being an input and a desired output, and over a series of iterative rounds, it is possible for the neural network to provide the desired output for a given input. Determine the weight value of the connection between the layers to increase the During each training round, the weights are adjusted to account for erroneous output values. Once trained, a neural network can be used to predict output based on the input provided.

In some embodiments, the neural network system includes a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one of the layers of the neural network is a convolutional layer. A convolutional layer is a layer in which the values of a layer are calculated based on applying a kernel function to a subset of the values of previous layers. Training a neural network may include adjusting weights of a kernel function based on training examples. Typically, the same kernel function may be used to compute each value in the convolutional layer. Therefore, the weights that have to be learned while training a convolutional layer can be applied to fully connected layers in a neural network (e.g., each value in one layer is adjusted independently of each value in the previous layer). (layers computed as a weighted combination). Because convolutional layers typically have fewer weights, training and using a convolutional layer may require less memory, processor cycles, and time than an equivalent fully connected layer.

After supplemental anchor identification engine 124 recognizes an entity or entity type in the image, a textual description of the entity or entity type may be generated. Additionally, entities or entity types may be mapped to supplemental anchors. In some embodiments, a supplemental anchor is associated with one or more digital supplements.

In some embodiments, supplemental anchor identification engine 124 determines confidence scores for recognized anchors. A higher confidence score indicates that content from the image (e.g., image, extracted text, barcode, QR code) is more associated with the determined anchor than if a lower confidence score were determined. This can indicate that there is a high possibility that the

Although the example of FIG. 1 shows supplemental anchor identification engine 124 as a component of application 122 on client computing device 102, certain embodiments include a supplemental anchor identification engine on search server 152. For example, client computing device 102 may send an image captured by camera assembly 112 to search server 152, and search server 152 may identify supplemental anchors within the image.

In some embodiments, supplemental anchor identification engine 124 identifies potential supplemental anchors. For example, supplemental anchor identification engine 124 may identify (recognize) various entities within an image. Identifiers of the recognized entities may then be sent to search server 152, which may determine whether any of the entities are associated with any supplemental anchors. In some embodiments, search server 152 may use the identified entity as context information even if the identified entity is not a supplemental anchor.

Digital supplement search engine 126 searches digital supplements. For example, digital supplement search engine 126 may search for digital supplements associated with supplemental anchors identified by supplemental anchor identification engine 124. In some embodiments, digital supplement search engine 126 searches for digital supplements from search server 152 or digital supplement server 172.

For example, after a supplemental anchor is identified, digital supplement search engine 126 may search for one or more digital supplements associated with the identified supplemental anchor. Digital supplementary search engine 126 may generate an image content query that includes an image (or an identifier of a supplementary anchor or entity within the image) and send the image content query to search server 152. The image content query may also include contextual information such as the location of client computing device 102. In some embodiments, data associated with the digital supplement, such as a name, image, or description, is retrieved and presented to the user (eg, by user interface engine 130). If multiple digital supplements are presented, the user may select one of the digital supplements via a user interface generated by user interface engine 130.

Digital supplement presentation engine 128 presents or causes digital supplements to be presented on client computing device 102. In some embodiments, the digital supplement presentation engine 128 causes the client computing device to start one of the other applications 140. In some embodiments, digital supplemental presentation engine 128 causes information or content to be displayed. For example, digital supplement presentation engine 128 may cause user interface engine 130 to generate a user interface that includes information or content from the digital supplement displayed by client computing device 102. In some embodiments, digital supplement presentation engine 128 is triggered by digital supplement search engine 126 searching for digital supplements. Digital supplement presentation engine 128 may then trigger display device 108 to display content associated with the digital supplement. In some embodiments, digital supplement presentation engine 128 causes digital supplements to be displayed at different times than when digital supplement search engine 126 searches for digital supplements. For example, the digital supplement may be retrieved in response to an image content query at a first point in time, and the digital supplement may be presented at a second point in time. For example, the digital supplement responds to an image content query based on images of household goods or furniture from a catalog or store at a first time (e.g., while the user is viewing a catalog or while in a store). can be searched. A digital supplement that includes AR content of household items or furniture may be presented at a second time (e.g., while the user is in a room where the household items or furniture may be installed).

User interface engine 130 generates a user interface. User interface engine 130 may also cause client computing device 102 to display the generated user interface. The generated user interface may display information or content from the digital supplement, for example. In some embodiments, user interface engine 130 generates a user interface that includes a plurality of user actuatable controls, each associated with a digital supplement. For example, the user can perform a user-actuatable One of the controls may be activated.

Search server 152 is a computing device. Search server 152 may respond to search requests, such as image content queries. The response may include one or more digital supplements potentially related to the image content query. In some embodiments, search server 152 includes memory 160, processor assembly 154, and communication module 156. Memory 160 may include a content crawler 162, a digital supplementary search engine 164, and a digital supplementary data storage 166.

Content crawler 162 may crawl network accessible resources to identify digital supplements. For example, content crawler 162 may access web pages accessible via the Internet, such as web pages provided by digital supplement server 172. Crawling a network-accessible resource may include requesting the resource from a web server and analyzing at least a portion of the resource. Digital supplements may be identified based on metadata provided by network-accessible resources, such as XML or JSON data that provides information about the digital supplement. In some embodiments, the crawler identifies network-accessible resources based on extracting links from previously crawled network-accessible resources. Content crawler 162 may also identify network-accessible resources to crawl based on receiving input submitted by users. For example, a user may submit a URL (or other information) to a network-accessible resource containing a digital supplement via a web form or application programming interface (API). In some embodiments, content crawler 162 generates an index of the identified digital supplements. Content crawler 162 may also generate a score associated with the digital supplement, such as a relevance score or popularity (fame) score.

Digital supplement search engine 164 receives a search query and generates a response that may include one or more potentially relevant digital supplements. For example, digital supplementary search engine 164 may receive image content queries from client computing device 102. Image content queries may include images. Digital supplement search engine 164 may identify supplement anchors within the image and may identify associated or potentially related digital supplements based on the identified supplement anchors. Digital supplement search engine 164 may send a response to client computing device 102 that includes the digital supplement or information that may be used to access the digital supplement. In some embodiments, digital supplement search engine 164 may return information associated with multiple digital supplements. For example, a list of digital supplements may be included in a response to a query. The list may be ordered based on relevance to the supplement anchor, popularity, or other characteristics of the digital supplement.

Image content queries may include, for example, images captured by camera assembly 112 or text or other data associated with images captured by camera assembly 112. The image content query may also include other information, such as the location of client computing device 102 or an identifier of the user of client computing device 102. In some embodiments, the search server 152 retrieves the client information from the user identifier (e.g., if the user has enabled location services on the client computing device 102 that associate information about the user's location with the user's account). An expected location of computing device 102 may be determined.

Digital supplement data storage unit 166 stores information regarding digital supplements. In some embodiments, digital supplement data storage 166 includes an index of digital supplements. For example, the index may be generated by content crawler 162. Digital supplementary search engine 164 may use the index to respond to search queries.

Digital supplementary server 172 is a computing device. Digital supplement server 172 provides digital supplements. In some embodiments, digital supplementary server 172 includes memory 180, processor assembly 174, and communication module 176. Memory 180 may include digital supplement 182 and metadata 184. In some embodiments, memory 180 may also include other network-accessible resources, such as web pages, which are not necessarily digital supplements. For example, memory 180 may store web pages that include metadata to provide details about one or more digital supplements and how to access those digital supplements. Additionally, memory 180 may include a resource serving engine, such as a web server, that responds to requests, such as HTTP requests, with network-accessible resources such as web pages and digital supplements.

Digital supplement 182 is any type of content that can be provided as a supplement to something in the user's physical environment. Digital supplement 182 may also include any type of content that can supplement stored images (eg, of a previous physical environment around the user). For example, a digital supplement may be associated with a supplemental anchor such as an image, an object or product identified within the image, or a location. Digital supplement 182 may include one or more images, audio content, textual data, video, games, data files, applications, or structured text documents. Examples of structured text documents include Hypertext Markup Language (HTML) documents, XML documents, and other types of structured text documents.

Digital supplement 182 may launch an application and may define parameters for the application. Digital supplement 182 may also cause requests to be sent to the server (eg, HTTP requests) and define parameters for the requests. In some embodiments, digital supplement 182 begins as a workflow for completing an activity, such as a workflow for completing a purchase. For example, digital supplement 182 may send an HTTP request to a server that adds a particular item to the user's shopping cart, adds a coupon code, and retrieves a purchase confirmation page.

Metadata 184 is data that describes the digital supplement. Metadata 184 may describe one or more digital supplements provided by digital supplement server 172 or provided elsewhere. Metadata 184 for a digital supplement may include one or more of the following: a type indicator, an anchor indicator, a name, a description, a preview snippet or excerpt, an associated image, a URL-like link to the digital supplement, and Identifier of the application associated with the digital supplement. The metadata may also include information about the digital supplement's publisher, such as a publisher name, a publisher description, and an image or icon associated with the publisher. In some embodiments, the metadata also includes contextual information about the digital supplement or that must be satisfied to provide the digital supplement. For example, the metadata may include conditions that must be met to access the digital supplement (eg, geographic conditions, client computing device requirements, required applications). Exemplary context information includes a location, an entity identified within an image, or multiple entities identified within an image (e.g., some digital supplements require a combination of entities recognized within an image). (in some cases). A recognized entity may be a supplementary anchor. In some embodiments, the recognized entity does not supplement the anchor, but instead provides context information. Metadata 184 may also include supplemental anchors (eg, text, code, entities, or types of entities) associated with the digital supplement.

Metadata 184 may be stored in a variety of formats. In some embodiments, metadata 184 is stored in a database. Metadata 184 may be stored as an XML file, JSON file, or other format file. In some embodiments, digital capture server 172 retrieves metadata 184 from a database, formats metadata 184 as XML, JSON, or otherwise provides a response to a request from a client or search server 152. For example, search server 152 may access metadata 184 to generate data that is stored in digital supplemental data store 166 and used to respond to search requests from client computing device 102. .

Communication module 106 includes one or more devices for communicating with other computing devices, such as search server 152 or digital supplement server 172. Communication module 106 may communicate via a wireless or wired network, such as network 190. Communication module 156 of search server 152 and communication module 176 of digital supplement server 172 may be similar to communication module 106.

Display device 108 may be, for example, an LCD (Liquid Crystal Display) screen, an LED (Light Emitting Diode) screen, an OLED (Organic Light Emitting Diode) screen, a touch screen, or any other screen or display for displaying images or information to a user. may include. In some embodiments, display device 108 includes a light projector configured to project light onto a portion of the user's eye.

Memory 120 may include one or more non-transitory computer-readable storage media. Memory 120 is configured to generate an image content query based on the acquired images, transmit the image content query, receive a response to the image content query, and present digital supplements identified in the response to the image content query. may store instructions and data that can be used by client computing device 102 to implement the techniques described herein, such as. Memory 160 of search server 152 and memory 180 of digital supplementation server 172 may be similar to memory 120 and may store data instructions that can be used to implement the techniques of search server 152 and digital supplementation server 172, respectively. .

Processor assembly 104 executes one or more devices capable of executing instructions, such as instructions stored by memory 120, to perform various tasks associated with digital supplementary association and retrieval for visual search. include. For example, processor assembly 104 may include a central processing unit (CPU) and/or a graphics processor unit (GPU). For example, if a GPU is present, some image/video rendering tasks may be offloaded from the CPU to the GPU, such as generating and displaying a user interface or displaying portions of a digital supplement. In some embodiments, some image recognition tasks may also be offloaded from the CPU to the GPU.

Although not shown in FIG. 1, certain embodiments include a head-mounted display device (HMD). The HMD may be a separate device from client computing device 102, or client computing device 102 may include an HMD. In some embodiments, client computing device 102 communicates with the HMD via a cable. For example, the client computing device 102 may send video and/or audio signals to the HMD for display to the user, and the HMD may transmit motion, position, and/or orientation information to the client computing device 102. can be sent to.

Client computing device 102 may also include various user input components (not shown), such as a controller that communicates with client computing device 102 using a wireless communication protocol. In some embodiments, the client computing device 102 connects via a wired connection (e.g., Universal Serial Bus (USB) cable) or via a wireless communication protocol (e.g., any WiFi protocol, any BlueTooth protocol, Zigbee® ), etc.) may communicate with an HMD (not shown). In some embodiments, client computing device 102 is a component of an HMD and may be housed within a housing of the HMD.

Network 190 may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and/or any other network. Client computing device 102 may, for example, receive audio/video signals, which in one example embodiment may be provided over a network as part of a digital supplement.

FIG. 2 is a third-person view of an exemplary physical space 200 in which an embodiment of client computing device 102 is accessing digital supplements. In this example, physical space 200 includes object 222 . Here, object 222 is artwork on the wall of physical space 200. Object 222 is included within field of view 204 of camera assembly 112 of client computing device 102.

An example user interface screen 206 is also shown. User interface screen 206 may be generated by user interface engine 130 of client computing device 102, for example. User interface screen 206 includes an image display panel 208 and a digital supplement selection panel 210. Image display panel 208 displays images. For example, image display panel 208 may show an image corresponding to a real-time feed from camera assembly 112 of client computing device 102. In some embodiments, image display panel 208 shows previously captured images or images retrieved from memory 120 of client computing device 102 .

In some embodiments, user interface screen 206 is presented to a user on a display device of client computing device 102. In some embodiments, the user interface screen 206 may be so overlaid onto an image of the physical space (or a video feed being captured by a camera of the computing device). Additionally, user interface screen 206 may be displayed as AR content across the user's field of view using an HMD worn by the user.

Image display panel 208 may also include annotations or user interface elements associated with the image. For example, image display panel 208 may include an indicator that an object in the image (eg, object 222) has been recognized as a supplemental anchor. The indicator may include user actuatable controls to access or view information regarding the digital supplement associated with the identified supplemental anchor. In some situations, images displayed on image display panel 208 may include multiple objects that are recognized as supplemental anchors, and image display panel 208 may include multiple annotations overlaying the images to identify those supplemental anchors. may include.

The supplemental anchor may be recognized by the supplemental anchor identification engine of client computing device 102. In some embodiments, supplemental anchors are identified by sending images to search server 152. Search server 152 may then analyze the image and identify supplemental anchors within the image. In some embodiments, search server 152 may send one or more of the location (eg, image coordinates) or dimensions of any identified object associated with the supplemental anchor to client computing device 102. Client computing device 102 may then update the user interface screen to show annotations that identify supplementary anchors (or associated objects) within the image. In some embodiments, client computing device 102 may track the position of a supplemental anchor (or associated object) within a video stream (e.g., a sequentially acquired image sequence) acquired by camera assembly 112 ( For example, supplemental anchor identification engine 124 may track supplemental anchors identified by search server 152).

Digital supplement selection panel 210 allows the user to select digital supplements for presentation. For example, digital supplement selection panel 210 may include a menu that each includes user actuatable controls associated with a digital supplement. In this example, digital supplement selection panel 210 includes user-actuatable controls 212 and user-actuatable controls 214, each containing information about an associated digital supplement. For example, the user-actuatable control may display one or more of a name (or title), a short description, and an image associated with the digital supplement that may be received from search server 152. Upon actuation of user-actuatable control 212 or user-actuatable control 214, associated digital supplemental content may be presented to the user. Presenting a digital supplement to a user may include causing client computing device 102 to display a user interface screen that includes images, video, text, other content, or a combination thereof from the digital supplement. In some embodiments, the digital supplemental content is displayed as an overlay on the image display panel 208 over the image or camera feed. The digital supplemental content may be three-dimensional augmented reality content.

In some embodiments, presenting the digital supplement includes launching an application (eg, one of the other applications 140) installed on the client computing device 102. Presenting the digital supplement may also include sending a request to a URL associated with the digital supplement. The request may include parameters associated with the digital supplement, such as an identifier for the product or object identified in the image. In some embodiments, images (or other content) from an image content query are passed parameters with the request. Images may also be provided via an API associated with digital supplementation server 172. In some embodiments, client computing device 102 transmits the image to digital capture server 172. In some embodiments, search server 152 may send the images to digital supplementation server 172. For example, in response to a user selecting a digital supplement, client computing device 102 can send an indicator of the selection to search server 152, which in turn sends the image to the corresponding digital supplement server. obtain. Client computing device 102 may also send a URL to a location on search server 152 that digital supplementation server 172 can use to access the images. Beneficially, these implementations may reduce the amount of data that the client computing device needs to transmit.

Digital supplements associated with user actuatable controls 212 may cause information about object 222 to be displayed, such as information from a museum. Digital supplements associated with user actuatable controls 214 may cause information related to the museum to be displayed. For example, the presentation of a digital supplement may cause a stop on a museum tour to be marked as complete and information about the next stop may be displayed.

FIG. 3 is an illustration of an example method 300 of enabling a trigger for digital supplementation, according to embodiments described herein. The method 300 may be performed, for example, by the content crawler 162 of the search server 152 to allow users to access digital supplements based on image content queries.

At operation 302, data specifying digital supplementation is received. The data may identify digital supplements and situations in which digital supplements should be provided. Data specifying digital supplements may be received in a variety of ways. For example, data specifying a digital supplement may be received from a network-accessible resource such as a web page that includes metadata about the digital supplement. Data specifying digital supplements may also be received via an API or form provided by search server 152, for example. Data specifying the digital supplement may also be received from a memory location or data store.

Data regarding the digital supplement may include access data usable by a client computing device to access the digital supplement. For example, the access data may include a URL of the digital supplement and parameters passed to the URL. The access data may also include an identifier of the application and parameters for the application. Data regarding the digital supplement may also include descriptive data regarding the digital supplement. The descriptive data may be used by the client computing device to present information about the digital supplement to the user (eg, on a menu from which the user may select the digital supplement). Such descriptive data may include, for example, a name (or title), a description, a publisher's name, and an image. Data regarding the digital supplement may also include an identifier of the supplement anchor.

At act 304, a data structure instance is generated based on the received data. A data structure may be, for example, a record in a database. The database may be a relational database, and a data structure instance may be linked (eg, via an external key) to one or more records associated with a supplementary anchor.

At operation 306, after generating the data structure instance, searching for the digital supplement with an image content query is enabled. For example, a database field associated with a data structure instance may be set to active so that digital supplement search engine 164 can access and return the associated digital supplement. In some embodiments, triggering a digital supplement may include saving or committing a record to a database. In some embodiments, enabling retrieval of the digital supplement includes enabling triggering of the digital supplement by the client computing device. For example, after the instance is generated, the digital supplement may be returned to, activated or presented by the client computing device in response to the search.

FIG. 4 is an illustration of an example method 400 of enabling a trigger for digital supplementation, according to embodiments described herein. The method 400 may be performed, for example, by the content crawler 162 of the search server 152 to allow users to access digital supplements based on image content queries.

At operation 402, network accessible resources are analyzed. In some embodiments, the network-accessible resource is, for example, a web page presented by digital supplementary server 172. In some embodiments, a set of network accessible resources is analyzed. The set of network-accessible resources may be generated based on submission via a form or API. In some embodiments, the set of network-accessible resources may be generated by crawling other network-accessible resources to identify URLs. This crawling process may be performed recursively.

At operation 404, metadata associated with the digital supplement within the network-accessible resource is identified. In some embodiments, the network-accessible resource may include an indicator of metadata associated with the digital supplement. For example, a network-accessible resource may include a tag that identifies a portion of the network-accessible resource that includes metadata. The tag may be an XML tag with specific types or attributes. The tag can be an HTML tag, such as a script tag that includes a JSON data structure that includes metadata.

At operation 406, a metadata-based digital supplementary data structure instance is generated. Act 406 may be similar to act 304.

At operation 408, an image content query is received. Image content queries may be sent by a client computing device, such as client computing device 102, for example. In some embodiments, the image content query includes images. The image content query may also include textual data that describes the image. For example, the text data may include an identifier of a supplemental anchor within an image captured by a camera assembly of the client computing device. In some embodiments, the image content query also includes other information, such as the location of the client computing device or an identifier of a user account associated with the client computing device.

At operation 410, multiple digital supplementary data structure instances are identified based on the image content query. In some embodiments, supplemental anchors are identified within images provided in an image content query. The supplement anchor may then be used to query an index or database for the associated digital supplement. In some embodiments, other data provided with the query may also be used to identify digital supplements, such as the location of the client computing device or information associated with a user account. In some embodiments, multiple supplemental anchors may be used to identify associated supplemental anchors.

At operation 412, an ordering of the plurality of digital supplementary data structure instances is determined. The ordering may be based on various scores associated with the digital supplements or the relevance of the digital supplements to the image content query. In some embodiments, a relevance score corresponding to the relevance of a digital supplement to an image content query may be used to order multiple digital supplement data structure instances. Content on a Network-Accessible Resource (or a Network-Accessible Resource associated with a Digital Supplement) that links to, link text, or nearby content that links to a Digital Supplement on other Network-Accessible Resources. may be determined from multiple factors, such as one or more of the following:

Scores may also be based on popularity metrics. A reputation metric is an example of a popularity metric. The prestige metric may be based on a combination of how many other network resources link to the digital supplement and the prestige scores of those other network-accessible resources. In some embodiments, the popularity score may be based on how often the digital resource is or has been selected. In some embodiments, the popularity score may correspond to how often a digital resource is selected for image content queries.

The score may be determined or retrieved from a data store or API. In some embodiments, the API is accessed to retrieve the score of the digital supplement. For example, scores may be retrieved from search engines that have determined the relevance and/or popularity of digital resources with respect to search terms based on supplementary anchors.

The plurality of digital supplementary data structures may also be ordered based on frequency of use or recency of use by a particular user (eg, a user of a client computing device). In some embodiments, the plurality of digital supplementary data structures are randomly ordered.

At operation 414, the image content query is answered based on a plurality of digital supplementary data structure instances. For example, information associated with multiple digital supplementary data structure instances may be sent to the client computing device in the order determined in operation 412. In some embodiments, the information includes descriptive data that may be presented within a menu or other type of user interface configured to receive user selections of digital supplements. The information may also include access data that may be used by the client computing device to access or present the digital supplement.

FIG. 5 is an illustration of an example method 500 for searching and presenting digital supplements, according to embodiments described herein. The method 500 may be performed, for example, by the application 122 of the client computing device 102 to identify and access digital supplements based on an image content query.

At act 502, an image-based image content query is sent to a server computing device (eg, search server 152). For example, images may be captured with camera assembly 112 of client computing device 102. The image may also be a stored image, such as an image previously captured by camera assembly 112. In some embodiments, the image content query includes only images. In some embodiments, the image content query includes additional information. For example, the image content query may include information such as the location of client computing device 102 or an identifier of an account associated with a user of client computing device 102. Application 122 may also identify anchors within the image (eg, using supplemental anchor identification engine 124). The image content query may include an identifier (eg, text, number, or other type of identifier) of the identified anchor. In at least some embodiments, the image content query does not include images.

In some embodiments, sending the image content query to the server includes calling an API. In some embodiments, sending the image content query to the server includes calling an API provided by the server. In some embodiments, sending the image content query to the server includes submitting a form using the HTTP protocol (eg, by submitting a GET or POST request).

At operation 504, a response to an image content query that identifies a digital supplement is received. The response may be received from search server 152 via network 190 . The response may include one or more digital supplements identified by search server 152 based on the image content query. For example, the response may include an array of data associated with the digital supplement. In some embodiments, data associated with a digital supplement may include descriptive data that may be used to present digital supplement options for a user to select. For example, descriptive data may include a name, short description, publisher name, and image. The data may also include access data such as the URL and parameters to include in the request via the URL, or the application name and associated parameters. The data may also include the location, coordinates, or dimensions of the supplemental anchor in the image sent in the image content query (eg, if the supplemental anchor is identified by search server 152).

At operation 506, a user interface screen is displayed that includes information associated with the digital supplement. In some embodiments, the user interface screen includes annotations that overlay identified supplemental anchors (eg, based on the provided coordinates). The annotation may provide information about objects within the image associated with the identified supplemental anchor. The annotation may include user actuatable controls that may be activated to present or launch a digital supplement. The user interface screen may also include a digital supplement selection panel that may be used to select from multiple digital supplements identified in the response received in operation 504. In some embodiments, the user interface screen may be generated by a web browser opening a URL specified by the digital supplement. User interface screens may also be generated by another application launched to provide digital supplements.

FIG. 6 is an illustration of an example method 600 for identifying and presenting digital supplements based on images, according to embodiments described herein. The method 600 may be performed, for example, by the application 122 of the client computing device 102 to identify and access digital supplements based on an image content query.

At operation 602, an image is acquired. For example, images may be captured by camera assembly 112 of client computing device 102. In some embodiments, a series of images (ie, video) may be captured by camera assembly 112.

At operation 604, the image-based image content query is sent to a server computing device, such as search server 152. Act 604 may be similar to act 502. In embodiments where a sequence of images is obtained, the image content query may include multiple images or a sequence of images. In some embodiments, the sequence of images may be streamed to the server computing device.

At operation 606, a response to an image content query that identifies a plurality of digital supplements is received. Act 606 may be similar to act 504 described above.

At operation 608, a user interface screen is displayed that includes user actuatable controls for selecting a digital supplement from a plurality of digital supplements. For example, a digital supplemental selection panel may be displayed. The digital supplement selection panel may include a plurality of user actuatable controls each associated with one of a plurality of digital supplements identified in the response. Digital supplement selection may provide user actuatable controls based on the ordering or ranking of digital supplements provided by the server computing device. The digital supplemental selection panel may arrange the user actuatable controls vertically, horizontally, or otherwise. The user actuatable controls may be associated with or include information about the associated digital supplement that the user may consider in determining whether to select the digital supplement. For example, the displayed information may include one or more of the digital supplement's name, description, image, and publisher name.

At operation 610, user input is received to select a digital supplement. User input may be a click with a mouse or other device. User input may also be touch input from a stylus or finger. Another example of user input is near-touch input (eg, holding a finger or pointing device close to the screen). In some embodiments, user input may also include hand gestures, head movements, eye movements, or speech input.

At operation 612, information is provided to resources associated with the selected digital supplement. For example, information about a user of a client computing device may be sent to a server that provides digital supplements (if permission to provide the information is provided). Information may also be provided to applications that provide digital supplements. Various types of information may be provided. For example, the information may include user information such as a user name, user preferences, or location.

The information may also include information associated with an image content query, such as an image or sequence of images. The information may also include an identifier and/or location of one or more supplemental anchors within the image. This information may be used to provide digital supplements to the user. For example, digital supplementary AR content may be created and positioned based on the image.

Information may be sent by client computing device 102 directly to a resource associated with the digital supplement (eg, digital supplement server 172). In some embodiments, information is provided by search server 152 to resources associated with the digital supplement (eg, the client computing device does not need to send as much data). In at least some of these implementations, client computing device 102 may transmit selection information identifying the selected digital supplement to search server 152. Upon receiving the selection and confirming that the user has authorized sharing of the information, search server 152 may send the information to the resource that provides the digital supplement. Client computing device 102 may also prompt users to share information. In some embodiments, search server 152 may determine information to send to a resource based on a digital supplement data structure instance, which may be based on metadata associated with the digital supplement.

At operation 614, the user interface is updated based on the selected digital supplement. Act 614 may be similar to act 506.

7A-7C are schematic illustrations of user interface screens displayed by an embodiment of client computing device 102 to perform image content searches and display digital supplements. FIG. 7A shows a user interface screen 700a. User interface screen 700a includes an image display panel 708 and an information panel 730. In this example, image display panel 708 is displaying an image of a shelf full of wine bottles (such as those that might be found in a store). Image display panel 708 also includes an indicator 740 and an indicator 742. Each of these indicators indicates that the wine bottle shown in the image below the indicator is recognized as a complementary anchor (eg, in this case as a recognized product). Indicator 740 and indicator 742 are examples of user actuatable controls. Within the information panel 730, instructions are provided to "tap on what interests you."

In FIG. 7B, user interface screen 700b is shown after the user activates indicator 740. After activation, annotations 744 from the digital supplement are displayed. Annotations 744 include information regarding wine ratings that may assist the user in selecting a bottle of wine to purchase.

In FIG. 7C, another user interface screen 700c is shown after the user activates indicator 740. User interface screen 700c may be shown instead of or in addition to user interface screen 700b being displayed (e.g., after actuation of annotation 744 or when the user swipes up on information panel 730 of FIG. 7B). It will be done. An expanded information panel 732 is shown in FIG. 7C. Expanded information panel 732 occupies more user interface screen 700c than information panel 730 featured in FIGS. 7A and 7B.

Extended information panel 732 includes a digital supplement selection panel 710 and a digital supplement content display panel 734. Digital supplementary selection panel 710 includes user-actuatable controls 712, user-actuatable controls 714, and (only partially visible) user-actuatable controls 716. In some embodiments, when a user swipes on digital supplemental selection panel 710, additional user-actuatable controls may be displayed. The user actuatable controls of digital supplementary selection panel 710 may be arranged in a ranked order. User actuatable controls 712 are associated with digital supplements for meal pairing. When user actuatable control 712 is activated, a digital supplement may be displayed that displays food and meal pairing information for the selected wine. User actuatable controls 714 are associated with a digital supplement that stores photos. When activated, a photo storage application may be activated to provide the image. Additional information may be saved with the photo, such as identified supplemental anchors.

Digital supplement content display panel 734 may display content from the digital supplement. Digital supplement content display panel 734 may display the default digital supplement or the highest ranking digital supplement associated with the identified supplement anchor. In this example, digital supplemental content display panel 734 includes product information about the product associated with the selected supplemental anchor. In this case, the wine name, rating, country of origin, image, and comments are provided.

8A-8C are schematic illustrations of user interface screens displayed by an embodiment of client computing device 102 to perform image content searches and display digital supplements. In this example, the image content search is based on images of receipts.

FIG. 8A shows a user interface screen 800a. User interface screen 800a includes an image display panel 808 and an information panel 830. In this example, image display panel 808 is displaying an image of a receipt from a restaurant. Image display panel 808 also includes indicators 840, indicators 842, annotations 844, and highlight overlays 846. In this case, indicator 840 is associated with the receipt as a document, and indicator 842 is associated with the particular restaurant named on the receipt. The identified receipt document and the identified restaurant name are both examples of supplemental anchors.

Annotations 844 are associated with digital supplements that provide advanced calculators. In this example, an exemplary tip calculator is included in annotation 844 and superimposed on the image display panel 808 at the appropriate location. In some embodiments, the digital supplement may be selected by default and displayed upon identifying the appropriate supplement anchor. A highlight overlay 846 is overlaid on a portion of the receipt document containing information used by the digital supplement of the advanced calculator.

In this example, the items displayed on information panel 830 relate to the receipt as a document, as if indicator 840 had been activated. In some embodiments, the identified supplementary anchor may be based on, for example, the user's past behavior, the behavior of other users on similar images, the confidence score of the supplementary anchor, or the location or size of the portion of the image to which the supplemental anchor relates. Based on the user's likely relevance or interest. Information panel 830 may then display items related to the highest ranked supplemental anchor, at least in some embodiments. If indicator 842 is activated instead, information panel 830 may include items regarding the particular restaurant.

Here, information panel 830 includes digital supplement selection panel 810. The digital supplement panel includes user-actuatable controls 812 , user-actuatable controls 814 , and user-actuatable controls 816 . In this example, user-actuatable controls 812 are associated with the Tip Calculator Digital Supplement, user-activated controls 814 are associated with the Check Split Digital Supplement, and user-activable controls 816 are associated with the Expense Report Digital Supplement. It is being For example, when user-actuatable control 812 is activated, user interface controls for adjusting parameters of the tip calculator (eg, to adjust percentages) may be displayed.

In FIG. 8B, user interface screen 800b is shown after the user has activated user-actuatable controls 814. In FIG. After activation, an expanded information panel 832 is shown that includes items to help the user calculate how to split the bill. For example, the number of people splitting the bill can be entered to determine the amount each should pay.

In FIG. 8C, user interface screen 800c is shown after the user activates user-actuatable controls 816. After activation, an expanded information panel 834 is shown that includes items to help the user store receipts in an expense report. For example, a user can select an expense report (eg, "Sydney Trip 2018") with which the receipt should be associated. Once the expense report is selected, an image of the receipt may be uploaded to the expense report submission or management system. In some embodiments, all images displayed on image display panel 808 are uploaded. In some embodiments, a portion of the image is uploaded (eg, the image is cropped to include only the receipt).

9A and 9B are schematic illustrations of user interface screens displayed by an embodiment of client computing device 102 to perform image content searches and display digital supplements. In this example, the image content search is based on images of faces.

FIG. 9A shows a user interface screen 900a. User interface screen 900a includes an image display panel 908 and an information panel 930. In this example, image display panel 908 is displaying an image of a face. Here, the face is an example of a supplementary anchor. Information panel 930 includes user-actuatable controls 912 for digital supplementation identified for supplemental anchors within the image (ie, faces). User actuatable controls 912 are associated with a digital supplement for coupling to the glasses.

In FIG. 9B, user interface screen 900b is shown after the user activates user-actuatable controls 912. In FIG. After activation, an expanded information panel 932 is shown that includes items to help the user visually try out the glasses on the face in the image. Here, multiple eyeglass styles are displayed and the user can select a pair to try. When a pair of glasses is selected, AR content 960 is superimposed on image display panel 908. Here, the AR content 960 corresponds to the selected glasses and is sized to match the face in the image. In some embodiments, when a digital supplement for trying on glasses is selected, the image shown on the image display panel 908 is sent to a server that provides the digital supplement, such that the image includes AR content 960. It can be analyzed to determine where and how to place it and what size it should be or whether it should be recommended to try on the glasses.

10A-10C are schematic illustrations of user interface screens displayed by an embodiment of client computing device 102 to perform image content searches and display digital supplements. In this example, the image content search is based on images of furniture in the catalog.

FIG. 10A shows a user interface screen 1000a. User interface screen 1000a includes an image display panel 1008. In this example, the image display panel 1008 is displaying an image of a portion of a page of a furniture catalog. The image display panel also includes an indicator 1040, an indicator 1042, and an indicator 1044. In this example, indicator 1040 is associated with the bed, indicator 1042 is associated with the ornament, and indicator 1044 is associated with the rug. Images of beds, decorations, and rugs in catalogs are examples of supplemental anchors.

In FIG. 10B, user interface screen 1000b is shown after the user selects indicator 1040 (eg, by touching the screen at or near where indicator 1040 is displayed). User interface screen 1000b includes a digital supplement selection panel 1010 and an information panel 1030. Information panel 1030 includes information about the supplemental anchor associated with the selected indicator (eg, product name, description, image).

Digital supplementary selection panel 1010 includes user-actuatable controls 1012 and user-actuatable controls 1014. User actuatable controls 1012 are associated with digital supplements that provide an in-home view. User actuatable controls 1014 are associated with another digital supplement (eg, a digital supplement for posting to a social media site).

In FIG. 10C, user interface screen 1000c is shown after actuation of user actuatable controls 1012. In FIG. User interface screen 1000c includes an image display panel 1008, a digital supplement selection panel 1010, and a reduced information panel 1032. The reduced information panel 1032 may include user actuatable controls that, when activated, can cause the information panel to pop up and display.

Here, the image display panel 1008 displays an image of the room and includes AR content 1060. AR content 1060 includes a 3D model of a bed associated with indicator 1040 overlaid on the image panel. The user may adjust the position of the AR content 1060 in the room to see how the bed fits into the room. In some embodiments, when a digital supplement for an in-home view is selected, the image shown on the image display panel 1008 is sent to a server that provides the digital supplement, such that the image includes AR content 1060. It can be analyzed to determine where and how it should be located and what size it should be. In some embodiments, AR content 1060 may be provided after the image content query.

11A-11C are schematic diagrams of user interface screens displayed by an embodiment of client computing device 102 to perform various image content searches within a store. In this example, the image content search is based on images of products obtained within the store.

FIG. 11A shows a user interface screen 1100a. User interface screen 1100a includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 displays images taken inside the store. Image display panel 1108 also includes an indicator 1140 associated with the vase. The vase displayed on image display panel 1108 is an example of a supplementary anchor. Information panel 1130 displays digital supplements including product information about the vase and functionality for purchasing the vase. The digital supplement may include, for example, a workflow to initiate the purchase of a vase. In this example, the digital supplement is identified based on the image content and the location of the client computing device, thereby identifying the digital supplement published by (or associated with) the store from which the image was obtained. and may be provided as a high ranking result for image content queries when the client computing device is in-store. In some embodiments, a different digital supplement will be provided for the same image if the location of the client computing device changes.

FIG. 11B shows a user interface screen 1100b. User interface screen 1100b includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 is displaying another image captured within the store. Image display panel 1108 also includes indicators 1142 associated with the lugs. The lag displayed on image display panel 1108 is an example of a supplemental anchor. Information panel 1130 displays a digital supplement that includes product information about the rug and the functionality to select a size and purchase the rug. As in FIG. 11A, the digital supplement is identified based on the image content and the location of the client computing device.

FIG. 11C shows a user interface screen 1100c. User interface screen 1100c includes an image display panel 1108 and an information panel 1130. In this example, image display panel 1108 is displaying another image captured within the store. Image display panel 1108 also includes an indicator 1144 associated with the vase. The vase displayed on image display panel 1108 is an example of a supplementary anchor. Information panel 1130 displays a digital supplement containing product information about the vase. Information panel 1130 also includes a coupon indicator 1132 and functionality for redeeming coupons. Redeeming a coupon may include purchasing merchandise at a discounted price from a website associated with the retailer. In some embodiments, a coupon code is presented that can be used to secure a discount during checkout. Similar to FIGS. 11A and 11B, digital supplements are identified based on image content and the location of the client computing device.

12A-12C are schematic diagrams of user interface screens displayed by embodiments of client computing device 102 during various image content searches. In this example, the image content search is based on images of movie posters (eg, as may be obtained at a movie theater).

FIG. 12A shows a user interface screen 1200a. User interface screen 1200a includes an image display panel 1208. In this example, image display panel 1208 is displaying an image of a movie poster. Image display panel 1208 also includes an indicator 1240 associated with the movie poster identified within the image. A movie poster is an example of a supplemental anchor. Indicator 1240 may include a user actuatable control that, when activated, displays a digital supplement or a menu for selecting a digital supplement.

FIG. 12B shows a user interface screen 1200b. Image display panel 1208 also includes a preview digital supplement 1242 associated with the movie poster identified within the image. For example, preview digital supplement 1242 may be shown after actuation of indicator 1240 (of FIG. 12A). Preview digital supplement 1242 may superimpose images or footage from the movie associated with the identified movie poster onto the image of the movie poster.

FIG. 12C shows a user interface screen 1200c. Image display panel 1208 also includes rating indicator 1244 and rating indicator 1246. Rating indicator 1244 and rating indicator 1246 may be generated by one or more digital supplements in response to an image content query that includes a movie poster. The digital supplement may overlay rating information within the image, for example, for a movie associated with a movie poster. Rating indicator 1244 and rating indicator 1246 may include user actuatable controls that, when activated, cause the rating and additional information about the associated movie to be indicated.

FIG. 13 illustrates a computer device 1300 and a mobile computer device 1350 that may be used with the techniques described herein (e.g., to implement client computing device 102, search server 152, and digital supplement server 172). Give an example. Computing device 1300 includes a processor 1302, memory 1304, storage 1306, a high speed interface 1308 for connecting to memory 1304 and high speed expansion port 1310, and a low speed interface for connecting to low speed bus 1314 and storage 1306. 1312. Each of the components 1302, 1304, 1306, 1308, 1310 and 1312 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 1302 includes instructions stored in memory 1304 or storage 1306 for displaying graphical information for a GUI on an external input/output device, such as a display 1316 coupled to high-speed interface 1308. Instructions can be processed for execution within computing device 1300. In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and types of memory, as desired. Also, multiple computing devices 1300 may be connected, each device providing a portion of the required operation (eg, as a server bank, group of blade servers, or multiprocessor system).

Memory 1304 stores information within computing device 1300. In one embodiment, memory 1304 is one or more volatile memory units. In another embodiment, memory 1304 is one or more non-volatile memory units. Memory 1304 may also be another form of computer readable storage medium, such as a magnetic or optical disk.

Storage device 1306 may provide mass storage for computing device 1300. In one embodiment, the storage device 1306 is a floppy disk drive, hard disk drive, optical disk drive, or tape drive, flash memory or other similar solid state memory device, or within a storage area network or other configuration. may be or may include a computer-readable storage medium such as an array of devices including apparatus. A computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable storage medium or machine-readable medium, such as memory 1304 , storage device 1306 , or memory on processor 1302 .

A high-speed controller 1308 manages bandwidth-intensive operations for the computing device 1300, while a low-speed controller 1312 manages less bandwidth-intensive operations. Such functional assignments are exemplary only. In one embodiment, high speed controller 1308 is coupled to memory 1304, to display 1316 (e.g., via a graphics processor or accelerator), and to high speed expansion port 1310 that can accept various expansion cards (not shown). has been done. In this embodiment, low speed controller 1312 is coupled to storage device 1306 and low speed expansion port 1314. Low-speed expansion ports, which may include a variety of communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), can be used to connect keyboards, pointing devices, scanners, etc. via network adapters. , or to one or more input/output devices, such as a networking device such as a switch or router.

Computing device 1300 may be implemented in several different forms, as illustrated. For example, it may be implemented multiple times as a standard server 1320 or in a group of such servers. Also, it may be implemented as part of rack server system 1324. Additionally, it can be implemented as a personal computer, such as a laptop computer 1322. Alternatively, components from computing device 1300 may be combined with other components within a mobile device (not shown), such as device 1350. Each such device may include one or more of the computing devices 1300, 1350, and the entire system may be comprised of multiple computing devices 1300, 1350 in communication with each other.

Computing device 1350 includes a processor 1352, memory 1364, input/output devices such as a display 1354, a communication interface 1366, and a transceiver 1368, among other components. Device 1350 may also include storage, such as a microdrive or other device, to provide additional storage. Each of the components 1350, 1352, 1364, 1354, 1366 and 1368 are interconnected using various buses, with some of the components on a common motherboard or on other method, it can be installed.

Processor 1352 can execute instructions within computing device 1350, including instructions stored in memory 1364. The processor may be implemented as a chipset of chips that includes separate analog and digital processors. The processor may provide coordination of other components of device 1350, such as control of a user interface, applications executed by device 1350, and wireless communications by device 1350.

Processor 1352 may communicate with a user via a control interface 1358 and a display interface 1356 coupled to display 1354. Display 1354 may be, for example, a Thin-Film-Transistor Liquid Crystal display (TFT), a Light Emitting Diode (LED), an Organic Light Emitting Diode (OLED) display, or other suitable display technology. Display interface 1356 may include suitable circuitry to drive display 1354 to present graphics and other information to the user. Control interface 1358 may receive commands from a user, convert and submit them to processor 1352. Additionally, an external interface 1362 may be provided in communication with processor 1352 to enable close range communication between device 1350 and other devices. External interface 1362 may be provided for wired communication in some embodiments or wireless communication in other embodiments, for example, and multiple interfaces may also be used.

Memory 1364 stores information within computing device 1350. Memory 1364 may be implemented as one or more of one or more computer-readable storage media, one or more volatile memory units, or one or more non-volatile memory units. Expansion memory 1374 may also be provided and connected to device 1350 via expansion interface 1372, which may include, for example, a Single In-Line Memory Module (SIMM) card interface. Such expanded memory 1374 can provide additional storage space for device 1350 or store applications or other information for device 1350. In particular, expanded memory 1374 may include instructions to perform or supplement the processes described above, and may also include secure information. Thus, for example, expanded memory 1374 may be provided as a security module for device 1350 and may be programmed with instructions to enable secure use of device 1350. Additionally, secure applications may be provided via the SIMM card with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable storage medium or machine-readable medium, such as memory 1364, expansion memory 1374, or memory on processor 1352, which may be received via transceiver 1368 or external interface 1362, for example.

Device 1350 may communicate wirelessly via communication interface 1366, which may optionally include digital signal processing circuitry. The communication interface 1366 can communicate under various modes or protocols such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. may provide communications. Such communication may occur via radio frequency transceiver 1368, for example. Additionally, short range communications may occur, such as using Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 1370 may provide additional navigation and location-related wireless data to the device 1350, which may be used as needed by applications running on the device 1350. .

Device 1350 may also communicate audibly using audio codec 1360, which may receive spoken information from a user and convert it into usable digital information. Audio codec 1360 may also generate audible sound for the user, such as, for example, via a speaker in the handset of device 1350. Such sounds may include sounds from voice calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 1350. .

Computing device 1350 may be implemented in several different forms, as shown. For example, it may be implemented as a mobile phone 1380. Also, it may be implemented as part of a smartphone 1382, personal digital assistant, or other similar mobile device.

Various embodiments of the systems and techniques described herein include digital electronic circuits, integrated circuits, specially designed ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and/or the like. It can be realized by a combination of. These various implementations may be specialized or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. may include implementation in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that obtains the information.

These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and are written in high-level procedural and/or object-oriented programming languages and/or in assembly. /Can be implemented in machine language. As used herein, the terms "machine-readable medium" and "computer-readable storage medium" include a machine-readable medium that receives machine instructions as machine-readable signals. Refers to any computer program product, apparatus, and/or device (eg, magnetic disk, optical disk, memory, programmable logic device (PLD)) that is used to provide instructions and/or data to a programmable processor. The term "machine readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide user interaction, the systems and techniques described herein use display devices (LEDs (light emitting diodes) or OLEDs (organic LEDs) or LCDs (liquid crystal displays) to display information to the user. The computer may be implemented on a computer that has a monitor/screen) and a keyboard and pointing device (eg, a mouse or trackball) through which a user may provide input to the computer. Other types of devices may also be used to provide interaction with the user, for example, the feedback provided to the user may include any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form including acoustic input, voice input, or tactile input.

The systems and techniques described herein include back-end components (e.g., as data servers), or middleware components (e.g., as application servers), or front-end components (e.g., user a client computer having a graphical user interface or a web browser that can interact with embodiments of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. may be implemented in a computing system including. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact via a communications network. The relationship between clients and servers is created by computer programs running on their respective computers and having a client-server relationship with each other.

In some embodiments, the computing device shown in FIG. 13 includes sensors for interfacing with an AR headset/HMD device 1390 to create an augmented environment for viewing inserted content within a physical space. may be included. For example, one or more sensors included in computing device 1350 shown in FIG. 13 or other computing device may provide input to AR headset 1390, or may provide input to AR space in general. . Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. Computing device 1350 can determine the absolute position and/or detected rotation of the computing device in AR space using the sensor, and the absolute position and/or detected rotation is connected to AR space. Can be used as input. For example, computing device 1350 may be incorporated into the AR space as a virtual object such as a controller, laser pointer, keyboard, weapon, etc. When embedded in an AR space, positioning of a computing device/virtual object by a user may allow the user to position the computing device to view the virtual object in a particular way within the AR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were an actual laser pointer. A user can move the computing device from side to side, up and down, and in a circular motion, allowing the user to use the device in a manner similar to using a laser pointer. In some embodiments, a user may use a virtual laser pointer to aim at a target.

In some embodiments, one or more input devices included in or coupled to computing device 1350 may be used as input to the AR space. Input devices include a touch screen, keyboard, one or more buttons, trackpad, touch pad, pointing device, mouse, trackball, joystick, camera, microphone, and earphones or earphones with input capabilities. (buds), game controllers, or other connectable input devices. When a computing device is embedded in an AR space, a user interacting with input devices included in the computing device 1350 can cause certain actions in the AR space.

In some embodiments, the touch screen of computing device 1350 may be rendered as a touchpad within the AR space. A user may interact with the touch screen of computing device 1350. The interaction is rendered as movement on a touchpad rendered in AR space, for example in an AR headset 1390. The rendered motion can control virtual objects within the AR space.

In some embodiments, one or more output devices included in computing device 1350 can provide output and/or feedback to a user of AR headset 1390 within an AR space. Output and/or feedback may be visual, tactile, or audio. Outputs and/or feedback may include vibrations, turning on and off or flashing and/or emitting one or more lights or strobes, sounding an alarm, sounding a chime, playing a musical piece, and playing an audio file. may include, but are not limited to, musical performances. Output devices may include, but are not limited to, vibrating motors, vibrating coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

In some embodiments, computing device 1350 may appear as a separate object in a computer-generated 3D environment. User interactions with computing device 1350 (e.g., rotating, shaking, touching the touch screen, swiping a finger across the touch screen) are interpreted as interactions with objects within the AR space. can be done. In the example of a laser pointer in AR space, computing device 1350 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates the computing device 1350, the user in the AR space sees movement of the laser pointer. A user receives feedback from interactions with computing device 1350 in an AR environment on computing device 1350 or AR headset 1390. A user's interaction with a computing device may be translated into interaction with a user interface generated in an AR environment for the controllable device.

In some embodiments, computing device 1350 may include a touch screen. For example, a user can interact with a touch screen to interact with a user interface for a controllable device. For example, a touch screen may include user interface elements such as sliders that can control characteristics of a controllable device.

Computing device 1300 represents various forms of digital computers and devices, including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. It is intended that Computing device 1350 is intended to represent various forms of mobile devices, such as personal digital assistants, cell phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions depicted herein are exemplary only and are not intended to limit the embodiments of the invention described and/or claimed herein. is not intended.

Several embodiments have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the specification.

Additionally, the logic flows illustrated in the figures do not require the particular order shown or sequential order to achieve desirable results. In addition, other steps may be provided or steps may be omitted from the described flow, and other components may be added to or modified from the described system. Can be removed. Accordingly, other embodiments are within the scope of the following claims.

Although certain features of the illustrated embodiments have been described as described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the scope of the embodiments. It is to be understood that they are presented by way of example only and not as a limitation, and that various changes in form and detail may be made. Any portions of the apparatus and/or methods described herein may be combined in any combination, except in mutually exclusive combinations. The embodiments described herein may include various combinations and/or subcombinations of the features, components, and/or features of different described embodiments.

Claims (18)

A computer-implemented method, the method comprising:
a search server receiving a web page from a digital supplementary server ;
the search server obtains an image supplement anchor and a digital supplement associated with the image supplement anchor by analyzing metadata of the web page ;
the search server generating a data structure instance associating the digital supplement with the image supplement anchor , the digital supplement being virtual content ;
storing the data structure instance in a database of the search server ;
the search server receiving from a client computing device an image content query that includes an image of a physical object, the database including a plurality of other data structure instances, and the database including a plurality of other data structure instances; each includes a digital supplement and one or more image supplement anchors associated with each digital supplement, the method further comprising:
the search server , in response to receiving the image content query , obtains digital supplements associated with image supplement anchors identified from images included in the image content query;
transmitting the obtained digital supplement to the client computing device to cause the client computing device to display the digital supplement as augmented reality content overlaid on an image of the physical object; Method. 2. The computer-implemented method of claim 1 , further comprising crawling a plurality of web pages including the web page. 3. The computer-implemented method of claim 1 or 2 , further comprising associating the data structure instance with a popularity score of the web page. 4. The computer-implemented method of any of claims 1-3 , further comprising associating the data structure instance with a relevance score of the image supplement anchor based on a web page. 5. The computer-implemented method of any preceding claim, wherein the metadata includes a name, description, image, and uniform resource locator. 6. The computer-implemented method of claim 5 , wherein the metadata includes an application identifier. Sending the obtained digital supplements to the client computing device includes sending a list of digital supplements, the list including the digital supplements from the data structure instance and the plurality of other data structures. 4. The computer-implemented method of claim 3 , including digital supplements from one of the instances, ordered based on the popularity score . Sending the obtained digital supplements to the client computing device includes sending a list of digital supplements, the list including digital supplements from the data structure instance and the plurality of other data structure instances. 5. The computer-implemented method of claim 4, wherein the computer-implemented method includes a digital supplement from one of the digital supplements, and is ordered based on the relevance score. A computing device,
at least one processor;
a memory storing instructions, the instructions, when executed by the at least one processor, cause the computing device to:
receive a web page from a digital supplementary server ;
obtaining an image supplement anchor and a digital supplement associated with the image supplement anchor by analyzing metadata of the web page ;
generating a data structure instance that associates the digital supplement with the image supplement anchor, the digital supplement being virtual content;
storing the data structure instance in a database;
an image content query including an image of a physical object is received from a client computing device, the database including a plurality of other data structure instances, each of the plurality of other data structure instances including a digital supplement and a respective one or more image supplement anchors associated with the digital supplement;
in response to receiving the image content query , retrieving a digital supplement associated with an image supplement anchor identified from an image included in the image content query;
A computing device that causes the acquired digital supplement to be transmitted to the client computing device to cause the client computing device to display the digital supplement as augmented reality content overlaid on an image of the physical object. The instructions further cause the computing device to:
10. The computing device of claim 9, configured to crawl a plurality of web pages including the web page. The instructions further cause the computing device to:
11. The computing device of claim 9 or 10, further comprising associating the data structure instance with a popularity score of the web page. The instructions further cause the computing device to:
A computing device according to any of claims 9 to 11, further causing associating the data structure instance with a relevance score of the image supplement anchor based on a web page. A computing device according to any of claims 9 to 12 , wherein the metadata includes a name, a description, an image, and a uniform resource locator. 14. The computing device of claim 13 , wherein the metadata includes an application identifier. Sending the obtained digital supplements to the client computing device includes sending a list of digital supplements, the list including digital supplements from the data structure instance and the plurality of other data structure instances. 12. The computing device of claim 11 , wherein the computing device includes a digital supplement from one of the digital supplements and is ordered based on the popularity score . Sending the obtained digital supplements to the client computing device includes sending a list of the digital supplements, the list including the digital supplements from the data structure instance and the plurality of other data structures. and a digital supplement from one of the instances, ordered based on the relevance score. A computing device,
at least one processor;
a memory storing instructions, the instructions, when executed by at least one processor, causing the computing device to perform the method of any of claims 1 to 8 . . 9. A computer program product comprising instructions, said instructions, when executed by one or more processors, causing said one or more processors to perform the method of any of claims 1 to 8 . program.

Images