JP2022110057A - Digital supplement association and retrieval for visual search - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system and a method for identification and retrieval of a content for visual search.

SOLUTION: A method includes receiving data that specifies digital supplement. The data may identify the digital supplement and a supplement anchor for associating the digital supplement with an image content. The method also includes generating a data structure instance that specifies the digital supplement and the supplement anchor and, after generating the data structure instance, enabling triggering of the digital supplement by an image based at least on storing the data structure instance in a database that includes a plurality of other data structure instances. The other data structure instances can each specify digital supplement and one or more supplement anchors.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2022,JPO&INPIT

Description

REFERENCE TO RELATED APPLICATIONS This application is entitled "DIGITAL SUPPLEMENT ASSOCIATION AND
No. 16/014,520, entitled RETRIEVAL FOR VISUAL SEARCH,” to which priority is claimed, the disclosure of which is hereby incorporated by reference in its entirety.

Background Mobile computing devices, such as smartphones, often include a camera. These cameras can be used to capture images of entities in the environment surrounding the computing device. Various types of content or experiences related to these entities may be available to users via such mobile computing devices.

Overview This disclosure describes systems and methods for digital supplemental association and retrieval for visual search. For example, the systems and techniques described herein can be used to provide digital supplements, such as augmented reality (AR) content or experiences, that are responsive to visual searches. A visual search may be based, for example, on images or entities identified within images. Digital supplementation can include, for example, providing information or functionality associated with the image.

One aspect is a computer-implemented method that includes receiving data specifying a digital supplement, the data identifying the digital supplement and a supplemental anchor for associating the digital supplement with image content. The method also includes creating a data structure instance that specifies the digital supplement and the supplemental anchor. The method further includes, after generating the data structure instance, enabling triggering of digital capture with the image based on storing at least the data structure instance in a database containing a plurality of other data structure instances. Each of the other data structure instances specify a digital supplement and one or more supplemental anchors.

Another aspect is a computing device that includes at least one processor and a memory that stores instructions. The instructions, when executed by the at least one processor, cause the computing device to receive data specifying a digital supplement, the data including a digital supplement and a supplement for associating the digital supplement with image content. Receive anchors and contextual information. The instructions also cause the computing device to generate a data structure instance that specifies the digital supplement, the supplemental anchor, and the contextual information. The instructions further cause the computing device to trigger digital supplementation with the image based at least upon generating the data structure instance and then storing the data structure instance in a database containing a plurality of other data structure instances. Activate. Each such other data structure instance specifies a digital supplement and one or more supplemental anchors.

Yet another aspect is a computer-implemented method that includes receiving an image content query from a computing device and identifying a supplemental anchor based on the image content query. The method also includes generating an ordered list of digital supplements based on the identified supplemental anchors and transmitting the ordered list to the client computing device.

The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

1 is a block diagram of a system according to an exemplary embodiment; FIG. 3 is a third-person view of an exemplary physical space in which the client computing device embodiment of FIG. 1 is accessing a digital supplement. FIG. 4 is a diagram of an exemplary method of enabling triggering of digital capture, according to embodiments described herein. FIG. 4 is a diagram of an exemplary method of enabling triggering of digital capture, according to embodiments described herein. FIG. 3 is an illustration of an exemplary method of searching and presenting digital supplements according to embodiments described herein. FIG. 3 is an illustration of an exemplary method of identifying and presenting digital supplements based on images, according to embodiments described herein. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic diagram of a user interface screen displayed by an embodiment of the client computing device of FIG. 1 for conducting image content searches and displaying digital supplements; FIG. 2 is a schematic illustration of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for conducting various image content searches within a store; FIG. 2 is a schematic illustration of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for conducting various image content searches within a store; FIG. 2 is a schematic illustration of a user interface screen displayed by the embodiment of the client computing device of FIG. 1 for conducting various image content searches within a store; FIG. 2A-2D are schematic illustrations of user interface screens displayed by embodiments of the client computing device of FIG. 1 during various image content searches; 2A-2D are schematic illustrations of user interface screens displayed by embodiments of the client computing device of FIG. 1 during various image content searches; 2A-2D are schematic illustrations of user interface screens displayed by embodiments of the client computing device of FIG. 1 during various image content searches; 1 is a schematic diagram of an example computing device and mobile computing device that may be used to implement the techniques described herein; FIG.

DETAILED DESCRIPTION This disclosure describes technical improvements that simplify the identification and presentation of digital supplements based on image content. Certain embodiments of the technology described herein generate an index of digital supplements associated with particular types of image content and their digital content in response to image content queries received from client computing devices. Provide supplements. This index allows users to access relevant digital supplements provided by network-accessible resources (eg, web pages) located around the world. This can provide a functional data structure that allows for more efficient retrieval of information.

For example, a client computing device, such as a smart phone, may obtain images of supplemental anchors, such as entities. The client computing device may then send an image-based image content query to the server computing device to retrieve digital supplements associated with the identified supplemental anchors. In some embodiments, the supplemental anchor is based on the physical environment around the client computing device, and the digital supplement is virtual content that can supplement the user's experience in the physical environment.

The image content query may include images or data determined from the image (eg, indicators of identified supplemental anchors). An example of data determined from an image is text extracted from the image using, for example, optical character recognition. Other examples of data extracted from an image include barcodes, QR codes, etc. in the image, an identifier or description of an entity, product, or type of entity identified in the image.

Entities, products, or entity types can be identified in images using, for example, a neural network system, such as a convolutional neural network system. An entity, product, or entity type identifier or description may include metadata or references to records in a database associated with the entity, product, or entity type. Non-limiting examples of entities include buildings, works of art, products, books, posters, photographs, catalogs, signs, documents (eg, business cards, receipts, coupons, catalogs), people, and body parts.

Various types of digital supplements associated with supplemental anchors may be available. Digital supplements may be provided by network-accessible resources, such as web pages available on the Internet. What is needed is a way to locate and provide these digital supplements in response to image content queries. Certain embodiments generate and maintain an index of digital supplements associated with entities for use in responding to image content queries. The index may, for example, crawl a network-accessible resource to determine whether the network-accessible resource includes or provides digital supplements, and determine supplemental anchors associated with those digital supplements. , can be added.

For example, the network-accessible resource may include metadata that identifies a supplemental anchor (eg, text, code, entity, or type of entity) with which the digital supplement is associated. Metadata may be included by a network-accessible resource in response to a Hypertext Transfer Protocol (HTTP) request. Metadata may be provided in various formats, such as extensible markup language (XML), JavaScript Object Notation (JSON), or other formats.

Metadata for Digital Supplements may include one or more of the following: type indicator, anchor indicator, name, description, snippets of content (i.e., excerpts or previews of portions of content), associated images, digital supplements. and an identifier of the application associated with the digital supplement. Metadata may also include information about the publisher of the digital supplement. For example, metadata may include one or more of a publisher name, a publisher description, and an image or icon associated with the publisher. In some embodiments, the metadata includes contextual information associated with providing the digital supplement. For example, metadata may also include conditions associated with providing the digital supplement or accessing the digital capture (eg, geographic conditions, required applications).

The identified digital supplement can be added to the index stored in memory. In at least some embodiments, the associated supplemental anchor for the digital supplement is used as the key to the index. Digital supplements may also be associated with various scores. For example, a digital supplement may indicate how many other links were found (e.g., while crawling network-accessible resources) that refer to the digital supplement or network-accessible resources associated with the digital supplement, and may be associated with a prestige score based on the prestige of the network-accessible resource that provides those links. As another example, a digital supplement may be associated with one or more relevance scores corresponding to the relevance of the digital supplement (or associated network-accessible resource) to a particular anchor. Relevance scores can also be associated with keywords or subjects. The relevance score is one of the content of digital supplements, the content of network-accessible resources, the content of sites that link to network-accessible resources, and the content (e.g., text) of links to network-accessible resources or may be determined based on more than one.

FIG. 1 is a block diagram illustrating system 100 in accordance with an exemplary embodiment. System 100 can associate digital supplements with entities or entity types and can retrieve digital supplements in response to visual searches. A visual search is a search based on image content. For example, visual searches can be performed based on image content queries. Image content queries are queries based on images or other image content. For example, an image content query may include images. In some embodiments, image content queries may include text or data based on images. For example, text or data may be generated by recognizing one or more entities within an image. Some image content queries do not include images (eg, image content queries may include only data or text generated from images). In one embodiment, system 100 includes client computing device 102 , search server 152 , and digital supplement server 172 . Also shown is network 190 with which client computing device 102, search server 152, and digital supplement server 172 may communicate.

Client computing device 102 may include processor assembly 104 , communication module 106 , sensor system 110 , and memory 120 . Sensor system 110 may include various sensors such as camera assembly 112 , inertial motion unit (IMU) 114 , and global positioning system (GPS) receiver 116 . Embodiments of sensor system 110 may also include, for example, optical sensors, audio sensors, image sensors, distance and/or proximity sensors, contact sensors such as capacitive sensors, timers, and/or other sensors and/or sensors. Other sensors with different combinations may be included. In some embodiments, client computing device 102 is a mobile device (eg, smart phone).

Camera assembly 112 captures images or videos of the physical space around client computing device 102 . Camera assembly 112 may include one or more cameras. Camera assembly 112 may also include an infrared camera. Images captured by camera assembly 112 may be used to capture anchors and identify to form image content queries.

In some embodiments, the images acquired by camera assembly 112 are also captured by the client computing device in a physical space, such as an interior space, based on a representation of the physical space received from memory 120 or an external computing device. 102 can be used to determine the position and orientation. In some embodiments, a representation of a physical space may include visual features of the physical space (eg, features extracted from images of the physical space). Such representations may also be associated with those features and used by a visual positioning system to determine a position and/or location within a physical space based on one or more images of the physical space. It may contain decision data. The representation may also include a three-dimensional model of at least some structure within the physical space. In some embodiments, the representation does not include a 3D model of physical space.

IMU 114 may detect motion, motion, and/or acceleration of the client computing device. IMU 114 may include a variety of different types of sensors such as, for example, accelerometers, gyroscopes, magnetometers, and other such sensors. The orientation of client computing device 102 may be detected and tracked based on data provided by IMU 114 or GPS receiver 116 .

GPS receiver 116 may receive signals emitted by GPS satellites. The signal contains the time and position of the satellite. Based on receiving signals from several (eg, at least four) satellites, GPS receiver 116 may determine the global position of client computing device 102 .

Memory 120 may include applications 122 , other applications 140 , and device positioning system 142 . Other applications 140 include any other applications installed or otherwise available for execution on client computing device 102 . In some embodiments, application 122 may cause one of the other applications 140 to provide the digital supplement. In some embodiments, some digital supplements may only be available if other applications 140 include the particular application associated with or required to provide the digital supplement. .

Device positioning system 142 determines the location of client computing device 102 . Device positioning system 142 may use sensor system 110 to determine the position and orientation of client computing device 102 globally or within physical space. In one embodiment, device positioning system 142 determines the position of client computing device 102 based, for example, on cellular triangulation.

In some embodiments, the client computing device 102 compares the images acquired by the camera assembly 112 (or features extracted from those images) to known arrangements of features within a representation of the physical space, and may include a visual positioning system that determines the six degrees of freedom pose (eg, position and orientation) of the client computing device 102 in physical space.

Applications 122 may include supplemental anchor identification engine 124 , digital supplemental search engine 126 , digital supplemental presentation engine 128 , and user interface engine 130 . Certain embodiments of application 122 may include fewer, additional, or other components.

Supplemental anchor identification engine 124 identifies supplemental anchors based on images acquired by camera assembly 112, for example. In one embodiment, supplemental anchor identification engine 124 analyzes images to identify text. That text can then be used to identify the anchor. For example, text can be mapped to nodes in the knowledge graph. For example, text may be recognized as the name of an entity, such as a person, place, product, building, artwork, movie, or other type of entity. In some embodiments, the text may be recognized as phrases commonly associated with or describing a particular entity. For example, the text can then be recognized as an anchor associated with that particular entity.

In some embodiments, supplemental anchor identification engine 124 identifies one or more codes, such as barcodes, QR codes, or another type of code, within the image. The codes can then be mapped to supplemental anchors.

Supplemental anchor identification engine 124 may include a machine learning module capable of recognizing at least some types of entities in images. For example, a machine learning module may include a neural network system. A neural network is a computational model used in machine learning and consists of nodes organized in layers with weighted connections. Training a neural network uses multiple training examples, each example being an input and a desired output, and over a series of iteration rounds the neural network can provide the desired output for a given input. Determine the weight value of the connection between layers that increases the resilience. During each training round, the weights are adjusted to accommodate erroneous output values. Once trained, neural networks can be used to predict outputs based on provided inputs.

In some embodiments, the neural network system includes a convolutional neural network (CNN). A convolutional neural network (CNN) is a neural network in which at least one of the layers of the neural network is a convolutional layer. A convolutional layer is a layer whose layer values are computed based on applying a kernel function to a subset of the previous layer's values. Training the neural network may include adjusting kernel function weights based on training examples. Typically, the same kernel function can be used to compute each value in a convolutional layer. Therefore, the weights that must be learned while training a convolutional layer are fully connected layers in a neural network (e.g., each value in one layer is adjusted independently of each value in the previous layer). layers calculated as weighted combinations). Training and using a convolutional layer may require less memory, processor cycles, and time than an equivalent fully-connected layer because the convolutional layer usually has less weight.

After the supplemental anchor identification engine 124 recognizes an entity or entity type within an image, a textual description of the entity or entity type may be generated. Additionally, entities or entity types can be mapped to supplemental anchors. In some embodiments, a supplemental anchor is associated with one or more digital supplements.

In some embodiments, the supplemental anchor identification engine 124 determines confidence scores for recognized anchors. A higher confidence score indicates that content from the image (e.g., image, extracted text, barcode, QR code) is associated with the determined anchor than if a lower confidence score were determined. It can be shown that there is a high possibility that

Although the example of FIG. 1 shows supplemental anchor identification engine 124 as a component of application 122 on client computing device 102 , certain embodiments include a supplemental anchor identification engine on search server 152 . For example, client computing device 102 may send images captured by camera assembly 112 to search server 152, and search server 152 may identify supplemental anchors in the images.

In one embodiment, the supplemental anchor identification engine 124 identifies potential supplemental anchors. For example, the supplemental anchor identification engine 124 may identify (recognize) various entities within the image. Identifiers of recognized entities can then be sent to search server 152, which can determine whether any of the entities are associated with any supplemental anchors. In some embodiments, search server 152 may use the identified entity as contextual information even if the identified entity is not a supplemental anchor.

A digital supplement search engine 126 searches for digital supplements. For example, digital supplement search engine 126 may search for digital supplements associated with supplemental anchors identified by supplemental anchor identification engine 124 . In some embodiments, digital supplement search engine 126 retrieves digital supplements from search server 152 or digital supplement server 172 .

For example, after a supplemental anchor is identified, digital supplement search engine 126 may search for one or more digital supplements associated with the identified supplemental anchor. Digital supplemental search engine 126 may generate image content queries that include images (or identifiers of supplemental anchors or entities within images) and may transmit the image content queries to search server 152 . Image content queries may also include contextual information such as the location of client computing device 102 . In some embodiments, data associated with the digital supplement, such as a name, image, or description, is retrieved and presented to the user (eg, by user interface engine 130). If multiple digital supplements are presented, the user may select one of the digital supplements via the user interface generated by user interface engine 130 .

The digital supplement presentation engine 128 presents or causes the digital supplement to be presented on the client computing device 102 . In some embodiments, the digital supplemental presentation engine 128 causes the client computing device to start one of the other applications 140 . In some embodiments, the digital supplemental presentation engine 128 causes the information or content to be displayed. For example, digital supplement presentation engine 128 may cause user interface engine 130 to generate a user interface that includes information or content from the digital supplement displayed by client computing device 102 . In one embodiment, digital supplement presentation engine 128 is triggered by digital supplement search engine 126 searching for digital supplements. Digital supplement presentation engine 128 may then trigger display device 108 to display the content associated with the digital supplement. In some embodiments, digital supplement presentation engine 128 causes the digital supplement to be displayed at a different time than when digital supplement search engine 126 searches for the digital supplement. For example, a digital supplement may be retrieved in response to an image content query at a first time, and a digital supplement may be presented at a second time. For example, the digital supplement responds to image content queries based on images of household items or furniture from the catalog or store at a first time (eg, while the user is viewing the catalog or at the store). can be retrieved by A digital supplement containing AR content of household items or furniture may be presented at a second time (eg, while the user is in a room where the household items or furniture may be installed).

User interface engine 130 generates a user interface. User interface engine 130 can also cause client computing device 102 to display the generated user interface. The generated user interface may display information or content from the digital supplement, for example. In one embodiment, user interface engine 130 generates a user interface that includes a plurality of user-actuatable controls each associated with a digital supplement. For example, a user can select a user-actuatable One of the controls can be activated.

Search server 152 is a computing device. Search server 152 may respond to search requests, such as image content queries. The response may include one or more digital supplements potentially related to the image content query. In one embodiment, search server 152 includes memory 160 , processor assembly 154 , and communication module 156 . Memory 160 may include content crawler 162 , digital supplemental search engine 164 , and digital supplemental data store 166 .

Content crawler 162 may crawl network-accessible resources to identify digital supplements. For example, content crawler 162 may access web pages accessible over the Internet, such as web pages provided by digital supplement server 172 . Crawling the network-accessible resource may include requesting the resource from a web server and analyzing at least a portion of the resource. A digital supplement may be identified based on metadata provided by a network-accessible resource, such as XML or JSON data that provides information about the digital supplement. In some embodiments, the crawler identifies network-accessible resources based on extracting links from previously crawled network-accessible resources. The content crawler 162 may also identify network-accessible resources to crawl based on receiving user-submitted input. For example, users may submit URLs (or other information) to network-accessible resources, including digital supplements, via web forms or application programming interfaces (APIs). In some embodiments, content crawler 162 generates an index of the identified digital supplements. The content crawler 162 may also generate scores associated with the digital supplement, such as relevance scores or popularity (fame) scores.

The digital supplement search engine 164 receives search queries and generates responses that may include one or more potentially relevant digital supplements. For example, digital supplemental search engine 164 may receive image content queries from client computing device 102 . Image content queries may include images. The digital supplement search engine 164 may identify supplemental anchors within the images and may identify associated or potentially related digital supplements based on the identified supplemental anchors. Digital supplement search engine 164 may send a response to client computing device 102 that includes the digital supplement or information that can be used to access the digital supplement. In some embodiments, the digital supplement search engine 164 may return information associated with multiple digital supplements. For example, a list of digital supplements can be included in responses to queries. The list may be ordered based on relevance to supplemental anchors, popularity, or other characteristics of digital supplements.

An image content query may include, for example, an image captured by camera assembly 112 or text or other data associated with an image captured by camera assembly 112 . The image content query may also include other information such as the location of the client computing device 102 or the identifier of the user of the client computing device 102 . In some embodiments, search server 152 retrieves the client from the user identifier (eg, if the user has enabled location services on client computing device 102 that associates information about the user's location with the user's account). A probable location of computing device 102 may be determined.

The digital supplemental data storage unit 166 stores information regarding digital supplementation. In some embodiments, digital supplemental data store 166 includes an index of digital supplements. For example, the index may be generated by content crawler 162 . Digital supplemental search engine 164 may use the index to respond to search queries.

Digital supplement server 172 is a computing device. Digital supplement server 172 provides digital supplements. In one embodiment, digital supplement server 172 includes memory 180 , processor assembly 174 , and communication module 176 . Memory 180 may include digital supplement 182 and metadata 184 . In some embodiments, memory 180 may also include other network-accessible resources such as web pages that are not necessarily digital supplements. For example, memory 180 may store web pages containing metadata for providing details about one or more digital supplements and how to access those digital supplements. Additionally, memory 180 may include a resource serving engine, such as a web server, that responds to requests, such as HTTP requests, with network-accessible resources, such as web pages and digital supplements.

Digital supplement 182 is any type of content that can be provided as a supplement to something in the user's surrounding physical environment. Digital supplement 182 may also include any type of content that can supplement stored images (eg, of the previous physical environment around the user). For example, a digital supplement may be associated with an image, an object or product identified within the image, or a supplemental anchor such as a location. Digital supplement 182 may include one or more images, audio content, text data, videos, games, data files, applications, or structured text documents. Examples of structured text documents include Hypertext Markup Language (HTML) documents, XML documents, and other types of structured text documents.

A digital supplement 182 can launch an application and define parameters for that application. Digital supplement 182 may also cause a request to be sent to the server (eg, an HTTP request) and define parameters for the request. In some embodiments, digital supplement 182 begins as a workflow for completing an activity, such as a workflow for completing a purchase. For example, digital supplement 182 may add a particular item to a user's shopping cart, add a coupon code, and send an HTTP request to a server to retrieve a purchase confirmation page.

Metadata 184 is data that describes the digital supplement. Metadata 184 may describe one or more digital supplements provided by digital supplement server 172 or provided elsewhere. Metadata 184 for digital supplements may include one or more of the following: type indicators, anchor indicators, names, descriptions, preview snippets or excerpts, related images, URL-like links to digital supplements, and An identifier for the application associated with the digital supplement. Metadata may also include information about the publisher of the digital supplement, such as publisher name, publisher description, and an image or icon associated with the publisher. In some embodiments, the metadata also includes contextual information about the digital supplement, or contextual information that must be satisfied in order to provide the digital supplement. For example, metadata may include conditions (eg, geographic conditions, client computing device requirements, required applications) that must be met in order to access the digital supplement. Exemplary contextual information may be a location, an entity identified in the image, or multiple entities identified in the image (e.g., some digital supplements require a combination of recognized entities in the image). may be included). A recognized entity can be a supplemental anchor. In some embodiments, recognized entities do not supplement anchors, but instead provide contextual information. Metadata 184 may also include supplemental anchors (eg, text, code, entity, or type of entity) associated with the digital supplement.

Metadata 184 may be stored in various formats. In some embodiments, metadata 184 is stored in a database. Metadata 184 may be stored as XML files, JSON files, or other format files. In some embodiments, digital supplement server 172 retrieves metadata 184 from a database, formats metadata 184 as XML, JSON, or otherwise provides responses to requests from clients or search servers 152 . For example, search server 152 may access metadata 184 to generate data that is stored in digital supplemental data store 166 and used to respond to search requests from client computing devices 102 . .

Communication module 106 includes one or more devices for communicating with other computing devices, such as search server 152 or digital supplement server 172 . Communication module 106 may communicate via a wireless or wired network, such as network 190 . Communication module 156 of search server 152 and communication module 176 of digital supplement server 172 may be similar to communication module 106 .

Display device 108 may be, for example, an LCD (liquid crystal display) screen, an LED (light emitting diode) screen, an OLED (organic light emitting diode) screen, a touch screen, or any other screen or display for displaying images or information to a user. can include In some embodiments, display device 108 includes a light projector configured to project light onto a portion of the user's eye.

Memory 120 may include one or more non-transitory computer-readable storage media. Memory 120 is for generating an image content query based on the acquired image, transmitting the image content query, receiving a response to the image content query, and presenting the digital supplement identified in the response to the image content query. , and may store instructions and data that can be used by the client computing device 102 to implement the techniques described herein. Memory 160 of search server 152 and memory 180 of digital supplement server 172 may be similar to memory 120 and may store data instructions that may be used to implement the techniques of search server 152 and digital supplement server 172, respectively. .

Processor assembly 104 implements one or more devices capable of executing instructions, such as those stored by memory 120, to perform various tasks associated with digital supplemental association and retrieval for visual search. include. For example, processor assembly 104 may include a central processing unit (CPU) and/or a graphics processor unit (GPU). For example, if a GPU is present, some image/video rendering tasks may be offloaded from the CPU to the GPU, such as user interface generation and display, or display of portions of the digital supplement. In some embodiments, some image recognition tasks may also be offloaded from the CPU to the GPU.

Although not shown in FIG. 1, one embodiment includes a head-mounted display device (HMD). The HMD may be a separate device from the client computing device 102, or the client computing device 102 may include the HMD. In some embodiments, client computing device 102 communicates with the HMD via a cable. For example, the client computing device 102 can transmit video and/or audio signals to the HMD for display to the user, and the HMD transmits movement, position, and/or orientation information to the client computing device 102. can be sent to

Client computing device 102 may also include various user input components (not shown) such as a controller that communicates with client computing device 102 using a wireless communication protocol. In some embodiments, the client computing device 102 communicates via a wired connection (eg, Universal Serial Bus (USB) cable) or wireless communication protocol (eg, any WiFi protocol, any BlueTooth protocol, Zigbee® ), etc.), the HMD
(not shown). In some embodiments, the client computing device 102 is a component of the HMD and may be contained within the HMD's housing.

Network 190 may be the Internet, a local area network (LAN), a wireless local area network (WLAN), and/or any other network. Client computing device 102 can, for example, receive audio/video signals, which in one exemplary embodiment can be provided over a network as part of a digital supplement.

FIG. 2 is a third-person view of an exemplary physical space 200 in which an embodiment of client computing device 102 is accessing a digital supplement. In this example, physical space 200 includes object 222 . Here, object 222 is the artwork on the walls of physical space 200 . Object 222 is contained within field of view 204 of camera assembly 112 of client computing device 102 .

An exemplary user interface screen 206 is also shown. User interface screen 206 may be generated by user interface engine 130 of client computing device 102, for example. User interface screen 206 includes image display panel 208 and digital supplement selection panel 210 . The image display panel 208 displays images. For example, image display panel 208 may show images corresponding to real-time feeds from camera assembly 112 of client computing device 102 . In some embodiments, image display panel 208 displays previously acquired images or images retrieved from memory 120 of client computing device 102 .

In some embodiments, user interface screen 206 is presented to the user on the display device of client computing device 102 . In some embodiments, the user interface screen 206 may be so overlaid over an image of the physical space (or a video feed being captured by the computing device's camera). Additionally, user interface screen 206 may be displayed as AR content across the user's field of view using an HMD worn by the user.

Image display panel 208 may also include annotations or user interface elements associated with the image. For example, image display panel 208 may include an indicator that an object (eg, object 222) within the image has been recognized as a supplemental anchor. The indicator may include user-actuatable controls for accessing or viewing information regarding the digital supplement associated with the identified supplemental anchor. In some situations, an image displayed on the image display panel 208 may include multiple objects that are recognized as supplemental anchors, and the image display panel 208 displays multiple annotations that overlay the image to identify those supplemental anchors. can include

Supplemental anchors may be recognized by the supplemental anchor identification engine of client computing device 102 . In some embodiments, supplemental anchors are identified by sending images to search server 152 . Search server 152 may then analyze the image and identify supplemental anchors within the image. In an embodiment, search server 152 may transmit one or more of the locations (eg, image coordinates) or dimensions of any identified objects associated with the supplemental anchors to client computing device 102 . Client computing device 102 may then update the user interface screen to show annotations identifying supplemental anchors (or associated objects) within the image. In some embodiments, client computing device 102 may track the position of supplemental anchors (or associated objects) within a video stream (e.g., a sequentially acquired image sequence) acquired by camera assembly 112 ( For example, supplemental anchor identification engine 124 may track supplemental anchors identified by search server 152).

A digital supplement selection panel 210 allows the user to select a digital supplement for presentation. For example, digital supplement selection panel 210 may include a menu containing user-actuatable controls each associated with a digital supplement. In this example, digital supplement selection panel 210 includes user-actuatable controls 212 and 214, each containing information about an associated digital supplement. For example, the user-actuatable control may display one or more of the name (or title), brief description, and image associated with the digital supplement, which may be received from search server 152 . Upon actuation of user-actuatable control 212 or user-actuatable control 214, the content of the associated digital supplement may be presented to the user. Presenting a digital supplement to a user may include causing client computing device 102 to display a user interface screen that includes images, videos, text, other content, or a combination thereof from the digital supplement. In some embodiments, the digital supplemental content is displayed as an overlay on the image display panel 208 over the image or camera feed. The digital supplemental content can be three-dimensional augmented reality content.

In some embodiments, presenting the digital supplement includes launching an application installed on the client computing device 102 (eg, one of the other applications 140). Presenting the digital supplement may also include sending a request to a URL associated with the digital supplement. The request may include parameters associated with the digital supplement, such as product or object identifiers identified in the image. In some embodiments, the image (or other content) from the image content query is passed parameters with the request. Images may also be provided via an API associated with digital supplement server 172 . In one embodiment, client computing device 102 transmits the image to digital supplement server 172 . In some embodiments, search server 152 may transmit images to digital supplement server 172 . For example, in response to a user selecting a digital supplement, client computing device 102 can send an indicator of the selection to search server 152, which then sends the image to the corresponding digital supplement server. obtain. Client computing device 102 may also send a URL to a location on search server 152 that digital supplement server 172 can use to access the image. Beneficially, these implementations can reduce the amount of data that the client computing device needs to transmit.

A digital supplement associated with user-actuatable control 212 may cause information about object 222, such as information from a museum, to be displayed. A digital supplement associated with user-actuatable control 214 may cause information related to the museum to be displayed. For example, presentation of a digital supplement may cause a stop on the museum tour to be marked as completed and information for the next stop to be displayed.

FIG. 3 is a diagram of an exemplary method 300 of enabling triggering of digital capture, according to embodiments described herein. This method 300 may be performed, for example, by content crawler 162 of search server 152 to allow users to access digital supplements based on image content queries.

At operation 302, data specifying a digital supplement is received. The data may identify the digital supplement and the circumstances under which the digital supplement should be provided. Data specifying digital supplements may be received in a variety of ways. For example, data specifying a digital supplement may be received from a network-accessible resource such as a web page containing metadata about the digital supplement. Data specifying digital supplements may also be received, for example, via an API or form provided by search server 152 . Data specifying the digital supplement may also be received from a memory location or data store.

Data regarding the digital supplement may include access data usable by the client computing device to access the digital supplement. For example, access data may include the URL of the digital supplement and the parameters passed to that URL. Access data may also include an identifier for the application and parameters for that application. Data relating to the digital supplement may also include descriptive data relating to the digital supplement. Such descriptive data may be used by the client computing device to present information about the digital supplement to the user (eg, on a menu from which the user may select the digital supplement). Such descriptive data may include, for example, name (or title), description, publisher's name, and image. Data relating to digital supplements may also include identifiers of supplemental anchors.

At operation 304, a data structure instance is created based on the received data. A data structure can be, for example, a record in a database. The database may be a relational database, and the data structure instance may be linked (eg, via an outer key) to one or more records associated with the supplemental anchor.

At operation 306, after creating the data structure instance, the digital supplement is enabled for retrieval by image content query. For example, a database field associated with a data structure instance may be set active so that the digital supplement search engine 164 can access and return the associated digital supplement. In some embodiments, triggering digital supplementation may include saving or committing a database record. In some embodiments, enabling search of the digital supplement includes enabling triggering of the digital supplement by the client computing device. For example, after an instance is generated, the digital supplement may be returned to the client computing device and activated or presented by the client computing device in response to a search.

FIG. 4 is a diagram of an example method 400 of enabling triggering of digital capture, according to embodiments described herein. This method 400 may be performed, for example, by the content crawler 162 of the search server 152 to allow users to access digital supplements based on image content queries.

At operation 402, network accessible resources are analyzed. In one embodiment, the network-accessible resource is, for example, a web page served by digital supplement server 172 . In one embodiment, a set of network-accessible resources is analyzed. The set of network-accessible resources can be generated based on submissions via forms or APIs. In some embodiments, the set of network-accessible resources may be generated by crawling other network-accessible resources to identify URLs. This crawling process may be performed recursively.

At operation 404, metadata associated with the digital supplement within the network-accessible resource is identified. In some embodiments, the network-accessible resource may include an indicator of metadata associated with the digital supplement. For example, a network-accessible resource may include tags that identify portions of the network-accessible resource that contain metadata. The tag can be an XML tag with a specific type or attribute. The tags can be HTML tags, such as script tags, which contain JSON data structures containing metadata.

At operation 406, a metadata-based digital supplemental data structure instance is generated. Operation 406 may be similar to operation 304 .

At operation 408, an image content query is received. An image content query may be sent by a client computing device, such as client computing device 102, for example. In some embodiments, the image content query includes images. Image content queries may also include text data describing the image. For example, the text data may include identifiers of supplemental anchors within images captured by the camera assembly of the client computing device. In some embodiments, the image content query also includes other information, such as the location of the client computing device or an identifier of a user account associated with the client computing device.

At operation 410, a plurality of digital supplemental data structure instances are identified based on the image content query. In some embodiments, supplemental anchors are identified within the images provided in the image content query. The supplemental anchor can then be used to query an index or database for the associated digital supplement. In some embodiments, other data provided with the query may also be used to identify digital supplements, such as information associated with the client computing device's location or user account. In some embodiments, multiple supplemental anchors may be used to identify associated supplemental anchors.

At operation 412, an ordering of the plurality of digital supplemental data structure instances is determined. Such ordering may be based on various scores associated with the digital supplements or the relevance of the digital supplements to the image content query. In some embodiments, a relevance score corresponding to the relevance of a digital supplement to an image content query can be used to order a plurality of digital supplement data structure instances. Content of a Network-Accessible Resource (or a Network-Accessible Resource associated with a Digital Supplement) that links to, link text, or content near a link to a Digital Supplement on another Network-Accessible Resource It can be determined from multiple factors, such as one or more of:

Scores may also be based on popularity metrics. A prestige metric is an example of a popularity metric. The prestige metric may be based on how many other network resources link to the digital supplement combined with the prestige scores of those other network-accessible resources. In some embodiments, the popularity score may be based on how often the digital resource is or has been selected. In some embodiments, popularity scores may correspond to how often digital resources are selected for image content queries.

Scores may be determined or retrieved from a data store or API. In one embodiment, an API is accessed to retrieve the score of the digital supplement. For example, scores may be retrieved from search engines that have determined the relevance and/or popularity of digital resources for search terms based on supplemental anchors.

A plurality of digital supplemental data structures may also be ordered based on frequency of use or recent recency by a particular user (eg, a user of a client computing device). In some embodiments, the plurality of digital supplemental data structures are randomly ordered.

At operation 414, the image content query is answered based on the plurality of digital supplemental data structure instances. For example, information associated with multiple digital supplemental data structure instances may be sent to the client computing device in the order determined in operation 412 . In some embodiments, such information includes descriptive data that may be presented within a menu or another type of user interface configured to receive user selections of digital supplements. Such information may also include access data that may be used by the client computing device to access or present the digital supplement.

FIG. 5 is a diagram of an exemplary method 500 for searching and presenting digital supplements, according to embodiments described herein. This method 500 may be performed, for example, by application 122 of client computing device 102 to identify and access digital supplements based on image content queries.

At operation 502, an image content query based on the image is sent to a server computing device (eg, search server 152). For example, images may be captured with camera assembly 112 of client computing device 102 . The image may also be a stored image, such as an image previously captured by camera assembly 112 . In some embodiments, the image content query includes only images. In some embodiments, the image content query includes additional information. For example, the image content query may include information such as the location of the client computing device 102 or an identifier of an account associated with the user of the client computing device 102 . Application 122 may also identify anchors in the image (eg, using supplemental anchor identification engine 124). The image content query may include an identifier (eg, text, numeric, or other type of identifier) for the identified anchor. In at least some embodiments, the image content query does not include images.

In some embodiments, sending the image content query to the server includes calling an API. In some embodiments, sending the image content query to the server includes calling an API provided by the server. In some embodiments, sending the image content query to the server includes submitting the form using the HTTP protocol (eg, by submitting a GET or POST request).

At operation 504, a response to the image content query that identifies the digital supplement is received. The response may be received from search server 152 over network 190 . The response may include one or more digital supplements identified by search server 152 based on the image content query. For example, the response may include an array of data associated with the digital supplement. In some embodiments, data associated with a digital supplement may include descriptive data that may be used to present digital supplement options for a user to select from. For example, descriptive data may include a name, short description, publisher name, and image. Data may also include access data such as the URL and parameters to include in a request via the URL, or the application name and associated parameters. Such data may also include the location, coordinates, or dimensions of supplemental anchors in images sent in image content queries (eg, if supplemental anchors are identified by search server 152).

At operation 506, a user interface screen containing information associated with the digital supplement is displayed. In some embodiments, the user interface screen includes annotations overlaying the identified supplemental anchors (eg, based on the provided coordinates). Annotations may provide information about objects in the image associated with the identified supplemental anchors. Such annotations may include user-actuatable controls that may be actuated to present or launch the digital supplement. The user interface screen may also include a digital supplement selection panel that may be used to select from multiple digital supplements identified in the response received at operation 504 . In some embodiments, the user interface screen may be generated by a web browser that opens the URL specified by the digital supplement. A user interface screen may also be generated by another application launched to provide the digital supplement.

FIG. 6 is an illustration of an exemplary method 600 for identifying and presenting digital supplements based on images, according to embodiments described herein. This method 600 may be performed, for example, by application 122 of client computing device 102 to identify and access digital supplements based on image content queries.

At operation 602, an image is acquired. For example, images may be captured by camera assembly 112 of client computing device 102 . In some embodiments, a series of images (ie, videos) may be captured by camera assembly 112 .

At operation 604 , the image-based image content query is sent to a server computing device, such as search server 152 . Operation 604 may be similar to operation 502 . In embodiments in which a sequence of images is obtained, the image content query may include multiple images or sequences of images. In some embodiments, the sequence of images may be streamed to the server computing device.

At operation 606, a response to the image content query that identifies a plurality of digital supplements is received. Operation 606 may be similar to operation 504 described above.

At operation 608, a user interface screen is displayed that includes user actuable controls for selecting a digital supplement from a plurality of digital supplements. For example, a digital supplement selection panel may be displayed. The digital supplement selection panel may include a plurality of user-actuatable controls each associated with one of the plurality of digital supplements identified in the response. The digital supplement selection may provide user-actuatable controls based on the ordering or ranking of the digital supplements provided by the server computing device. The digital supplemental selection panel allows the user to arrange the actuatable controls vertically, horizontally, or otherwise. A user-actuatable control may be associated with or include information about the associated digital supplement that the user may consider in deciding whether to select the digital supplement. For example, the displayed information may include one or more of the digital supplement's name, description, image, and publisher name.

At operation 610, user input is received for selecting a digital supplement. A user input may be a click with a mouse or other device. User input can also be touch input from a stylus or finger. Another example of user input is near-touch input (eg, holding a finger or pointing device in close proximity to the screen). In some embodiments, user input may also include hand gestures, head movements, eye movements, or speech input.

At operation 612, information is provided to the resource associated with the selected digital supplement. For example, information about the user of the client computing device may be sent to the server that provides the digital supplement (if permission is provided to provide the information). Information may also be provided to applications that provide digital supplements. Various types of information may be provided. For example, the information may include user information such as username, user preferences, or location.

Information may also include information associated with an image content query, such as an image or sequence of images. The information may also include identifiers and/or locations of one or more supplemental anchors within the image. This information can be used to provide digital supplements to the user. For example, digital supplemental AR content can be formed and positioned based on the image.

Information may be sent by client computing device 102 directly to a resource associated with the digital supplement (eg, digital supplement server 172). In some embodiments, information is provided by search server 152 to resources associated with the digital supplement (eg, client computing devices do not need to send as much data). In at least some of these implementations, client computing device 102 may send selection information to search server 152 that identifies the selected digital supplement. Upon receiving the selection and confirming that the user has authorized sharing of the information, search server 152 may transmit the information to the resource that provides the digital supplement. Client computing device 102 may also prompt the user to share information. In some embodiments, search server 152 may determine the information to send to the resource based on the digital supplement data structure instance (which can be based on metadata associated with the digital supplement).

At operation 614, the user interface is updated based on the selected digital supplement. Operation 614 may be similar to operation 506 .

7A-7C are schematic illustrations of user interface screens displayed by embodiments of client computing device 102 for conducting image content searches and displaying digital supplements. FIG. 7A shows a user interface screen 700a. User interface screen 700 a includes image display panel 708 and information panel 730 . In this example, image display panel 708 is displaying an image of a shelf full of wine bottles (eg, you might find them in a store). Image display panel 708 also includes indicators 740 and 742 . Each of these indicators indicates that the wine bottle shown in the image below the indicator is recognized as a complementary anchor (eg, in this case as a recognized product). Indicators 740 and 742 are examples of user-actuatable controls. Within the information panel 730, an instruction is provided to "tap what interests you."

In FIG. 7B, user interface screen 700 b is shown after the user has activated indicator 740 . After activation, annotations 744 from the digital supplement are displayed. Annotations 744 contain information about the wine rating that can help the user select a bottle of wine to purchase.

In FIG. 7C, another user interface screen 700c is shown after the user has activated indicator 740. In FIG. User interface screen 700c may be shown instead of or in addition to user interface screen 700b being displayed (e.g., after activation of annotation 744 or when the user swipes up on information panel 730 of FIG. 7B). be An expanded information panel 732 is shown in FIG. 7C. Expanded information panel 732 occupies more user interface screen 700c than information panel 730 featured in FIGS. 7A and 7B.

Extended information panel 732 includes digital supplement selection panel 710 and digital supplement content display panel 734 . Digital supplemental selection panel 710 includes user-actuatable controls 712, user-actuatable controls 714, and (only partially visible) user-actuatable controls 716. FIG. In some embodiments, when the user swipes over the digital supplemental selection panel 710, additional user-actuatable controls may be displayed. The user-actuatable controls of digital supplemental selection panel 710 may be arranged in a ranked order. User actuatable controls 712 are associated with digital supplements for meal pairing. Upon actuation of user actuatable control 712, a digital supplement displaying food and meal pairing information for the selected wine may be displayed. User actuatable control 714 is associated with digital supplements that store photographs. Upon activation, an application that saves photos may be activated to provide the image. Additional information may be stored with the photo, such as identified supplemental anchors.

Digital supplemental content display panel 734 may display content from the digital supplement. Digital supplemental content display panel 734 may display the default digital supplement or the highest ranked digital supplement associated with the identified supplemental anchor. In this example, digital supplemental content display panel 734 includes product information for the product associated with the selected supplemental anchor. In this case, the wine name, rating, origin, image and comments are provided.

8A-8C are schematic illustrations of user interface screens displayed by embodiments of client computing device 102 for conducting image content searches and displaying digital supplements. In this example, the image content search is based on receipt images.

FIG. 8A shows a user interface screen 800a. User interface screen 800 a includes image display panel 808 and information panel 830 . In this example, image display panel 808 displays an image of a receipt from a restaurant. Image display panel 808 also includes indicators 840 , indicators 842 , annotations 844 and highlight overlays 846 . In this case, indicator 840 is associated with the receipt as a document and indicator 842 is associated with the particular restaurant named on the receipt. Both the identified receipt document and the identified restaurant name are examples of supplemental anchors.

Annotations 844 are associated with digital supplements that provide tip calculators. In this example, an exemplary tip calculator is included in annotation 844 and overlaid at the appropriate location on image display panel 808 . In some embodiments, a digital supplement may be selected by default and displayed upon identifying an appropriate supplemental anchor. A highlight overlay 846 is overlaid on a portion of the receipt document containing information used by the digital supplement of the tip calculator.

In this example, the items displayed in information panel 830 relate to the receipt as a document, as if indicator 840 had been activated. In some embodiments, the identified supplemental anchors are, for example, the user's past actions, the actions of other users on similar images, the confidence scores of the supplemental anchors, or the location or size of the portion of the image to which the supplemental anchors relate. based on the user's possible relevance or interest. Information panel 830 may then, in at least some embodiments, display items associated with the highest ranked supplemental anchors. If, instead, indicator 842 is activated, information panel 830 may include items relating to a particular restaurant.

Here, information panel 830 includes digital supplemental selection panel 810 . The digital supplement panel includes user-actuatable controls 812 , user-actuatable controls 814 , and user-actuatable controls 816 . In this example, user operable control 812 is associated with the tip calculator digital supplement, user operable control 814 is associated with the check split digital supplement, and user operable control 816 is associated with the expense report digital supplement. It is For example, when user actuatable control 812 is activated, user interface controls for adjusting parameters of the tip calculator (eg, to adjust percentages) may be displayed.

In FIG. 8B, user interface screen 800b is shown after the user has activated user-actuatable control 814. In FIG. After activation, an extended information panel 832 is shown containing items to help the user calculate how to split the bill. For example, the number of people to split the bill may be entered to determine how much each should pay.

In FIG. 8C, user interface screen 800c is shown after the user has activated user-actuatable control 816. In FIG. After activation, an extended information panel 834 is shown containing items to help the user store the receipt on the expense report. For example, the user can select an expense report with which the receipt should be associated (eg, "Sydney Trip 2018"). Once the expense report is selected, the image of the receipt can be uploaded to the expense report submission or management system. In some embodiments, all images displayed on image display panel 808 are uploaded. In some embodiments, a portion of the image is uploaded (eg, the image is cropped to include only the receipt).

9A and 9B are schematic illustrations of user interface screens displayed by embodiments of client computing device 102 for performing image content searches and displaying digital supplements. In this example, the image content search is based on images of faces.

FIG. 9A shows a user interface screen 900a. User interface screen 900 a includes image display panel 908 and information panel 930 . In this example, image display panel 908 displays an image of a face. Here, the face is an example of a supplemental anchor. Information panel 930 includes user-actuatable controls 912 for identified digital supplements for supplemental anchors in the image (ie, face). User actuatable controls 912 are associated with digital supplements for tying to eyeglasses.

In FIG. 9B, user interface screen 900b is shown after the user has activated user-actuatable control 912. In FIG. After activation, an extended information panel 932 is shown containing items to help the user visually try out the glasses on the face in the image. Here, multiple eyeglass styles are displayed and the user can select a pair to try. Upon selection of a pair of glasses, AR content 960 is overlaid on image display panel 908 . Here, the AR content 960 corresponds to the selected glasses and is sized to match the face in the image. In one embodiment, when the digital supplement for trying on eyeglasses is selected, the image display panel 90
The image shown in 8 is sent to a server that provides a digital supplement whereby the image tells where and how the AR content 960 should be placed and what size it should be, or It can be analyzed to determine if it should be recommended to try on glasses.

10A-10C are schematic illustrations of user interface screens displayed by embodiments of client computing device 102 for conducting image content searches and displaying digital supplements. In this example, the image content search is based on images of furniture in the catalog.

FIG. 10A shows a user interface screen 1000a. User interface screen 1000 a includes an image display panel 1008 . In this example, the image display panel 1008 displays a partial image of the furniture catalog page. The image display panel also includes indicators 1040 , 1042 , and 1044 . In this example, indicator 1040 is associated with a bed, indicator 1042 is associated with a decorative item, and indicator 1044 is associated with a rug. Images of beds, upholstery, and rugs in catalogs are examples of complementary anchors.

In FIG. 10B, user interface screen 1000b is shown after a user selects indicator 1040 (eg, by touching the screen at or near where indicator 1040 is displayed). User interface screen 1000 b includes digital supplement selection panel 1010 and information panel 1030 . Information panel 1030 includes information (eg, product name, description, image) regarding supplemental anchors associated with the selected indicator.

Digital supplemental selection panel 1010 includes user-actuatable controls 1012 and user-actuatable controls 1014 . User-actuatable control 1012 is associated with a digital supplement that provides an in-home view. User-actuatable control 1014 is associated with another digital supplement (eg, a digital supplement for posting to social media sites).

10C, user interface screen 1000c is shown after actuation of user-actuatable control 1012. In FIG. User interface screen 1000 c includes image display panel 1008 , digital supplement selection panel 1010 , and reduced information panel 1032 . Shrink information panel 1032 may include user actuatable controls that, when activated, may cause an information panel to pop up and be displayed.

Here, image display panel 1008 displays an image of the room and includes AR content 1060 . AR content 1060 includes a 3D model of the bed associated with indicator 1040 overlaid on the image panel. The user can adjust the position of the AR content 1060 in the room to see how the bed fits in the room. In an embodiment, when the digital supplement for home view is selected, the image shown in the image display panel 1008 is sent to the server that provides the digital supplement, such that the image renders the AR content 1060. It can be analyzed to determine where and how it should be positioned and sized. In some embodiments, the AR content 1060 may be served after the image content query.

11A-11C are schematic illustrations of user interface screens displayed by embodiments of client computing device 102 for conducting various image content searches within a store. In this example, the image content search is based on images of products captured in the store.

A user interface screen 1100a is shown in FIG. 11A. User interface screen 1100 a includes image display panel 1108 and information panel 1130 . In this example, the image display panel 1108 displays images taken inside the store. Image display panel 1108 also includes an indicator 1140 associated with the vase. A vase displayed on image display panel 1108 is an example of a supplemental anchor. Information panel 1130 displays a digital supplement containing merchandise information about the vase and functionality for purchasing the vase. A digital supplement may include, for example, a workflow to initiate the purchase of a vase. In this example, the digital supplement is identified based on the image content and the location of the client computing device, thereby identifying the digital supplement published by (or associated with) the store from which the image was obtained. and may be provided as high ranking results for image content queries when the client computing device is in-store. In some embodiments, different digital supplements may be provided for the same image if the location of the client computing device changes.

A user interface screen 1100b is shown in FIG. 11B. User interface screen 1100 b includes image display panel 1108 and information panel 1130 . In this example, image display panel 1108 displays another image captured inside the store. Image display panel 1108 also includes an indicator 1142 associated with the lug. A lug displayed in image display panel 1108 is an example of a complementary anchor. Information panel 1130 displays a digital supplement that includes product information about the rug and the functionality to select a size and purchase the rug. As in FIG. 11A, the digital supplement is identified based on the image content and location of the client computing device.

A user interface screen 1100c is shown in FIG. 11C. User interface screen 1100 c includes image display panel 1108 and information panel 1130 . In this example, image display panel 1108 displays another image captured inside the store. Image display panel 1108 also includes an indicator 1144 associated with the vase. A vase displayed on image display panel 1108 is an example of a supplemental anchor. Information panel 1130 displays a digital supplement containing product information about the vase. Information panel 1130 also includes a coupon indicator 1132 and functionality for redeeming coupons. Redeeming the coupon may include purchasing merchandise at a discounted price from a website associated with the merchant. In some embodiments, a coupon code is presented that can be used to secure the discount during checkout. Similar to FIGS. 11A and 11B, digital supplements are identified based on image content and location of the client computing device.

12A-12C are schematic illustrations of user interface screens displayed by embodiments of client computing device 102 during various image content searches. In this example, the image content search is based on images of movie posters (eg, which may be obtained at a movie theater).

FIG. 12A shows a user interface screen 1200a. User interface screen 1200 a includes an image display panel 1208 . In this example, image display panel 1208 displays an image of a movie poster. Image display panel 1208 also includes an indicator 1240 associated with the movie poster identified in the image. A movie poster is an example of a supplemental anchor. Indicator 1240 may include a user-actuatable control that, when activated, displays a digital supplement or a menu for selecting a digital supplement.

A user interface screen 1200b is shown in FIG. 12B. Image display panel 1208 also includes a preview digital supplement 1242 associated with the movie poster identified in the image. For example, preview digital supplement 1242 may be shown after activation of indicator 1240 (of FIG. 12A). The preview digital supplement 1242 may overlay an image or video from the movie associated with the identified movie poster over the image of the movie poster.

A user interface screen 1200c is shown in FIG. 12C. Image display panel 1208 also includes rating indicator 1244 and rating indicator 1246 . Rating indicators 1244 and rating indicators 1246 may be generated by one or more digital supplements in response to image content queries that include movie posters. A digital supplement may overlay rating information within an image, for example, for a movie associated with a movie poster. Rating indicator 1244 and rating indicator 1246 may include user-actuatable controls that, when activated, cause the rating and additional information regarding the associated movie to be shown.

FIG. 13 illustrates a computing device 1300 and a mobile computing device 1350 that may be used with the techniques described herein (eg, to implement client computing device 102, search server 152, and digital supplement server 172). Give an example. Computing device 1300 includes processor 1302 , memory 1304 , storage 1306 , high speed interface 1308 for connecting to memory 1304 and high speed expansion port 1310 , low speed bus 1314 and low speed interface for connecting to storage 1306 . 1312. Each of the components 1302, 1304, 1306, 1308, 1310 and 1312 are interconnected using various buses and may be mounted on a common motherboard or otherwise as desired. Processor 1302 includes instructions stored in memory 1304 or storage device 1306 for displaying graphical information for the GUI on an external input/output device, such as display 1316 coupled to high speed interface 1308. It can process instructions for execution within computing device 1300 . In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and types of memory, as appropriate. Also, multiple computing devices 1300 may be connected, each device providing a portion of the required operations (eg, as a server bank, group of blade servers, or multi-processor system).

Memory 1304 stores information within computing device 1300 . In one embodiment, memory 1304 is one or more volatile memory units. In another embodiment, memory 1304 is one or more non-volatile memory units. Memory 1304 may also be another form of computer-readable storage medium, such as a magnetic or optical disk.

Storage device 1306 may provide mass storage for computing device 1300 . In one embodiment, storage device 1306 is a floppy disk drive, hard disk drive, optical disk drive, or tape drive, flash memory or other similar solid state memory device, or in a storage area network or other configuration. may be or include a computer readable storage medium such as an array of devices including an apparatus for A computer program product may be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods such as those described above. The information carrier is memory 1304
, storage device 1306, or memory on processor 1302.

High speed controller 1308 manages bandwidth-intensive operations for computing device 1300, while low speed controller 1312 manages lower bandwidth-intensive operations. Such functional assignments are exemplary only. In one embodiment, high-speed controller 1308 is coupled to memory 1304, to display 1316 (eg, via a graphics processor or accelerator), and to high-speed expansion port 1310, which can accept various expansion cards (not shown). It is In this embodiment, low speed controller 1312 is coupled to storage device 1306 and low speed expansion port 1314 . Low-speed expansion ports, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), for example, through network adapters, keyboards, pointing devices, scanners , or may be coupled to one or more input/output devices, such as a networking device such as a switch or router.

Computing device 1300 may be embodied in a number of different forms, as shown. For example, it may be implemented multiple times as a standard server 1320 or in a group of such servers. It can also be implemented as part of the rack server system 1324 . Additionally, it may be implemented as a personal computer, such as laptop computer 1322 . Alternatively, components from computing device 1300 may be combined with other components in a mobile device (not shown), such as device 1350 . Each such device may include one or more of the computing devices 1300, 1350, and the overall system may consist of multiple computing devices 1300, 1350 communicating with each other.

Computing device 1350 includes, among other components, processor 1352 , memory 1364 , input/output devices such as display 1354 , communication interface 1366 , and transceiver 1368 . Device 1350 may also include storage, such as a microdrive or other device, to provide additional storage. Each of the components 1350, 1352, 1364, 1354, 1366 and 1368 are interconnected using various buses, some of the components on a common motherboard, or other In a way it can be installed.

Processor 1352 can execute instructions within computing device 1350 , including instructions stored in memory 1364 . The processor may be implemented as a chipset of chips containing separate analog and digital processors. Such processor may provide coordination of other components of device 1350 , such as, for example, control of the user interface, applications executed by device 1350 , and wireless communications by device 1350 .

Processor 1352 may communicate with a user via control interface 1358 and display interface 1356 coupled to display 1354 . The display 1354 is, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal display), an LED (Light Emitting Diode), an OLED (Organic Light Emitting Diode).
) display, or other suitable display technology. Display interface 1356 may include suitable circuitry for driving display 1354 to present graphics and other information to a user. Control interface 1358 may receive commands from a user, translate them and submit them to processor 1352 . Additionally, an external interface 1362 may be provided in communication with processor 1352 to enable near field communication between device 1350 and other devices. External interface 1362 may, for example, provide for wired communication in some embodiments or wireless communication in other embodiments, and multiple interfaces may also be used.

Memory 1364 stores information within computing device 1350 . Memory 1364 may be implemented as one or more of one or more computer-readable storage media, one or more volatile memory units, or one or more non-volatile memory units. Expansion memory 1374 may also be provided and connected to device 1350 via expansion interface 1372, which may include, for example, a SIMM (Single In-Line Memory Module) card interface. Such expanded memory 1374 may provide additional storage space for device 1350 or may store applications or other information for device 1350 . Specifically, expansion memory 1374 may contain instructions for performing or supplementing the processes described above, and may also contain secure information. Thus, for example, expansion memory 1374 may be provided as a security module for device 1350 and may be programmed with instructions that enable secure use of device 1350 . In addition, secure applications can be provided through the SIMM card with additional information, such as placing identification information on the SIMM card unhackable.

Such memory may include, for example, flash memory and/or NVRAM memory, as described below. In one embodiment, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable storage medium or machine-readable medium, such as memory 1364 , expansion memory 1374 , or memory on processor 1352 , which may be received via transceiver 1368 or external interface 1362 , for example.

Device 1350 may communicate wirelessly via communication interface 1366, which may include digital signal processing circuitry as appropriate. The communication interface 1366 can operate under various modes or protocols such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. can provide communication. Such communication may occur via radio frequency transceiver 1368, for example. Additionally, short-range communication can occur, such as with Bluetooth, Wi-Fi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 1370 may provide additional navigation and location-related wireless data to the device 1350 that may be used as needed by applications running on the device 1350. .

Device 1350 may also communicate audibly using speech codec 1360, which may receive spoken information from a user and convert it into usable digital information. Audio codec 1360 may also generate audible sounds for the user, such as through a speaker within a handset of device 1350, for example. Such sounds may include sounds from voice calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 1350. .

Computing device 1350 may be embodied in a number of different forms, as shown. For example, it can be implemented as a mobile phone 1380 . It may also be implemented as part of a smart phone 1382, personal digital assistant, or other similar mobile device.

Various embodiments of the systems and techniques described herein may be digital electronic circuits, integrated circuits, specially designed ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and/or can be realized by a combination of These various implementations are dedicated or general purpose, coupled to receive data and instructions from and send data and instructions to the storage system, at least one input device, and at least one output device. implementation in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor.

These computer programs (also known as programs, software, software applications or code) contain machine instructions for programmable processors, are written in high-level procedural and/or object-oriented programming languages, and/or are written in assembly / can be implemented in machine language. As used herein, the terms "machine-readable medium" and "computer-readable storage medium" include machine-readable media that receive machine instructions as machine-readable signals. Refers to any computer program product, apparatus, and/or device (eg, magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide instructions and/or data to a programmable processor. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide user interaction, the systems and techniques described herein use a display device (LED (light emitting diode) or OLED (organic LED) or LCD (liquid crystal display)) to display information to the user. monitor/screen), and a keyboard and pointing device (eg, a mouse or trackball) through which a user may provide input to the computer. Other types of devices can also be used to provide interaction with the user, e.g., the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be received in any form including acoustic, speech, or tactile input.

The systems and techniques described herein may include back-end components (eg, as data servers), or middleware components (eg, as application servers), or front-end components (eg, user a client computer having a graphical user interface or web browser capable of interacting with embodiments of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components may be implemented in a computing system including The components of the system can be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks (“LAN”), wide area networks (“WAN”), and the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

In one embodiment, the computing device shown in FIG. 13 interfaces with an AR headset/HMD device 1390 to generate an enhanced environment for viewing inserted content within a physical space. can contain. For example, one or more sensors included in the computing device 1350 shown in FIG. 13 or other computing devices can provide input to the AR headset 1390, or to the AR space in general. . Sensors may include, but are not limited to, touch screens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. Computing device 1350 can use sensors to determine the absolute position and/or detected rotation of the computing device in AR space, and that absolute position and/or detected rotation into AR space. can be used as input. For example, the computing device 1350 can be incorporated into the AR space as virtual objects such as controllers, laser pointers, keyboards, weapons. When incorporated into the AR space, positioning of the computing device/virtual object by the user may allow the user to position the computing device to view the virtual object in a particular way within the AR space. For example, if the virtual object represents a laser pointer, the user may manipulate the computing device as if it were a real laser pointer. The user can move the computing device left and right, up and down, in a circle, and use the device similar to using a laser pointer. In some embodiments, a user may aim at a target using a virtual laser pointer.

In some embodiments, one or more input devices included in or coupled to computing device 1350 may be used as inputs to AR space. Input devices include touch screens, keyboards, one or more buttons, trackpads, touchpads, pointing devices, mice, trackballs, joysticks, cameras, microphones, earphones with input capabilities, or in-ear earphones. (buds), game controllers, or other connectable input devices. When a computing device is incorporated into an AR space, a user interacting with an input device included in computing device 1350 can cause certain actions in the AR space.

In some embodiments, the touchscreen of computing device 1350 may be rendered as a touchpad in AR space. A user may interact with the touch screen of computing device 1350 . Interactions are rendered as movements on a touchpad rendered in AR space, for example in an AR headset 1390 . The rendered motion can control virtual objects in AR space.

In some embodiments, one or more output devices included in computing device 1350 can provide output and/or feedback to the user of AR headset 1390 in AR space. The output and/or feedback can be visual, tactile, or audio. The output and/or feedback may include vibrating, turning on and off or flashing and/or emitting one or more lights or strobes, sounding alarms, chiming, playing music, and playing audio files. It may include, but is not limited to playing. Output devices may include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

In some embodiments, computing device 1350 may appear as another object in a computer-generated 3D environment. A user's interaction with the computing device 1350 (e.g., rolling, shaking, touching the touchscreen, swiping a finger across the touchscreen) is interpreted as interacting with objects in the AR space. can be In the example of a laser pointer in AR space, computing device 1350 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates the computing device 1350, the user in the AR space sees the movement of the laser pointer. The user receives feedback from interactions with computing device 1350 in an AR environment on computing device 1350 or AR headset 1390 . A user's interaction with a computing device can be translated into interaction with a user interface generated in an AR environment for the controllable device.

In some embodiments, computing device 1350 may include a touch screen. For example, a user can interact with a touch screen to interact with the user interface for the controllable device. For example, a touch screen may include user interface elements such as sliders that can control properties of the controllable device.

Computing device 1300 represents various forms of digital computers and devices including, but not limited to, laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. is intended. Computing device 1350 is intended to represent various forms of mobile devices such as personal digital assistants, mobile phones, smart phones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are exemplary only and are not intended to limit the embodiments of the invention described and/or claimed herein. not intended to

A number of embodiments have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of this specification.

Additionally, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve desirable results. Additionally, other steps may be provided or steps may be omitted from the described flows, and other components may be added to or removed from the described systems. can be removed. Accordingly, other embodiments are within the scope of the following claims.

While certain features of the described embodiments have been described as described herein, many modifications, permutations, changes, and equivalents will occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of the embodiments. It is to be understood that they have been presented only by way of example and not limitation, and that various changes in form and detail may be made. Any part of the devices and/or methods described herein may be combined in any combination, except mutually exclusive combinations. Embodiments described herein may include various combinations and/or subcombinations of the features, components, and/or features of different described embodiments.

Claims (51)

A computer implemented method comprising:
receiving data designating a digital supplement, said data identifying a digital supplement and a supplemental anchor for associating said digital supplement with image content; said method further comprising:
creating a data structure instance that specifies the digital supplement and the supplemental anchor;
after generating the data structure instance, enabling triggering of digital capture with images based on storing at least the data structure instance in a database containing a plurality of other data structure instances; A method wherein each of the other data structure instances specifies a digital supplement and one or more supplemental anchors. Receiving data specifying the digital supplement includes:
analyzing web pages;
and identifying metadata associated with the digital supplement expressed within the web page. Receiving data specifying the digital supplement includes:
3. The computer-implemented method of claim 2, further comprising crawling a plurality of web pages containing the web page before analyzing the web page. Receiving data specifying the digital supplement includes:
4. The computer-implemented method of claim 2 or 3, further comprising receiving an identifier of the web page prior to analyzing the web page. The computer-implemented method of any of claims 2-4, further comprising associating the data structure instance with a popularity score for the web page. The computer-implemented method of any of claims 2-5, further comprising associating the data structure instance with a relevance score of a supplemental anchor based on the web page. 4. The computer-implemented method of any preceding claim, wherein the digital supplement includes a name, description, image, and uniform resource locator. 4. The computer-implemented method of any preceding claim, wherein the digital supplement includes an identifier for an application. After activating the digital acquisition trigger,
receiving an image content query that includes an image;
determining that the image matches the supplemental anchor specified by the generated data structure instance;
4. Any preceding claim, further comprising providing a digital supplement from the data structure instance in response to the image content query in response to determining that the image matches the supplemental anchor. computer. 10. The computer-implemented method of claim 9, wherein determining that the image matches the supplemental anchor comprises identifying entities within the image. The data structure instance also specifies context information, and providing the digital supplement from the data structure instance provides the data from the data structure instance based on determining that the context is satisfied. 11. The computer-implemented method of claim 9 or claim 10, comprising: Providing the digital supplement from the data structure instance in response to the image content query includes providing a list of digital supplements, the list including digital supplements from the data structure instance and the other digital supplements. and a digital supplement from one of the data structure instances, ordered based on one or more of a popularity score and a relevance score. how to be a computing device,
at least one processor;
a memory storing instructions that, when executed by at least one processor, cause the computing device to:
receiving data specifying a digital supplement, said data identifying said digital supplement, a supplemental anchor for associating said digital supplement with image content, and contextual information;
generating a data structure instance that specifies the digital supplement, the supplemental anchor, and the contextual information;
after generating the data structure instance, enabling triggering of digital capture with images based on storing at least the data structure instance in a database containing a plurality of other data structure instances; A computing device, wherein each structure instance specifies a digital supplement and one or more supplemental anchors. 14. The computing device of claim 13, wherein the contextual information includes location. 15. A computing device according to claim 13 or 14, wherein the contextual information includes entities identified in images. A computing device according to any of claims 13-15, wherein said contextual information comprises a plurality of entities identified within an image. A computer implemented method comprising:
receiving an image content query from a computing device;
identifying supplemental anchors based on the image content query;
generating an ordered list of digital supplements based on the identified supplemental anchors;
Sending the ordered list to a computing device. 18. The computer-implemented method of claim 17, wherein the image content query includes an image and the digital supplement includes a URL to a web page presenting information about entities identified within the image. 19. The computer-implemented method of claim 17 or claim 18, wherein the digital supplement includes a URL to a web page presenting coupon codes for entities identified in the image. The computer-implemented method of any of claims 17-19, wherein the digital supplement includes video content identified based on the image. The computer-implemented method of any of claims 17-20, wherein the digital supplement comprises a request to a server and parameters for the request. 22. The computer-implemented method of claim 21, wherein the parameter initiates a workflow. The computer-implemented method of any of claims 17-22, wherein the digital supplements in the list are ordered based on one or more of popularity scores and relevance scores. a computing device,
at least one processor;
and a memory storing instructions which, when executed by at least one processor, cause the computing device to perform the method of any of claims 1-12 or 17-23. , computing devices. A computer program product comprising instructions which, when executed by one or more processors, causes the one or more processors to perform the method of any of claims 1-12 or 17-23. A computer program product that causes the execution of A computer readable storage medium containing instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any of claims 1-12 or 17-23. A computer-readable storage medium for execution. sending an image content query to a server computing device;
receiving a response to an image content query that identifies a digital supplement;
and causing a user interface containing information associated with said digital supplement to be displayed. 28. The computer-implemented method of claim 27, wherein sending an image content query comprises sending an image to the server computing device. 29. The computer-implemented method of claim 27 or claim 28, wherein sending an image content query comprises sending identifiers of entities within an image to the server computing device. 30. The computer-implemented method of any of claims 27-29, further comprising transmitting contextual information to the server computing device. 31. The computer-implemented method of claim 30, wherein the contextual information includes location. The computer-implemented method of any of claims 27-31, wherein receiving a response to the image content query comprises receiving a URL associated with the digital supplement. 33. The computer-implemented method of any of claims 27-32, wherein receiving a response to the image content query comprises receiving an identifier of an application associated with the digital supplement. 34. The computer-implemented method of any of claims 27-33, wherein receiving a response to the image content query includes receiving a name, description, and image associated with a digital supplement. 35. The computer-implemented method of any of claims 27-34, wherein receiving a response to the image content query comprises receiving a list of digital supplements containing the digital supplement. 36. The computer-implemented method of claim 35, wherein the user interface includes information regarding a plurality of digital supplements from the list of digital supplements. 37. The computer-implemented method of claim 36, further comprising determining an order for the list of digital supplements based on whether an application associated with the plurality of digital supplements is installed. 38. The computer-implemented method of any of claims 27-37, further comprising obtaining an image, wherein the image content query is based on the image. a computing device,
at least one processor;
a memory storing instructions which, when executed by the at least one processor, cause the computing device to:
get the image,
cause an image content query based on the image to be sent to a server computing device;
receiving a response to the image content query that identifies a digital supplement;
A computing device for displaying a user interface including information associated with said digital supplement. 40. Computing according to claim 39, wherein instructions for causing said computing device to receive a response to said image content query comprise instructions for causing said computing device to receive an ordered list of digital supplements including said digital supplement. device. 41. The computing device of claim 40, wherein the user interface includes a digital supplement selection panel containing user-actuatable controls associated with a plurality of digital supplements from the ordered list. 42. The computing device of claim 41, wherein the user-actuatable controls are ordered on the digital supplemental selection panel based on an order provided by the ordered list. The digital supplement is associated with a digital supplement server, and the instructions, when executed by the at least one processor, cause the computing device to further:
cause information to be sent to the digital supplement server;
43. A computing device as claimed in any of claims 39 to 42, adapted to receive digital supplemental content from said digital supplemental server, said digital supplemental content being based on said transmitted information. 44. The computing device of claim 43, wherein the transmitted information includes contextual information. obtaining an image;
sending an image content query based on the image to a search server;
receiving a response to an image content query that identifies a digital supplement server;
causing the image to be sent to the digital supplement server;
receiving digital supplemental content from the digital supplemental server;
and causing the digital supplemental content to be displayed. 46. The method of claim 45, wherein the image content query includes the image, and causing the image to be sent to the digital supplement server comprises sending to the search server instructions to provide the image to the digital supplement server. The described computer-implemented method. 47. The computer-implemented method of claim 45 or 46, wherein causing the image to be transmitted to the digital supplemental server comprises causing the digital supplemental server to transmit the image. 48. The computer-implemented method of any of claims 45-47, wherein the digital supplement comprises augmented reality content sized and positioned based on the image. a computing device,
at least one processor;
and a memory storing instructions which, when executed by said at least one processor, cause said computing device to perform the method of any of claims 27-38 or 45-48. computing device that allows A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of any of claims 27-38 or 45-48. A computer program product that causes a method to be performed. A computer readable storage medium containing instructions which, when executed by one or more processors, causes the one or more processors to perform any of claims 27-38 or 45-48. A computer-readable storage medium for performing the method described in .

Images