KR101329102B1 - augmented reality - image retrieval system using layout descriptor and image feature. - Google Patents

Abstract

The present invention relates to a real-time augmented reality image retrieval system using a layout descriptor and an image feature point, more specifically, to extract a desired page from a PDF file of a print medium, a newspaper, etc., which is a target of an augmented reality service, and efficiently stores a desired page. When searching by the entire layout of the page, it matches by using a layout descriptor, and matches by using image feature point vectors within a limited page range and augments it from a document image management server under a mobile terminal and a network environment. A real time augmented reality image retrieval system using a layout descriptor for retrieving a page surface that is a target of a real service and an image feature point.
According to the present invention, it is possible to efficiently detect a desired page in a stored database extracted from a PDF file of a print medium, a newspaper, etc., which is a target of an augmented reality service, and in particular, when looking up the entire layout of a page (Layout descriptor) ), And by using image feature point vectors within a limited page range, it is possible to search the page surface of the augmented reality service from the document image management server under the mobile terminal and the network environment.

Description

Augmented reality-image retrieval system using layout descriptor and image feature.}

The present invention relates to a real-time augmented reality image retrieval system using a layout descriptor and an image feature point, more specifically, to extract a desired page from a PDF file of a print medium, a newspaper, etc., which is a target of an augmented reality service, and efficiently stores a desired page. When searching by the entire layout of the page, it matches by using a layout descriptor, and matches by using image feature point vectors within a limited page range and augments it from a document image management server under a mobile terminal and a network environment. A real time augmented reality image retrieval system using a layout descriptor for retrieving a page surface that is a target of a real service and an image feature point.

In general, an e-book refers to a digital book that is produced through XML, digital images, multimedia, etc., rather than a book printed on paper, and viewed through terminal screens such as computers, PDAs, and mobile phones.

Recently, various e-books that can be used as terminals such as computers, mobile phones, and PMPs have been developed. Second-generation e-books that combine multimedia such as PDF, XML, and first generation e-books have been actively developed. It has been in the spotlight as a means to replace paper books.

This can be obtained through a variety of multimedia content information that can not be obtained from a paper book, there is an advantage that can provide a variety of content by combining a variety of multimedia scenarios according to the purpose and situation in the same paper book.

Since the above technology provides visual information by using augmented reality technology, page recognition and tracking techniques become the most important part.

Conventionally, natural feature point matching techniques based on markers and textures have been applied to recognize pages.

Marker-based techniques are capable of page recognition and tracking, but there is a problem in that the content is not augmented when a part of the marker is obscured, as well as attaching a marker of a specific pattern to the printed publication.

In addition, it is difficult to apply to a book having a large amount of pages due to the limitation of the marker pattern that can be implemented.

Particularly, in order to perform a matching process in a database using only image feature points by photographing printed pages with a user's camera, a problem of fast and accurate page recognition is difficult due to the similarity between target pages and the amount of feature points.

Therefore, there is a need for a technology that provides a method for convenient recognition and efficient page matching.

none.

Therefore, the present invention has been proposed in view of the above-described problems of the prior art, and an object of the present invention is to efficiently detect a desired page from a database stored in a PDF file extracted from a print medium, a newspaper, etc., which is an object of an augmented reality service. In particular, when looking up the entire layout of a page, it is matched by using a layout descriptor, and it is matched by using the image feature point vector within the limited page range from the document image management server under the mobile terminal and the network environment. It is to be able to search the page surface of the augmented reality service.

In order to achieve the object of the present invention,

Real-time augmented reality image retrieval system using a layout descriptor and image feature point in accordance with an embodiment of the present invention,

Capture the image from the camera and extract the feature points of the image into a vector, create a layout block by setting the distance between adjacent pixels as a threshold value, and generate a layout descriptor XML (XML), the generated image A mobile terminal 100 for transmitting the feature point vector and the layout descriptor XML to the document image management server and requesting an image inquiry to render an image provided from the document image management server and output it to the screen;

It includes the image information DB that stores the layout descriptor item and the image feature point vector information, and searches for the image in the image information DB by referring to the layout descriptor XML provided when the image is requested from the mobile terminal. A document image management server 200 for creating a layout matching result list and providing a corresponding image extracted by searching for an image matching the image feature point vector in the layout matching result list to the mobile terminal. The problem of the invention is solved.

Real-time augmented reality image retrieval system using a layout descriptor and image feature point according to the present invention having the above configuration and action,

It extracts from a PDF file such as a print media or newspaper, which is the target of augmented reality service, to efficiently detect a desired page in a stored database. Especially, when searching the entire layout of a page, it matches using a layout descriptor. By using image feature point vectors within a limited page range, the page surface which is augmented reality service target can be retrieved from document image management server under mobile terminal and network environment.

1 is an overall configuration diagram of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an embodiment of the present invention.
2 is a block diagram of a mobile terminal of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an embodiment of the present invention.
3 is a block diagram of a document image management server of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an embodiment of the present invention.
4 is a block diagram of an image information DB construction unit of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an exemplary embodiment of the present invention.
5 is a flowchart illustrating an augmented reality of a real-time augmented reality image retrieval system using a layout descriptor and an image feature point according to an embodiment of the present invention.
6 is a flowchart illustrating a flow of storing information in an image information DB of a real-time augmented reality image retrieval system using a layout descriptor and an image feature point according to an embodiment of the present invention.
7 is an exemplary diagram illustrating a data field of an image information DB of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an embodiment of the present invention.
8 is an exemplary diagram illustrating an example of converting a PD file to an image of a real-time augmented reality image retrieval system using a layout descriptor and an image feature point according to an embodiment of the present invention.
9 is an exemplary view showing the entire page layout of a real-time augmented reality image search system using a layout descriptor and image feature points according to an embodiment of the present invention.
FIG. 10 is an exemplary diagram illustrating a layout descriptor XML of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an embodiment of the present invention.

Hereinafter, an embodiment of a real-time augmented reality image search system using a layout descriptor and an image feature point according to the present invention will be described in detail.

1 is an overall configuration diagram of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an embodiment of the present invention.

As shown in FIG. 1, a real-time augmented reality image retrieval system using a layout descriptor and an image feature point according to an embodiment of the present invention is provided.

Capture the image from the camera and extract the feature points of the image into a vector, create a layout block by setting the distance between adjacent pixels as a threshold value, and generate a layout descriptor XML (XML), the generated image A mobile terminal 100 for transmitting the feature point vector and the layout descriptor XML to the document image management server and requesting an image inquiry to render an image provided from the document image management server and output it to the screen;

It includes the image information DB that stores the layout descriptor item and the image feature point vector information, and searches for the image in the image information DB by referring to the layout descriptor XML provided when the image is requested from the mobile terminal. A document image management server 200 for creating a layout matching result list and providing a corresponding image extracted by searching for an image matching the image feature point vector in the layout matching result list to the mobile terminal. .

When receiving the augmented reality content (image) from the document image management server through the mobile terminal is to query the page surface that is augmented reality.

The mobile terminal processes the query request query image to form a block and records the layout descriptor as XML.

The layout descriptor XML includes the total number of blocks, the number of blocks for each central reference position, the number and positions of separators, and the like.

When a request is made to the document image management server using the layout descriptor and the extracted image feature point information, the document image management server parses the layout descriptor XML content as a database inquiry condition and performs the inquiry.

Due to the similarity of layout between pages, a plurality of pages may be retrieved as a result of the matching, and the image feature point matching operation is performed within the retrieved page range, and the page having the highest similarity is presented to the mobile terminal as a recognition result.

2 is a block diagram of a mobile terminal of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an embodiment of the present invention.

As shown in Figure 2, the mobile terminal 100 of the present invention,

An image capture unit 110 for capturing an image obtained from a camera;

An image preprocessing unit 120 for acquiring the captured image and performing preprocessing of downsampling, noise removal, and tilt correction;

An image feature point extracting unit 130 for extracting a feature point by acquiring an image having undergone a preprocessing process by the image preprocessing unit;

A layout block generator 140 for generating a layout block by setting a distance between adjacent pixels of an image as a threshold value;

An image feature point vector converter 150 for acquiring an image feature point extracted by the image feature point extractor and converting the image feature point into a vector;

A layout descriptor XML generation unit 160 for generating a layout descriptor XML from the layout block generated by the layout block generation unit;

An image lookup request unit 170 for acquiring an image feature point vector converted by the image feature point vector converting unit and a layout descriptor XML generated by the layout descriptor XMl generation unit and requesting an image search to a document image management server; ,

Tracking module unit 180 for tracking the three-dimensional coordinates on the image page sent from the document image management server,

A rendering module unit 190 for performing rendering with reference to the 3D coordinates tracked by the tracking module unit;

It is configured to include a terminal control unit 195 for controlling the signal flow between the respective units.

The image capture unit 110 is to capture an image in a continuous frame from the camera.

The image preprocessing unit 120 acquires the captured image and performs preprocessing such as downsampling, noise removal, and tilt correction.

The downsampling, noise removal, and tilt correction are well known to those skilled in the art, and thus detailed descriptions thereof will be omitted.

The characteristic parts of the present invention are the feature point extraction and feature point vector transformation and layout block generation and layout descriptor XML generation.

This makes it possible to efficiently perform search accuracy and shorten processing time.

The image feature point extractor 130 obtains an image that has been preprocessed by the image preprocessing unit and extracts feature points, and the layout block generator 140 sets distances between adjacent pixels of the image based on a threshold value. To create the layout block.

Thereafter, the image feature point vector converter 150 acquires the image feature points extracted by the image feature point extractor and converts the image feature points into a vector. The layout descriptor X-Mn generator 160 generates a layout generated by the layout block generator. The layout descriptor XML (XML) is generated from the block.

Techniques for extracting feature points on the image and converting the feature points into vectors are well known to those skilled in the art, and thus detailed descriptions thereof will be omitted.

Generating layout descriptor XML from the layout block by the layout descriptor XML generating unit is illustrated in FIG. 10.

After completing the above processing, the image lookup request unit 170 obtains the image feature point vector converted by the image feature point vector converting unit and the layout descriptor XML generated by the layout engineer XML generation unit to obtain a document image management server. You will be asked to retrieve an image.

Thereafter, the tracking module unit 180 tracks the three-dimensional coordinates on the image page sent from the document image management server, and performs rendering with reference to the three-dimensional coordinates tracked from the tracking module unit by the rendering module unit 190. To provide augmented reality information.

Since the tracking processing technique and the rendering technique are generally used in augmented reality, detailed description thereof will be omitted.

3 is a block diagram of a document image management server of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an embodiment of the present invention.

As shown in Figure 3, the document image management server 200,

Parsing unit 210 for parsing in order to obtain the layout descriptor XML transmitted to the image information request from the mobile terminal to convert the image information DB query conditions, and

A matching result list creating unit 220 for searching an image corresponding to the layout condition from the image information DB and creating a layout matching result list;

An image matching query unit 230 for retrieving an image feature point vector transmitted from the mobile terminal and retrieving an image matching the image feature point vector from a layout matching result list created by the matching result list creating unit;

An image information DB 240 for storing and managing an original storage path, an image storage path, a layout descriptor item, and image feature point vector information;

An image transmitting unit 250 for transmitting an image queried by the image matching inquiry unit to a mobile terminal;

It characterized in that it comprises a server control unit 260 for controlling the signal flow between the respective units and the DB.

That is, the parsing unit 210 acquires the layout descriptor XML (XML) transmitted when the image inquiry request is received from the mobile terminal and parses it to convert the image information into the DB information retrieval condition.

The matching result list creating unit 220 inquires an image matching the layout condition from the image information DB 240 and creates a layout matching result list.

Thereafter, when the image matching query unit 230 requests an image search from the mobile terminal, the image matching query unit 230 obtains an image feature point vector to be sent and searches an image matching the image feature point vector in the layout matching result list created by the matching result list creating unit. The characteristic is that if an image with similarity rate above the threshold is obtained, the search result is sent to the image transmitter. If an image with a similarity rate below the threshold is obtained, another image is obtained. It performs a matching operation on the image.

The image transmitting unit 250 transmits the image inquired by the image matching inquiry unit to the mobile terminal, and the mobile terminal tracks and renders the image.

4 is a block diagram of an image information DB construction unit of a real-time augmented reality image retrieval system using a layout descriptor and image feature points according to an exemplary embodiment of the present invention.

As shown in FIG. 4, the image information DB construction unit 270 of the present invention extracts the layout descriptor item and the image feature point vector information from the original image file and stores the extracted information in the image information DB.

Specifically, the image information DB building unit 270,

An image converter 271 for converting a PD file into an image,

An image blocker 272 for acquiring an image converted by the image converter and forming a block;

An adjacent block integrating unit 273 for integrating into one block when the vertical distance between the image blocks formed by the image blocking unit is less than or equal to a threshold;

A layout descriptor extracting unit 274 for extracting a layout descriptor describing a relative position of each block based on the entire layout obtained by the image block forming unit and the adjacent block integrating unit;

Performing preprocessing of downsampling, noise removal, and tilt correction by acquiring the image converted by the image conversion unit 271, and extracting feature point extractor 275 for extracting feature points from the preprocessed image. )Wow,

An invariant coordinate system conversion unit 276 for acquiring image information obtained by performing the preprocessing and feature point extraction and converting the image information into an invariant coordinate system;

A hash index calculation unit 277 for calculating a hash index by acquiring image information obtained by performing the preprocessing and feature point extraction;

And a DB building module unit 278 for generating a hash table by acquiring an image ID, an original storage path, an image storage path, layout descriptor items, and image feature point vector information, and storing the image in an image information DB.

The image conversion unit 271 converts the PD file into an image, as shown in FIG. 8, and the PD file is individually given per page.

The image blocker 272 forms a block by acquiring an image converted by the image converter.

Thereafter, the adjacent block integrating unit 273 merges into one block when the vertical distance between the image blocks formed by the image block forming unit is less than or equal to the threshold.

That is, in order to further simplify the layout descriptor, it is necessary to collect image blocks formed in the image block unit, and when the vertical distance between the blocks is less than or equal to the threshold, as shown in FIG. 9, the blocks are integrated into one block.

Thereafter, the layout descriptor extracting unit 274 extracts a layout descriptor describing a relative position of each block based on the entire layout obtained by the image block forming unit and the adjacent block integrating unit.

For example, the number of blocks on the left and right and the quadrant position of the separator between paragraphs are extracted based on the middle of the page, and these attributes are used as an index of the database.

The feature storing point extractor 275 for storing the image is obtained by converting the image converted by the image converter 271 to perform presampling of downsampling, noise removal, and tilt correction, and extracts feature points from the preprocessed image. Done.

In general, the feature point extraction generates a blurred image using a binary image generation and a Gaussian filter, and extracts the center of each of the resulting word regions as a feature point.

This is the same process as the preprocessing of the mobile terminal.

The invariant coordinate system conversion unit 276 obtains image information obtained by performing a preprocessing process and feature point extraction and converts the image information into a invariant coordinate system.

Since the invariant calculation method is already known to those skilled in the art, it will be briefly described.

That is, seven adjacent points are found for each feature point extracted in the feature point extraction process, and the cross ratio between two triangles connecting four adjacent points is calculated to represent invariant relations between adjacent points. In the same manner as above, the cross ratio is calculated by combining all adjacent points.

The hash index calculation unit 277 calculates a hash index by obtaining image information obtained by performing a preprocessing process and feature point extraction.

Since the hash index calculation process is already known to those skilled in the art, a detailed description thereof will be omitted.

In the present invention, a hash scheme is adopted for fast searching.

Thereafter, the DB building module unit 278 acquires an image ID, an original storage path, an image storage path, layout descriptor items, and image feature point vector information, generates a hash table, and stores the hash table in the image information DB. See FIG. 7)

5 is a flowchart illustrating an augmented reality of a real-time augmented reality image retrieval system using a layout descriptor and an image feature point according to an embodiment of the present invention.

As shown in FIG. 5, the image capture unit 110 captures an image obtained from a camera (S100), and the image preprocessing unit 120 acquires the captured image to downsample, remove noise, and correct the tilt. The pretreatment process is performed (S110).

Thereafter, the image feature point extractor 130 obtains an image that has been preprocessed by the image preprocessing unit to extract the feature point (S120), and the image feature point vector converter 150 is extracted by the image feature point extractor. The image feature point is obtained and converted into a vector (S125).

In addition, the layout block generation unit 140 sets the distance between adjacent pixels of the image based on a threshold value to generate a layout block (S130), and the layout descriptor XM generation unit 160 generates the layout block by the layout block generation unit. The layout descriptor XML (XML) is generated from the used layout block (S135).

Subsequently, the image lookup request unit 170 obtains the image feature point vector converted by the image feature point vector converting unit and the layout descriptor XML generated by the layout engineer XML generation unit, and requests an image inquiry to the document image management server. (S140) will be.

Subsequently, when the parsing unit 210 of the document image management server 200 requests an image inquiry from the mobile terminal, the parsing unit 210 parses to obtain the layout descriptor XML that is sent and converts it into an image information DB inquiry condition (S150). The matching result list creation unit 220 inquires an image matching the layout condition from the image information DB (S160) to create a layout matching result list (S170), and the image matching query unit 230 is sent from the mobile terminal. By obtaining the image feature point vector, an image matching the image feature point vector is searched in the layout matching result list created by the matching result list generator (S180).

At this time, the image matching inquiry unit 230 determines whether or not having similarity above the threshold value (S190), and obtains an image having similarity above the threshold value from the image transmitting unit and transmits it to the mobile terminal.

In this case, in the mobile terminal, the tracking module 180 tracks the 3D coordinates on the image page (S200), and the rendering module 190 performs rendering by referring to the 3D coordinates tracked by the tracking module unit. (S210) to output the augmented reality information on the screen.

6 is a flowchart illustrating a flow of storing information in an image information DB of a real-time augmented reality image retrieval system using a layout descriptor and an image feature point according to an embodiment of the present invention.

As shown in FIG. 6, after the PD file is loaded (S300), the image converter 271 converts the PD file to an image (S310).

Thereafter, the image blocker 272 acquires the image converted by the image converter to form a block (S350), and the adjacent block integrator 273 vertical distances between the image blocks formed by the image blocker. Is less than or equal to the threshold (S360) is integrated into one block, the layout descriptor extraction unit 274 is a layout for explaining the relative position of each block based on the overall layout obtained by the image block unit and the adjacent block integration unit A descriptor is extracted (S370).

Meanwhile, in addition to the layout descriptor extraction, the feature point extraction process should be performed in the conversion of the PD file into an image (S310).

That is, the feature storing unit extractor 275 for storing the data acquires the image converted by the image converting unit 271 to perform preprocessing of downsampling, noise removal, and tilt correction, and then extracts the feature points from the preprocessing image. In operation S320, the invariant coordinate system converter 276 acquires the image information obtained by performing the preprocessing and the feature point extraction, and converts the information into the invariant coordinate system S330.

Subsequently, the hash index calculation unit 277 obtains image information obtained by performing the preprocessing and feature point extraction to calculate a hash index (S340).

In this case, the DB building module unit 278 obtains an image ID, an original storage path, an image storage path, layout descriptor items, and image feature point vector information, generates a hash table, and stores the hash table in the image information DB.

Through the configuration and operation as described above, it extracts from a PDF file such as a print medium or a newspaper, which is a target of augmented reality service, to efficiently detect a desired page in a stored database. (Layout descriptor) is matched and image feature point vectors within a limited page range can be matched to search the page surface of the augmented reality service from the document image management server under the mobile terminal and the network environment. .

It will be appreciated by those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Therefore, the above-described embodiments are to be understood as illustrative in all respects and not restrictive.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: mobile terminal
200: document image management server

Claims (8)

delete delete delete In the real-time augmented reality image retrieval system using a layout descriptor and image feature points,
Capture the image from the camera and extract the feature points of the image into a vector, create a layout block by setting the distance between adjacent pixels as a threshold value, and generate a layout descriptor XML (XML), the generated image A mobile terminal 100 for transmitting the feature point vector and the layout descriptor XML to the document image management server and requesting an image inquiry to render an image provided from the document image management server and output it to the screen;
Parsing unit 210 for parsing in order to obtain the layout descriptor XML transmitted to the image information request from the mobile terminal to convert the image information DB query conditions, and
A matching result list creating unit 220 for searching an image corresponding to the layout condition from the image information DB and creating a layout matching result list;
An image matching query unit 230 for retrieving an image feature point vector transmitted from the mobile terminal and retrieving an image matching the image feature point vector from a layout matching result list created by the matching result list creating unit;
An image information DB 240 for storing and managing an original storage path, an image storage path, a layout descriptor item, and image feature point vector information;
An image transmitting unit 250 for transmitting an image queried by the image matching inquiry unit to a mobile terminal;
And a document image management server 200 including a server controller 260 for controlling the signal flow between the respective units and DBs.
The image matching inquiry unit 230,
If an image with a similarity rate above the threshold is obtained, the search result is sent to the image transmitter. If an image with a similarity rate below the threshold is acquired, matching is made with other images. Real-time augmented reality image retrieval system using a layout descriptor and an image feature point characterized in that performing the operation.
delete In the real-time augmented reality image retrieval system using a layout descriptor and image feature points,
Capture the image from the camera and extract the feature points of the image into a vector, create a layout block by setting the distance between adjacent pixels as a threshold value, and generate a layout descriptor XML (XML), the generated image A mobile terminal 100 for transmitting the feature point vector and the layout descriptor XML to the document image management server and requesting an image inquiry to render an image provided from the document image management server and output it to the screen;
Parsing unit 210 for parsing in order to obtain the layout descriptor XML transmitted to the image information request from the mobile terminal to convert the image information DB query conditions, and
A matching result list creating unit 220 for searching an image corresponding to the layout condition from the image information DB and creating a layout matching result list;
An image matching query unit 230 for retrieving an image feature point vector transmitted from the mobile terminal and retrieving an image matching the image feature point vector from a layout matching result list created by the matching result list creating unit;
An image information DB 240 for storing and managing an original storage path, an image storage path, a layout descriptor item, and image feature point vector information;
An image transmitting unit 250 for transmitting an image queried by the image matching inquiry unit to a mobile terminal;
An image converter 271 for converting a PD file into an image,
An image blocker 272 for acquiring an image converted by the image converter and forming a block;
An adjacent block integrating unit 273 for integrating into one block when the vertical distance between the image blocks formed by the image blocking unit is less than or equal to a threshold;
A layout descriptor extracting unit 274 for extracting a layout descriptor describing a relative position of each block based on the entire layout obtained by the image block forming unit and the adjacent block integrating unit;
Performing preprocessing of downsampling, noise removal, and tilt correction by acquiring the image converted by the image conversion unit 271, and extracting feature point extractor 275 for extracting feature points from the preprocessed image. )Wow,
An invariant coordinate system conversion unit 276 for acquiring image information obtained by performing the preprocessing and feature point extraction and converting the image information into an invariant coordinate system;
A hash index calculation unit 277 for calculating a hash index by acquiring image information obtained by performing the preprocessing and feature point extraction;
Image information DB comprising a DVD construction module unit 278 that obtains an image ID, an original storage path, an image storage path, layout descriptor items, image feature point vector information, and generates a hash table and stores the image in the image information DB. The construction unit 270,
Document image management server 200 including a server control unit 260 for controlling the signal flow between the respective units and DB; Real-time augmented reality using a layout descriptor and the image feature point comprising a Image Search System.
The method according to claim 6,
The layout descriptor extraction unit 274,
A real-time augmented reality image retrieval system using a layout descriptor and image feature points, characterized by extracting the quadrant positions of the left and right blocks of the reference and the separator between paragraphs and using them as indexes of the image information DB.
delete

Images