JPH10260983A - Device and method for image retrieval - Google Patents

Abstract

PROBLEM TO BE SOLVED: To retrieve a similar image in consideration of the arrangement of feature quantities of an image and to enable similar image retrieval that absorb differences due to changes of photographic conditions etc. SOLUTION: An image feature quantity extraction part 14 divides an image into blocks and calculates feature quantities of the blocks. A feature label matrixing part 15 gives labels to the respective blocks according to the feature quantities calculated by the blocks and rearranges them in specific block order to generate a label matraix. An image management DB 18 stores the image data stored in an image storage part 17 and the label matrix corresponding to respective image data so that they correspond to each other. A label matrix index 19 stores label-series indexes having the image data registered by using the label matrix as keys. For retrieval, the similarity between the label array of the source image and the label array stored in the label matrix index 19 is calculated and an image whose similarity exceeds a specific value is outputted as a retrieval result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image retrieval apparatus and method for retrieving an image.

[0002]

2. Description of the Related Art Conventionally, various techniques for searching for similar images have been proposed. Techniques that have been implemented to some extent to perform similar image searches on natural images include:
Many use color information as an image feature amount. In most cases, a histogram relating to color information is obtained, and in most cases, retrieval is performed using a combination of RGB ratios and colors which are frequently present in an image.

[0003]

However, in the above-mentioned method, the search accuracy is not always high because the color position information is lost. Also, for example, see
No. 249349 discloses pattern matching in which an image is divided into a plurality of blocks and each feature amount (representative color) is used. However, in this method, it is necessary to calculate the distance of the feature value between each block for two images to be matched, which requires a huge amount of calculation. In particular, when a representative color is used as a feature amount, RGB
Since three data must be handled, the calculation becomes more complicated. Further, since the comparison is performed using the feature amount itself, the accuracy of the comparison is improved, but there is a problem that a similar image search cannot be performed simply because the angle of the image changes or the position of the object changes. That is,
It is a so-called robust method that performs an appropriate image search while having a certain degree of ambiguity, such as changing the angle of an image, changing the position of an object, or absorbing a certain difference in image feature amount due to shooting conditions. No similar image search could be performed.

Therefore, when searching for a natural image in the prior art, it is common to assign a keyword to an image and perform an image search using this keyword. However, this keyword assigning operation is a labor-intensive operation, and for images for which no keyword is assigned, an operation of presenting a reduced image and manually selecting the image occurs, making the search operation complicated. Was.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and has as its object to provide an image search apparatus and a method capable of searching for a similar image in consideration of the arrangement of image feature amounts.

Another object of the present invention is to search for a similar image in consideration of the arrangement of the feature amounts of the image, and to enable an image search apparatus capable of searching for a similar image by absorbing a difference due to a change in photographing conditions. And a method.

[0007]

An image retrieval apparatus according to the present invention for achieving the above object has the following arrangement. That is, the image is divided into a plurality of blocks, a labeling unit that labels the blocks in accordance with the characteristic amount acquired for each block, and the labels provided by the labeling unit are arranged based on a predetermined block order. Generating means for generating a sequence, storing means for storing the label sequence generated by the generating means in association with the image, and searching means for performing an image search based on the label sequence stored in the storing means. Prepare.

Further, the image search method of the present invention for achieving the above object has the following steps. That is, an image is divided into a plurality of blocks, and a labeling step is performed by providing a label in accordance with the characteristic amount obtained for each block, and the labels provided in the providing step are arranged based on a predetermined block order. A generation step of generating a column, a storage step of storing the label row generated in the generation step in a memory in association with the image, and a search step of performing an image search based on the label row stored in the memory Is provided.

[0009]

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 is a block diagram showing a control configuration of the image retrieval apparatus according to the present embodiment. In the figure, 101 is C
It is a PU and executes various controls in the image search device of the present embodiment. Reference numeral 102 denotes a ROM, which stores a boot program executed when the apparatus is started and various data. A RAM 103 stores a control program to be processed by the CPU 101 and
1 provides a work area when executing various controls. 10
A keyboard 4 and a mouse 105 provide various input operation environments for the user.

Reference numeral 106 denotes an external storage device, which comprises a hard disk, a floppy disk, a CD-ROM, or the like. A network interface 107 enables communication with each device on the network. Reference numeral 109 denotes an interface, and 110 denotes a scanner for reading an image. A bus 111 connects the above components.

In the above configuration, the scanner 110
Alternatively, the external storage device 106 may be replaced with a device arranged on a network.

FIG. 2 is a block diagram showing a functional configuration of the image search device of the present embodiment. In the figure, reference numeral 11 denotes a user interface unit, which detects various operation inputs from a user using a display 107, a keyboard 104, and a mouse 105. An image input unit 12 reads an image by the scanner 110. An image memory 13 stores image data obtained by the image input unit 12 in a predetermined area of the RAM 103. Reference numeral 14 denotes an image feature amount extraction unit, which extracts a feature amount of an image stored in the image memory 13 according to a procedure described later. A feature label labeling unit 15 generates a label sequence based on the feature obtained by the image feature extracting unit 14. 16
Is a pattern matching unit that calculates a similarity based on a label sequence of a designated image and a label sequence of an image stored in the image storage unit 17 and searches for a similar image.

Reference numeral 17 denotes an image storage unit, and the image input unit 12
And the like are accumulated. FIG. 3 is a diagram for explaining a storage state of the image data in the image storage unit 17. Each image data 112 is provided with an image ID 111, and these are stored as a pair in the image storage unit 17. Reference numeral 18 denotes an image management database (hereinafter, image management D)
B) and the image storage unit 1 in the data form shown in FIG.
7 is managed. Reference numeral 19 denotes a label sequence index, which stores the label sequence index shown in FIG. 9 and the label component index file shown in FIG.

An example of the operation of the image retrieval apparatus according to the present embodiment having the above configuration will be described below. In the following example, three colors of red (R), green (G), and blue (B) are adopted as image feature values focusing on color, and the processing is performed using a three-dimensional color space.

[Image Registration Processing] First, processing performed at the time of image registration will be described. FIG. 4 is a flowchart illustrating the procedure of the image registration process according to the present embodiment. First, in step S <b> 11, an image using the image input unit 12 is read in accordance with an instruction through the user interface unit 11 and stored in the image memory 13. Next, step S1
In 2, the image is divided into a plurality of blocks. In the present embodiment, an image is divided into a plurality of vertical and horizontal blocks.
FIG. 5 is a diagram illustrating an example of image block division according to the present embodiment. As shown in the figure, in the present embodiment, 3 ×
It is assumed that the image is divided into a total of nine images of three. Next, in step S13, the feature amount of each divided block is calculated, and the obtained feature amount is labeled by the following procedure.

FIG. 6 is a diagram for explaining a multidimensional feature quantity space according to the present embodiment. As shown in FIG. 6, the multidimensional feature space (RGB color space) is divided into a plurality of blocks (color blocks), that is, cells (color cells), and each cell (color cell) is unique by a serial number. Give a label.
The reason why the multidimensional feature space (RGB color space) is divided into a plurality of blocks is to absorb subtle differences in feature amounts (colors).

In the multi-dimensional feature amount space, the image feature amount is not used as it is, but the average and variance of each parameter are obtained by experiment and normalized (normalized).
For example, it is conceivable to perform orthogonal transformation such as principal component analysis and use a significant dimension. In addition, the “significant dimension” is a dimension configured by a principal component axis having a large contribution rate in the principal component analysis.

In step S13, a predetermined image feature amount calculation process is performed on each of the divided blocks obtained in step S12 to determine which cell in the multidimensional feature amount space belongs, and a corresponding label is determined. Ask for. This process is performed for all blocks. That is, for each divided image block, a calculation process is performed to determine which color cell all the pixels belong to, and the label of the most frequent color cell is determined as the parameter label (color label) of the divided image block. Processing is performed on all blocks.

When the parameter labels are assigned to the respective blocks as described above, in step S14, the parameter labels assigned to the respective blocks are arranged in a predetermined block order, thereby forming a parameter label string (hereinafter referred to as a label). Column) is generated. FIG. 7 is a diagram illustrating an example of a block order when a label string is generated.
The above-mentioned parameter labels are arranged according to the numbers in the cells of the divided image block in FIG.

Here, in FIG. 7A, the divided block is scanned obliquely from upper right to lower left.
This is in order to obtain a label string that is as continuous as possible along the similar search target object with a high expected value in order to reduce the influence of the delicate difference in the angles of the images to be compared and the deviation. As a result, in combination with the operation of the pattern matching unit 16 to be described later, it is possible to compare label strings that are less affected by any of vertical, horizontal, and horizontal displacements.

The scanning method applicable to this embodiment is as follows: horizontal direction (scanning from left to right from top to bottom;
There are four possible scanning methods, such as scanning from left to right from bottom to top.) ・ Vertical direction (four possible scanning methods, such as scanning from top to bottom from left to right). ), Diagonal direction (there are two diagonal scans for each starting point of the four corners, and there are eight different scan methods including (a) to (c) in FIG. 7), zigzag scan (used in JPEG, etc.) Scanning method, there are two types of zigzag scanning for each starting point of the four corners, and there are a total of eight types of scanning). In the present embodiment, a scan method to be adopted is determined from the following viewpoints. That is, (1) In this embodiment, the comparison is a chronological comparison between label strings, and it is not preferable that the order is reversed. Therefore, it is necessary to scan all the images by a predetermined scanning method and perform label arraying. (2) It is desirable that a block whose position is close be located close in the label string. (3) The fact that the label of a block caught by an object to be searched appears as quickly as possible and lasts for a long time facilitates matching. (4) Even if the object moves or the angle changes,
Make sure that the label arrangement does not change significantly. Is adopted. Especially,
Assuming that most of the object of interest is located at the center of the image, a block including the object of interest appears as early as possible in the scan, and as a method of increasing the expected value of scanning the object for a long time, in the present embodiment, oblique scanning is performed. Has adopted. In the present embodiment, an oblique scan from the upper right to the lower left as shown in FIG. 7A is employed, but naturally, an example as shown in FIG. 7B and a case as shown in FIG. Various scanning methods may be employed.

Subsequently, in step S15, the label string and the image data obtained as described above are stored in the image storage section 17,
The image is stored in the image management DB 18 and the label column index 19. That is, an image ID is acquired for the image data read in step S11, and these are paired and stored in the image storage unit 17. Then, an image management DB record shown in FIG. 8 is generated in association with the image ID, and registered in the image management DB 18. Further, in step S16, a record (label sequence index in FIG. 9) in which the image ID group is stored in a variable length record using the label string as a search key is created and registered in the label string index 19.
If the label string has not been registered, a new record is generated, a label string ID is assigned, and the label string and the image ID are registered. On the other hand, if the label string has already been registered, the image ID is additionally registered in the image ID group. By using such a label sequence index, when a label sequence is given, an image ID corresponding to the label sequence can be obtained at high speed.

The above is the processing performed at the time of image registration.

[Similar Image Search Process] Next, a similar image search process will be described with reference to the flowchart of FIG. FIG.
0 is a flowchart for explaining the processing procedure of similar image search. In this embodiment, at the time of initialization, a label sequence group of already registered images is obtained from the label sequence index, and a label component index file is generated with each label component as a key. 19 is stored. Note that the initialization at this time may be either when the system is started or when the application is started. Also, when a new image is registered and is registered in the image DB, the label component index is generated. FIG. 11 is a diagram illustrating a data configuration example of the label component index. As shown in FIG. 11, the label component index has, for each label component, an address group (column ID group) to a label column having the label therein. It is not necessary to re-create the label component index file until it becomes necessary to reflect the registration, deletion, and change of the image.

First, in step S21, when a similar search source image is specified from the user interface unit 11, the image ID of the specified similar search source image is obtained in step S22. (A color label string in this example) is obtained.

Next, in step S23, a label sequence group (label sequence in the label sequence index) including a label that is at least somewhat the same as the label sequence of the similar search source image is acquired with reference to the label component index file. This is because if the comparison with all the label strings of the registered image is performed, the processing becomes slow. This is to improve the processing speed by making a one-to-one comparison with the label string of the similar search source image after narrowing down.
Of course, if the processing can be delayed, comparison with all the label strings of the registered image may be performed to perform a highly accurate search (in this case, step S23 is omitted).

Next, in step S24, each label string acquired in step S23 is compared with the label string of the similar search source image, and the similarity is calculated. Then, the search result is output as a search result together with the similarity in order from the label string closest to the label string of the similar search source image.

Here, a method of performing similarity comparison (calculation of similarity) between label strings will be described.

FIG. 12 is a diagram showing an example of a penalty matrix between labels used for comparing label strings and calculating similarity. The smaller the value in the matrix, the more similar. For example, the penalty for label 2 and label 6 is “7”. The penalty between the same labels is “0” as a matter of course. The purpose of using this matrix is to make a distance determination according to the similarity of labels. That is, in the present embodiment, since the RGB color space is used as the feature amount space, it is possible to determine the distance according to the similarity of the color.

For example, in order to reduce the penalty (distance) between cells adjacent to each other in pattern matching between labels and to give a large penalty to distant cells, FIG.
Introduce a penalty matrix between labels as shown in FIG. In step S24, this penalty matrix is taken into account, and the label strings are compared with each other.
You may make it perform matching which can compare label sequences, such as automaton, vaguely. By using such an ambiguity technique, a low penalty is given to the addition of an extra label, a missing label, or a repetition of the same label. By calculating the distance between label strings using the penalty matrix, ambiguous pattern matching can be performed. As the automaton, a "fuzzy non-deterministic finite automaton" described in "A fuzzy character string search method and system using a fuzzy non-deterministic finite automaton" of Japanese Patent Application Laid-Open No. 8-241335 can be applied. In this automaton, the distance between each symbol (penalty) can be multi-valued, and the label position to be compared can be moved ambiguously back and forth, so that the total distance is minimum (similarity is maximum). In addition to the above-described automaton, there is a DP matching used in speech recognition and the like as a technique for realizing the comparison of the label strings, and this technique can be suitably applied to the embodiment.

Further, in addition to the above-described vague pattern matching, by using the rules of the block order shown in FIGS. It is possible to compare the color label rows that are less affected by any of the vertical and horizontal displacements. That is, DP matching and fuzzy non-deterministic automata are matchings that allow ambiguity before and after a label string, and have a property of absorbing the influence of image displacement. In addition, it is expected that the position of the object changes due to a difference in angle and the position of the object cut by the block changes, so that the hue of the block may be slightly different, but this difference is absorbed by the above penalty matrix. Will be. Thus, by the synergistic effect of the DP matching or the matching that allows the ambiguity by the fuzzy automaton and the allowance of the ambiguity of the feature amount by the penalty matrix,
Matching with little effect on vertical and horizontal displacements is enabled. Furthermore, since the change in the label position due to the change in the position of the object is reduced by the oblique scanning as shown in FIGS. 7A to 7C, the influence of the object shift at the time of matching can be reduced more effectively.

Next, in step S25, a search is performed from the label sequence index using a label sequence having a high degree of similarity as a key, and a corresponding image ID is obtained. Hereinafter, this process is repeated for each label string output in descending order of similarity, and as a result, an image ID group of similar images is obtained.
In step S26, a full path file name is acquired for each image ID in the image ID group with reference to the image management DB 18, and this is presented to the user.

By the above-described processing, a similar image which is robust, such as absorbing a certain difference in color caused by a change in the angle of the image, a change in the position of the object, or a change in the photographing conditions, is obtained. The search can be performed at high speed.

In the above embodiment, an example in which a natural image search is performed has been described. However, it is known to those skilled in the art that the present invention is a technique applicable to search for artificial images such as CG and CAD. it is obvious.

In the above embodiment, the color information is selected as the image feature value. However, the present invention is not limited to this, and the present invention can be implemented by obtaining other image parameters for each image division block. .

In the present embodiment, an example of recognition using one feature has been described. However, a high-speed search from a plurality of features is performed by performing a logical operation on a search result using other features. It is also possible to do.

In the case of a similarity search in which one parameter is added to one image that cannot be blocked,
It is also possible to perform a search based on a statistical distance scale, using the similarity obtained by the present invention (created using the sum of penalties) as one new feature amount. Further, in the above embodiment, similar images having similarities exceeding a predetermined value are obtained as search results. However, it is also possible to output a predetermined number of images in order from images having high similarity as search results. Needless to say.

The present invention can be applied to a system including a plurality of devices such as a host computer, an interface device, a reader, and a printer. Etc.).

Another object of the present invention is to provide a storage medium storing a program code of software for realizing the functions of the above-described embodiments to a system or an apparatus, and to provide a computer (or CPU) of the system or the apparatus.
And MPU) read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instruction of the program code. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the program code is read based on the instruction of the program code. It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

As described above, according to the present embodiment, a feature amount group (a group of feature amounts obtained by dividing the feature amount space) is represented by one symbol (ie, labeling), and the label The distance based on the similarity is given by a penalty matrix. For this reason, the amount of calculation of the distance between the blocks of the two images can be significantly reduced, and similar feature amounts are represented by the same label, so that a similar image can be searched well. it can.

In addition, (1) the concept of distance between labels by the penalty matrix is introduced, and (2) the position of the label to be compared, such as DP matching or fuzzy non-deterministic automaton, can be moved ambiguously back and forth, and the total distance Introduces a method of comparing label strings that minimizes (similarity is maximum), so that it is possible to search even if the angle of the image slightly changes, and to search for images with similar atmosphere become able to.

Further, according to the above embodiment, the use of an index database (label sequence index or label component index) further speeds up image retrieval.

[0048]

As described above, according to the present invention, it is possible to search for a similar image in consideration of the arrangement of the feature amounts of the image.

Further, according to the present invention, a similar image is searched for in consideration of the arrangement of image feature amounts, and a similar image can be searched for by absorbing a difference due to a change in photographing conditions, which has been difficult in the past. The angle of the image changes,
Robust similar image search can be performed, for example, by absorbing a certain difference in the feature amount of an image due to a change in the position of an object or a change in other imaging conditions.

[0050]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a control configuration of an image search device according to an embodiment.

FIG. 2 is a block diagram illustrating a functional configuration of the image search device according to the embodiment;

FIG. 3 is a diagram illustrating a storage state of image data in an image storage unit 17;

FIG. 4 is a flowchart illustrating a procedure of an image registration process according to the embodiment.

FIG. 5 is a diagram illustrating an example of image block division according to the present embodiment.

FIG. 6 is a diagram illustrating a multidimensional feature amount space according to the present embodiment.

FIG. 7 is a diagram illustrating an example of a block order when a label string is generated.

FIG. 8 is a diagram illustrating a data configuration example of an image management DB record.

FIG. 9 is a diagram showing a data order of a label series index.

FIG. 10 is a flowchart illustrating a similar image search processing procedure.

FIG. 11 is a diagram illustrating a data configuration example of a label component index.

FIG. 12 is a diagram illustrating an example of a penalty matrix between labels used when comparing label strings and calculating similarity.

Claims (15)

[Claims] An image dividing unit that divides an image into a plurality of blocks and assigns a label according to a feature amount obtained for each block; and arranges the labels assigned by the assigning unit based on a predetermined block order. Generating means for generating a label string by the above-mentioned method, storing means for storing the label string generated by the generating means in association with the image, and searching for performing an image search based on the label string stored in the storing means An image search device comprising: 2. The label is a unique label assigned to each of the obtained cells by dividing a multidimensional feature amount space into a plurality of cells, and the assigning unit assigns a feature amount to each of the blocks. 2. The image search device according to claim 1, wherein a label assigned to a cell to which the calculated feature amount belongs is assigned to the block. 3. The plurality of blocks are obtained by dividing an image into a plurality of vertical and horizontal blocks, and a block order used by the generating unit is an order in which the plurality of blocks are scanned in an oblique direction. The image search device according to claim 1, wherein: 4. The plurality of blocks are obtained by dividing an image into a plurality of vertical and horizontal blocks, and the block order used by the generating means is an order in which the plurality of blocks are scanned in a horizontal or vertical direction. 2. The image search device according to claim 1, wherein: 5. A search means for calculating a similarity between a label string of an original image and a label string stored in the storage means, and searching for an image having a similarity exceeding a predetermined value by the calculation means. The image retrieval apparatus according to claim 1, further comprising an output unit that outputs the result. 6. The computing means has a table for holding a penalty value based on a distance between cells for each label value pair, and calculates a label string of the original image and a label string stored in the storage means. 6. The image search device according to claim 5, wherein a penalty value is obtained with reference to the table for each obtained label value pair, and a similarity is calculated based on the obtained penalty value. 7. The calculating means calculates a similarity between a label string of a search source image and a label string stored in the storage means, and further provides a penalty value for excess or deficiency of a label. The image search device according to claim 6, wherein 8. The plurality of blocks are obtained by dividing an image into a plurality of vertical and horizontal blocks, and a block order used by the generating unit is an order in which the plurality of blocks are scanned in an oblique direction. The image search device according to claim 7, wherein: 9. The image search device according to claim 7, wherein the penalty value for the excess or deficiency of the label is obtained based on an automaton theory. 10. A first table for registering an image ID group using a label string generated by the generation means as a key, a label string of an original image and a label string stored in the storage means, Calculating means for calculating the degree of similarity, and extracting means for obtaining a label string whose similarity exceeds a predetermined value by the calculating means and extracting an image corresponding to the obtained label string with reference to the first table. The image search device according to claim 1, further comprising: 11. The image retrieval apparatus according to claim 10, wherein a label string to be subjected to similarity calculation by the arithmetic means is a label string serving as a key of the first table. 12. A label sequence to be subjected to similarity calculation by the arithmetic means is the same as the label component included in the label sequence of the original image among the label sequences stored in the storage means. The image retrieval device according to claim 10, wherein the label sequence includes a predetermined number or more. 13. A label table stored in the storage means, further comprising a second table in which each label component is used as a key and a label sequence group including the label component is registered. 13. The method according to claim 12, wherein a label string including a predetermined number or more of the same label component is acquired by referring to the second table, and the acquired label string is set as a target of similarity calculation in the arithmetic unit. Image search device. 14. An image dividing apparatus, wherein an image is divided into a plurality of blocks, and a label is assigned in accordance with a feature amount obtained for each block; and the labels assigned in the assigning step are arranged based on a predetermined block order. Generating a label string by performing the above-described steps; storing the label string generated in the generating step in a memory in association with the image; and performing an image search based on the label string stored in the memory An image search method comprising: a search step. 15. A storage medium for storing a control program for image search, the control program divides an image into a plurality of blocks, and assigns a label according to a feature amount obtained for each block. Assigning means, generating means for generating a label sequence by arranging the labels assigned by the adding means based on a predetermined block order, and storing the label string generated by the generating means in association with the image Storage means for performing an image search based on a label string stored in the storage means.