JPH06318256A - Image retrieving device - Google Patents

Abstract

PURPOSE:To provide the image retrieving device which enables retrieval by designating the rough structure of an image. CONSTITUTION:A reference image memory 101 stores a reference image as the arrangement of picture elements defined as the color vector features of R, G and B. The reference image is inputted to a reference vector generator 102, and a reference vector set expressing 'what color is distributed at what area' is outputted by clustering the feature vector of a picture element coupling a color vector and a position vector. A correlative value computing unit 104 reads the feature vector from a retrieval object image 106 at random and calculates the value of correlation with the reference vector. By repeating such an operation concerning the retrieval object image stored in a retrieval object image memory 106, the image is retrieved. Finally, a maximum correlation image index output equipment 105 outputs the index of the retrieval object image for which the correlation value is maximum. Therefore, the image can be retrieved as 'an image which center is red and peripheral part is blue', for example.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image search device used for searching and editing an image database.

[0002]

2. Description of the Related Art As a conventional image retrieval apparatus, an apparatus in which a person visually attaches retrieval information to a moving image in advance and later retrieves it by the retrieval information is conceivable.
Ueda: Proposal of interactive video editing method, IEICE technical report, IE90-6, 1990). However, when developing an apparatus for retrieving images from a large amount of moving images, an apparatus that does not require manual addition of retrieval information is desirable.
An example of such an automated image search device is an image search device that searches for an image based on this feature, using the colors of two adjacent areas in the image as a pair (this is called a color pair). Nagasaka, Tanaka: Automatic Search Indexing and Object Search in Color Video Images, Transactions of Information Processing Society of Japan, Vol.3
3, No. 4, 1992. ". FIG. 7 is a block diagram for explaining an image search device based on the color pair. The operation will be described below.

The reference image stored in the reference image memory 701 is read by the cell unit histogram calculating means 702,
After dividing into 16 × 16 grid cells, a color histogram is calculated for each cell. The color histogram is 51
The pixels in the cell are classified into two colors, and the total number of pixels in each color is obtained. Using this histogram, the reference color pair calculation means 703 sets 16 characteristic color pairs in adjacent cells. This represents the reference image.
On the other hand, the cell unit histogram calculation means 705 reads out the search target image from the search target image memory 704,
After dividing into 30 × 30 cells, a color histogram is calculated for each cell. Although the number of cells is larger than that of the reference image, the method of constructing the histogram is the same. Based on this histogram, the color bitmap creating means 7
At 06, a 30 × 30 bitmap is created for each of the 512 colors. That is, 512 bitmaps are used as image 1
Create about a frame. For each color, 1 is set if the number of pixels in the cell is greater than or equal to a certain threshold value, and 0 is set if the number is less than a certain threshold value. The color pair scanning means 707 is the 30 × 30 unit.
Of the 16 reference color pairs in 8 neighborhoods in the bitmap of
Scan the entire map to see if any exist,
If there is, the bit of the map that stores the operation result is set to 1, and if there is not at all, it is set to 0, and the operation result is output as a 30 × 30 bitmap as a correlation map. In the correlation map,
The larger the number of bits that are 1, the stronger the correlation between the reference image and the search target image.

The above-mentioned color pair apparatus expresses the structure of an image as a color pair of adjacent areas, and has a feature that it is resistant to movement, rotation and scaling of an object.

[0005]

However, the above-mentioned conventional image retrieval apparatus has a problem that it is not possible to perform a retrieval focusing on the structure of the entire image because only the adjacent colors of the image are focused on. For example, it is not possible to correspond to a case where it is desired to perform a search by designating a rough feature depending on the position as an “image with red in the center and blue in the peripheral portion”.

Further, in order to characterize the reference image, it is necessary to have many features in which highly saturated regions are adjacent to each other, or when the image to be searched contains many color components, it is an erroneous detection. There is also a problem.

An object of the present invention is to provide an image search apparatus capable of searching by designating a rough structure of an image in consideration of the problems of the conventional image search apparatus.

[0008]

According to the present invention of claim 1, the pixels of the reference image are clustered with the feature quantity and position of the pixel as a feature vector to generate a reference vector set consisting of distribution parameters of the feature vector. A plurality of pixels from the image to be searched, and using the feature amount and position of the pixel as the feature vector, a distance corresponding to the error to the feature vector distribution of the pixel described by the reference vector is obtained. An image search apparatus that has a correlation calculation unit that obtains a correlation value of an image based on it, holds results of the correlation calculation unit for a plurality of search target images, and outputs a search target image having a large correlation value as a search result. .

The present invention according to claim 2 clusters pixels of the reference image and the image to be searched with the feature amount and position of the pixel as a feature vector to generate a reference vector set consisting of distribution parameters of the feature vector. A vector generation means, a reference vector set of the reference image, and a correlation calculation means for calculating a correlation value between the reference vector set of the search target image, holding the results of the correlation calculation means for a plurality of search target images, The image retrieval apparatus is characterized by outputting a retrieval target image having a large set correlation value as a retrieval result.

[0010]

According to the first aspect of the present invention, the reference vector generating means clusters the feature vector consisting of the feature amount and the position of the pixel from the reference image, and the reference image is represented by the distribution parameter of the feature vector. In order to cluster feature vectors including positions, neighboring pixels with similar feature quantities form one cluster, and the feature vector distribution in that cluster determines which feature quantity at which position in the image. It is expressed whether there are pixels. There are as many distribution parameters as there are clusters, and this is called a reference vector set. With respect to the obtained reference vector set, the distance from the search target image to the feature vector including the feature amount and position of the pixel and the reference vector is calculated. Distance calculation is performed for some feature vectors, and a correlation value is obtained based on this distance. Correlation values are obtained for a plurality of search target images, and the image with a large value is used as the search result.

Next, in the second aspect of the present invention, similarly to the first aspect of the present invention, the reference vector generating means clusters the feature vector including the feature amount and the position of the pixel from the reference image, and It is represented by a reference vector set together with the position whether it is composed of such image features. Similarly, the search target image is also represented by a reference vector set by the reference vector generation means. The correlation calculation means evaluates the correlation between the search target image and the reference image by calculating the correlation value between the reference vector set of the reference image and the reference vector set of the search target image. For a plurality of search target images, an image having a large correlation value is set as a search result.

Therefore, an effective image search can be performed by the structure of the image, that is, the distribution of the features depending on the position of the pixel.

[0013]

Embodiments of the present invention will be described below with reference to the drawings. The images used in the following two examples are RGB
It is a color image composed of the three primary colors.

A first embodiment corresponding to claim 1 is shown in FIG.
This will be described with reference to FIGS. 2, 3, 4, and 5. Figure 1 is the first
2 is a functional block diagram of the image retrieval apparatus in the embodiment of FIG.
Is a functional block diagram of the reference vector generator in the first and second embodiments, FIG. 3 is a functional block diagram of the correlation calculator in the first embodiment, and FIG. 4 is an explanation of the operation in the first embodiment. FIG. 5 is an explanatory diagram of the assumption of the feature vector distribution in the first and second embodiments.

In FIG. 1, 101 is a reference image memory storing a reference image, 102 is a reference vector generator for generating a reference vector, 103 is a reference vector changing means for changing the reference vector, and 104 is a correlation value calculation. A correlation value calculator 105, a maximum correlation image index output device 105 for indexing and outputting an image having a maximum correlation value, and a search target image memory 106. Color is used as the feature amount of the pixel. Let this be x, which is a column vector of RGB. The pixel position is p, which is a column vector of the horizontal position ξ and the vertical position η. Then, the feature vector of the pixel is set to (x, p).
The reference image memory 101 stores a reference image as an array of pixels featuring the RGB color vector x.
The reference image is input to the reference vector generator 102, and a reference vector set is obtained. The reference vector is a parameter consisting of a sequence of mean x-covariance X of color vector x and covariance X and mean p-and covariance P of position vector p. The reference vector generator 102 classifies the pixels of the reference image from the feature vector (x, p). An example of this result is shown as an ellipse in FIG. The result of clustered pixel clusters is not always guaranteed to form an elliptical region, but since position is used as a feature vector in addition to color, a cluster of pixels with similar colors and close distances is one cluster. To form. This is based on the assumption that the color vector and the position vector are independently distributed in the cluster shown in FIG.

In the example shown in FIG. 4, one reference vector is assigned to each of the black portion of the hair and the skin color portion of the face. In this way, by clustering the feature vectors of pixels in which the color vector and the position vector are combined, "where and what color distribution is present" are expressed. A procedure for obtaining this reference vector set will be described with reference to FIG. FIG. 2 shows the reference vector generator 10 in FIG.
2 is a functional block diagram of No. 2, 201 is a frame memory that stores a reference image, 202 is a random address generation unit that randomly generates a position vector p of an image, and 203 is a position vector p obtained by the random address generation unit 202.
By this, the color vector x of the pixel at that position is read out, and the feature vector synthesis unit that synthesizes and outputs the feature vector (x, p), 204 is the reference vector determination unit, 205 is the reference vector change unit, and 206 is the reference vector. It is a storage memory. The reference vector storage memory 206 has 1 to N slots for storing N reference vectors. Let the reference vector set be θ _i = (x− _i , X _i , p− _{i i} , P _i ),
It is expressed as 1 ≦ i ≦ N. i corresponds to the slot number. Here, each slot of the reference vector memory 206 is associated with a valid flag that takes a binary value of 1,0. When 1 is valid, when 0 is invalid. The clustering procedure is shown below.

Step 1: First, the valid flags of c slots (c≤N) of N slots are set to 1 and the others are set to 0. Let θ be the set of reference vectors at discrete time t.
_{i (t) = (x¯ i} (t), X i (t), p¯ i (t), P
_i (t)), 1 ≦ i ≦ N. Initialize the memory contents of the reference vector slot which is 1 of the valid flag. Initialization is performed assuming that the covariance matrices X _i (0) and P _i (0) are unit matrices, and the average x− _i (0) of the color vectors x.
And the average p- _i (0) of the position vector p are set at random. The discrete time t at the time of initialization is 0.

Step 2: The feature vector synthesizing unit 203 reads the color vector x of the pixel at that position by the position vector p obtained by the random address generating unit 202, synthesizes the feature vector, and outputs it. The reference vector determination unit 204 calculates the distance from the feature vector for the reference vector in the slot whose valid flag is 1 among the N slots. This distance is evaluated by (Equation 1) (t means transposition of matrix and vector).

[0019]

[Equation 1]

Here, | X | and | P | are determinants of each covariance matrix. In (Equation 1), the pixel distribution is the color vector x
And, assuming that the position vector p is normally distributed with the mean x ￣ _i , p ￣ _i and the covariances X _i , P _i , it can be obtained from the logarithm (log likelihood) of the probability density function. It is a statistical measure that can Of the N slots, the distance is calculated with respect to the reference vector whose effective flag is 1, and the reference vector determination unit 204 uses the number k and the sample of the feature vector as the reference vector. Send to the changing unit 205. Reference vector changing unit 205
Then, using the reference vector of slot number k as a sample (Equation 1)
Under the distance function of, it becomes closer (Equations 2, 3, 4,
Change in 5). (Equation 6) is an experimentally determined gain coefficient.

[0021]

[Equation 2]

[0022]

[Equation 3]

[0023]

[Equation 4]

[0024]

[Equation 5]

[0025]

[Equation 6]

Then, the time t is incremented by 1 and the above step 2 is repeated a predetermined number of times (for example, 10,000 times), whereby clustering is performed by self-organization of pixels.

That is, one cluster is formed by modifying the distribution parameter data so that the reference vector closest to the randomly selected feature vector is closer to the sample. Random address generation is
This is to prevent bias when changing the sequential reference vector. In the repetition of step 2, the reference vector changing unit 205 satisfies the condition 1: (total number of reference vector slots N) − (currently valid reference vector slot number c)> 0, and condition 2: ln | X _{If i} | + ln | P _i | is greater than or equal to a predetermined threshold value, the cluster represented by the reference vector θ _i is divided. For this purpose, the reference vector θ _i is duplicated in the reference vector slot j whose valid flag is 0 (invalid), and θ _j = θ _i is created. Then, the average vector is adjusted by ε, respectively. The value of ε is experimentally determined. This is shown in (Equations 7, 8, 9, 10).

[0028]

[Equation 7]

[0029]

[Equation 8]

[0030]

[Equation 9]

[0031]

[Equation 10]

Then, the valid flag of the slot j is set to 1 and the number of valid reference vector slots c is increased by 1, and used for the repetition of self-organization thereafter. As a result, ln | X
_A cluster having a large _i | + ln | P _i | is divided.

Further, during the repetition of step 2, the valid flag of the slot of the reference vector for which the randomly selected sample vector has not been obtained even once for the predetermined period, that is, the shortest distance is not set, is set to 0. , The effective reference vector slot number c is decremented by 1. This is necessary to cancel the parameters of the reference vector that have converged to the wrong value.

By the above processing, a reference vector set including N reference vectors including invalid reference vectors is generated for the reference image and sent to the reference vector changing means 103 in FIG. Here, a part of the reference vector set is invalidated. In the example of FIG. 4, it is desired to search for an image including the person image in the center. At this time, since the background image is not the search target, the reference vector belonging to this area is manually invalidated. That is, the valid flag is set to 0. In FIG. 4, this is indicated by a dashed ellipse. The reference vector set thus changed is sent to the correlation value calculator 104. FIG. 3 is a diagram for explaining the operation of the correlation calculator 104. FIG. 3 is a functional block diagram of the correlation calculator 104. Reference numeral 301 is a frame memory that stores a search target image, 302 is a random address generator that randomly generates a position vector p of the image, and 303 is a random address generator 302. A feature vector combining unit that reads out the color vector x of the pixel at that position from the position vector p obtained in step 1, and combines and outputs the feature vector (x, p),
Reference numeral 304 is a reference vector determination unit, 305 is a correlated pixel counting unit, and 306 is a reference vector storage memory. The reference vector set changed by the reference vector changing unit 205 is stored in the reference vector storage memory 306 of FIG. The search target image data is stored in the frame memory 301. The feature vector synthesis unit 303 reads the color vector x of the pixel at that position from the frame memory 301 based on the position vector p obtained by the random address generation unit 302, synthesizes the feature vector (x, p), and outputs it. . The reference vector determination unit 304 calculates the distance from the feature vector for the reference vector in the slot whose valid flag is 1 among the N slots. This distance is evaluated by (Equation 1). In the distance calculation of (Equation 1), the distance is calculated for the reference vector whose valid flag is 1 in N slots, and the reference vector k which is the shortest distance is obtained. Then, in (Equation 1), only the component shown in (Equation 11) excluding the value of the covariance determinant is sent to the correlated pixel counter 305 as the distance χ.

[0035]

[Equation 11]

Here, the correlation pixel counter 305 counts pixels with the shortest distance χ <threshold value T. Specifically, the initial value of the correlation value is set to 0, and when the shortest distance equal to or less than the threshold value T is input from the reference vector determination unit 304, the number is incremented by 1. Here, the distance χ is calculated from the feature vector as an average x ￣ _k ,
a Mahalanobis distance is the normalized distance to the p¯ _k. The Mahalanobis distance follows a χ ² distribution, assuming that the feature vector is normally distributed. Therefore, as the threshold value T, the threshold value used for the χ ² test can be used. Read out the above random feature vector, determine the reference vector and calculate the normalized distance χ, normalized distance χ
The correlation value is calculated by repeating the counting of the correlation pixels by the threshold processing of.

The correlation value is output from the correlation calculator 104 thus constructed and stored in the maximum correlation image index output unit 105. At the same time, index information for the search target image 106 is also recorded. For the search target image stored in the search target image memory 106, the correlation calculator 104 is repeatedly operated to search for the image. Finally, the maximum correlation image index output unit 105 outputs the index of the search target image having the maximum correlation value.

According to the present embodiment, the local color distribution of the image to be searched is stored as the reference vector, so that the image search depending on the position and the color can be performed. Taking FIG. 4 as an example, in the search target image, an image having a black hair region slightly above the center of the screen, a flesh color region at the center, and a clothing color region at the bottom has a high correlation value. You can search not only for color correlation but also for images that have been specified by position.
On the contrary, the present embodiment can also deal with the case where the search is not performed for only the color that does not depend on the position or the position is uncertain. In addition to invalidating the reference vector belonging to the background area by the reference vector changing means, the eigenvalue of the covariance P relating to the position of the reference vector belonging to the person area is changed to be large, whereby It can also be used to search for images when you are moving.

Next, a second embodiment corresponding to claim 2 will be described with reference to FIG. FIG. 6 is a functional block diagram of the image search apparatus according to the second embodiment. In FIG. 6, 6
Reference numeral 01 is a search target image memory, 602 and 606 are luminance normalizers for normalizing RGB color vectors by their sizes, 6
Reference numerals 03 and 607 are reference vector generators, 604 is an indexed search target reference vector memory, 605 is a reference image memory, 608 is a correlation value calculator, and 609 is a maximum correlation image index output device.

In the second embodiment configured as described above, search target images are input in advance from the reference vector generator 603, a reference vector set is obtained for each image, and an indexed search target reference vector memory is obtained. An index is attached to 604 and stored. Also in the second embodiment, RGB color vectors are stored for both the reference image and the search target image. The difference from the first embodiment is that the color vector x is (Equation 1)
As shown in 2), the one normalized by the brightness is used.

[0041]

[Equation 12]

This is performed by the brightness normalizers 602 and 606. The position vector is the same as in the first embodiment. The reference image is processed by the reference vector generator 607, and the reference vector set of the reference image is calculated by the correlation value calculator 608.
Is output to. Reference vector generators 603 and 607
Has exactly the same configuration as the reference vector generator 102 used in the first embodiment, and therefore its description is omitted. Here, the reference vector set of the reference images is S = {θ _i | θ _i = (x− _i , X _i , p− _{i i} , P _i ), 1 together with the valid flag.
≦ i ≦ N), the reference vector set of the search target image is O = {θ _j | θ _j = (x− _{j j} ,
X _j , p − _j , P _j ), 1 ≦ i ≦ N).
In this embodiment, the correlation value calculator 608 calculates the distance between the reference vector sets S and O based on (Equations 13 and 14).

[0043]

[Equation 13]

[0044]

[Equation 14]

Specifically, -f (S, O) obtained by inverting the sign of f (S, O) in (Equation 13) is output to the maximum correlation image index output unit 609 as a correlation value. The correlation value calculator 608 is repeatedly operated on a plurality of reference vector sets stored in the indexed search target reference vector memory 604 to search an image. Finally, the maximum correlation image index output unit 609 outputs the index of the search target image having the maximum correlation value.

According to the second embodiment, by calculating the correlation value between the sets of the reference vector set of the search target image and the reference vector set of the reference image which are obtained in advance, the first embodiment It can be expected to search images faster than Further, since the color vector x obtained by normalizing the RGB color features with the brightness is used as the feature, it is possible to perform the image search based on the color similarity, which is resistant to the change in the brightness.

The pixel feature amount of the present invention uses color in the above embodiment, but other feature amount may be used.

Each means of the present invention can be realized by software using a computer, or by using a dedicated hardware circuit having each of these functions.

[0049]

As is apparent from the above description,
According to the present invention, the following effects can be obtained.

(1) It is possible to search for an image that reflects the structure of the image as compared with a conventional image search device. For example, an image can be searched for by using a rough sketch image or the like.

(2) A search in which the position of the key feature of the search is uncertain can be dealt with by changing the distribution parameter regarding the position of the reference vector.

[Brief description of drawings]

FIG. 1 is a block diagram of a first embodiment of an image search device according to the present invention.

FIG. 2 is a block diagram of a reference vector generator in the first and second embodiments of the image search device of the invention.

FIG. 3 is a block diagram of a correlation placement calculator in the first embodiment.

FIG. 4 is an explanatory diagram of an operation in the first embodiment.

FIG. 5 is an explanatory diagram of a feature vector distribution in the first embodiment and the second embodiment.

FIG. 6 is a block diagram of a second embodiment of the image search device of the invention.

FIG. 7 is a block diagram of a conventional image search device.

[Explanation of symbols]

101 Reference Image Memory 102 Reference Vector Generator 103 Reference Vector Change Unit 104 Correlation Value Calculator (Correlation Calculator) 105 Maximum Correlation Image Index Output Unit 106 Search Target Image Memory 201 Reference Image Frame Memory 202 Random Address Generator 203 Feature Vector Synthesis Part 204 Reference vector determination part 205 Reference vector change part 206 Reference vector storage memory 301 Search target image frame memory 302 Random address generation part 303 Feature vector synthesis part 304 Reference vector determination part 305 Correlated pixel counting part 306 Reference vector storage memory 601 Search target Image memory 602, 606 Luminance normalizer 603, 607 Reference vector generator 604 Indexed search target reference vector memory 605 Reference image memory 608 Correlation value calculator (correlation Calculation means) 609 maximum correlation image index output device

Claims (2)

[Claims] 1. A reference vector generation means for clustering pixels of a reference image using the feature amount and position of the pixel as a feature vector to generate a reference vector set consisting of distribution parameters of the feature vector, and a search target image. From the above, a plurality of pixels are extracted, the feature amount and position of the pixel are used as the feature vector, and the correlation value of the image is obtained based on the distance corresponding to the error to the feature vector distribution of the pixel described by the reference vector. An image search apparatus comprising: a correlation calculation unit, holding results of the correlation calculation unit for the plurality of search target images, and outputting a search target image having a large correlation value as a search result. 2. A reference vector generating means for clustering pixels of a reference image and a search target image using the feature amount and position of the pixel as a feature vector to generate a reference vector set including distribution parameters of the feature vector. A reference vector set of the reference image, and a correlation calculation unit that calculates a correlation value between the reference vector set of the search target image, holding the result of the correlation calculation unit for the plurality of search target images, An image search apparatus, which outputs a search target image having a large set correlation value as a search result.