JP2020190861A - Image search apparatus and image search method - Google Patents

Abstract

To provide an image search apparatus and an image search method for displaying, at the top of search results, images which can be easily compared while realizing highly accurate similar image search.SOLUTION: An image search apparatus 1000 comprises: an input unit 100 that accepts input of images including a search target and annotation data of the images; a feature extraction unit 102 that extracts a feature quantity of the image; a comparison easiness calculation unit 104 that calculates comparison easiness between the images; a learning unit 103 that learns a method for extracting the feature quantity of the image and a method for calculating the comparison easiness using the images and the annotation data; a search unit 101 that determines the order of search results to be displayed on a display screen based on the feature quantity of the image learned by the learning unit 103 and the comparison easiness between the images; and a display unit 105 that displays the search results on the display screen in the order determined by the search unit 101.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image search device and an image search method.

Patent Document 1 is a background technique in this technical field. In the publication, the rotation angle is calculated in the process of calculating the local feature amount required for the search, and when the image is displayed as the search result, the necessity of rotation normalization is determined based on the estimated rotation angle. When the judgment is made and the rotation angle is equal to or more than a predetermined value, the sample image is rotated and normalized based on the query image and then displayed to reduce the load due to the mental rotation when checking the search result. At the same time, a technique for obtaining search results while suppressing the amount of calculation is disclosed.

Japanese Unexamined Patent Publication No. 2016-47031

There is a monitoring system as an application of similar image search. For example, if a observer finds a suspicious person, similar image retrieval can be used to help the observer track the suspicious person. In addition, for example, when a watcher finds a baggage left behind, the owner of the baggage can be identified by using a similar image search. In such an application, it is preferable that the same individual such as the same person or the same baggage appears at the top of the search results, and a highly accurate search is required.

Conventionally, as a method for performing a highly accurate similar image search, a method has been proposed in which a feature amount is extracted from an image using a convolutional neural network (CNN) and the similarity degree of the feature amount is calculated. In this method, CNN is learned so as to extract similar features for various images of the same individual having different search target positions and sizes in the lighting environment and images, so that it is highly accurate. You can search. On the other hand, since learning is performed so as to extract similar features for any image, an image that is easy to compare with the query image does not always appear at the top of the search results. In order to instantly determine whether the query image and the comparison image of the search result are the same individual or different individuals, it is preferable to display an image that is easy to compare at the top. Also, for example, if an individual is tracked in the camera and there are multiple images that are already known to be the same individual, an image that is easy to compare with the query image is selected from them and used as the search result. It is preferable to display.

Patent Document 1 describes a method of calculating the rotation angle of a comparison image based on a query image and rotating the comparison image when displaying a search result to improve the ease of comparison. The described method uses local features to calculate similarity and rotation angle, and it is said that the ease of comparison is improved by aligning the orientations of the same individuals in the image. However, this method cannot be applied to a method such as CNN in which pre-learning is performed to perform a similar image search, and it is not possible to improve the ease of comparison while realizing a highly accurate search. In addition, it is not easy to calculate the angle when the appearance of the object is different in the image, such as when the object is photographed in a different direction or a part of the object is shielded, and the ease of comparison is improved. It is difficult.

One aspect of the present invention is to provide an image search device capable of displaying an image that can be easily compared at a higher level of a search result while realizing a highly accurate similar image search.

One aspect of the present invention is an input unit that accepts input of an image including a search target and annotation data of the image, a feature extraction unit that extracts a feature amount of the image, and a comparison that calculates the ease of comparison between the images. Using the ease calculation unit, the image and the annotation data, a learning unit that learns a method for extracting features of the image and a method for calculating the comparison ease, and a learning unit that learns the image. A search unit that determines the order of search results to be displayed on the display screen based on the feature amount and the ease of comparison between the images, and a display that displays the search results on the display screen in the order determined by the search unit. It is configured as an image search device characterized by including a unit.

According to one aspect of the present invention, it is possible to display an image that can be easily compared at the top of the search result while realizing a highly accurate similar image search.

Basic configuration diagram showing an embodiment Image search flowchart showing an example Flow chart for learning comparison ease calculation showing one example Annotation for learning comparison ease calculation showing an example Annotation method for learning comparison ease calculation showing an example Search result display screen showing an example Annotation method for learning comparison ease calculation showing an example Configuration diagram showing an embodiment Tracking flowchart showing an example Image search flowchart showing an example

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and are appropriately omitted and simplified for the sake of clarification of the description. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range and the like disclosed in the drawings.

In the following description, various information may be described by expressions such as "table" and "list", but various information may be expressed by a data structure other than these. The "XX table", "XX list", etc. may be referred to as "XX information" to indicate that they do not depend on the data structure. When describing the identification information, expressions such as "identification information", "identifier", "name", "ID", and "number" are used, but these can be replaced with each other.

When there are a plurality of components having the same or similar functions, they may be described by adding different subscripts to the same reference numerals. However, when it is not necessary to distinguish between these plurality of components, the subscripts may be omitted for explanation.

Further, in the following description, a process performed by executing a program may be described, but the program is executed by a processor (for example, CPU, GPU) to appropriately store a predetermined process as a storage resource (a storage resource (for example, CPU, GPU). Since it is performed using, for example, a memory) and / or an interface device (for example, a communication port), the main body of processing may be a processor. Similarly, the main body of processing for executing a program may be a controller, a device, a system, a computer, or a node having a processor. The main body of the processing performed by executing the program may be an arithmetic unit, and may include a dedicated circuit (for example, FPGA or ASIC) that performs a specific processing.

The program may be installed from the program source into a device such as a calculator. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server includes a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Further, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

In the present embodiment, as an example of the image search device, an example of searching a luggage image such as a suitcase will be described. The search target does not have to be a baggage, and may be a target that can be imaged (for example, an image target such as an object).

FIG. 1 is a diagram showing a configuration example of an image search device used in the present embodiment. As shown in the figure, the image search device 1000 of the present embodiment includes an input unit 100, a search unit 101, a feature extraction unit 102, a learning unit 103, a comparison ease calculation unit 104, and a display unit 105. .. Each block will be described below. The hardware of the image search device 1000 includes, for example, an input device such as a keyboard and a mouse, a display device such as an LCD (Liquid Crystal Display), an input / output device such as a touch panel, an HDD (Hard Disk Drive), and an SSD (Solid). It is composed of a computer such as a general PC or a server equipped with a storage device such as State Drive) and a storage medium such as a memory.

The input unit 100 receives data necessary for executing each process when executing search or learning. The input unit 100 receives the query image and the image group to be compared when executing the search, outputs the data to the search unit 101, and outputs the image and annotation data necessary for learning when executing the learning. It receives and outputs those data to the learning unit 103.

The search unit 101 manages data for searching and performs calculation processing. In the search process, the search unit 101 first receives the query image and the image group to be compared from the input unit 101, and outputs the data to the feature extraction unit 102 and the comparison ease calculation unit 104. The search unit 101 acquires the feature amount extracted from the image by the feature extraction unit 102 and the comparison ease calculated by the comparison ease calculation unit 104 from the image, and determines the display order of the comparison image group from them. Finally, the search unit 101 outputs the query image, the comparison image group, and the display order to the display unit 105. The comparison ease is information indicating the ease of comparison between images including the search target, that is, the ease of comparison and a certain degree of comparison. Hereinafter, the ease of comparison is described as indicating the ease of comparison between the above images, but the ease of comparison between the search targets included in the images may be referred to as the ease of comparison.

The feature extraction unit 102 extracts the feature amount required for the similarity calculation from the image. When executing the search, the feature extraction unit 102 extracts the feature amount from the image acquired from the search unit 101, and outputs the extracted feature amount to the search unit 101. When executing learning, the feature extraction unit 102 extracts a feature amount from the image acquired from the learning unit 103, and outputs the extracted feature amount to the learning unit 103. Further, the feature extraction unit 102 holds a parameter necessary for extracting the feature amount, and sends and acquires this parameter to the learning unit 103 when executing learning.

The learning unit 103 updates the parameters used when the feature extraction unit 102 extracts the feature amount and the parameters used when the comparison ease calculation unit 104 calculates the comparison ease by learning. When executing learning, the learning unit 103 acquires images and annotation data necessary for learning from the input unit 100. When updating the parameters for extracting the feature amount, the learning unit 103 outputs an image to the feature extraction unit 102, acquires the feature amount extracted from the image by the feature extraction unit 102, and obtains the feature amount and the feature amount. Calculate the update value of the parameter from the annotation data. The calculated update value is output to the feature extraction unit 102. When updating the parameters for calculating the comparison ease, the learning unit 103 outputs an image to the comparison ease calculation unit 104, and acquires the comparison ease calculated by the comparison ease calculation unit 104 from the image. Then, the parameter update value is calculated from this comparison ease and the annotation data. The calculated update value is output to the comparison ease calculation unit 104.

The display unit 105 displays the search result. The display unit 105 acquires the query image, the comparison image group, and the display order to be displayed from the search unit 101 when executing the search, and arranges the results according to the display order. In addition, the display unit 105 can acquire the similarity and comparison ease of images from the search unit 101 and display them.

Each part of the image search device is realized by executing the program. For example, the CPU of the image search device 1000 reads a program from the ROM and executes it to realize the functions of the above-mentioned parts. The program may be provided to the image search device 1000 by being read from a storage medium such as a USB (Universal Serial Bus) memory or being downloaded from another computer via a network.

FIG. 2 is a diagram showing an example of an image search flow used in the present embodiment. The search flow will be described below.

In step 200, the search unit starts the search process by the search start operation of the search executor. The search executor does not have to be a person, and may be a program that executes the search.

In step 201, the input unit 100 receives input of an image to be a query for the search executor to operate the input device to perform a search and a comparison image group to be searched.

In step 202, the input unit 100 outputs the image to be the query input from the search executor in step 201 and the comparison image to be searched to the search unit 101.

In step 203, the search unit 101 outputs these images acquired from the input unit 100 in step 202 to the feature extraction unit 102.

In step 204, the feature extraction unit 102 extracts the feature amount from these images acquired from the search unit 101 in step 203. For example, CNN is used for extracting the feature amount. CNN inputs an image and outputs a numerical vector as a feature. The numerical vector is calculated by a convolution operation using the parameters held by the CNN. This parameter can be set to any value, and a similar vector is output for the image of the same individual by learning.

In step 205, the feature extraction unit 102 outputs the feature amount extracted in step 204 to the search unit 101.

In step 206, the search unit 101 outputs the query image and the comparison image to be searched obtained from the input unit 100 in step 202 to the comparison ease calculation unit 104.

In step 207, the comparison ease calculation unit 104 calculates the comparison ease with respect to these images acquired from the search unit 101 in step 206. For example, CNN is used to calculate the ease of comparison. CNN inputs two images and outputs the value of the ease of comparison for the two images. The value is, for example, in the range of 0.0 to 1.0, and the larger the value, the easier the comparison. The comparison ease is calculated by a convolution operation using the parameters held by the CNN. This parameter can be set to any value, and a large value is output for a combination of images that can be easily compared by learning.

In step 208, the comparison ease calculation unit 104 outputs the comparison ease calculated in step 207 to the search unit 101.

In step 209, the search unit 101 determines the display order of the search results using the feature amount acquired from the feature extraction unit 102 in step 205 and the comparison ease acquired from the comparison ease calculation unit 104 in step 208. To calculate the similarity between the query image and the comparison image using the feature amount, for example, the Euclidean distance of the numerical vector which is the feature amount is calculated. By normalizing the numerical vector so that the magnitude of the vector becomes 1.0 in advance, the range of the Euclidean distance is limited to the range of 0.0 to 2.0. Furthermore, the similarity is limited to the range of 0.0 to 1.0 by the following formula, and the larger the value, the higher the similarity.

ただし、f₁およびf₂は比較する二つの画像の特徴量である数値ベクトルを表し、sは類似度を表す。表示順は例えば以下のスコアを計算することにより決定する。

However, f ₁ and f ₂ represent the numerical vector which is the feature quantity of the two images to be compared, and s represents the similarity. The display order is determined, for example, by calculating the following scores.

However, p is the score, s is the similarity calculated in the above process, c is the comparison ease calculated in step 207, α is the importance for determining the display order of similarity and comparison ease, and is from 0.0. Set in the range of 1.0. When α is small, the items with the highest degree of similarity are displayed in order, and when α is large, the items with the highest degree of comparison are displayed in order. If α can be dynamically changed on the search result display screen, the importance of similarity and comparison ease can be changed according to the request of the search executor, and flexible display becomes possible.

In step 210, the search unit 101 outputs the query image, the comparison image group, and the score in the display order calculated in step 209 to the display unit 105.

In step 211, the display unit 105 determines the display order using the query image, the comparison image group, and the score of the display order acquired from the search unit 101 in step 210, and displays the search result. To display the search results, the display unit 105 displays, for example, a query image, and displays the comparison image group next to the query image group on the display device by arranging them from top to bottom so that the display scores are in descending order.

In step 212, the input unit 100 receives from the search executor a decision as to whether to continue or end the search.

In step 213, when the input unit 100 accepts the decision to end the search (step 212; Yes), the search ends. On the other hand, when the input unit 100 does not accept the decision to end the search (step 212; No), the process returns to step 201 and the subsequent processing is repeated.

By using the image search method as in this embodiment, the image of the same individual as the query image is found from the comparison image group, and the query image is displayed while realizing a highly accurate search to be displayed at the top of the search results. On the other hand, images that are easy to compare can be selected and tabulated, and the confirmation time of search results can be reduced. Although the method of Patent Document 1 can improve the ease of comparison, the accuracy of the search is lowered because the method of extracting the feature amount cannot be learned. In addition, when using a method for learning a general feature extraction method, learning is performed so that the similarity is high for all the images of the same individual, so the ease of comparison of the comparison image with respect to the query image is taken into consideration. Not done. In contrast to these methods, this method learns not only the feature amount extraction method but also the comparison ease calculation method and reflects it in the display order, so that it becomes easy to compare the query image of the search result with the comparison image.

In this flow, it is premised that the query image and the comparison image are one independent image, but these do not have to be one, and may be a plurality of images of the same individual. In such a case, a plurality of numerical vectors representing the feature amount can be created for one individual. For example, the average value of those numerical vectors may be calculated and combined into one numerical vector, or the query image and the query image. The similarity of all combinations may be calculated for a plurality of numerical vectors of the comparison image, and the minimum value or the average value of the similarity may be calculated. By using such a method, it becomes possible to perform a similarity calculation that is more robust to changes in the image than a one-to-one similarity calculation.

Further, in this flow, when performing a search, the search executor inputs an image and extracts the feature amount and calculates the comparison ease one by one. For example, the feature amount is extracted and the comparison ease is calculated in advance. It is possible to reduce the processing time when executing the search by saving the image, the extracted features, and the calculated comparison ease in the storage medium, and using the one saved in the storage medium at the time of searching. it can.

In addition, in this flow, a general CNN was used as a feature extraction method, but a feature is extracted using another conventionally known method, for example, a CNN that is robust against rotation. As a result, the feature amount does not change even when the luggage image is rotated, and the search accuracy can be improved.

FIG. 3 is a diagram showing an example of a flowchart for learning the comparison ease calculation used in the present embodiment. The learning flow will be described below.

In step 300, the learning unit starts the learning process by the learning start operation of the learning executor. The learning executor does not have to be a person, and may be a program that executes learning.

In step 301, the input unit 100 receives input of image and annotation data for the learning executor to operate the input device for learning.

In step 302, the input unit 100 outputs the image and annotation data input from the learning executor in step 301 to the learning unit 103.

In step 303, the learning unit 103 outputs the image acquired from the input unit 100 in step 302 to the comparison ease calculation unit 104.

In step 304, the comparison ease calculation unit 104 calculates the comparison ease with respect to the image acquired from the learning unit 103 in step 303. The method of calculating the ease of comparison is the same as in step 207 in FIG.

In step 305, the comparison ease calculation unit 104 outputs to the learning unit 103 the comparison ease calculated in step 304 and the parameters used when calculating the comparison ease.

In step 306, the learning unit 103 uses the comparison ease acquired from the comparison ease calculation unit 104 in step 305 and the annotation data acquired from the input unit 100 in step 302, and the comparison ease calculation unit 104 in step 305. Update the parameters for calculation of comparison ease obtained from. For updating the parameters, for example, the stochastic gradient descent method is used. The objective function of the stochastic gradient descent method uses, for example, the mean square error of the estimated value of the comparative ease acquired from the comparative ease calculation unit 104 and the correct answer value of the comparative ease included in the annotation data acquired from the input unit 100. .. The mean square error E is calculated as follows.

ただし、ｃ_ｇは比較容易度の正解値、ｃ_ｐは比較容易度の推定値である。平均二乗誤差Ｅを減少させるようにパラメータを更新する。

However, c _g is the correct answer value of the ease of comparison, and c _p is the estimated value of the ease of comparison. Update the parameters to reduce the mean square error E.

In step 307, the learning unit 103 outputs the parameters for comparison ease calculation updated in step 306 to the comparison ease calculation unit 104.
In step 308, the comparison ease calculation unit 104 is set to use the comparison ease calculation parameter acquired from the learning unit 103 in step 307 for the comparison ease calculation.
In step 309, the input unit 100 receives a decision from the learning executor as to whether to continue or end the learning.
In step 310, when the input unit 100 accepts the decision to end learning (step 309; Yes), the learning ends. On the other hand, when the input unit 100 has not received the determination of the end of learning (step 309; No), the process returns to step 301, and the subsequent processing is repeated.

When the comparison ease calculation is learned as in the present embodiment, various factors can be considered for the ease of comparison. In the method of Patent Document 1, the ease of comparison can be considered for the rotation of the image, but other factors cannot be considered. On the other hand, in this method, in addition to rotation, for example, shielding of luggage can be considered. Such images will be excluded from the top of the search results if you learn to reduce the ease of comparison when your luggage is shielded by other luggage and you cannot see the important parts when comparing your luggage. can do.

The learning method of the comparison ease calculation in this flow can also be used to update the parameters used for learning the feature extraction by changing the objective function. An objective function in the learning of feature extraction includes, for example, a method of learning to identify an individual by Softmax cross entropy. The training data set at this time does not have to be one, and a plurality of data sets can be used. When using multiple datasets, instead of learning to identify each individual from all the individuals included in the multiple datasets, each individual from among the individuals in each dataset is shared while sharing the features. The accuracy can be improved by learning to identify the individual.

FIG. 4 is a diagram showing an example of annotation data for learning the comparison ease calculation in the present embodiment. The annotation data 400 is composed of a combination of two image IDs including a search target and a set of ease of comparison between the two images. The image ID is an identifier given to one image so as not to overlap with other images, and is used to uniquely identify each image. The ease of comparison between the one image and the other image is a target value estimated by the comparison ease calculation unit 104, and a numerical value indicating that the larger the value in the range of 0 to 1.0, the easier the comparison. Is. The order of image ID1 and image ID2 is irrelevant, and the same combination of image ID1 and image ID2 is treated as the same annotation data.

FIG. 5 is a diagram showing an example of an annotation method for learning the comparison ease calculation in the present embodiment. The display unit 104 displays different images 501-504 of the same individual and a comparison ease level 505 as the annotation screen 500 to be displayed on the display device. The displayed numerical value is the comparison ease of each image input by the annotation executor. For example, the numerical value 505 is the comparison ease 0.8 input by the annotation executor operating the input device for the image 502 and the image 504. It shows that it was done.

As in the present embodiment, in a situation where a human confirms a search result and compares images in the same situation, a human sense of ease of comparison is searched by creating annotation data by inputting the degree of comparison. It can be reflected in the result.

In this display screen, the images to be annotated are arranged at the four corners of the screen. For example, a table display screen in which the two images and the ease of comparison are arranged in the row direction and each annotation data is arranged in the column direction may be used. By using such a display screen, it is possible to deal with a large number of images to be annotated.

FIG. 6 is a diagram showing an example of a search result display screen according to the present embodiment. In the following, the search unit 101 receives an operation from the user by the input device, changes the order of displaying the search results according to the similarity of the feature amount of the image and the importance of the comparison ease, and changes the display unit. 105 displays the result of the change by the search unit 101 on the display screen.

For example, the display unit 105 uses the search result display screen 600 to be displayed on the display device as the query image 601 used for the search, the search result 602 of the comparison image group, the similarity 603 between the query image and the comparison image, and the query image and the comparison image. The comparison ease 604 and the α value 605 that sets the importance of the similarity and the comparison ease are displayed. On this display screen, the α value 605 is set to 0.5, and the score is calculated assuming that the similarity and the ease of comparison are of equal importance. The search results 602 are arranged so that the scores are in descending order.

As in the present embodiment, the input unit 100 receives the setting operation of the α value 605 from the search executor and enables the α value 605 to be arbitrarily set on the display screen, so that the similarity and the comparison ease can be improved. You can change the importance and check the search results flexibly.

Although the α value is input as a numerical value on this display screen, it may be set by using, for example, a slide bar. By using the slide bar, it becomes possible to easily input the α value 605.

In the present embodiment, as an example of the image search device, an example of searching a luggage image such as a suitcase will be described. The search target does not have to be a baggage, and may be a target that can be imaged as in the case of the first embodiment. In the present embodiment, the comparison ease is calculated from the rotation angle of the image, and the rotation angle of the image is used as the annotation data for learning the comparison ease calculation. Others are the same as in Example 1.

FIG. 7 is a diagram showing an example of an annotation method for learning the comparison ease calculation in the present embodiment.
The display unit 104 displays different images 701-702 and rotation angles 703 of the same individual as the annotation screen 700 to be displayed on the display device. The input unit 100 receives input of the rotation angle of the luggage of the reference image 701 and the comparison image 702 from the annotation executor. The comparison ease calculation unit 104 calculates the comparison ease from the input rotation angle as follows.

ただし、ｃは比較容易度、θは回転角度を表している。

However, c represents the ease of comparison and θ represents the rotation angle.

As in the present embodiment, by limiting the ease of comparison to the rotation angle of the image, there is a feature that the annotation executor can easily annotate.

In the present embodiment, as an example of the image search device, an example of searching a luggage image such as a suitcase will be described. The search target does not have to be a baggage, and may be a target that can be imaged as in the case of the first embodiment. In the present embodiment, the tracking of the object is performed in advance, and when there are a plurality of images of each individual, an image that can be easily compared with the query image is selected from the plurality of images. Others are the same as in Example 1.

FIG. 8 is a diagram showing a configuration example used in the present embodiment. As shown in the figure, the image search device 8000 of the present embodiment is configured by adding the tracking unit 800 and the tracking data storage unit 801 to the configuration example of FIG. Each block added below will be described.

The tracking unit 800 detects a package from each of a plurality of images that are continuous in time series, and tracks the package in the time series direction. When executing tracking, first, the tracking unit 800 acquires continuous images in time series from the input unit 100. Next, the tracking unit 800 performs tracking processing on the acquired image. Finally, the result of the tracking process is output to the tracking data storage unit 801.

When the tracking unit 800 finishes the tracking process, the tracking data storage unit 801 acquires the tracking result and stores it in a storage device or a storage medium.

FIG. 9 is a diagram showing an example of a tracking flowchart used in the present embodiment. The tracking flow will be described below.

In step 900, the tracking unit starts the tracking process by the tracking start operation of the tracking executor. The tracker does not have to be a person, but may be a program that executes the track.

In step 901, the input unit 100 receives input of a time-series continuous image for the tracking executor to operate the input device to perform tracking.

In step 902, the input unit 100 outputs the time-series continuous images input from the tracking executor in step 901 to the tracking unit 800.

In step 903, the tracking unit 800 performs tracking processing on the time-series continuous images acquired from the input unit 100 in step 902. For tracking, for example, first, the package of each image is detected by using various conventionally known methods. The detection result is the type of luggage and the position and size in the image. Next, the tracking unit 800 connects the packages detected as the same type that are spatially close to each other with respect to the detection results of the images adjacent to each other in time series, and assigns the same tracking ID to the images including the linked packages. For the calculation of the given distance, for example, the Euclidean distance of the detection position is used. The detection result generated in this way is the type of the package, the image ID in which the package is reflected, the tracking ID of the image detected as the same type of package, and the position and size of the package in each image.

In step 904, the tracking unit 800 outputs the detection result generated in step 903 to the tracking data storage unit 801 as the tracking result.

In step 905, the tracking data storage unit 801 stores data indicating the tracking result acquired from the tracking unit 800 in step 904 in a storage device or a storage medium.

In step 906, the input unit 100 receives a decision from the tracking executor whether to continue or end the tracking.

In step 907, when the input unit 100 accepts the decision to end the tracking (step 906; Yes), the tracking ends. On the other hand, when the input unit 100 does not accept the determination of the end of tracking (step 906; No), the process returns to step 901 and the subsequent processing is repeated.

When tracking as in the present embodiment is performed, a plurality of images of the same individual can be collected as a set before the search is executed, so that the number of images to be compared can be reduced and the search accuracy can be improved. .. Further, when calculating the similarity, since a plurality of images can be used for each of the query image and the comparison image, it is possible to perform the similarity calculation that is robust against changes in the images, and it is possible to improve the search accuracy.

FIG. 10 is a diagram showing an example of an image search flowchart used in the present embodiment. Only the steps different from the search flow of FIG. 2 will be described below.

In step 1000, the input unit 100 receives an input of an image as a query for the search executor to operate the input device to perform a search.

In step 1001, the tracking data storage unit 801 outputs the image group and the tracking data to be compared with the search to the search unit 101.

In step 1002, the search unit 101 outputs the query image acquired from the input unit 100 in step 1000 and the comparison image group acquired from the tracking data storage unit 801 in step 1001 to the feature amount extraction unit 102.

In step 1003, the search unit 101 outputs the query image acquired from the input unit 100 in step 1000 and the comparison image group acquired from the tracking data storage unit 801 in step 1001 to the comparison ease calculation unit 104.

In step 1004, the search unit 101 determines the display order of the search results using the feature amount acquired from the feature extraction unit 102 in step 205. The search unit 101 calculates the similarity from the feature amount by using, for example, the similarity calculation method of step 209 as a score for determining the display order, and uses the similarity.

In step 1005, the search unit 101 selects an image to be displayed using the ease of comparison from a plurality of images showing each individual to be compared. The search unit 101 selects, for example, the image having the highest degree of comparison with respect to the query image from a plurality of images.

By using the image search method as in the present embodiment, it is possible to select an image that is easy to compare with the query image from a plurality of images indicating the same individual to be compared, and the confirmation time of the search result is reduced. be able to.

As described above, according to each of the above-described embodiments, it is possible to provide an image search device capable of displaying an image that can be easily compared at the top of the search result while realizing a highly accurate similar image search. Also, learn to calculate the comparison ease of images together with the extraction method of similar features, and sort the search results using both the similarity of features and the ease of comparison when searching. Can provide a method for improving the ease of comparison.

The present invention is not limited to the above-mentioned examples, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

1000, 8000 Image search device 100 Input unit 101 Search unit 102 Feature extraction unit 103 Learning unit 104 Comparison ease calculation unit 105 Display unit 800 Tracking unit 801 Tracking data storage unit

Claims (10)

An input unit that accepts input of an image including a search target and annotation data of the image,
A feature extraction unit that extracts the feature amount of the image,
A comparison ease calculation unit that calculates the comparison ease between the images,
A learning unit that learns a method for extracting features of the image and a method for calculating the degree of comparison using the image and the annotation data.
A search unit that determines the order of search results to be displayed on the display screen based on the feature amount of the image learned by the learning unit and the ease of comparison between the images.
A display unit that displays the search results on the display screen in the order determined by the search unit, and
An image search device characterized by comprising. The image search device according to claim 1.
The input unit receives input of data including one image, another image, and ease of comparison between the one image and the other image as the annotation data.
The learning unit performs the learning using the data input from the input unit.
An image search device characterized by this. The image search device according to claim 1.
The search unit receives an operation from the user by the input device, and changes the order of displaying the search results according to the similarity of the feature amount of the image and the importance of the comparison ease.
The display unit displays the result of the change by the search unit on the display screen.
An image search device characterized by this. The image search device according to claim 1.
The comparison ease calculation unit calculates the comparison ease using the rotation angle of the image.
An image search device characterized by this. The image search device according to claim 1.
A tracking unit that tracks the search target using the images that are continuous in chronological order,
A tracking data storage unit that stores the results of tracking by the tracking unit is provided.
The search unit searches for an image including the search target from the tracking result accumulated by the tracking data storage unit, and based on the searched image and the comparison ease, displays an image to be displayed on the display screen. select,
An image search device characterized by this. The input unit accepts the input of the image including the search target and the annotation data of the image,
The feature extraction unit extracts the feature amount of the image and
The comparison ease calculation unit calculates the comparison ease between the images, and then
The learning unit learns the method of extracting the feature amount of the image and the method of calculating the degree of comparison using the image and the annotation data.
The search unit determines the order of the search results to be displayed on the display screen based on the feature amount of the image learned by the learning unit and the ease of comparison between the images.
The display unit displays the search results on the display screen in the order determined by the search unit.
An image search method characterized by that. The image search method according to claim 6.
The input unit receives input of data including one image, another image, and ease of comparison between the one image and the other image as the annotation data.
The learning unit performs the learning using the data input from the input unit.
An image search method characterized by that. The image search method according to claim 6.
The search unit receives an operation from the user by the input device, and changes the order of displaying the search results according to the similarity of the feature amount of the image and the importance of the comparison ease.
The display unit displays the result of the change by the search unit on the display screen.
An image search method characterized by that. The image search method according to claim 6.
The comparison ease calculation unit calculates the comparison ease using the rotation angle of the image.
An image search method characterized by that. The image search method according to claim 6.
The tracking unit tracks the search target using the images that are continuous in chronological order.
The tracking data storage unit accumulates the results of tracking by the tracking unit,
The search unit searches for an image including the search target from the tracking result accumulated by the tracking data storage unit, and based on the searched image and the comparison ease, displays an image to be displayed on the display screen. select,
An image search method characterized by that.

Images