JP5743938B2 - Associative search system, associative search server, and program - Google Patents

Description

  The present invention relates to an associative search system for searching for a document related to a document given as a search request, and in particular, an associative search system, an associative search server using information on the appearance position of a feature word in a given document, and those It relates to the program to be realized.

  With the spread of computers and the Internet, computerization of document information is rapidly progressing. On the other hand, with the increase of available information, it has become an important issue to search for necessary information from them. In addition, there is an increasing demand for examining the relationship between document groups among a plurality of document databases. For example, there are many requests to search for articles of encyclopedia related to newspaper articles of interest.

  In the case of keyword search technology that is currently in practical use, it is possible to search by switching between multiple document databases, but for a document group included in a document database, the related document group is assigned to the same document database. Alternatively, it is impossible to search from another document database (a search method called document associative search).

  As long as it is limited to the same document database, a document associative search using a document group as a search input can be realized by calculating the similarity between documents in advance. However, when realizing a document associative search between a plurality of document databases, the number of combinations of relevance levels between documents to be calculated in advance increases explosively as the number of document databases increases. For this reason, the reality of the document associative search by the method of calculating the similarity between documents in advance is impossible.

  On the other hand, Patent Document 1 discloses a method for efficiently retrieving a document group related to a document group from an arbitrary document database for an arbitrary document group in a document database designated by a user. Has been.

  The method disclosed in Patent Literature 1 uses only a characteristic word group (characteristic word group) in a search input input as a document group, and realizes a high-speed document associative search. By using this method, the user can check the relevance of the document group while switching between a plurality of different types of document databases, and can search the document with high accuracy and efficiency. Also, this method extracts feature word groups appearing in the document group obtained as a search result, and presents them to the user as an overview (summary) of the search result, thereby determining whether or not the search result by the user is acceptable. It also provides technology to support judgment.

JP 2000-155758 A

  In general, in document retrieval based on words, documents are indexed by words appearing in the document to realize document retrieval. The same applies to Patent Document 1, and when extracting a feature word group from a document, the importance is calculated using a statistical measure (typically tf * idf method) of words included in the document, Extract words in descending order of importance and realize associative search.

  However, in the conventional associative search, a target word is extracted from the entire document. For this reason, when a document includes a plurality of topics, the words are extracted while the feature words of the plurality of topics are mixed. That is, a similar document is searched by comprehensively judging a plurality of topics. For this reason, the result desired by the user is not always obtained.

  In view of this technical problem, the present invention provides an associative search system capable of searching for similar documents for each topic included in a document group serving as a search input.

  For this reason, in the present invention, when extracting a feature word group from a search input document in an associative search, not only the importance of each word but also position information of the word in the search input document is added and extracted. Execute the process. Next, the extracted feature word group is classified based on the importance and appearance position of each word. The classification number of the feature word group may be automatically set by setting a threshold value for the classification score when classifying according to the importance and distance of the feature word in the search input document. The number of classifications may be arbitrarily set on the interface. Finally, the search is executed using each of the feature word groups obtained as a classification result as a search input.

  According to the present invention, even when a document group including a plurality of topics is used as a search input, a group of characteristic words (corresponding to topics included in the document) is not a similar document as a whole document group, but a group of feature words. A similar document can be obtained as an associative search result every time. Thereby, a result closer to the user's request can be presented. Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments.

The figure which shows the conceptual structural example of an associative search system. The figure which shows the structural example of an associative search slave server. The figure which shows the structural example of a search means. The figure which shows the structural example of a characteristic word extraction means. The figure which shows the example of distribution of the characteristic word in a document. The figure which shows the example of the extraction and classification method of a feature word. The figure which shows the example of the extraction and classification method of a feature word. The figure which shows the example of the initial screen in a search client. The figure which shows the example of a display of the search result in a search client. The figure which shows the example of a display of the search result in a search client. The figure which shows the example of a display of the search result in a search client. The figure which shows the example of a display of the confirmation screen of a characteristic word group classification result. The figure which shows an example of indexing. The figure which shows the flow of data and a process between a search client, an associative search master server, and an associative search slave server. The figure which shows the flow of data and a process between a search client, an associative search master server, and an associative search slave server. The figure which shows the flow of data and a process between a search client, an associative search master server, and an associative search slave server.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment of the present invention is not limited to the embodiments described later, and various modifications are possible within the scope of the technical idea.

  FIG. 1 shows a schematic configuration of an associative search system according to an embodiment. This system includes a search client 20 used for inputting a search request and displaying a search result by a user, an associative search slave server 40, 50, 60 for searching a document database, an associative search client 20 and an associative search slave server. The associative search master server 30 that mediates 40, 50, and 60 and the communication network 10 that connects them are configured.

  In the example of FIG. 1, the case where three associative search slave servers that search the document database are connected to the communication network 10 is shown, but the number of associative search slave servers connected to the communication network 10 may be arbitrary. . The number of search clients 20 is also arbitrary.

  In the example of FIG. 1, the search client 10, the associative search master server 30, and the associative search slave servers 40, 50, and 60 are connected via the communication network 10. Alternatively, all may be configured on the same computer.

  FIG. 2 shows a configuration example of the associative search slave server 40. The configuration of the other associative search slave servers 50 and 60 is the same as that of the associative search slave server 40. The associative search slave server 40 includes a memory device 491, an arithmetic processing device 492, an interface device 493, an auxiliary storage device 494, an input device 495, and an output device 496, which are connected to each other via a bus 490.

  The memory device 491 is a storage device such as a RAM (Random Access Memory) that reads and stores a program from the auxiliary storage device 494. The memory device 491 stores programs corresponding to the search means 410 and the feature word extraction means 420, files and data corresponding to the search index 430 and the document database 440 necessary for the execution.

  The arithmetic processing device 492 is an arithmetic processing device such as a CPU (Central Processing Unit) that executes a program stored in the memory device 491. The interface device 493 is an interface device for connecting to an external network or the like. The auxiliary storage device 494 is a storage device such as an HDD (Hard Disk Drive) that stores programs corresponding to the search means 410 and the feature word extraction means 420, files and data corresponding to the search index 430 and the document database 440, and the like. The input device 495 is a device that provides a user interface (for example, a keyboard and a mouse). The output device 496 is an output device (for example, a display device) that provides a user interface.

  FIG. 2 is a diagram showing the configuration of the associative search slave server, but the configurations of the search client 20 and the associative search master server 30 are also configured in the same manner except for the difference in programs and data stored in the auxiliary storage device. .

  FIG. 3 shows a functional block configuration of the search means 410 provided in the associative search slave server 40. The search means 410 as a program is constituted by the functions of a word frequency acquisition means 411, a position information acquisition means 412, a relevance calculation means 413, a proximity calculation means 414, and a score calculation means 415. Each means constituting these search means 410 is also provided through program processing.

  The associative search slave server 40 searches the document database 440 for a document group having a high degree of relevance in response to the search request sent from the search request issuing means 320 provided in the associative search master server 30, and the search result is obtained from the relevance degree. It returns to the associative search master server 30 with a score. The search here can be realized by, for example, a known keyword search method.

  In the keyword search method, in order to increase the efficiency of the search process, the documents contained in the document database are divided into words (morphological analysis is performed for Japanese documents and stemming processing is performed for English documents). A search index is created in advance indicating which words are included in which documents. When position information is also used at the time of search as in the search method of this embodiment described later, the appearance position of each word is also stored in the index. When a search is executed, the search process can be executed at high speed by using a search index created in advance.

  In the case of FIG. 1, search indexes 430, 530, and 630 are created in advance for each of the document databases 440, 540, and 640 included in the associative search slave servers 40, 50, and 60, and are used for search processing.

  The calculation of the degree of association between the search request and the search target document is executed in the following procedure. First, the search means 410 receives the search request transmitted from the search request issuing means 320 of the associative search master server 30. The search unit 410 searches for a document including a word group included in the received search request. The word frequency acquisition unit 411 acquires the frequency information of the word group included in the search request among the word groups included in each document for each of the documents obtained as the search results. Next, the relevance calculation means 413 calculates the relevance between the search request and the document. The calculation method of the relevance may be arbitrary. For example, the importance of words is calculated by a known technique, tf * idf method, and the sum is used as the relevance. When the word proximity is reflected in the search score, the position information acquisition unit 412 acquires the appearance position information of the word group included in the search request among the word groups included in each document, and the proximity calculation unit 414. Calculates the proximity score. The calculation method of the proximity score may be arbitrary. For example, how densely the word group included in the search request appears, and the calculation result is used as the proximity score. The score calculation unit 415 integrates the scores obtained from each of the relevance calculation unit 413 and the proximity calculation unit 414, and gives the integrated score to the document as the relevance.

  FIG. 4 shows a functional block configuration of the feature word extraction unit 420 included in the associative search slave server 40. The feature word extraction unit 420 as a program is configured by the functions of a word frequency acquisition unit 421, a position information acquisition unit 422, an importance degree calculation unit 423, a proximity clustering unit 424, and a word addition unit 425. Each means constituting these characteristic word extracting means 420 is also provided through program processing.

  The feature word extraction unit 420 extracts, from the document database 440, the feature words for the document group sent from the feature word request unit 330 included in the associative search master server 30. The feature word extraction unit 420 uses the search index 430 in the same manner as the search unit 410 in order to realize high-speed extraction of feature words. That is, the feature word extracting unit 420 checks which word is included in a certain document with reference to the search index 430.

  The extraction of feature words is performed in the following procedure. First, the feature word extraction unit 420 receives the document group transmitted from the feature word request unit 330 of the associative search master server 30. The word frequency acquisition unit 421 acquires frequency information of each word included in the received document group. Based on the acquired frequency information, the importance calculation means 423 calculates the importance of each word. The calculation method of importance may be arbitrary. For example, the importance of a word is calculated by a known technique tf * idf method. In the case of an associative search that does not use position information, the feature word extraction unit 420 returns to the associative search master server 30 as feature words in order from words with high importance.

  In the present embodiment, the position information acquisition unit 422 acquires appearance position information for each word with importance. Further, the proximity clustering means 424 classifies the searched word group based on the importance level and the position information. Further, the word adding means 425 adds words that are close to the word group included in each of the classification results. The feature word extraction unit 420 returns the set of feature words obtained in this way to the associative search master server 30. Use of the word adding means 425 may be arbitrary.

  Next, the operation of the proximity clustering means 424 will be described with reference to FIGS. FIG. 5 illustrates two documents 1 and 2 that include the same word. In document 1, each word (term1 to term6) is distributed and distributed throughout the document, whereas in document 2, term1 to term3 are concentrated in the first half of the document and term4 to term6 are concentrated in the second half of the document. Distributed.

  Even in such a case, in the conventional associative search, the appearance position of the feature word is not taken into consideration. Therefore, the result of executing the associative search using the document 1 as a search input and the associative search using the document 2 as a search input are executed. The result is the same. However, if the distribution of feature words in the document is biased, it may be written about a plurality of topics, so it is desirable to classify feature word groups into individual topics.

  FIG. 6 is an example of extracting a feature word group from the document 1. In this case, since each characteristic word is distributed and distributed throughout the document, it is considered that one feature word is written. Therefore, even if the feature word group is classified based on the position information, it cannot be classified and becomes one feature word group.

  FIG. 7 shows an example of extracting a feature word group from the document 2. In this case, each characteristic word is considered to be written on two topics because term1 to term3 are concentrated in the first half of the document and term4 to term6 are concentrated in the second half of the document. Therefore, when the feature word group is classified based on the position information, two feature word groups, that is, the feature word group of term1 to term3 and the feature word group of term4 to term6 are extracted.

  For classification of the feature word group by the proximity clustering unit 424, for example, a hierarchical clustering method using the appearance position of the word and its weight may be applied. For words that appear multiple times, the position of the center of gravity is obtained in advance. Then, based on the position of each word, the closest words are put together. At that time, a new center of gravity is determined in consideration of the weight of each word. A clustering result is obtained by repeating this process.

  Alternatively, as another method, it is possible to determine which range in the document is covered by a word that appears multiple times, and classify the feature word group at a location with a low coverage in the entire document.

  In the above description, two methods have been described as the feature word classification method in the proximity clustering unit 424. However, any method may be used as long as it is a method for classifying a feature word group based on position information.

  By performing an associative search using the feature word group obtained in this way, it is possible to obtain a search result desired by the user even when a document includes a plurality of topics.

  FIG. 8 shows an example of a screen provided by the search request input means 210 provided in the search client 20. The user inputs a search request in the search request input area 211 and clicks the search instruction button 212 to instruct the search client 20 to execute the search.

  FIG. 9 is a display example of a search result by the search client 20. The search result is displayed by the search result display unit 220, and the feature word group extracted from the search result is displayed by the feature word display unit 230. Whether or not to use the feature word display means 230 is arbitrary. The search result display unit 220 also serves as a document group designation unit. When an associative search instruction button 213 is clicked in a state where an arbitrary number of documents are selected by the document selection check box 221, a document related to the selected document can be searched. The feature word display means 230 also serves as a word group designation means. When an associative search instruction button 213 is clicked in a state where an arbitrary number of words are selected by the word selection check box 231, a search from a characteristic word can be executed. The classification number designation means 240 is used to designate and input the number of topics to be divided into when a document is selected and an associative search is executed. The number of classifications may be directly specified as a numerical value, or may be specified using a slide bar, a button, or the like. Further, the number of classifications may be automatically set through a comparison between the classification score and a threshold value. The classification score is defined as a score obtained by integrating the importance score and the proximity score of the feature word. When the number of classifications is automatically set based on a threshold value, the classification number designation unit 240 does not have to be displayed on the screen.

  FIG. 10 is an example of a search result when two topics are included in a document given as a search input. In this case, the search result display means 220 displays the search results for each topic in two columns. Articles 1 to 5 in the left column correspond to topic 1, and articles A to E in the right column correspond to topic 2. In the case of FIG. 10, the feature word display means 230 displays the result of extracting the feature word group from the search results of the two topics integrated.

  On the other hand, FIG. 11 is the same as FIG. 10 in that the document given as the search input includes two topics, but the feature word group is displayed in the feature word display means 230 for each topic search result. Extracted and displayed in two columns. Feature terms 1 to 5 in the left column correspond to topic 1, and feature terms A to E in the right column correspond to topic 2. As in the case of FIG. 9, whether or not to use the feature word display means 230 is arbitrary.

  FIG. 12 is a screen for confirming the feature word group classified by the proximity clustering means 424. Using this screen, the user determines whether or not the classified feature word group is appropriate, and if so, clicks the search instruction button 213. If not appropriate, the user designates a new classification number in the classification number designation means 240, and then clicks the classification number change instruction button 241 to confirm the classified feature word group again. Use of this screen is arbitrary.

  FIG. 13 is an example of a search index when the search indexes 430, 530, and 630 are created from documents included in the document databases 440, 540, and 640. The document ID column stores an identifier for identifying each document, and information on the appearance position of a word included in the document corresponding to the identifier.

  Next, the flow of processing executed by the associative search system according to the embodiment will be described with reference to the sequence diagram of FIG. Below, the case where the associative search slave server 40 is used as an associative search slave server is demonstrated.

  The user uses the search request input means 210 provided in the search client 20 to input a search request. The input search request is transmitted from the search client 20 to the associative search master server 30 (T11).

  The search request analysis unit 310 of the associative search master server 30 analyzes the search request and creates a search request for transmission to the associative search slave server 40. The search request issuing means 320 transmits a search request to the associative search slave server 40 (T12).

  The search means 410 provided in the associative search slave server 40 searches the document database 440 using the search index 430 and returns the result to the associative search master server 30 (T13).

  The feature word requesting unit 330 of the associative search master server 30 transmits a feature word extraction request to the associative search slave server 40 in order to extract a feature word from the obtained search result (T14).

  The feature word extraction means 420 included in the associative search slave server 40 extracts a feature word group using the search index 430 and returns it to the associative search master server 30 (T15).

  Finally, the search result and the feature word group are transmitted from the associative search master server 30 to the search client 20 (T16) and presented to the user by the search result display means 220 and the feature word display means 230 of the search client 20.

  Next, the sequence diagram shown in FIG. 15 will be described. This sequence diagram shows a flow of processing when an associative search is executed from a document group obtained as a search result.

  The user uses the document group specifying means 220 provided in the search client 20 to select a document group as a search input. The identifier of the selected document group is transmitted to the associative search master server 30 (T21).

  The feature word requesting unit 330 of the associative search master server 30 transmits a feature word extraction request to the associative search slave server 40 in order to extract a feature word from the selected document group (T22).

  The feature word extraction unit 420 included in the associative search slave server 40 extracts a feature word group using the search index 430 and returns it to the associative search master server 30 (T23).

  The search request issuing unit 320 of the associative search master server 30 transmits the obtained feature word group to the associative search slave server (T24).

  The search means 410 provided in the associative search slave server 40 searches the document database 440 using the search index 430, and returns the result to the associative search master server 30 (T25).

  The feature word requesting unit 330 of the associative search master server 30 transmits a feature word extraction request to the associative search slave server 40 in order to extract a feature word from the obtained search result (T26).

  The feature word extraction means 420 included in the associative search slave server 40 extracts a feature word group using the search index 430 and returns it to the associative search master server 30 (T27).

  Finally, the search result and the feature word group are transmitted from the associative search master server 30 to the search client 20 (T28) and presented to the user by the search result display means 220 and the feature word display means 230 of the search client 20.

  Next, the sequence diagram shown in FIG. 16 will be described. This sequence diagram shows a flow of processing when an associative search is executed from a document group obtained as a search result, and shows a case where two topics are included in the obtained document group. .

  The user uses the document group specifying means 220 provided in the search client 20 to select a document group as a search input. The identifier of the selected document group is transmitted from the search client 20 to the associative search master server 30 (T31).

  The feature word requesting unit 330 of the associative search master server 30 transmits a feature word extraction request to the associative search slave server 40 in order to extract a feature word from the selected document group (T32).

  The feature word extraction means 420 included in the associative search slave server 40 extracts a feature word group using the search index 430 and returns it to the associative search master server 30 (T33).

  The search request issuing unit 320 of the associative search master server 30 transmits the feature word group corresponding to the first topic among the obtained two feature word groups to the associative search slave server 40 (T341).

  The search means 410 provided in the associative search slave server 40 searches the document database 440 using the search index 430 and returns the result to the associative search master server 30 (T351).

  The feature word requesting unit 330 of the associative search master server 30 transmits a feature word extraction request to the associative search slave server 40 in order to extract a feature word from the obtained search result (T361).

  The feature word extraction unit 420 included in the associative search slave server 40 extracts a feature word group using the search index 430 and returns it to the associative search master server 30 (T371).

  Next, the search request issuing unit 320 of the associative search master server 30 transmits a feature word group corresponding to the second topic among the obtained two feature word groups to the associative search slave server 40 (T342).

  The search means 410 provided in the associative search slave server 40 searches the document database 440 using the search index 430 and returns the result to the associative search master server 30 (T352).

  The feature word requesting unit 330 of the associative search master server 30 transmits a feature word extraction request to the associative search slave server 40 in order to extract a feature word from the obtained search result (T362).

  The feature word extraction means 420 included in the associative search slave server 40 extracts a feature word group using the search index 430 and returns it to the associative search master server 30 (T372).

  Finally, the search result and the feature word group are transmitted from the associative search master server 30 to the search client 20 (T28) and presented to the user by the search result display means 220 and the feature word display means 230 of the search client 20.

  When there are three or more topics, the same processing as the search request issuing means after T33 → T341 → search means → T351 → feature word requesting means → T361 → feature word extracting means → T371 may be repeated as many times as necessary. .

  When extracting feature words from the entire search results of two topics as shown in FIG. 10, feature word requesting means after T351 → T361 → feature word extracting means → T371 is omitted in the sequence diagram of FIG. In the feature word requesting means after T352, the entire document group of the search results of the two topics may be transmitted to the associative search slave server 40.

  In addition, this invention is not limited to the form example mentioned above, Various modifications are included. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Moreover, it is possible to replace a part of a certain form example with the structure of another form example, and it is also possible to add the structure of another form example to the structure of a certain form example. Moreover, it is also possible to add, delete, or replace another structure with respect to a part of structure of each form example.

  In addition, each of the above-described configurations, functions, processing units, processing means, and the like can be realized in part or in whole as, for example, an integrated circuit or other hardware.

10: Communication network 20: Search client 210: Search request input means 211: Search request input area 212: Search instruction button 213: Associative search instruction button 220: Search result display means (document group specification means)
221: Document selection check box 230: Feature word display means (word group designation means)
231: Word selection check box 240: Classification number designation means 241: Classification number change instruction button 30: Associative search master server 310: Search request analysis means 320: Search request issue means 330: Feature word request means 40: Associative search slave server 410 : Search means 411: word frequency acquisition means 412: position information acquisition means 413: relevance calculation means 414: proximity calculation means 415: score calculation means 420: feature word extraction means 421: word frequency acquisition means 422: position information acquisition means 423: Importance calculation means 424: Proximity clustering means 425: Word addition means 430: Search index 440: Document database 490: Bus 491: Memory device 492: Arithmetic processing device 493: Interface device 494: Auxiliary storage device 495: Input device 496: Output device 50: Virtual search slave server 510: search means 520: feature word extraction unit 530: search index 540: the document database 50: the associative search slave server 510: search means 520: feature word extraction unit 530: search index 540: document database

Claims (15)

A search client having at least input means for inputting a search request document, and search result display means for displaying the searched search results;
A document database storing multiple documents;
Search means for searching the document database for a document having a high degree of relevance to the received search request document, and extracting a feature word group from the given document group, and extracting the extracted feature word group and the importance of each word And a feature word extracting means for classifying into one or more feature word groups based on the appearance position information, and when a plurality of feature word groups are extracted, the degree of relevance for each of the classified feature word groups An associative search system comprising: an associative search server for searching a high document from the document database. The associative search system according to claim 1,
An associative search system in which the number of classifications of feature words is arbitrarily input by the user through an interface. The associative search system according to claim 2,
The interface includes a button for instructing a change in the number of classifications of the feature word group. The associative search system according to claim 1,
The associative search system is characterized in that the number of classifications of the feature word group is automatically set by comparing the classification score with a threshold value. The associative search system according to claim 1,
The associative search system characterized in that the feature word extraction means obtains a range covered by a word that appears multiple times and classifies the feature word group at a location with a low coverage. The associative search system according to claim 1,
The associative search system characterized in that the feature word extraction means obtains the centroid position of a word that appears multiple times and classifies the feature word group around the centroid position. The associative search system according to claim 1,
The associative search system, wherein the search client includes feature word display means for displaying a feature word of a searched document group. In an associative search server that searches a document database storing a plurality of documents for documents similar to a search request document input from a search client,
Search means for searching the document database for documents having a high degree of relevance to the received search request document;
Feature word extraction means for extracting a feature word group from a given document group and classifying the extracted feature word group into one or more feature word groups based on the importance of each word and its appearance position information And an associative search server that searches the document database for documents having a high degree of relevance for each of the classified feature word groups. The associative search server according to claim 8,
The associative search server characterized in that the feature word extracting means classifies the feature word group based on the number of classifications arbitrarily designated by the user through the interface. The associative search server according to claim 9,
The interface includes a button for instructing a change in the number of classifications of the feature word group. The associative search server according to claim 8,
The feature word extraction unit automatically sets the number of classifications of the feature word group by comparing the classification score with a threshold value.
An associative search server characterized by that. To a computer functioning as an associative search server that searches a document database storing a plurality of documents for documents similar to a search request document input from a search client,
A first process for searching the document database for a document having a high degree of relevance to the received search request document;
A second process for extracting a feature word group from a given document group and classifying the extracted feature word group into one or more feature word groups based on the importance of each word and its appearance position information When,
When a plurality of feature word groups are extracted, a program that executes a third process of searching the document database for a document having a high degree of association with each of the classified feature word groups. The program according to claim 12,
The program in which the second processing classifies the feature word group based on the number of classifications arbitrarily designated by the user through the interface. The program according to claim 13, wherein
The said interface has a button which instruct | indicates the change of the classification number of a characteristic word group. The program characterized by the above-mentioned. The program according to claim 12,
In the second process, the classification number of the feature word group is automatically set by comparing the classification score with a threshold value.

Images