US20120240032A1 - System and method for document collection, grouping and summarization - Google Patents
System and method for document collection, grouping and summarization Download PDFInfo
- Publication number
- US20120240032A1 US20120240032A1 US13/435,176 US201213435176A US2012240032A1 US 20120240032 A1 US20120240032 A1 US 20120240032A1 US 201213435176 A US201213435176 A US 201213435176A US 2012240032 A1 US2012240032 A1 US 2012240032A1
- Authority
- US
- United States
- Prior art keywords
- articles
- cluster
- content
- documents
- summarization
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
Definitions
- the present invention relates generally to automatic document collection and summarization and more particularly relates to systems and methods for generating summaries of multiple documents grouped as clusters including documents of various degrees of relatedness.
- a more difficult task, yet equally desirable, is that of providing a summary of multiple documents in a document collection which are directed to a common event, person, theme and the like.
- a collection of documents can span numerous sources, ranges in time and focus.
- the ability to generate a readable summary which conveys the content of the document collection is important to enable researchers to determine if the collection of documents pertains to the research question at hand.
- the MultiGen system developed by Barzilay et al. and available from Columbia University, Department of Computer Science, New York, N.Y., is a known system which performs well at generating summaries of a set of documents which are closely related, such as documents concerning a single event. While the performance of the MultiGen system is suitable for use with documents which are closely related, this system was not intended to generate summaries for document collections which are less closely related, such as collections of documents addressing multiple events, issues and biographical documents. Documents in these forms of diverse collections present additional challenges for generating readable, meaningful summaries.
- a system for generating a summary of a cluster of documents includes a computer readable document collection containing related documents stored in electronic form.
- a number of different forms of document summarization engines are provided.
- a router is interposed between the document collection and the summarization engines. The router determines a relationship of at least a subset of the documents in the collection and selects one of the document summarization engines to generate a summary of the subset of documents based on the relationship of the documents.
- the various summarization engines include a single event engine, a biography engine and a multi-event engine.
- the single-event engine generally takes the form of a similarity based summarization engine whereas the biography engine and a multi-event engine are preferably dissimilarity engines which have been optimized to the particular summarization tasks.
- the router selects the single event engine if a predetermined number of documents in the subset of the collection are generated within a predetermined time period.
- the router selects the biography engine if the documents in the collection are not generated within a predetermined time period, but the number of capitalized words and the number of personal pronouns each exceed a predetermined threshold value. Otherwise, the router selects the multi-event engine to generate the summary.
- the system can also include a document collection program which is operatively coupled to a number of content sources, such as content provider websites on the internet.
- the document collection program is configured to search the content of a predetermined set of content sources and gather documents for the document collection. Preprocessing can be performed to extract desired content from the documents and to conform the desired content to a common document format.
- Preprocessing can also include grouping the documents in the collection into clusters of related documents.
- the clusters can be assigned to predetermined document categories and labels for the clusters can be established by extracting keywords from the documents or summaries.
- a user interface can be provided to display the categories, cluster titles, summaries and related images.
- Also in accordance with the present invention is a method of gathering, organizing and presenting groups of content, such as news articles, to a user.
- the method includes the step of gathering a collection of news articles from a number of content providers. Next, the collection is analyzed to determine clusters of at least a portion of the articles. Then one among a number of multiple document summarization engines is selected for each cluster of articles and the selected summarization engine is used to generate a summary for the respective cluster of articles.
- the available summarization engines include a single event engine and at least one dissimilarity engine for multiple document summarization.
- the step of selecting one of the available summarization engines can include using a temporal relationship to determine whether the documents in a cluster relate to a single event.
- an additional step of extracting a set of keywords from the articles in the clusters is performed and the keywords are used as labels for the cluster summaries.
- the method can include the further step of sorting each of the summaries into one of a number of different categories.
- FIG. 1 is a simplified block diagram illustrating an overview of the operation of a system for generating a summary of multiple documents in accordance with the present invention
- FIG. 2 is a flow chart illustrating an overview of the operation of a system for generating a summary of multiple documents, such as news articles in accordance with the present invention
- FIG. 3 is a flow chart illustrating the operation of a router program for determining which summarization engine to apply to a cluster of documents
- FIG. 4 is a pictorial representation of an exemplary graphical user interface (GUI) for use in connection with the present multidocument summarization system;
- GUI graphical user interface
- FIG. 5 is a pictorial representation of an exemplary graphical user interface (GUI) for use in connection with the present multidocument summarization system while displaying a selected summary;
- GUI graphical user interface
- FIG. 6 is a simplified flow chart illustrating the operation of a dissimilarity engine for multidocument summarization (DEMS).
- DEMS multidocument summarization
- FIG. 7 is a flow chart illustrating a method of assigning scores to sentences in the documents to be summarized in accordance with one method of the present invention.
- FIG. 8 is a flow chart illustrating a method of performing named entity substitution for generating a summary of multiple documents in accordance with the present invention.
- FIG. 1 is a simplified block diagram of a system for generating a document collection, generating summaries of document collections containing multiple documents having various relationships between the documents and presenting the summaries to a user in a comprehensible format.
- the system generally includes a document collection engine 103 , a document collection database 105 , a preprocessing program 110 , a router program 115 and a number of document-type specific summarization engines tailored to the characteristics of expected document collection types having varying degrees and relationships of similarity.
- the preprocessing program 110 performs a number of tasks which, depending on application, can include content and image extraction, grouping documents into document clusters, assigning clusters to various categories and generating suitable labels for both the clusters and categories.
- a user interface 135 generally in the form of a graphical user interface such as an internet web browser program, is provided to display the system output, which can include the generated summaries, category labels, cluster labels and associated images.
- the summarizer router program 115 evaluates the type of documents which make up the document collection, or clusters of documents which are a subset of the collection, and selects one of a number of summarization engine types which is best suited for the particular summarization task.
- the summarization engine types can include a single-event engine 120 , a biography engine 125 and a multi-event engine 130 .
- the term engine refers to a computer program or subroutine which can be selected to perform the specific processing task.
- the single event summarization engine 120 is generally used for closely related documents whereas the biography engine 125 and multi-event engine 130 are variations of a dissimilarity engine for summarizing multiple documents (DEMS) which are optimized for documents that are less closely related.
- DEMS dissimilarity engine for summarizing multiple documents
- a document collection refers to any set of documents which have been grouped together by some similarity criteria.
- the documents can be grouped as a collection by either automatic similarity measures or by grouping by a human operator.
- the document collection 105 is assumed to be in electronic form and is generally stored in computer readable media for convenient access by a computer based system.
- the documents in the document collection database can be gathered by using the document collection engine 103 to search a number of document sources 100 which are expected to contain related information.
- a document collection engine in the form of a well known web crawler program can be used to gather content from a predetermined set of internet sites which are known to carry news stories, such as the internet websites for newspapers, television newscasts, radio newscasts and the like.
- a script or text file can be used to provide a list of internet sites that will be searched by the web crawler program.
- internet content sites generally also include content which is not desirable for inclusion into a summary, such as advertising content.
- the system can determine whether a particular piece of content should be included in a document collection by applying one or more heuristics. For example, the largest from of the content being evaluated can be analyzed to determine whether the quantity of text is greater or equal to a predetermined quantity. Since advertising content generally includes less than 512 characters, content which includes 512 characters or more and which is located on a news content source 100 can be regarded as a news article. Such sorting of content can take place in the preprocessing program 110 .
- the documents in a collection may come from a variety of sources and are not necessarily in a common document format.
- the pre-processing program 110 can also be used to convert the various document formats into a common XML format for further processing.
- data associated with the document such as the publication date, may be represented in a variety of formats, such as DATE, DT, PUBLISHDATE and the like.
- Preprocessing translates these SGML tags into a uniform XML format.
- Each of the processing operations such as the preprocessing operation, router 115 and the various summarization engines can be operating on a common computer platform, such as a single desktop computer, or can be distributed among a number of computers which are in communication with one another, such as by a local area network or global network such as the internet.
- each of the summarization engines 120 , 125 and 130 operate on an independent computer processor and operate in parallel.
- the summarization engines 120 , 125 and 130 each take a different form which is optimized for a particular multiple document summarization task.
- the single-event engine 120 generally takes the form of a known summarization engine, such as MultiGen, which is available through Columbia University, Department of Computer Science, New York, N.Y.
- MultiGen a known summarization engine
- a system and method for performing multidocument summarization on documents which are closely related, such as those relating to a single event, is also described in International Patent Application PCT US00/04118, which was published on Aug. 24, 2000 as publication WO 00/49517, which is hereby incorporated by reference in its entirety.
- the MultiGen summarizer extracts phrases from the documents in the cluster based on various thematic relevancy criteria and generates a summary by synthesizing these phrases into sentences using language generation to cut and paste together phrases from theme sentences.
- the biography engine 125 and multi-event engine 130 are each variations of a dissimilarity engine for multiple document summarization (DEMS), which are described in further detail below, in connection with FIGS. 6 , 7 and 8 .
- DEMS document summarization
- FIG. 2 is a flow chart illustrating the operation of an example of the system of FIG. 1 configured to gather and summarize news stories from a number of news content locations distributed on a computer data network, such as the internet.
- the document collection engine 103 takes the form of a web crawler program which uses a predetermined list of internet content provider websites which are expected to provide news content such as the internet web sites for newspapers, television newscasts, radio newscasts and the like.
- a script or text file can be used to provide the list of internet sites that will be searched by the web crawler program in the form of addresses for sites, such as Uniform Resource Locators (URL).
- the pages of content on such sites are generally stored as tables in HTML format and include text, images and other content, such as links to other internet resources.
- the pages which are encountered by the web crawler program are stored in a temporary file for processing, such as in the document collection database 105 .
- the temporary file of collected content is a large file which can exceed hundreds of megabytes of storage.
- each page in this file is evaluated to determine whether the content represents content of interest, in this example news stories, or content which is not of interest, such as advertising.
- the largest cell in the HTML page is identified and the text of this cell extracted from the page.
- the content of this cell is then evaluated to determine whether the content of the page is of interest.
- the content of the largest cell of each page can be analyzed to determine whether the quantity of text is greater or equal to a predetermined quantity. For example, it has been determined that advertising content generally includes less than 512 characters.
- content which includes 512 characters or more which is located on a news content source can be regarded as a news article.
- the page is reevaluated to determine is there are images and captions associate with the page.
- Image selection takes place using a rules based technique which favors precision over recall. This recognizes that the lack of an image is less detrimental than the insertion of an unrelated image.
- selecting an image to be displayed in connection with selected content such as a news article, the following rules have been found to provide acceptable results.
- the URL of the image is evaluated for words such as “ad,” “advertisement,” and the like.
- Image files including these terms, or variants of these terms will generally be considered advertising rather than an associated image and will be discarded.
- images require a significant amount of storage capacity, the images are not generally stored by the summarization system. Instead, a link to the image file, such as an associated URL for the image file, is recorded and is associated with the page in the document collection database.
- Such sorting and extraction of content and images takes place in step 210 , which is generally performed in the preprocessing program 110 ( FIG. 1 ).
- the articles are evaluated to group the articles into clusters which represent a single story, event, person or theme (step 215 ).
- a cluster represents a group of articles for which a summary will generated.
- a minimum size can be set for the clusters so that a certain minimum number of related documents are required to define a cluster that will be summarized. It has been found that four is a reasonable minimum size for a cluster, but more or less documents can be used as the minimum.
- the clusters can then be evaluated to create super-clusters, which are groups of related clusters. In the majority of cases, each super-cluster will only contain one cluster.
- the system uses agglomerative clustering with a groupwise average similarity function.
- This employs TF*IDF weighted words as well as linguistically motivated features including terms, noun phrase heads and proper nouns, which tend to correlate with events.
- a log-linear statistical model can be used to adjust the weights of the different features.
- clustering techniques such as single-link, complete-link, groupwise-average and single-pass hierarchical clustering methods can also be used to perform the clustering step.
- clustering algorithms which employ linguistically motivated features for generating clusters can also be used.
- a suitable clustering method is disclosed in “An Investigation of Linguistic Features and Clustering Algorithms for Topical Document Clustering,” V.
- Clustering algorithms are discussed, for example, in “Information Retrieval: Data Structures and Algorithms.” W. B. Frakes and R. Baeza-Yates, editors, Prentice Hall, New Jersey, 1992, the portions of which related to clustering algorithms is hereby incorporated by reference. Clustering techniques are also disclosed in “Finding Groups in Data: An Introduction to Cluster Analysis” by Kaufman et al., Wiley, New York, 1990, which is incorporated herein by reference.
- the super-clusters are used to group and label clusters.
- Each super-cluster will generally be displayed with a label formed with a group of selected keywords from the content in the clusters.
- Each cluster in the super-cluster will contain a summary of at least the minimum number of documents. For a typical day of news stories, thousands of articles will be downloaded for analysis and several hundred of these articles will be grouped into several dozen clusters and summaries of these clusters will be generated. The remaining articles which are not grouped in a cluster having at least the minimum number of documents are discarded and do not become part of a summary.
- the document clusters to be summarized are evaluated and are routed to an appropriate cluster summarization engine.
- the system generally employs variants of at least two summarization engine types. Documents which are closely related in time and subject matter are considered documents relating to a single event. Documents in a cluster concerning a single event are expected to include a high degree of thematic similarity and the cut and paste summarization methods of the MultiGen summarizer, described above, are well suited for extracting phrases and generating summaries for these documents.
- DEMS multidocument summarization
- One method of operating the router program 115 for performing the step of selecting the appropriate cluster summarization engine (step 220 ) is further illustrated in the flow chart of FIG. 3 .
- the method of FIG. 3 first determines the temporal relationship of documents within a cluster to determine if the documents relate to a single-event. Generally, when a large number of documents in a cluster are published on the same day, or when all of the documents in the cluster are published within a relatively short time frame, the documents in the cluster are expected to relate to a single event.
- step 305 the publication date and time for each article in the cluster is evaluated. This information is generally appended to the HTML page of the article or is available in some form of meta-data associated with the article.
- a threshold time window, X is established, such as three days. If in step 310 it is determined that all of the articles in the cluster have been published within X days, the router selects the single event engine in step 315 . Alternatively, if in step 320 it is determined that a certain percentage of the documents in the cluster, Y, are published on the same day, the single-event engine is also selected in step 315 .
- the router program 115 will select among one or more dissimilarity summarization engines which are optimized for a particular class of document clusters. In the example of FIG. 3 , it is assumed that there are two dissimilarity engines, a biography engine and a multi-event engine. If the test in step 320 fails, flow in the router continues to step 325 where the number of capitalized words in the articles (step 325 ) and the number of personal pronouns in the articles of the cluster (step 330 ) are determined. A high occurrence of both capitalized words, which generally indicate proper names, and personal pronouns are indicative of content which relates to a particular person or place.
- step 335 if the occurrence of capitalized words exceeds a threshold value, T 1 , and the number of pronouns exceeds a threshold, T 2 , then the router selects the biography dissimilarity summarization engine 125 ( FIG. 1 ).
- the router program 115 selects a more generalized multi-event dissimilarity summarization engine in step 345 .
- that engine is used to generate a summary of the documents in the clusters in step 225 .
- keywords which are common to the articles in the individual clusters are extracted (step 230 ).
- those keywords which are common to all of the clusters of the super-cluster are then selected.
- the selected keywords up to a predetermined maximum subset of the keywords, is then used as a label for the super-cluster. For documents which are related to a single event, the article in the cluster most closely related to the summary can be determined and the title of that article can be selected as the cluster label.
- the labeled clusters are then evaluated to sort the super-clusters into a predetermined set of categories. While the categories used are not critical, it has been found desirable to categorize documents into categories of US News, World News, Sports, Finance, Science and Technology, and Entertainment.
- the step of categorization can be performed by calculating TF*IDF vectors for each category classification.
- the category for each article in a cluster is determined by comparing the TF*IDF vector for each article in a cluster to that of each of the defined categories.
- the category for each cluster is determined by assigning the category to which the most articles in the cluster are assigned.
- the estimated frequencies can be smoothed using smoothing bins, such as is described by Sable et al. in “Using Bins to Estimate Term Weights for Text Categorization,” Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2001, which is hereby incorporated by reference in its entirety.
- the summaries are archived (step 240 ).
- the user interface is then updated in order to present the output of the system to the user (Step 245 ).
- the user interface can take the form of a web browser program and the content of the system output can be provided in the form of HTML pages, such as illustrated in FIGS. 4 and 5 . Updating the user interface includes providing the cluster labels, titles, summaries and associated image information to the user in a usable format.
- FIGS. 4 and 5 illustrate a typical graphical user interface (GUI) for the system in accordance with FIGS. 1 and 2 .
- FIG. 4 illustrates an example of an internet homepage user interface for reviewing and accessing the summaries generated in accordance with the method of FIG. 2 .
- the display is generally provided in a conventional web browser program, such as Netscape Navigator® or Window's Explorer®, which is directed to the internet address for the service provider of the summarization service.
- the categories of super-clusters are listed across the top portion of the display, including U.S. 405 , World 410 , Finance 415 , Entertainment 420 , Science/technology 425 and Sports 430 .
- Each of these labels preferably includes a hypertext link which when selected routes the user to summaries in the selected category.
- the categories are listed on the page as dividing bars, such as U.S. 435 . It will be appreciated that the arrangement of the content on the display can be altered without diminishing the functionality of the interface.
- each super-cluster keyword label 440 are titles of the clusters within the super-cluster.
- beneath label 440 are the titles “Egan Says He Apologizes if He Made Mistakes in Handling Sex Abuse Cases” 445 and “In Setback To Eastern Russia Blocks Bishops Reentry” 450 .
- each cluster is an indication of the number of articles in the cluster and the date range of the articles in the cluster 455 .
- the titles 445 and 450 represent hypertext links which will direct the user to the generated summary when the link is selected.
- FIG. 5 illustrates an example of the GUI after selection of a summary.
- the summary page of FIG. 5 includes the summary title 505 , associated image files 510 , the text summary 515 and a list of the source articles 520 .
- each entry in the list of source articles includes a link to the source article so that the source material can be conveniently accessed by the user.
- an aspect of the present invention is the use of multiple summarization engines which are adapted to various summarization tasks.
- One such summarization engine is used for documents related to a single event in which the documents in the cluster are expected to possess a high degree of similarity.
- a second class of summarization engine which uses a different summarization strategy, is used for generating summaries of documents which are less closely related. This is referred to as a dissimilarity engine for multidocument summarization, or DEMS.
- DEMS is a sentence extraction based summarization program.
- DEMS searches the documents in the cluster for sentences that include new information and generates summaries by extracting the topped ranked sentences from the articles until the desired summary length is satisfied.
- Sentences are ranked by scanning all sentences in the cluster and assigning a score to the sentences based on importance features.
- FIG. 6 is a simplified flow diagram illustrating the overall operation of the DEMS.
- the cluster of documents to be summarized is input either directly into an embodiment of a DEMS summarizer or into a router for selection of one DEMS engine among a plurality of engines, each optimized for a particular summarization task (step 605 ). If a router is used, then one of the engines is selected in step 610 .
- the sentences in the articles to be summarized are evaluated and each sentence is assigned an importance score (step 615 ). The assignment of a sentence score is discussed below in connection with FIG. 7 .
- the sentences are then selected, starting with the sentence having the highest score, until a predetermined summary length is achieved (step 620 ).
- the sentences that are extracted in step 620 are then placed in an order to improve readability of the summary (step 625 ) and then duplicate sentences are removed (step 630 ).
- a named entity substitution module can be used to improve the readability of the resulting summary (step 635 ).
- FIG. 7 is a simplified flow diagram illustrating one method of assigning sentence scores in accordance with the present invention. This method recognizes that features which are indicative of importance can include words which signal a lead sentence in an article, high content verbs, terms which are indicative of dominant concepts and other heuristic measures of importance.
- step 705 the sentences of the articles in the cluster are evaluated to determine whether they include words which arc associated with lead sentences of articles.
- two separate lexicons of over 4,000 lead words each were generated by evaluating a corpus of 1996 New York Times articles as well as a corpus of 1996 Reuters articles. Those words which exhibited a statistical significant measure of inclusion in a lead sentence as opposed to the full text of the article were selected for inclusion in the lead words corpus.
- the selection criteria was based on the equation: p(Winlead)/(pWanywhere)>1. In order to satisfy the requirement for statistical significance a pvalue ⁇ 0.05 was required.
- the words in the sentences of the articles in the cluster are evaluated to determine if they are lead words in the lexicon. For each word that is included in the lead word lexicon, the lead word score for that sentence is incremented. Thus, sentences which include the highest number of lead words receive the highest lead value score.
- the DEMS In addition to evaluating the lead sentence values, the DEMS also evaluates a measure of verb specificity in step 710 . It is known that certain verbs are highly specific to a select group of subjects and therefore, are capable of conveying information or importance about a sentence. For example, the verb “arrest” is often suggestive of police activity.
- the concept of verb specificity has been used in connection with a biographical summarization engine, such as described by Schiffman et al. in “Producing Biographical Summaries: Combining Linguistic Knowledge With Corpus Statistics,” Proceedings European Association for Computational Linguistics, 2001, which is hereby incorporated by reference in its entirety.
- the verb specificity measure is determined by ranking how closely related subject-verb pairs are in a large corpus.
- the highest verb specificity in a sentence is used as a feature of the sentence and increases the sentence score.
- Such sentences are generally considered sentences that convey substantial information apart from the surrounding context.
- the DEMS can provide a score to sentences based upon concept sets conveyed by the sentences in the cluster (step 715 ).
- a concept set can be built for each sentence by generalizing noun-verb pairs into classes of words which refer to one another. For example, a lexicon of synonyms, hypemyms an hyponyms, such as the WordNet lexicon can be used to expand a noun-verb pairing in sentence into a concept set. In the case of words having more than five senses, a large number of synonyms will be encountered and the results, if not constrained, may become ambiguous. However, by applying such constraints can lead to words being dropped from the concept set.
- a negative value can be used to penalize sentences which occur late in the document. This feature recognizes the concept of primacy as an importance measure.
- the publication date can be used to increase the score of sentences which occur in more recent documents, assuming that the most up-to-date information has more value.
- Target Adjusts the score of a sentence if the presence of a central personage in the document cluster is referenced in the sentence.
- Length The score of a sentence can be penalized if the length is less than a minimum word count, such as 15 words, or is in excess of a maximum word count, such as 30 words.
- the score of a sentence can be increased based on the frequency of occurrences of any named entity occurring in the document.
- Pronoun A negative value is assigned to sentences that have pronouns in the beginning of the sentence.
- Role A positive value in cases where a pronoun follows a named entity.
- the sentence scores are generated in step 725 by aggregating the values of the features from steps 705 through 720 in a weighted sum.
- the weighting for each factor is determined experimentally based on the focus of the various DEMS configurations. For example, in the case of a biographical engine, the Target and Publication date features are given higher weighting than the other features.
- the sentences with the highest weighted scores are selected for inclusion in the summary (step 620 ).
- a simple ordering algorithm can be used to present the sentences in a more readable format.
- sentences extracted from the same document are grouped together in order of appearance in the document.
- the sentence groups can then be further ordered, such as in reverse chronological order based on the publication date of the articles from which the sentences are extracted.
- the clarity of a summary can be further improved by removing duplicate sentences in step 630 .
- exact matching sentences not only can exact matching sentences be removed, but through the use of concept sets, as described above, two sentences can be found to be matching in meaning without having exact matching words.
- an overlap threshold value such as in the range of 40% or more overlap, sentences which convey the same information can be removed as redundant.
- a further method for improving the quality of the resulting DEMS summary is to use a named entity substitution algorithm in step 635 .
- a method for performing named entity substitution is illustrated in the simplified flow chart of FIG. 8 .
- Named entities can be referred to by first name, last name, title, nickname, description and the like.
- the first reference to the named entity in an article is the most complete with subsequent references being some shortened form. Therefore, when a sentence is extracted from within an article, the reference to the named entity may not be clear.
- the named entity substitution algorithm identifies occurrences of variants of each named entity occurring in the cluster.
- the NOMINATOR system by International Business Machines, is suitable for extracting the named entity references.
- the longest variant of the name is determined in step 810 .
- This may include a title, first name and last name and possibly a description, such as “George W. Bush, President of the United States of America.”
- the shortest most common variant of the name is determined in step 815 by counting the occurrences of the various short form references and selecting the one with the highest frequency of occurrence in the cluster.
- the first reference to the named entity in the summary is replaced with the longest variant determined in step 810 .
- each subsequent reference to the named entity is replaced with the shortest variant which was determined in step 815 .
- the methods described herein are generally embodied in computer programs.
- the programming language and computer hardware on which the methods are performed is not critical to the present invention. It will be appreciated by those skilled in the art that such programs are embodied on computer readable media, such as optical or magnetic media, such as CD-ROMS, magnetic diskettes and the like. Such programs can also be distributed by downloading the programs over a digital data network.
Landscapes
- Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- Strategic Management (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Resources & Organizations (AREA)
- Operations Research (AREA)
- Economics (AREA)
- Marketing (AREA)
- Data Mining & Analysis (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Document Processing Apparatus (AREA)
Abstract
System for generating a summary of a plurality of documents is provided. The system includes a computer readable document collection containing a plurality of related documents stored in electronic form therein, a plurality of forms of multiple document summarization engines, and a router for determining a temporal relationship of at least a subset of the documents in the collection and selecting one of the plurality of forms of multiple document summarization engines for generating a summary of the subset of documents based on the temporal relationship.
Description
- The present invention relates generally to automatic document collection and summarization and more particularly relates to systems and methods for generating summaries of multiple documents grouped as clusters including documents of various degrees of relatedness.
- The desirability of generating a summary of a document, such as an abstract, is well known. A more difficult task, yet equally desirable, is that of providing a summary of multiple documents in a document collection which are directed to a common event, person, theme and the like. Generally, such a collection of documents can span numerous sources, ranges in time and focus. The ability to generate a readable summary which conveys the content of the document collection is important to enable researchers to determine if the collection of documents pertains to the research question at hand.
- A number of methods for generating a summary of multiple related documents have been considered. For example, the MultiGen system developed by Barzilay et al. and available from Columbia University, Department of Computer Science, New York, N.Y., is a known system which performs well at generating summaries of a set of documents which are closely related, such as documents concerning a single event. While the performance of the MultiGen system is suitable for use with documents which are closely related, this system was not intended to generate summaries for document collections which are less closely related, such as collections of documents addressing multiple events, issues and biographical documents. Documents in these forms of diverse collections present additional challenges for generating readable, meaningful summaries.
- One important application for multidocument summarization is in the area of summarizing news stories published from multiple sources. In this regard, it would be useful to have a system which could gather news stories from a number of sources, group these stories into clusters of related documents and then generate a readable summary of the documents in the cluster. Such a system would enable a user to browse large quantities of content quickly and efficiently.
- It is an object of the invention to provide a system for generating a summary of multiple documents in a document collection which employs a plurality of summarization subsystems which are optimized for a particular document type or class.
- It is another object of the invention to provide a system and method for collecting numerous documents from a number of content providers and present summaries of the content, grouped into categories and clusters with associated images.
- It is a further object of the invention to provide a system which enables a user to browse large quantities of content, such as news articles, through categorization, clustering and summarization of the content.
- It is a further object of the present invention to provide a system for generating a summary of multiple documents in a document collection by selecting and ordering sentences from the documents based on selection and ordering heuristics.
- In accordance with the present invention, a system for generating a summary of a cluster of documents is provided that includes a computer readable document collection containing related documents stored in electronic form. A number of different forms of document summarization engines are provided. A router is interposed between the document collection and the summarization engines. The router determines a relationship of at least a subset of the documents in the collection and selects one of the document summarization engines to generate a summary of the subset of documents based on the relationship of the documents.
- Preferably, the various summarization engines include a single event engine, a biography engine and a multi-event engine. The single-event engine generally takes the form of a similarity based summarization engine whereas the biography engine and a multi-event engine are preferably dissimilarity engines which have been optimized to the particular summarization tasks. In this case, the router selects the single event engine if a predetermined number of documents in the subset of the collection are generated within a predetermined time period. The router selects the biography engine if the documents in the collection are not generated within a predetermined time period, but the number of capitalized words and the number of personal pronouns each exceed a predetermined threshold value. Otherwise, the router selects the multi-event engine to generate the summary.
- The system can also include a document collection program which is operatively coupled to a number of content sources, such as content provider websites on the internet. The document collection program is configured to search the content of a predetermined set of content sources and gather documents for the document collection. Preprocessing can be performed to extract desired content from the documents and to conform the desired content to a common document format.
- Preprocessing can also include grouping the documents in the collection into clusters of related documents. The clusters can be assigned to predetermined document categories and labels for the clusters can be established by extracting keywords from the documents or summaries. A user interface can be provided to display the categories, cluster titles, summaries and related images.
- Also in accordance with the present invention is a method of gathering, organizing and presenting groups of content, such as news articles, to a user. The method includes the step of gathering a collection of news articles from a number of content providers. Next, the collection is analyzed to determine clusters of at least a portion of the articles. Then one among a number of multiple document summarization engines is selected for each cluster of articles and the selected summarization engine is used to generate a summary for the respective cluster of articles. Preferably, the available summarization engines include a single event engine and at least one dissimilarity engine for multiple document summarization. The step of selecting one of the available summarization engines can include using a temporal relationship to determine whether the documents in a cluster relate to a single event.
- Preferably, an additional step of extracting a set of keywords from the articles in the clusters is performed and the keywords are used as labels for the cluster summaries. In addition, the method can include the further step of sorting each of the summaries into one of a number of different categories.
- Further objects, features and advantages of the invention will become apparent from the following detailed description taken in conjunction with the accompanying figures showing illustrative embodiments of the invention, in which:
-
FIG. 1 is a simplified block diagram illustrating an overview of the operation of a system for generating a summary of multiple documents in accordance with the present invention; -
FIG. 2 is a flow chart illustrating an overview of the operation of a system for generating a summary of multiple documents, such as news articles in accordance with the present invention; -
FIG. 3 is a flow chart illustrating the operation of a router program for determining which summarization engine to apply to a cluster of documents; -
FIG. 4 is a pictorial representation of an exemplary graphical user interface (GUI) for use in connection with the present multidocument summarization system; -
FIG. 5 is a pictorial representation of an exemplary graphical user interface (GUI) for use in connection with the present multidocument summarization system while displaying a selected summary; -
FIG. 6 is a simplified flow chart illustrating the operation of a dissimilarity engine for multidocument summarization (DEMS). -
FIG. 7 is a flow chart illustrating a method of assigning scores to sentences in the documents to be summarized in accordance with one method of the present invention; and -
FIG. 8 is a flow chart illustrating a method of performing named entity substitution for generating a summary of multiple documents in accordance with the present invention. - Throughout the figures, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of the illustrated embodiments. Moreover, while the subject invention will now be described in detail with reference to the figures, it is done so in connection with the illustrative embodiments. It is intended that changes and modifications can be made to the described embodiments without departing from the true scope and spirit of the subject invention as defined by the appended claims.
-
FIG. 1 is a simplified block diagram of a system for generating a document collection, generating summaries of document collections containing multiple documents having various relationships between the documents and presenting the summaries to a user in a comprehensible format. The system generally includes adocument collection engine 103, adocument collection database 105, apreprocessing program 110, arouter program 115 and a number of document-type specific summarization engines tailored to the characteristics of expected document collection types having varying degrees and relationships of similarity. Thepreprocessing program 110 performs a number of tasks which, depending on application, can include content and image extraction, grouping documents into document clusters, assigning clusters to various categories and generating suitable labels for both the clusters and categories. Auser interface 135, generally in the form of a graphical user interface such as an internet web browser program, is provided to display the system output, which can include the generated summaries, category labels, cluster labels and associated images. - The
summarizer router program 115 evaluates the type of documents which make up the document collection, or clusters of documents which are a subset of the collection, and selects one of a number of summarization engine types which is best suited for the particular summarization task. The summarization engine types can include a single-event engine 120, abiography engine 125 and amulti-event engine 130. - The term engine, as used herein, refers to a computer program or subroutine which can be selected to perform the specific processing task. The single
event summarization engine 120 is generally used for closely related documents whereas thebiography engine 125 andmulti-event engine 130 are variations of a dissimilarity engine for summarizing multiple documents (DEMS) which are optimized for documents that are less closely related. - A document collection refers to any set of documents which have been grouped together by some similarity criteria. The documents can be grouped as a collection by either automatic similarity measures or by grouping by a human operator. The
document collection 105 is assumed to be in electronic form and is generally stored in computer readable media for convenient access by a computer based system. - The documents in the document collection database can be gathered by using the
document collection engine 103 to search a number ofdocument sources 100 which are expected to contain related information. For example, in the case of a multidocument summarization system for providing summaries of news articles, a document collection engine in the form of a well known web crawler program can be used to gather content from a predetermined set of internet sites which are known to carry news stories, such as the internet websites for newspapers, television newscasts, radio newscasts and the like. A script or text file can be used to provide a list of internet sites that will be searched by the web crawler program. - In addition to desired content, such as news articles, internet content sites generally also include content which is not desirable for inclusion into a summary, such as advertising content. The system can determine whether a particular piece of content should be included in a document collection by applying one or more heuristics. For example, the largest from of the content being evaluated can be analyzed to determine whether the quantity of text is greater or equal to a predetermined quantity. Since advertising content generally includes less than 512 characters, content which includes 512 characters or more and which is located on a
news content source 100 can be regarded as a news article. Such sorting of content can take place in thepreprocessing program 110. - The documents in a collection may come from a variety of sources and are not necessarily in a common document format. To compensate for any variations in document format or metadata content or format, the
pre-processing program 110 can also be used to convert the various document formats into a common XML format for further processing. For example, since various publishers use various SGML formats, data associated with the document, such as the publication date, may be represented in a variety of formats, such as DATE, DT, PUBLISHDATE and the like. Preprocessing translates these SGML tags into a uniform XML format. - Each of the processing operations such as the preprocessing operation,
router 115 and the various summarization engines can be operating on a common computer platform, such as a single desktop computer, or can be distributed among a number of computers which are in communication with one another, such as by a local area network or global network such as the internet. Preferably, each of thesummarization engines - The
summarization engines event engine 120 generally takes the form of a known summarization engine, such as MultiGen, which is available through Columbia University, Department of Computer Science, New York, N.Y. A system and method for performing multidocument summarization on documents which are closely related, such as those relating to a single event, is also described in International Patent Application PCT US00/04118, which was published on Aug. 24, 2000 as publication WO 00/49517, which is hereby incorporated by reference in its entirety. The MultiGen summarizer extracts phrases from the documents in the cluster based on various thematic relevancy criteria and generates a summary by synthesizing these phrases into sentences using language generation to cut and paste together phrases from theme sentences. - The
biography engine 125 andmulti-event engine 130 are each variations of a dissimilarity engine for multiple document summarization (DEMS), which are described in further detail below, in connection withFIGS. 6 , 7 and 8. -
FIG. 2 is a flow chart illustrating the operation of an example of the system ofFIG. 1 configured to gather and summarize news stories from a number of news content locations distributed on a computer data network, such as the internet. Instep 205, thedocument collection engine 103 takes the form of a web crawler program which uses a predetermined list of internet content provider websites which are expected to provide news content such as the internet web sites for newspapers, television newscasts, radio newscasts and the like. A script or text file can be used to provide the list of internet sites that will be searched by the web crawler program in the form of addresses for sites, such as Uniform Resource Locators (URL). The pages of content on such sites are generally stored as tables in HTML format and include text, images and other content, such as links to other internet resources. The pages which are encountered by the web crawler program are stored in a temporary file for processing, such as in thedocument collection database 105. - The temporary file of collected content is a large file which can exceed hundreds of megabytes of storage. In
step 210, each page in this file is evaluated to determine whether the content represents content of interest, in this example news stories, or content which is not of interest, such as advertising. For each page, the largest cell in the HTML page is identified and the text of this cell extracted from the page. The content of this cell is then evaluated to determine whether the content of the page is of interest. In this regard, the content of the largest cell of each page can be analyzed to determine whether the quantity of text is greater or equal to a predetermined quantity. For example, it has been determined that advertising content generally includes less than 512 characters. Thus, content which includes 512 characters or more which is located on a news content source can be regarded as a news article. - If the content is of interest, the page is reevaluated to determine is there are images and captions associate with the page. Image selection takes place using a rules based technique which favors precision over recall. This recognizes that the lack of an image is less detrimental than the insertion of an unrelated image. In selecting an image to be displayed in connection with selected content, such as a news article, the following rules have been found to provide acceptable results. First, it is determined whether the image file is located within, or is embedded into, the same cell as the content. If so, the file format for the image file is evaluated. In the case of news articles, it is the general case that images associated with news content are generally presented as JPEG formatted files. Finally, in the case of image files stored on the site of an internet content provider, the URL of the image is evaluated for words such as “ad,” “advertisement,” and the like. Image files including these terms, or variants of these terms will generally be considered advertising rather than an associated image and will be discarded. Because images require a significant amount of storage capacity, the images are not generally stored by the summarization system. Instead, a link to the image file, such as an associated URL for the image file, is recorded and is associated with the page in the document collection database. Such sorting and extraction of content and images takes place in
step 210, which is generally performed in the preprocessing program 110 (FIG. 1 ). - After news articles are identified in
step 210, the articles are evaluated to group the articles into clusters which represent a single story, event, person or theme (step 215). A cluster represents a group of articles for which a summary will generated. A minimum size can be set for the clusters so that a certain minimum number of related documents are required to define a cluster that will be summarized. It has been found that four is a reasonable minimum size for a cluster, but more or less documents can be used as the minimum. Once the documents are grouped as clusters, the clusters can then be evaluated to create super-clusters, which are groups of related clusters. In the majority of cases, each super-cluster will only contain one cluster. - In one embodiment, the system uses agglomerative clustering with a groupwise average similarity function. This employs TF*IDF weighted words as well as linguistically motivated features including terms, noun phrase heads and proper nouns, which tend to correlate with events. A log-linear statistical model can be used to adjust the weights of the different features. A number of known techniques for clustering techniques, such as single-link, complete-link, groupwise-average and single-pass hierarchical clustering methods can also be used to perform the clustering step. In addition, clustering algorithms which employ linguistically motivated features for generating clusters can also be used. A suitable clustering method is disclosed in “An Investigation of Linguistic Features and Clustering Algorithms for Topical Document Clustering,” V. Hatzivassiloglou et al, Proceedings of the 23rd Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGR-00), 2000, pp. 224-231, which is hereby incorporated by reference. Clustering algorithms are discussed, for example, in “Information Retrieval: Data Structures and Algorithms.” W. B. Frakes and R. Baeza-Yates, editors, Prentice Hall, New Jersey, 1992, the portions of which related to clustering algorithms is hereby incorporated by reference. Clustering techniques are also disclosed in “Finding Groups in Data: An Introduction to Cluster Analysis” by Kaufman et al., Wiley, New York, 1990, which is incorporated herein by reference.
- The super-clusters are used to group and label clusters. Each super-cluster will generally be displayed with a label formed with a group of selected keywords from the content in the clusters. Each cluster in the super-cluster will contain a summary of at least the minimum number of documents. For a typical day of news stories, thousands of articles will be downloaded for analysis and several hundred of these articles will be grouped into several dozen clusters and summaries of these clusters will be generated. The remaining articles which are not grouped in a cluster having at least the minimum number of documents are discarded and do not become part of a summary.
- In
step 220, the document clusters to be summarized are evaluated and are routed to an appropriate cluster summarization engine. The system generally employs variants of at least two summarization engine types. Documents which are closely related in time and subject matter are considered documents relating to a single event. Documents in a cluster concerning a single event are expected to include a high degree of thematic similarity and the cut and paste summarization methods of the MultiGen summarizer, described above, are well suited for extracting phrases and generating summaries for these documents. - When documents in a cluster are less closely temporally related, the content, while still related, tends to be more divergent. In these cases, different summarization techniques are required. Rather than use similarity metrics to extract phrases and generate theme sentences as is done in the MultiGen engine, a dissimilarity engine for multidocument summarization (DEMS) is used to evaluate the sentences in the content, determine sentence scores, and the extract and arrange entire sentences in order to form the summary. The DEMS can be optimized using various heuristics for document clusters related to a person (biography engine), multiple related events (acts of terrorism) or other similarity measures of interest. The operation of the DEMS is discussed in further detail below.
- One method of operating the
router program 115 for performing the step of selecting the appropriate cluster summarization engine (step 220) is further illustrated in the flow chart ofFIG. 3 . The method ofFIG. 3 first determines the temporal relationship of documents within a cluster to determine if the documents relate to a single-event. Generally, when a large number of documents in a cluster are published on the same day, or when all of the documents in the cluster are published within a relatively short time frame, the documents in the cluster are expected to relate to a single event. Instep 305, the publication date and time for each article in the cluster is evaluated. This information is generally appended to the HTML page of the article or is available in some form of meta-data associated with the article. A threshold time window, X, is established, such as three days. If instep 310 it is determined that all of the articles in the cluster have been published within X days, the router selects the single event engine instep 315. Alternatively, if instep 320 it is determined that a certain percentage of the documents in the cluster, Y, are published on the same day, the single-event engine is also selected instep 315. - If the tests for temporal relatedness in
steps router program 115 will select among one or more dissimilarity summarization engines which are optimized for a particular class of document clusters. In the example ofFIG. 3 , it is assumed that there are two dissimilarity engines, a biography engine and a multi-event engine. If the test instep 320 fails, flow in the router continues to step 325 where the number of capitalized words in the articles (step 325) and the number of personal pronouns in the articles of the cluster (step 330) are determined. A high occurrence of both capitalized words, which generally indicate proper names, and personal pronouns are indicative of content which relates to a particular person or place. Instep 335, if the occurrence of capitalized words exceeds a threshold value, T1, and the number of pronouns exceeds a threshold, T2, then the router selects the biography dissimilarity summarization engine 125 (FIG. 1 ). - If the conditions in
step 335 for selecting the biography engine are not satisfied, therouter program 115 selects a more generalized multi-event dissimilarity summarization engine instep 345. - Returning to
FIG. 2 , after one of the available summarization engines is selected, that engine is used to generate a summary of the documents in the clusters instep 225. After summarization is complete, keywords which are common to the articles in the individual clusters are extracted (step 230). In the case of super-clusters which contain more than one cluster, those keywords which are common to all of the clusters of the super-cluster are then selected. The selected keywords, up to a predetermined maximum subset of the keywords, is then used as a label for the super-cluster. For documents which are related to a single event, the article in the cluster most closely related to the summary can be determined and the title of that article can be selected as the cluster label. - The labeled clusters are then evaluated to sort the super-clusters into a predetermined set of categories. While the categories used are not critical, it has been found desirable to categorize documents into categories of US News, World News, Sports, Finance, Science and Technology, and Entertainment. The step of categorization can be performed by calculating TF*IDF vectors for each category classification. The category for each article in a cluster is determined by comparing the TF*IDF vector for each article in a cluster to that of each of the defined categories. The category for each cluster is determined by assigning the category to which the most articles in the cluster are assigned. During these calculations, the estimated frequencies can be smoothed using smoothing bins, such as is described by Sable et al. in “Using Bins to Estimate Term Weights for Text Categorization,” Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2001, which is hereby incorporated by reference in its entirety.
- After the super-clusters are assigned to the various categories in the system, the summaries are archived (step 240). The user interface is then updated in order to present the output of the system to the user (Step 245). The user interface can take the form of a web browser program and the content of the system output can be provided in the form of HTML pages, such as illustrated in
FIGS. 4 and 5 . Updating the user interface includes providing the cluster labels, titles, summaries and associated image information to the user in a usable format. -
FIGS. 4 and 5 illustrate a typical graphical user interface (GUI) for the system in accordance withFIGS. 1 and 2 .FIG. 4 illustrates an example of an internet homepage user interface for reviewing and accessing the summaries generated in accordance with the method ofFIG. 2 . The display is generally provided in a conventional web browser program, such as Netscape Navigator® or Window's Explorer®, which is directed to the internet address for the service provider of the summarization service. In the exemplary interface ofFIG. 4 , the categories of super-clusters are listed across the top portion of the display, including U.S. 405,World 410,Finance 415,Entertainment 420, Science/technology 425 andSports 430. Each of these labels preferably includes a hypertext link which when selected routes the user to summaries in the selected category. The categories are listed on the page as dividing bars, such as U.S. 435. It will be appreciated that the arrangement of the content on the display can be altered without diminishing the functionality of the interface. - Listed beneath the category dividing bars 435 are the labels for the various super-clusters within the category. For example, the first super-category listed under dividing bar U.S. 435 is labeled by the group of keywords “Vatican, United States, Rome, Pope John Paul II, John Paul” 440. Beneath each
super-cluster keyword label 440 are titles of the clusters within the super-cluster. For example, beneathlabel 440 are the titles “Egan Says He Apologizes if He Made Mistakes in Handling Sex Abuse Cases” 445 and “In Setback To Vatican Russia Blocks Bishops Reentry” 450. Preferably, associated with each cluster is an indication of the number of articles in the cluster and the date range of the articles in thecluster 455. In this form of GUI, thetitles -
FIG. 5 illustrates an example of the GUI after selection of a summary. The summary page ofFIG. 5 includes thesummary title 505, associated image files 510, thetext summary 515 and a list of thesource articles 520. Preferably, each entry in the list of source articles includes a link to the source article so that the source material can be conveniently accessed by the user. - As noted above, an aspect of the present invention is the use of multiple summarization engines which are adapted to various summarization tasks. One such summarization engine is used for documents related to a single event in which the documents in the cluster are expected to possess a high degree of similarity. A second class of summarization engine, which uses a different summarization strategy, is used for generating summaries of documents which are less closely related. This is referred to as a dissimilarity engine for multidocument summarization, or DEMS. Unlike the MultiGen engine, which uses a sentence synthesis approach, DEMS is a sentence extraction based summarization program. DEMS searches the documents in the cluster for sentences that include new information and generates summaries by extracting the topped ranked sentences from the articles until the desired summary length is satisfied. Sentences are ranked by scanning all sentences in the cluster and assigning a score to the sentences based on importance features.
-
FIG. 6 is a simplified flow diagram illustrating the overall operation of the DEMS. The cluster of documents to be summarized is input either directly into an embodiment of a DEMS summarizer or into a router for selection of one DEMS engine among a plurality of engines, each optimized for a particular summarization task (step 605). If a router is used, then one of the engines is selected instep 610. The sentences in the articles to be summarized are evaluated and each sentence is assigned an importance score (step 615). The assignment of a sentence score is discussed below in connection withFIG. 7 . The sentences are then selected, starting with the sentence having the highest score, until a predetermined summary length is achieved (step 620). The sentences that are extracted instep 620 are then placed in an order to improve readability of the summary (step 625) and then duplicate sentences are removed (step 630). A named entity substitution module can be used to improve the readability of the resulting summary (step 635). -
FIG. 7 is a simplified flow diagram illustrating one method of assigning sentence scores in accordance with the present invention. This method recognizes that features which are indicative of importance can include words which signal a lead sentence in an article, high content verbs, terms which are indicative of dominant concepts and other heuristic measures of importance. - In
step 705, the sentences of the articles in the cluster are evaluated to determine whether they include words which arc associated with lead sentences of articles. To perform this analysis, two separate lexicons of over 4,000 lead words each were generated by evaluating a corpus of 1996 New York Times articles as well as a corpus of 1996 Reuters articles. Those words which exhibited a statistical significant measure of inclusion in a lead sentence as opposed to the full text of the article were selected for inclusion in the lead words corpus. The selection criteria was based on the equation: p(Winlead)/(pWanywhere)>1. In order to satisfy the requirement for statistical significance a pvalue<0.05 was required. - The words in the sentences of the articles in the cluster are evaluated to determine if they are lead words in the lexicon. For each word that is included in the lead word lexicon, the lead word score for that sentence is incremented. Thus, sentences which include the highest number of lead words receive the highest lead value score.
- In addition to evaluating the lead sentence values, the DEMS also evaluates a measure of verb specificity in
step 710. It is known that certain verbs are highly specific to a select group of subjects and therefore, are capable of conveying information or importance about a sentence. For example, the verb “arrest” is often suggestive of police activity. The concept of verb specificity has been used in connection with a biographical summarization engine, such as described by Schiffman et al. in “Producing Biographical Summaries: Combining Linguistic Knowledge With Corpus Statistics,” Proceedings European Association for Computational Linguistics, 2001, which is hereby incorporated by reference in its entirety. The verb specificity measure is determined by ranking how closely related subject-verb pairs are in a large corpus. The verb specificity measure can be stated as: VerbSpecificity=Count(Vmi>t)/Count(Vmi>0), which reflects how often the mutual information between a particular verb and one noun or another exceeds a threshold value. Instep 710, the highest verb specificity in a sentence is used as a feature of the sentence and increases the sentence score. Such sentences are generally considered sentences that convey substantial information apart from the surrounding context. - In addition to the lead value and verb specificity scores, the DEMS can provide a score to sentences based upon concept sets conveyed by the sentences in the cluster (step 715). A concept set can be built for each sentence by generalizing noun-verb pairs into classes of words which refer to one another. For example, a lexicon of synonyms, hypemyms an hyponyms, such as the WordNet lexicon can be used to expand a noun-verb pairing in sentence into a concept set. In the case of words having more than five senses, a large number of synonyms will be encountered and the results, if not constrained, may become ambiguous. However, by applying such constraints can lead to words being dropped from the concept set.
- Finally, a number of additional features of sentences which have been found to relate to sentence importance can be applied to determine the sentence score. This heuristic analysis is performed in
step 720. A non-exhaustive list of features includes: - Location: A negative value can be used to penalize sentences which occur late in the document. This feature recognizes the concept of primacy as an importance measure.
- Publication Date: The publication date can be used to increase the score of sentences which occur in more recent documents, assuming that the most up-to-date information has more value.
- Target: Adjusts the score of a sentence if the presence of a central personage in the document cluster is referenced in the sentence.
- Length: The score of a sentence can be penalized if the length is less than a minimum word count, such as 15 words, or is in excess of a maximum word count, such as 30 words.
- Other entity: The score of a sentence can be increased based on the frequency of occurrences of any named entity occurring in the document.
- Pronoun: A negative value is assigned to sentences that have pronouns in the beginning of the sentence.
- Role: A positive value in cases where a pronoun follows a named entity.
- The sentence scores are generated in
step 725 by aggregating the values of the features fromsteps 705 through 720 in a weighted sum. The weighting for each factor is determined experimentally based on the focus of the various DEMS configurations. For example, in the case of a biographical engine, the Target and Publication date features are given higher weighting than the other features. - Returning to
FIG. 6 , following determination of the sentence scores instep 615, the sentences with the highest weighted scores are selected for inclusion in the summary (step 620). To generate a summary using the selected sentences, a simple ordering algorithm can be used to present the sentences in a more readable format. In this regard, sentences extracted from the same document are grouped together in order of appearance in the document. The sentence groups can then be further ordered, such as in reverse chronological order based on the publication date of the articles from which the sentences are extracted. - The clarity of a summary can be further improved by removing duplicate sentences in
step 630. In this regard, not only can exact matching sentences be removed, but through the use of concept sets, as described above, two sentences can be found to be matching in meaning without having exact matching words. By using an overlap threshold value, such as in the range of 40% or more overlap, sentences which convey the same information can be removed as redundant. - A further method for improving the quality of the resulting DEMS summary is to use a named entity substitution algorithm in
step 635. A method for performing named entity substitution is illustrated in the simplified flow chart ofFIG. 8 . Named entities can be referred to by first name, last name, title, nickname, description and the like. Generally the first reference to the named entity in an article is the most complete with subsequent references being some shortened form. Therefore, when a sentence is extracted from within an article, the reference to the named entity may not be clear. The named entity substitution algorithm identifies occurrences of variants of each named entity occurring in the cluster. The NOMINATOR system by International Business Machines, is suitable for extracting the named entity references. - For each named entity, the longest variant of the name is determined in
step 810. This may include a title, first name and last name and possibly a description, such as “George W. Bush, President of the United States of America.” The shortest most common variant of the name is determined instep 815 by counting the occurrences of the various short form references and selecting the one with the highest frequency of occurrence in the cluster. Instep 820, the first reference to the named entity in the summary is replaced with the longest variant determined instep 810. Instep 825, each subsequent reference to the named entity is replaced with the shortest variant which was determined instep 815. - The methods described herein are generally embodied in computer programs. The programming language and computer hardware on which the methods are performed is not critical to the present invention. It will be appreciated by those skilled in the art that such programs are embodied on computer readable media, such as optical or magnetic media, such as CD-ROMS, magnetic diskettes and the like. Such programs can also be distributed by downloading the programs over a digital data network.
- Although the present invention has been described in connection with specific exemplary embodiments, it should be understood that various changes, substitutions and alterations can be made to the disclosed embodiments without departing from the spirit and scope of the invention as set forth in the appended claims.
Claims (49)
1. A system for generating a summary of a plurality of documents comprising:
a computer readable document collection containing a plurality of related documents stored in electronic form therein;
a plurality of forms of multiple document summarization engines; and
a router, the router determining a temporal relationship of at least a subset of the documents in the collection and selecting one of the plurality of forms of multiple document summarization engines for generating a summary of the subset of documents based on the temporal relationship.
2. The system for generating a summary of a plurality of documents of claim 1 , wherein the plurality of forms of multiple document summarization engines include a single event engine, a biography engine and a multi-event engine.
3. The system for generating a summary of a plurality of documents of claim 2 , wherein the router selects the single event engine if a predetermined number of documents in the subset of the collection are generated within a predetermined time period.
4. The system for generating a summary of a plurality of documents of claim 2 , wherein the router selects the biography engine if the documents in the collection are not generated within a predetermined time period and the number of capitalized words and the number of personal pronouns each exceed a predetermined threshold value.
5. The system for generating a summary of a plurality of documents of claim 1 , further comprising a document collection program, the document collection program being configured to search the content of a predetermined set of content sources to gather documents for the document collection.
6. The system for generating a summary of a plurality of documents of claim 5 , wherein the content sources are websites of content providers on the Internet.
7. A computer-based method of presenting content, such as news articles, to a user, comprising:
gathering a plurality of articles from a plurality of content providers;
determining clusters of at least a portion of the articles;
selecting one of a plurality of forms of multiple document summarization engines for each cluster of articles based on the temporal relationship of the articles in each cluster;
generating a summary for each cluster of articles; and
displaying at least one summary to a user.
8. The computer-based method of presenting content, such as news articles, to a user of claim 7 , further comprising generating labels for the cluster summaries related to the cluster content.
9. The computer-based method of presenting content, such as news articles, to a user of claim 8 , wherein the step of generating labels further comprises extracting a set of keywords from the articles in the clusters.
10. The computer-based method of presenting content, such as news articles, to a user of claim 8 , further comprising sorting each of the summaries into one of a plurality of categories.
11. The computer-based method of presenting content, such as news articles, to a user of claim 10 , wherein the step of displaying the summary includes generating a display on a graphical user interface, the graphical user interface presenting the summaries organized in accordance with the categories and labels of cluster summaries.
12. The computer-based method of presenting content, such as news articles, to a user of claim 11 , further comprising the step of displaying images associated with the summaries.
13. The computer-based method of presenting content, such as news articles, to a user of claim 12 , wherein the step of gathering articles further comprises selecting images that are associated with the articles and wherein the selected images are the images displayed in association with the summaries.
14. The computer-based method of presenting content, such as news articles, to a user of claim 12 , wherein articles are in the form of HTML content and the step of selecting images associated with the articles includes determining whether the image is within a common HTML cell as the content.
15. The computer-based method of presenting content, such as news articles, to a user of claim 14 , wherein the step of selecting images associated with the articles includes determining whether the computer readable file format of the image file is representative of an image associated with the content.
16. The computer-based method of presenting content, such as news articles, to a user of claim 15 , wherein the step of selecting images associated with the articles includes rejecting image files having an address of the image file that is indicative of advertising content.
17. The computer-based method of presenting content, such as news articles, to a user of claim 7 , wherein the step of gathering articles further comprises the step of extracting the news articles from other content.
18. The computer-based method of presenting content, such as news articles, to a user of claim 7 , wherein the step of gathering articles further comprises selecting images that are associated with the articles.
19. The computer-based method of presenting content, such as news articles, to a user of claim 18 , wherein articles are in the form of HTML content and the step of selecting images associated with the articles includes determining whether the image is within a common HTML cell as the content.
20. The computer-based method of presenting content, such as news articles, to a user of claim 18 , wherein the step of selecting images associated with the articles includes determining whether the computer readable file format of the image file is representative of an image associated with the content.
21. The computer-based method of presenting content, such as news articles, to a user of claim 20 , wherein the step of selecting images associated with the articles includes rejecting image files having an address of the image file that is indicative of advertising content.
22. The computer-based method of presenting content, such as news articles, to a user of claim 7 , wherein the plurality of forms of multiple document summarization engines include a single event engine and at least one dissimilarity engine for multiple document summarization.
23. The computer-based method of presenting content, such as news articles, to a user of claim 7 , wherein the step of selecting one of a plurality of forms of multiple document summarization engines includes using a temporal relationship to determine whether the documents in a cluster relate to a single event.
24. A system for document collection, grouping and summarization comprising:
a document collection engine, the document collection engine being configured to be operatively coupled to a computer network to access a plurality of content providers;
a computer readable document collection database, the document collection database being operatively coupled to the document collection engine;
a cluster processing engine operatively coupled to the document collection database, the cluster processing engine grouping at least a portion of the documents in the document collection database into clusters having a plurality of related documents;
a plurality of forms of multiple document summarization engines;
a summarization router, the summarization router being interposed between the cluster processing engine and the plurality of forms of multiple document summarization engines, the summarization router determining a temporal relationship among the documents in the clusters and selecting one of the plurality of forms of multiple document summarization engines for generating a summary of the cluster based on, the temporal relationship; and
a graphical user interface, the graphical user interface being operatively coupled to the cluster processing engine and the plurality of summarization engines and providing a display including cluster summaries and cluster information.
25. The system for document collection, grouping and summarization of claim 24 , wherein the plurality of forms of multiple document summarization engines include a single event engine, a biography engine and a multi-event engine.
26. The system for document collection, grouping and summarization comprising claim 25 , wherein the summarization router selects the single event engine if a predetermined number of documents in the subset of the collection are generated within a predetermined time period.
27. The system for document collection, grouping and summarization of claim 26 , wherein the router selects the biography engine if the documents in the collection are not generated within a predetermined time period and the number of capitalized words and the number of personal pronouns each exceed a predetermined threshold value.
28. The system for document collection, grouping and summarization of claim 24 , wherein the document collection engine is configured to search the content of a predetermined set of content sources.
29. The system for document collection, grouping and summarization comprising of claim 28 , wherein the content sources are websites of content providers on the Internet.
30. The system for document collection, grouping and summarization of claim 24 , wherein the cluster processing engine establishes cluster labels for each of the clusters and wherein the cluster information displayed by the graphical user interface includes the cluster labels.
31. The system for document collection, grouping and summarization of claim 30 , wherein the cluster processing engine sorts the clusters into one of a plurality of predetermined document categories, and wherein cluster information displayed by the graphical user interface includes the predetermined categories.
32. The system for document collection, grouping and summarization of claim 31 , wherein the graphical user interface presents a display including the predetermined categories and cluster labels.
33. The system for document collection, grouping and summarization of claim 32 , wherein one of the document collection engine or cluster processing engine identifies image files associated with documents that are collected and wherein the graphical user interface displays the image files with the summary associated with the document cluster.
34. A method of generating a summary of multiple documents in a document cluster, comprising:
for each document in the cluster, determining a score for each sentence based on features of the sentences;
selecting a subset of sentences based on a weighted sentence score;
merging the selected sentences in accordance with a predetermined order for the selected sentences; and
removing selected sentences which are duplicative.
35. The method of generating a summary according to claim 34 , further comprising the steps of determining variants of named entity references in the summary and performing a named entity substitution by inserting a first variant for the first occurrence of the reference and inserting a second variant for the remaining references.
36. The method of generating a summary according to claim 34 , wherein the features of the sentences include a score based on lead values.
37. The method of generating a summary according to claim 34 , wherein the features of the sentences include a score based on verb specificity.
38. The method of generating a summary according to claim 34 , wherein the features of the sentences include a score based on noun-verb concept sets.
39. The method of generating a summary according to claim 34 , wherein the features of the sentences include a score based on lead values, verb specificity, and noun-verb concept sets.
40. The method of generating a summary according to claim 34 , wherein the features of the sentences include a score based on at least one heuristic relationship selected from the set including sentence location, publication date, target, sentence length, other entity, pronoun, and role.
41. Computer readable media encoded with instructions for a computer processor to perform the following steps:
gathering a plurality of articles from a plurality of content providers;
determining clusters of at least a portion of the articles;
selecting one of a plurality of forms of multiple document summarization engines for each cluster of articles based on the temporal relationship of the articles in each cluster;
generating a summary for each cluster of articles; and
displaying at least one summary to a user.
42. The computer-based method of presenting content, such as news articles, to a user of claim 7 , further comprising displaying a list of at least one of the articles in the cluster corresponding to the summary displayed to the user.
43. The computer-based method of presenting content, such as news articles, to a user of claim 42 , wherein each entry in the list includes a link to contents of the article.
44. The computer-based method of presenting content, such as news articles, to a user of claim 42 , wherein each entry in the list includes a date for each article.
45. The computer-based method of presenting content, such as news articles, to a user of claim 42 , wherein the entries in the list are grouped by date.
46. The computer readable media encoded with instructions for a computer processor of claim 41 , further comprising displaying a list of at least one of the articles in the cluster corresponding to the summary displayed to the user.
47. The computer readable media encoded with instructions for a computer processor of claim 46 , wherein each entry in the list includes a link to contents of the article.
48. The computer readable media encoded with instructions for a computer processor of claim 46 , wherein each entry in the list includes a date for each article.
49. The computer readable media encoded with instructions for a computer processor of claim 46 , wherein the entries in the list are grouped by date.
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/435,176 US20120240032A1 (en) | 2002-09-16 | 2012-03-30 | System and method for document collection, grouping and summarization |
US13/473,837 US20120240020A1 (en) | 2002-09-16 | 2012-05-17 | System and method for document collection, grouping and summarization |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/US2002/029271 WO2004025490A1 (en) | 2002-09-16 | 2002-09-16 | System and method for document collection, grouping and summarization |
US11/071,968 US8176418B2 (en) | 2002-09-16 | 2005-03-04 | System and method for document collection, grouping and summarization |
US13/435,176 US20120240032A1 (en) | 2002-09-16 | 2012-03-30 | System and method for document collection, grouping and summarization |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/071,968 Continuation US8176418B2 (en) | 2002-09-16 | 2005-03-04 | System and method for document collection, grouping and summarization |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/473,837 Continuation US20120240020A1 (en) | 2002-09-16 | 2012-05-17 | System and method for document collection, grouping and summarization |
Publications (1)
Publication Number | Publication Date |
---|---|
US20120240032A1 true US20120240032A1 (en) | 2012-09-20 |
Family
ID=31989882
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/071,968 Expired - Fee Related US8176418B2 (en) | 2002-09-16 | 2005-03-04 | System and method for document collection, grouping and summarization |
US13/435,176 Abandoned US20120240032A1 (en) | 2002-09-16 | 2012-03-30 | System and method for document collection, grouping and summarization |
US13/473,837 Abandoned US20120240020A1 (en) | 2002-09-16 | 2012-05-17 | System and method for document collection, grouping and summarization |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/071,968 Expired - Fee Related US8176418B2 (en) | 2002-09-16 | 2005-03-04 | System and method for document collection, grouping and summarization |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/473,837 Abandoned US20120240020A1 (en) | 2002-09-16 | 2012-05-17 | System and method for document collection, grouping and summarization |
Country Status (3)
Country | Link |
---|---|
US (3) | US8176418B2 (en) |
CA (1) | CA2496567A1 (en) |
WO (1) | WO2004025490A1 (en) |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090100454A1 (en) * | 2006-04-25 | 2009-04-16 | Frank Elmo Weber | Character-based automated media summarization |
WO2014058433A1 (en) * | 2012-10-12 | 2014-04-17 | Hewlett-Packard Development Company, L.P. | A combinatorial summarizer |
WO2015163857A1 (en) * | 2014-04-22 | 2015-10-29 | Hewlett-Packard Development Company, L.P. | Determining an optimized summarizer architecture for a selected task |
WO2015183246A1 (en) * | 2014-05-28 | 2015-12-03 | Hewlett-Packard Development Company, L.P. | Data extraction based on multiple meta-algorithmic patterns |
US10019525B1 (en) | 2017-07-26 | 2018-07-10 | International Business Machines Corporation | Extractive query-focused multi-document summarization |
WO2020171972A1 (en) * | 2019-02-21 | 2020-08-27 | Microsoft Technology Licensing, Llc | Topic based summarizer for meetings and presentations using hierarchical agglomerative clustering |
US11106876B1 (en) * | 2014-12-01 | 2021-08-31 | Massachusetts Institute Of Technology | Method and apparatus for summarization of natural language |
US12135737B1 (en) * | 2023-06-21 | 2024-11-05 | Sas Institute Inc. | Graphical user interface and pipeline for text analytics |
Families Citing this family (99)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9280603B2 (en) * | 2002-09-17 | 2016-03-08 | Yahoo! Inc. | Generating descriptions of matching resources based on the kind, quality, and relevance of available sources of information about the matching resources |
US7568148B1 (en) | 2002-09-20 | 2009-07-28 | Google Inc. | Methods and apparatus for clustering news content |
US8090717B1 (en) | 2002-09-20 | 2012-01-03 | Google Inc. | Methods and apparatus for ranking documents |
DE10248837A1 (en) * | 2002-10-19 | 2004-04-29 | Philips Intellectual Property & Standards Gmbh | System and method for processing electronic documents |
US7292972B2 (en) * | 2003-01-30 | 2007-11-06 | Hewlett-Packard Development Company, L.P. | System and method for combining text summarizations |
US7577655B2 (en) | 2003-09-16 | 2009-08-18 | Google Inc. | Systems and methods for improving the ranking of news articles |
US7814089B1 (en) | 2003-12-17 | 2010-10-12 | Topix Llc | System and method for presenting categorized content on a site using programmatic and manual selection of content items |
US8271495B1 (en) | 2003-12-17 | 2012-09-18 | Topix Llc | System and method for automating categorization and aggregation of content from network sites |
GB2413420A (en) * | 2004-04-23 | 2005-10-26 | Hewlett Packard Development Co | Interactive document reading |
US20110019916A1 (en) * | 2004-04-23 | 2011-01-27 | Andrew Mackenzie | Interactive document reading |
US7664735B2 (en) * | 2004-04-30 | 2010-02-16 | Microsoft Corporation | Method and system for ranking documents of a search result to improve diversity and information richness |
US7207004B1 (en) * | 2004-07-23 | 2007-04-17 | Harrity Paul A | Correction of misspelled words |
CN100462961C (en) * | 2004-11-09 | 2009-02-18 | 国际商业机器公司 | Method for organizing multi-file and equipment for displaying multi-file |
US20110208732A1 (en) * | 2010-02-24 | 2011-08-25 | Apple Inc. | Systems and methods for organizing data items |
US20070067293A1 (en) * | 2005-06-30 | 2007-03-22 | Hong Yu | System and methods for automatically identifying answerable questions |
US8433711B2 (en) * | 2005-09-09 | 2013-04-30 | Kang Jo Mgmt. Limited Liability Company | System and method for networked decision making support |
US7739254B1 (en) * | 2005-09-30 | 2010-06-15 | Google Inc. | Labeling events in historic news |
US7917519B2 (en) * | 2005-10-26 | 2011-03-29 | Sizatola, Llc | Categorized document bases |
WO2007057799A1 (en) * | 2005-11-15 | 2007-05-24 | Koninklijke Philips Electronics N.V. | Method, system and device for obtaining a representation of a text |
US8140261B2 (en) * | 2005-11-23 | 2012-03-20 | Alcatel Lucent | Locating sensor nodes through correlations |
US7930647B2 (en) * | 2005-12-11 | 2011-04-19 | Topix Llc | System and method for selecting pictures for presentation with text content |
US20070143300A1 (en) * | 2005-12-20 | 2007-06-21 | Ask Jeeves, Inc. | System and method for monitoring evolution over time of temporal content |
US8332386B2 (en) | 2006-03-29 | 2012-12-11 | Oracle International Corporation | Contextual search of a collaborative environment |
US8209729B2 (en) * | 2006-04-20 | 2012-06-26 | At&T Intellectual Property I, Lp | Rules-based content management |
US20070260586A1 (en) * | 2006-05-03 | 2007-11-08 | Antonio Savona | Systems and methods for selecting and organizing information using temporal clustering |
US8135699B2 (en) * | 2006-06-21 | 2012-03-13 | Gupta Puneet K | Summarization systems and methods |
US7930226B1 (en) | 2006-07-24 | 2011-04-19 | Intuit Inc. | User-driven document-based data collection |
CN101553816B (en) * | 2006-11-20 | 2012-02-22 | 株式会社爱可信 | Information display device and information display system |
US9405732B1 (en) * | 2006-12-06 | 2016-08-02 | Topix Llc | System and method for displaying quotations |
US12038958B1 (en) * | 2007-02-06 | 2024-07-16 | Dmitri Soubbotin | System, method, and user interface for a search engine based on multi-document summarization |
US8239358B1 (en) * | 2007-02-06 | 2012-08-07 | Dmitri Soubbotin | System, method, and user interface for a search engine based on multi-document summarization |
US9218414B2 (en) * | 2007-02-06 | 2015-12-22 | Dmitri Soubbotin | System, method, and user interface for a search engine based on multi-document summarization |
US11651039B1 (en) * | 2007-02-06 | 2023-05-16 | Dmitri Soubbotin | System, method, and user interface for a search engine based on multi-document summarization |
US20080262998A1 (en) * | 2007-04-17 | 2008-10-23 | Alessio Signorini | Systems and methods for personalizing a newspaper |
US20080270119A1 (en) * | 2007-04-30 | 2008-10-30 | Microsoft Corporation | Generating sentence variations for automatic summarization |
US20080281927A1 (en) * | 2007-05-11 | 2008-11-13 | Microsoft Corporation | Summarization tool and method for a dialogue sequence |
US8209617B2 (en) * | 2007-05-11 | 2012-06-26 | Microsoft Corporation | Summarization of attached, linked or related materials |
US8494944B2 (en) * | 2007-06-06 | 2013-07-23 | O2 Media, LLC | System, report, and method for generating natural language news-based stories |
US7818117B2 (en) * | 2007-06-20 | 2010-10-19 | Amadeus S.A.S. | System and method for integrating and displaying travel advices gathered from a plurality of reliable sources |
US8108398B2 (en) * | 2007-06-29 | 2012-01-31 | Microsoft Corporation | Auto-summary generator and filter |
US8442969B2 (en) * | 2007-08-14 | 2013-05-14 | John Nicholas Gross | Location based news and search engine |
WO2009026140A2 (en) | 2007-08-16 | 2009-02-26 | Hollingsworth William A | Automatic text skimming using lexical chains |
US20090070346A1 (en) * | 2007-09-06 | 2009-03-12 | Antonio Savona | Systems and methods for clustering information |
US20090100357A1 (en) * | 2007-10-11 | 2009-04-16 | Alessio Signorini | Systems and methods for visually selecting information |
US10255609B2 (en) | 2008-02-21 | 2019-04-09 | Micronotes, Inc. | Interactive marketing system |
US20090265307A1 (en) * | 2008-04-18 | 2009-10-22 | Reisman Kenneth | System and method for automatically producing fluent textual summaries from multiple opinions |
US20090319555A1 (en) * | 2008-06-24 | 2009-12-24 | Microsoft Corporation | Optimizing documents based on desired content |
US8984398B2 (en) * | 2008-08-28 | 2015-03-17 | Yahoo! Inc. | Generation of search result abstracts |
JP5098914B2 (en) * | 2008-09-11 | 2012-12-12 | 富士通株式会社 | Message pattern generation program, method and apparatus |
US8818803B2 (en) * | 2008-10-27 | 2014-08-26 | Frank Elmo Weber | Character-based automated text summarization |
US8200670B1 (en) * | 2008-10-31 | 2012-06-12 | Google Inc. | Efficient document clustering |
US8271502B2 (en) * | 2009-06-26 | 2012-09-18 | Microsoft Corporation | Presenting multiple document summarization with search results |
US8832102B2 (en) * | 2010-01-12 | 2014-09-09 | Yahoo! Inc. | Methods and apparatuses for clustering electronic documents based on structural features and static content features |
US8554542B2 (en) * | 2010-05-05 | 2013-10-08 | Xerox Corporation | Textual entailment method for linking text of an abstract to text in the main body of a document |
US8244724B2 (en) | 2010-05-10 | 2012-08-14 | International Business Machines Corporation | Classifying documents according to readership |
US20110307497A1 (en) * | 2010-06-14 | 2011-12-15 | Connor Robert A | Synthewiser (TM): Document-synthesizing search method |
US8204805B2 (en) | 2010-10-28 | 2012-06-19 | Intuit Inc. | Instant tax return preparation |
US9558521B1 (en) | 2010-07-29 | 2017-01-31 | Intuit Inc. | System and method for populating a field on a form including remote field level data capture |
US8560520B2 (en) * | 2010-08-26 | 2013-10-15 | Microsoft Corporation | Information retrieval using time |
JP2013003630A (en) * | 2011-06-13 | 2013-01-07 | Sony Corp | Information processing device, information processing method, program, and information processing system |
EP2551811A1 (en) * | 2011-07-26 | 2013-01-30 | Tata Consultancy Services Limited | Method and system for integrating event processes in investment banking and custody lines of business |
US20130060760A1 (en) * | 2011-09-02 | 2013-03-07 | Microsoft Corporation | Determining comprehensive subsets of reviews |
US20130157234A1 (en) * | 2011-12-14 | 2013-06-20 | Microsoft Corporation | Storyline visualization |
US8880390B2 (en) * | 2011-12-20 | 2014-11-04 | Yahoo! Inc. | Linking newsworthy events to published content |
US10235346B2 (en) * | 2012-04-06 | 2019-03-19 | Hmbay Patents Llc | Method and apparatus for inbound message summarization using message clustering and message placeholders |
US20130311967A1 (en) * | 2012-05-16 | 2013-11-21 | Wellpoint, Inc. | Method and System for Collapsing Functional Similarities and Consolidating Functionally Similar, Interacting Systems |
GB201210661D0 (en) * | 2012-06-15 | 2012-08-01 | Qatar Foundation | Unsupervised cross-media summarization from news and twitter |
AU2013300455B2 (en) * | 2012-08-08 | 2016-09-01 | Yoshimitsu Kagiwada | Transaction support system |
US20140289260A1 (en) * | 2013-03-22 | 2014-09-25 | Hewlett-Packard Development Company, L.P. | Keyword Determination |
US9727641B2 (en) | 2013-04-25 | 2017-08-08 | Entit Software Llc | Generating a summary based on readability |
US9940099B2 (en) * | 2014-01-03 | 2018-04-10 | Oath Inc. | Systems and methods for content processing |
US10394867B2 (en) * | 2014-06-11 | 2019-08-27 | Hewlett-Packard Development Company, L.P. | Functional summarization of non-textual content based on a meta-algorithmic pattern |
WO2016053314A1 (en) * | 2014-09-30 | 2016-04-07 | Hewlett-Packard Development Company, L.P. | Specialized language identification |
US9678947B2 (en) | 2014-11-21 | 2017-06-13 | International Business Machines Corporation | Pattern identification and correction of document misinterpretations in a natural language processing system |
US9998413B2 (en) * | 2014-12-23 | 2018-06-12 | Palo Alto Research Center Incorporated | Computer-implemented system and method for prioritizing electronic messages based on importance |
US10387550B2 (en) | 2015-04-24 | 2019-08-20 | Hewlett-Packard Development Company, L.P. | Text restructuring |
US10152478B2 (en) * | 2015-06-07 | 2018-12-11 | Apple Inc. | Apparatus, system and method for string disambiguation and entity ranking |
US11886477B2 (en) | 2015-09-22 | 2024-01-30 | Northern Light Group, Llc | System and method for quote-based search summaries |
US11544306B2 (en) | 2015-09-22 | 2023-01-03 | Northern Light Group, Llc | System and method for concept-based search summaries |
WO2017107010A1 (en) * | 2015-12-21 | 2017-06-29 | 浙江核新同花顺网络信息股份有限公司 | Information analysis system and method based on event regression test |
US11226946B2 (en) | 2016-04-13 | 2022-01-18 | Northern Light Group, Llc | Systems and methods for automatically determining a performance index |
US9767165B1 (en) | 2016-07-11 | 2017-09-19 | Quid, Inc. | Summarizing collections of documents |
US10430450B2 (en) * | 2016-08-22 | 2019-10-01 | International Business Machines Corporation | Creation of a summary for a plurality of texts |
US10657525B2 (en) | 2017-06-27 | 2020-05-19 | Kasisto, Inc. | Method and apparatus for determining expense category distance between transactions via transaction signatures |
US20190129942A1 (en) * | 2017-10-30 | 2019-05-02 | Northern Light Group, Llc | Methods and systems for automatically generating reports from search results |
CN108959432A (en) * | 2018-06-11 | 2018-12-07 | 颜彦 | It is a kind of that media public sentiment monitoring method is melted based on big data |
JP7053423B2 (en) * | 2018-09-19 | 2022-04-12 | 富士フイルム株式会社 | Image processing device, image processing method and image processing program |
US10831793B2 (en) | 2018-10-23 | 2020-11-10 | International Business Machines Corporation | Learning thematic similarity metric from article text units |
CH715990A1 (en) * | 2019-03-22 | 2020-09-30 | Jyad Murr | Computer-implemented method for broadcasting multimedia content information. |
US10936796B2 (en) * | 2019-05-01 | 2021-03-02 | International Business Machines Corporation | Enhanced text summarizer |
US11144719B2 (en) * | 2019-11-27 | 2021-10-12 | International Business Machines Corporation | System and method for argument retrieval |
CN113282745B (en) * | 2020-02-20 | 2023-04-18 | 清华大学 | Automatic generation method and device for event encyclopedia document |
CN111597327B (en) * | 2020-04-22 | 2023-04-07 | 哈尔滨工业大学 | Public opinion analysis-oriented unsupervised multi-document abstract generation method |
WO2021260650A1 (en) * | 2020-06-27 | 2021-12-30 | Glance Inmobi Pte. Ltd | Generating personalized content for presentation on user devices |
EP4179419A4 (en) * | 2020-07-13 | 2024-03-20 | AI21 Labs | Controllable reading guides and natural language generation |
US11328732B2 (en) | 2020-09-16 | 2022-05-10 | International Business Machines Corporation | Generating summary text compositions |
US11615245B2 (en) | 2021-02-02 | 2023-03-28 | International Business Machines Corporation | Article topic alignment |
CN113641817B (en) * | 2021-08-30 | 2024-05-07 | 浪潮智能物联技术有限公司 | Report generation method and device based on theme and timeline abstract |
US20230376687A1 (en) * | 2022-05-17 | 2023-11-23 | Adobe Inc. | Multimodal extraction across multiple granularities |
Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5297027A (en) * | 1990-05-11 | 1994-03-22 | Hitachi, Ltd. | Method of and apparatus for promoting the understanding of a text by using an abstract of that text |
US5751914A (en) * | 1995-10-10 | 1998-05-12 | International Business Machines Corporation | Method and system for correlating a plurality of events within a data processing system |
US5819260A (en) * | 1996-01-22 | 1998-10-06 | Lexis-Nexis | Phrase recognition method and apparatus |
US5926784A (en) * | 1997-07-17 | 1999-07-20 | Microsoft Corporation | Method and system for natural language parsing using podding |
US5983221A (en) * | 1998-01-13 | 1999-11-09 | Wordstream, Inc. | Method and apparatus for improved document searching |
US6023760A (en) * | 1996-06-22 | 2000-02-08 | Xerox Corporation | Modifying an input string partitioned in accordance with directionality and length constraints |
US6128635A (en) * | 1996-05-13 | 2000-10-03 | Oki Electric Industry Co., Ltd. | Document display system and electronic dictionary |
US20010021938A1 (en) * | 1996-03-29 | 2001-09-13 | Ronald A. Fein | Document summarizer for word processors |
US6336132B2 (en) * | 1997-02-07 | 2002-01-01 | About.Com, Inc. | Internet resource location system with identified and approved human guides assigned to specific topics to provide content related to the topic |
US20020023077A1 (en) * | 2000-06-09 | 2002-02-21 | Nguyen Thanh Ngoc | Method and apparatus for data collection and knowledge management |
US20020061021A1 (en) * | 1997-10-27 | 2002-05-23 | Dillon Douglas M. | System and method for multicasting multimedia content |
US20020072997A1 (en) * | 2000-09-13 | 2002-06-13 | Ip.Com, Inc. | Global information network product publication system |
US20020138528A1 (en) * | 2000-12-12 | 2002-09-26 | Yihong Gong | Text summarization using relevance measures and latent semantic analysis |
US20030035144A1 (en) * | 2000-05-19 | 2003-02-20 | Toshihiro Shima | Network printer for printing edited contents on a network and method for printing contents on the network |
US6766287B1 (en) * | 1999-12-15 | 2004-07-20 | Xerox Corporation | System for genre-specific summarization of documents |
US20050138056A1 (en) * | 2003-12-18 | 2005-06-23 | Xerox Corporation | System and method for multi-dimensional foraging and retrieval of documents |
US6968511B1 (en) * | 2002-03-07 | 2005-11-22 | Microsoft Corporation | Graphical user interface, data structure and associated method for cluster-based document management |
Family Cites Families (24)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US3790958A (en) * | 1971-09-09 | 1974-02-05 | Xerox Corp | Data communication terminal |
JPH07113810B2 (en) * | 1986-07-17 | 1995-12-06 | ソニー株式会社 | Learning device |
JPH05252379A (en) * | 1992-03-04 | 1993-09-28 | Fuji Xerox Co Ltd | Image processor |
US5548507A (en) * | 1994-03-14 | 1996-08-20 | International Business Machines Corporation | Language identification process using coded language words |
US6029195A (en) * | 1994-11-29 | 2000-02-22 | Herz; Frederick S. M. | System for customized electronic identification of desirable objects |
JP3571408B2 (en) * | 1995-03-31 | 2004-09-29 | 株式会社日立製作所 | Document processing method and apparatus |
US5787422A (en) * | 1996-01-11 | 1998-07-28 | Xerox Corporation | Method and apparatus for information accesss employing overlapping clusters |
SE9603753L (en) * | 1996-10-14 | 1998-04-06 | Mirror Image Internet Ab | Procedure and apparatus for information transmission on the Internet |
JP3579204B2 (en) * | 1997-01-17 | 2004-10-20 | 富士通株式会社 | Document summarizing apparatus and method |
US6185550B1 (en) * | 1997-06-13 | 2001-02-06 | Sun Microsystems, Inc. | Method and apparatus for classifying documents within a class hierarchy creating term vector, term file and relevance ranking |
US6038557A (en) * | 1998-01-26 | 2000-03-14 | Xerox Corporation | Method and apparatus for almost-constant-time clustering of arbitrary corpus subsets |
US6295529B1 (en) * | 1998-12-24 | 2001-09-25 | Microsoft Corporation | Method and apparatus for indentifying clauses having predetermined characteristics indicative of usefulness in determining relationships between different texts |
US6560647B1 (en) * | 1999-03-04 | 2003-05-06 | Bmc Software, Inc. | Enterprise management system and method which includes semantically correct summarization |
US6446110B1 (en) * | 1999-04-05 | 2002-09-03 | International Business Machines Corporation | Method and apparatus for representing host datastream screen image information using markup languages |
US7137065B1 (en) * | 2000-02-24 | 2006-11-14 | International Business Machines Corporation | System and method for classifying electronically posted documents |
US7693866B1 (en) * | 2000-03-07 | 2010-04-06 | Applied Discovery, Inc. | Network-based system and method for accessing and processing legal documents |
JP3562572B2 (en) * | 2000-05-02 | 2004-09-08 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Detect and track new items and new classes in database documents |
US6782393B1 (en) * | 2000-05-31 | 2004-08-24 | Ricoh Co., Ltd. | Method and system for electronic message composition with relevant documents |
AU2001280998A1 (en) * | 2000-08-03 | 2002-02-18 | Bruce A. Epstein | Information collaboration and reliability assessment |
US6718336B1 (en) * | 2000-09-29 | 2004-04-06 | Battelle Memorial Institute | Data import system for data analysis system |
US7185001B1 (en) * | 2000-10-04 | 2007-02-27 | Torch Concepts | Systems and methods for document searching and organizing |
JP3916124B2 (en) * | 2001-02-15 | 2007-05-16 | インターナショナル・ビジネス・マシーンズ・コーポレーション | Digital document browsing system, browser, digital document display method, program, and storage medium |
US6965900B2 (en) * | 2001-12-19 | 2005-11-15 | X-Labs Holdings, Llc | Method and apparatus for electronically extracting application specific multidimensional information from documents selected from a set of documents electronically extracted from a library of electronically searchable documents |
US7028036B2 (en) * | 2002-06-28 | 2006-04-11 | Microsoft Corporation | System and method for visualization of continuous attribute values |
-
2002
- 2002-09-16 WO PCT/US2002/029271 patent/WO2004025490A1/en active Application Filing
- 2002-09-16 CA CA002496567A patent/CA2496567A1/en not_active Abandoned
-
2005
- 2005-03-04 US US11/071,968 patent/US8176418B2/en not_active Expired - Fee Related
-
2012
- 2012-03-30 US US13/435,176 patent/US20120240032A1/en not_active Abandoned
- 2012-05-17 US US13/473,837 patent/US20120240020A1/en not_active Abandoned
Patent Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5297027A (en) * | 1990-05-11 | 1994-03-22 | Hitachi, Ltd. | Method of and apparatus for promoting the understanding of a text by using an abstract of that text |
US5751914A (en) * | 1995-10-10 | 1998-05-12 | International Business Machines Corporation | Method and system for correlating a plurality of events within a data processing system |
US5819260A (en) * | 1996-01-22 | 1998-10-06 | Lexis-Nexis | Phrase recognition method and apparatus |
US20010021938A1 (en) * | 1996-03-29 | 2001-09-13 | Ronald A. Fein | Document summarizer for word processors |
US6128635A (en) * | 1996-05-13 | 2000-10-03 | Oki Electric Industry Co., Ltd. | Document display system and electronic dictionary |
US6023760A (en) * | 1996-06-22 | 2000-02-08 | Xerox Corporation | Modifying an input string partitioned in accordance with directionality and length constraints |
US6336132B2 (en) * | 1997-02-07 | 2002-01-01 | About.Com, Inc. | Internet resource location system with identified and approved human guides assigned to specific topics to provide content related to the topic |
US5926784A (en) * | 1997-07-17 | 1999-07-20 | Microsoft Corporation | Method and system for natural language parsing using podding |
US20020061021A1 (en) * | 1997-10-27 | 2002-05-23 | Dillon Douglas M. | System and method for multicasting multimedia content |
US5983221A (en) * | 1998-01-13 | 1999-11-09 | Wordstream, Inc. | Method and apparatus for improved document searching |
US6766287B1 (en) * | 1999-12-15 | 2004-07-20 | Xerox Corporation | System for genre-specific summarization of documents |
US20030035144A1 (en) * | 2000-05-19 | 2003-02-20 | Toshihiro Shima | Network printer for printing edited contents on a network and method for printing contents on the network |
US20020023077A1 (en) * | 2000-06-09 | 2002-02-21 | Nguyen Thanh Ngoc | Method and apparatus for data collection and knowledge management |
US20020072997A1 (en) * | 2000-09-13 | 2002-06-13 | Ip.Com, Inc. | Global information network product publication system |
US20020138528A1 (en) * | 2000-12-12 | 2002-09-26 | Yihong Gong | Text summarization using relevance measures and latent semantic analysis |
US6968511B1 (en) * | 2002-03-07 | 2005-11-22 | Microsoft Corporation | Graphical user interface, data structure and associated method for cluster-based document management |
US20050138056A1 (en) * | 2003-12-18 | 2005-06-23 | Xerox Corporation | System and method for multi-dimensional foraging and retrieval of documents |
Non-Patent Citations (1)
Title |
---|
Barry Schiffman, Inderjeet Mani, Kristian J. Concepcion; Producing Biographical Summaries: Combining Linguistic Knowledge with Corpus Statistics; 2001 (found at https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.19.8263 last visited August 27, 2015) * |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8392183B2 (en) * | 2006-04-25 | 2013-03-05 | Frank Elmo Weber | Character-based automated media summarization |
US20090100454A1 (en) * | 2006-04-25 | 2009-04-16 | Frank Elmo Weber | Character-based automated media summarization |
US9977829B2 (en) | 2012-10-12 | 2018-05-22 | Hewlett-Packard Development Company, L.P. | Combinatorial summarizer |
WO2014058433A1 (en) * | 2012-10-12 | 2014-04-17 | Hewlett-Packard Development Company, L.P. | A combinatorial summarizer |
US10572525B2 (en) | 2014-04-22 | 2020-02-25 | Hewlett-Packard Development Company, L.P. | Determining an optimized summarizer architecture for a selected task |
WO2015163857A1 (en) * | 2014-04-22 | 2015-10-29 | Hewlett-Packard Development Company, L.P. | Determining an optimized summarizer architecture for a selected task |
WO2015183246A1 (en) * | 2014-05-28 | 2015-12-03 | Hewlett-Packard Development Company, L.P. | Data extraction based on multiple meta-algorithmic patterns |
US10366126B2 (en) | 2014-05-28 | 2019-07-30 | Hewlett-Packard Development Company, L.P. | Data extraction based on multiple meta-algorithmic patterns |
US11106876B1 (en) * | 2014-12-01 | 2021-08-31 | Massachusetts Institute Of Technology | Method and apparatus for summarization of natural language |
US10019525B1 (en) | 2017-07-26 | 2018-07-10 | International Business Machines Corporation | Extractive query-focused multi-document summarization |
US10127323B1 (en) | 2017-07-26 | 2018-11-13 | International Business Machines Corporation | Extractive query-focused multi-document summarization |
US11269965B2 (en) | 2017-07-26 | 2022-03-08 | International Business Machines Corporation | Extractive query-focused multi-document summarization |
WO2020171972A1 (en) * | 2019-02-21 | 2020-08-27 | Microsoft Technology Licensing, Llc | Topic based summarizer for meetings and presentations using hierarchical agglomerative clustering |
US12135737B1 (en) * | 2023-06-21 | 2024-11-05 | Sas Institute Inc. | Graphical user interface and pipeline for text analytics |
Also Published As
Publication number | Publication date |
---|---|
WO2004025490A1 (en) | 2004-03-25 |
CA2496567A1 (en) | 2004-03-25 |
US20050203970A1 (en) | 2005-09-15 |
US20120240020A1 (en) | 2012-09-20 |
US8176418B2 (en) | 2012-05-08 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8176418B2 (en) | System and method for document collection, grouping and summarization | |
US6295529B1 (en) | Method and apparatus for indentifying clauses having predetermined characteristics indicative of usefulness in determining relationships between different texts | |
Eirinaki et al. | SEWeP: using site semantics and a taxonomy to enhance the Web personalization process | |
Wang et al. | A machine learning based approach for table detection on the web | |
US8285724B2 (en) | System and program for handling anchor text | |
EP1622055B1 (en) | Phrase-based indexing in an information retrieval system | |
US7840577B2 (en) | Search query processing to identify related search terms and to correct misspellings of search terms | |
AU2010241249B2 (en) | Methods and systems for determining a meaning of a document to match the document to content | |
US7624093B2 (en) | Method and system for automatic summarization and digest of celebrity news | |
US20040133560A1 (en) | Methods and systems for organizing electronic documents | |
US7822752B2 (en) | Efficient retrieval algorithm by query term discrimination | |
US20060117002A1 (en) | Method for search result clustering | |
US20070185863A1 (en) | Methods and apparatus for characterizing a search result as potential spam | |
US20040098385A1 (en) | Method for indentifying term importance to sample text using reference text | |
US7523109B2 (en) | Dynamic grouping of content including captive data | |
US7024405B2 (en) | Method and apparatus for improved internet searching | |
EP2307951A1 (en) | Method and apparatus for relating datasets by using semantic vectors and keyword analyses | |
US20040139042A1 (en) | System and method for improving data analysis through data grouping | |
JP5146108B2 (en) | Document importance calculation system, document importance calculation method, and program | |
Bawakid et al. | A Semantic Summarization System: University of Birmingham at TAC 2008. | |
US20090157647A1 (en) | Method and Apparatus for Discovering and Classifying Polysemous Word Instances in Web Documents | |
WO2004025496A1 (en) | System and method for document collection, grouping and summarization | |
Toth | Innovative solutions in automatic classification: a brief summary | |
Rosell | Text Clustering Exploration | |
Hawking | Very large scale information retrieval |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |