Usage: . run.sh

Settings in config.py:

input_data: Path to input tsv file, e.g. data/ontonotes_en_name_entity.tsv
string_col: Name of text column, e.g. 'string' 
entities_col: Name of entities column, e.g. 'type'
ent_min_n: Counts threshold for removing scarce entities, e.g. 3
train: Run training stage (True/False)
test: Run testing stage (True/False)

Version 1:

• Sentence tokenization and iterative searching

Version 2:

• Simple search for lomgest match
• Compatible output with the name_entity.tsv

Version 3:

• Re-design for iterative search over token.tsv

Version 4:

• Scoring fixed
• Calculating scores by entity

Reading the dev data line by line and adding completed sentences to a new csv. Using the part_of_speech column to detect the end of sentences and empty lines

We can now read the sentences as one string and use the ids to search the train data for the true entity

First we extract all the possible ngrams from the sentence. We start searching from the longest ones and stop when we find a match.

We then update the sentence by removing the matched string and we add the remaining sentence(s) to the queue. We continue until no ngrams are left

Printing out the augmented sentences, using NOMATCH for empty tags. Below the lines we get the local scores for each sentence.

Printing the final results. The first table contains results by entity. After that we get the overall scores for the devset.

A boxplot with the True Positives (Matched) counts per entity

These are the most frequent matches in the subset