Part-of-Speech Tagging with Neural Networks for a conversational agent

This toolkit was implemented during my master's thesis, that aimed to improve the natural language understanding of an artificial conversational agent. This agent utilized a Hidden Markov Model to calculate Part-of-Speech tags for input words. In order to achieve better results, two different classification architectures are implemented and evaluated: a Feed-forward Neural Network and a Recurrent Neural Network.

This repository contains the toolkit to train and evaluate language models for POS tagging and tag input sentences according to a trained model. It provides the possibility to use the python scripts directly as well as an API and is licensed under GPL-3.0.

Check if Python version >= 3.5 is installed:

$ python --version
Python 3.6.3

Install dependencies (consider using a virtual environment):

pip install -r requirements.txt

If the installation was successful, change to the directory of the Tagger and everything should be ready to run properly:

cd fnn-tagger/

The single python scripts of this toolkit can be called directly, documented in the following.

Static settings are located in the settings.py script. It contains the following configuration options:

Training, evaluation and tagging can be executed using the tagger.py script, which represents the core script of this toolkit. Its general usage is:

python tagger.py [-h] [--train TRAIN] [--tag TAG] [--evaluate EVALUATE]
                 [--reset] [-p PASTWORDS] [-e EMBEDDINGSIZE] [-s HIDDENSIZE]
                 [-n NEPOCHS] [-t TIMESTEPS] [-f] [-q] [-i]

optional arguments:
  -h, --help            show this help message and exit
  --train TRAIN         Invokes training of a language model on given corpus
  --tag TAG             Tags a given sentence with the pretrained language
                        model
  --evaluate EVALUATE   Evaluates pretrained language model with a given
                        evaluation file
  --reset               Removes all stored training and log data
  -p PASTWORDS, --pastwords PASTWORDS
                        Number of preceding words to take into account
  -e EMBEDDINGSIZE, --embeddingsize EMBEDDINGSIZE
                        Dimension of the word embeddings
  -s HIDDENSIZE, --hiddensize HIDDENSIZE
                        Dimension of the hidden layer
  -n NEPOCHS, --nepochs NEPOCHS
                        Number of training epochs
  -t TIMESTEPS, --timesteps TIMESTEPS
                        Number of past trained words
  -f, --force           Force operation without confirmation
  -q, --quiet           No output messages
  -i, --inline          Only one line output

However, the following sections explain the usage of the specific flags and their apropriate combination for each action.

To train the Tagger call the tagger.py script with the --train flag. According to your static configuration, the batch training will start. Once you reached a sufficient accuracy, you can interrupt the training with CTRL+C or wait till the training process finishes.

$ python tagger.py --train data/test.corpus
Training starts...
Loading training data from "data/test.corpus"...
Generating vocabulary...
Generating tensors...
Initializing model...
Step 100: loss 0.9, accuracy 91% - saved model checkpoint to 'saved/model-100'
Step 200: loss 0.2, accuracy 98% - saved model checkpoint to 'saved/model-200'
Step 300: loss 0.0, accuracy 100% - saved model checkpoint to 'saved/model-300'

You can also call the script with inline configuration. To train a model using the FNN architecture, use the flags -p, -e, -s, and -n. It is required to use exactly these 4 flags, otherwise the static configuration from the settings.py will be used. An example call would be:

python tagger.py --train data/test.corpus -p 1 -e 250 -s 350 -n 5

To train a model using the RNN architecture, use the flags -t, -e, -s, and -n. It is required to use exaclty these 4 flags, otherwise the static configuration from the settings.py will be used. An example call would be:

python tagger.py --train data/test.corpus -t 8 -e 100 -s 100 -n 5

To tag a sentence with a pretrained model call the tagger.py script with the --tag parameter followed by sentence to be tagged. Now a tag is attached to every word.

$ python tagger.py --tag "Show all modules of Bachelor Informatics"
The tagged sentence is:
 Show     all       modules     of       Bachelor         Informatics
R_LIST   R_LIST   M_MTSModule    X    C_Program:degree   C_Program:name

Make sure that the settings.py is configured with the same values that were used to train the model, otherwise the tagger cannot load the pretrained model correctly.

If you don't want to be bothered by the settings.py, you can also call the script with inline configuration. To tag a sentence using the FNN architecture, use the flags -p, -e and -s. It is required to use exactly these 3 flags, otherwise the static configuration from the settings.py will be used. An example call would be:

python tagger.py --tag "Show all modules of Bachelor Informatics" -p 1 -e 250 -s 350

To tag a sentence using the RNN architecture, use the flags -t, -e and -s. It is required to use exactly these 3 flags, otherwise the static configuration from the settings.py will be used. An example call would be:

python tagger.py --tag "Show all modules of Bachelor Informatics" -t 8 -e 100 -s 100

To evaluate a pretrained model on an external test set call the tagger.py script with the --evaluate parameter with the path to the file which contains the evaluation data. The evaluation data file must contain one sentence per line, containing space separated word/tag tuples.

$ python tagger.py --evaluate data/evaluation.txt
Loading saved vocabulary...
Generating tensors...
Evaluation starts...

# RESULTS:

  20 / 29   0.690   sentences correct
207 / 208   0.995   words recognized
197 / 208   0.947   tags correct
            0.966   kappa score

# ERRORS:

count           expected                    computed
------------------------------------------------------------
4     bachelor/C_Program:degree   bachelor/C_Program:name
4     master/C_Program:degree     master/C_Program:name
1     institute/X_Chair:name      institute/C_Program:name
1     quality/C_Chair:name        quality/C_Program:name
1     and/C_Chair:name            <UNKNOWN_WORD>/X

Make sure that the settings.py is configured with the same values that were used to train the model, otherwise the evaluation cannot load the pretrained model.

If you don't want to be bothered by the settings.py, you can also call the script with inline configuration. To tag a sentence using the FNN architecture, use the flags -p, -e and -s. It is required to use exactly these 3 flags, otherwise the static configuration from the settings.py will be used. An example call would be:

python tagger.py --evaluate data/evaluation.txt -p 1 -e 250 -s 350

To tag a sentence using the RNN architecture, use the flags -t, -e and -s. It is required to use exactly these 3 flags, otherwise the static configuration from the settings.py will be used. An example call would be:

python tagger.py --evaluate data/evaluation.txt -t 8 -e 100 -s 100

If you don't need the list of errors in the evaluation result, you can also print it in one line, adding the -i flag, i.e.:

$ python tagger.py --evaluate data/evaluation.txt -p 1 -e 250 -s 350 -i
data/evaluation_known.txt: 197/208 (94.7%) tags correct, 0.966 kappa score

To reset the tagger and delete all previously created files call the tagger.py script with the --reset flag and confirm with 'Yes' (or Enter):

$ python tagger.py --reset
Really delete all training data and log files? [Yes/no]
Reset was executed. All files successfully deleted.

If you don't want to be bothered by a security question, you can use the -f flag to force a direct deletion:

python tagger.py --reset -f

If you even don't want to be bothered by any output messages, you can use the -q flag to force a quiet deletion:

python tagger.py --reset -q

This toolkit is designed to be used in other applictions. The API is documented in the following.

Import the tagging script properly according to your directory structure, i.e.:

import tagger as nn

To instantiate the tagger, just call the Tagger() class. Without any parameters, the static configuration from the settings.py will be used:

t = nn.Tagger()

If you prefer inline configuration, pass the corresponding parameters according to the neural network architecture (the FNN needs n_past_words, the RNN needs n_timesteps). See these two examples, one for each architecture:

t = nn.Tagger('FNN', n_past_words=1, embedding_size=250, h_size=350, n_epochs=5)
t = nn.Tagger('RNN', n_timesteps=8, embedding_size=100, h_size=100, n_epochs=5)

To train the initialized tagger, just call the train() method with the path to the corpus file, i.e.:

t.train('data/test.corpus')

The trained model will be stored in the saved/ directory.

A sentence can by tagged with a pretrained model by calling the tag() method. You have additional parameters to print the tagging output in tabular form to the console (pretty_print) or mute console messages concerning model loading completely (silent).

tagged_sentence = t.tag('Show all modules of Bachelor Informatics', format_list=False, pretty_print=True, silent=False)

If you want to process lists instead of strings, you can use format_list=True as parameter. Now the input sentence has to be a list of words, an the output will be a list of word tag tuples:

$ sentence = ['Show', 'all', 'modules', 'of', 'Bachelor', 'Informatics']
$ tagged_sentence = t.tag(sentence, format_list=True, pretty_print=False, silent=True)
$ print list(tagged_sentece)
[('Show', 'R_LIST'), ('all', 'R_LIST'), ('modules', 'M_MTSModule'), ('of', 'X'), ('Bachelor', 'C_Program:degree'), ('Informatics', 'C_Program:name')]

To evaluate a pretrained model, calle the evaluate() method. You have an additional parameter to print the main evaluation results in one single line.

t.evaluate('data/evaluation.txt', print_inline=False)

• https://www.tensorflow.org/get_started/mnist/pros
• https://www.tensorflow.org/versions/master/api_docs/python/tf/nn

• https://github.com/mrahtz/tensorflow-pos-tagger
• https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/recurrent_network.ipynb

If you like and use this POS tagger and want to give some love back, feel free to...