NAACL 2016 submission "Structured Prediction with Neural Context Features" By Pushpendre Rastogi, Ryan Cotterell, Jason Eisner
@conference{rastogi2016weighting,
Author = {Pushpendre Rastogi and Ryan Cotterell and Jason Eisner},
Booktitle = {Proceedings of NAACL},
Date-Added = {2016-04-09 07:28:31 -0400},
Date-Modified = {2016-04-09 07:32:46 -0400},
Keywords = {Neural Network,Neural Symbolic,Finite State Transducer},
Title = {Weighting Finite-State Transductions With Neural Context},
Year = {2016},
Abstract = {How should one apply deep learning to tasks such as morphological reinflection, which stochastically edit one string to get another? A recent approach to such sequence-to-sequence tasks is to compress the input string into a vector that is then used to generate the output string, using recurrent neural networks. In contrast, we propose to keep the traditional architecture, which uses a finite-state transducer to score all possible output strings, but to augment the scoring function with the help of recurrent networks. A stack of bidirectional LSTMs reads the input string from left-to-right and right-to-left, in order to summarize the input context in which a transducer arc is applied. We combine these learned features with the transducer to define a probability distribution over aligned output strings, in the form of a weighted finite-state automaton. This reduces hand-engineering of features, allows learned features to examine unbounded context in the input string, and still permits exact inference through dynamic programming. We illustrate our method on the tasks of morphological reinflection and lemmatization.}}
Run the following command to compile the WFST portion of the model:
$ cd src/python/transducer
$ make # Make transducer.so and copy to src
$ cd - # Go back to toplevel
The following command will train the neural transducer model on the 4th fold of the rP-pA morphological transduction task. The test file is not used at this point.
PYTHONPATH=$PWD/src/python THEANO_FLAGS=floatX=float32 python -c "import transducer_score; print (
transducer_score.main(train_fn='res/celex/rP-pA/0500/4/train',
dev_fn='res/celex/rP-pA/0500/4/dev',
test_fn='res/celex/rP-pA/0500/4/test',
folder='results/tmp'))"
Once the model is trained and stored in the tmp directory we can test the model as follows:
PYTHONPATH=$PWD/src/python THEANO_FLAGS=floatX=float32 python -c "import transducer_score; print (
transducer_score.main(train_fn='res/celex/rP-pA/0500/4/train',
dev_fn='res/celex/rP-pA/0500/4/dev',
test_fn='res/celex/rP-pA/0500/4/test',
folder='results/tmp2',
pretrained_param_pklfile='results/tmp/transducer.pkl',
perform_training=0,
perform_testing=1,
nepochs=-1)"
For more complicated usage, including the exact parameters that were used to obtain the ablation results in the paper, see the scripts
src/python/transducer_celex.sh
src/python/transducer_celex_test.sh
These scripts contain the parameter strings that were used to obtain all the results in the paper.
-
How to fix
ImportError: No module named transducer?If you get the following error:
File "neural_wfst/src/python/transducer_score.py", line 25, in <module> from transducer.src.transducer import Transducer ImportError: No module named transducerThen run the following command:
$ cd src/python/transducer $ make # Make transducer.so and copy to srcSee the
Makefileinsrc/python/transducerto understand what's going on. In case there are further errors during compilation, then please raise an issue. -
Can I get the model predictions used in Table 1 and Table 2 of the paper ?
See the files
lemmatization_results.tgzandcelex_results.zipfiles. These archives contain many folders and each folder contains a single filecurrent.test.txtwhich contains the predictions of our model. Each file contains lines formatted like:input prediction goldOutput ^schwefelte schwefele schwefle ^flunkerte flunkere flunkere ^kuesste kuesse kuesse ^erahnte erahne erahne ^maulte maule mauleFor example the
celex_results.ziparchive contains the following folders. The tasks are the13SIA 2PIE 2PKE rPtasks. And the folds range from0to4.drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=13SIA-13SKE_fold=0_2 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=13SIA-13SKE_fold=1_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=13SIA-13SKE_fold=2_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=13SIA-13SKE_fold=3_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=13SIA-13SKE_fold=4_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PIE-13PKE_fold=0_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PIE-13PKE_fold=1_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PIE-13PKE_fold=2_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PIE-13PKE_fold=3_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PIE-13PKE_fold=4_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PKE-z_fold=0_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PKE-z_fold=1_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PKE-z_fold=2_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PKE-z_fold=3_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=2PKE-z_fold=4_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=rP-pA_fold=0_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=rP-pA_fold=1_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=rP-pA_fold=2_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=rP-pA_fold=3_0 drwxr-xr-x 2 prastog3 fax 29 Dec 28 2015 transtest_transducer_deep_task=rP-pA_fold=4_0