Skip to content

Text simplification for a better world: Deep-Martin Transformer 🤗

Notifications You must be signed in to change notification settings

chrislemke/deep-martin

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

55 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Deep Martin

Text simplification for the democratization of knowledge

Danach ist das In-der-Welt-sein ein Sich-vorweg-schon sein-in-der Welt als Sein-bei-innerweltlich-begegnendem-Seienden
Martin Heidegger

Unsimplifiable, untranslatable

Language as a fundamental characteristic of man and society is the center of NLP. It has the potential of great enlightenment, as well as great concealment. Language and thinking must be brought into harmony. Simplification of language leads to the democratization of knowledge. Thus, it can provide access to knowledge that may otherwise be hidden. No more complex language!

Deep Martin aims to contribute to this. The project is dedicated to different models to make complicated and complex content accessible to all. It follows the approach of Simple Wikipedia.

About the project

How to use

Two different approaches are available. One is to use the super nice Hugging Face library. This can be used to create various state-of-the-art sequence to sequence models. The other part is a self-made transformer. Here it is mainly about trying out different approaches.

Hugging Face

For using the Hugging Face implementation you need to provide a dataset. It needs to have one column with the normal version (Normal) and one for the simplified version (Simple). The HuggingFaceDataset class can help you with it.
To train a model you then simply run something like:

python /your/path/to/deep-martin/src/hf_transformer_trainer.py \
--eval_steps 5000 \ # This number should be based on the size of the dataset. 
--warmup_steps 800 \ # This number should be based on the size of the dataset.
--ds_path /path/ \ # Path to you dataset.
--save_model_path /path/ \ # Path to where the trained model should be stored.
--training_output_path /path/ \ # Path to where the checkpoints and the training data should be stored.
--tokenizer_id bert-base-cased # Path or identifier to Hugging Face tokenizer.

There are a lot more parameters. Check out hf_transformer_trainer.py to get an overview.

Self-made-transformer

This transformer is more for experimenting. Have a look at the code and get an overview of what is going on. To train the self-made-transformer, a train and a test dataset as CSV is needed. This will be transformed to a suitable dataset at the beginning of the training. Same as with the transformers from above the dataset needs to have one column with the normal version (Normal) and one for the simplified version (Simple)
To start the training you can run:

python /your/path/to/deep-martin/src/custom_transformer_trainer.py \
--ds_path /path \ # Path of the folder which contains the `train_file.csv` and the `test_file.csv`
--train_file train_file.csv \
--test_file test_file.csv \
--epochs 3 \
--save_model_path /path/ # Path to where the trained model should be stored.

Challenges

Let's talk about the problems in this project.

Dataset

As so often, one problem lies in obtaining high-quality data. Multiple datasets were used for this project. You can find them here.
While the ASSET dataset provides a very good quality due to the multiple simplification of each record, its size is simply too small for training a transformer. This problem is also true for other datasets. The two datasets based on Wikipedia unfortunately suffer from lack of quality. Either a record is not a simplification, but simply the same article. Or the simplification is of poor quality. In both cases, using it meant worse results. To increase the overall quality, the records were compared and filtered out using Doc2Vec and cosine distance.

Model size and computation

Transformers are huge, need a lot of data and a lot of time to train. Google colab can help, but it is not the most convenient way. With the help of AWS EC2, things can be sped up a lot and, training of larger models is also possible.

Next steps

Since the self-made-transformer is a work-in-progress project, it is never finished. It is made for learning and trying out. One interesting idea is to use the transformer as a generator in a GAN to improve the overall output.