This is a TensorFlow implementation of the model described in:
Jamie Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler. Skip-Thought Vectors. In NIPS, 2015.
Code author: Chris Shallue
Pull requests and issues: @cshallue
- Model Overview
- Getting Started
- Training a Model
- Expanding the Vocabulary
- Evaluating a Model
- Encoding Sentences
The Skip-Thoughts model is a sentence encoder. It learns to encode input sentences into a fixed-dimensional vector representation that is useful for many tasks, for example to detect paraphrases or to classify whether a product review is positive or negative. See the Skip-Thought Vectors paper for details of the model architecture and more example applications.
A trained Skip-Thoughts model will encode similar sentences nearby each other in the embedding vector space. The following examples show the nearest neighbor by cosine similarity of some sentences from the movie review dataset.
Input sentence | Nearest Neighbor |
---|---|
Simplistic, silly and tedious. | Trite, banal, cliched, mostly inoffensive. |
Not so much farcical as sour. | Not only unfunny, but downright repellent. |
A sensitive and astute first feature by Anne-Sophie Birot. | Absorbing character study by André Turpin . |
An enthralling, entertaining feature. | A slick, engrossing melodrama. |
First ensure that you have installed the following required packages:
- Bazel (instructions)
- TensorFlow (instructions)
- NumPy (instructions)
- scikit-learn (instructions)
- Natural Language Toolkit (NLTK)
- First install NLTK (instructions)
- Then install the NLTK data (instructions)
- gensim (instructions)
- Only required if you will be expanding your vocabulary with the word2vec model.
You can download model checkpoints pretrained on the BookCorpus dataset in the following configurations:
- Unidirectional RNN encoder ("uni-skip" in the paper)
- Bidirectional RNN encoder ("bi-skip" in the paper)
# Directory to download the pretrained models to.
PRETRAINED_MODELS_DIR="${HOME}/skip_thoughts/pretrained/"
mkdir -p ${PRETRAINED_MODELS_DIR}
cd ${PRETRAINED_MODELS_DIR}
# Download and extract the unidirectional model.
wget "https://download.tensorflow.org/models/skip_thoughts_uni_2017_02_02.tar.gz"
tar -xvf skip_thoughts_uni_2017_02_02.tar.gz
rm skip_thoughts_uni_2017_02_02.tar.gz
# Download and extract the bidirectional model.
wget "https://download.tensorflow.org/models/skip_thoughts_bi_2017_02_16.tar.gz"
tar -xvf skip_thoughts_bi_2017_02_16.tar.gz
rm skip_thoughts_bi_2017_02_16.tar.gz
You can now skip to the sections Evaluating a Model and Encoding Sentences.
To train a model you will need to provide training data in TFRecord format. The
TFRecord format consists of a set of sharded files containing serialized
tf.Example
protocol buffers. Each tf.Example
proto contains three
sentences:
encode
: The sentence to encode.decode_pre
: The sentence precedingencode
in the original text.decode_post
: The sentence followingencode
in the original text.
Each sentence is a list of words. During preprocessing, a dictionary is created
that assigns each word in the vocabulary to an integer-valued id. Each sentence
is encoded as a list of integer word ids in the tf.Example
protos.
We have provided a script to preprocess any set of text-files into this format. You may wish to use the BookCorpus dataset. Note that the preprocessing script may take 12 hours or more to complete on this large dataset.
# Comma-separated list of globs matching the input input files. The format of
# the input files is assumed to be a list of newline-separated sentences, where
# each sentence is already tokenized.
INPUT_FILES="${HOME}/skip_thoughts/bookcorpus/*.txt"
# Location to save the preprocessed training and validation data.
DATA_DIR="${HOME}/skip_thoughts/data"
# Build the preprocessing script.
cd tensorflow-models/skip_thoughts
bazel build -c opt //skip_thoughts/data:preprocess_dataset
# Run the preprocessing script.
bazel-bin/skip_thoughts/data/preprocess_dataset \
--input_files=${INPUT_FILES} \
--output_dir=${DATA_DIR}
When the script finishes you will find 100 training files and 1 validation file
in DATA_DIR
. The files will match the patterns train-?????-of-00100
and
validation-00000-of-00001
respectively.
The script will also produce a file named vocab.txt
. The format of this file
is a list of newline-separated words where the word id is the corresponding 0-
based line index. Words are sorted by descending order of frequency in the input
data. Only the top 20,000 words are assigned unique ids; all other words are
assigned the "unknown id" of 1 in the processed data.
Execute the following commands to start the training script. By default it will run for 500k steps (around 9 days on a GeForce GTX 1080 GPU).
# Directory containing the preprocessed data.
DATA_DIR="${HOME}/skip_thoughts/data"
# Directory to save the model.
MODEL_DIR="${HOME}/skip_thoughts/model"
# Build the model.
cd tensorflow-models/skip_thoughts
bazel build -c opt //skip_thoughts/...
# Run the training script.
bazel-bin/skip_thoughts/train \
--input_file_pattern="${DATA_DIR}/train-?????-of-00100" \
--train_dir="${MODEL_DIR}/train"
Optionally, you can run the track_perplexity
script in a separate process.
This will log per-word perplexity on the validation set which allows training
progress to be monitored on
TensorBoard.
Note that you may run out of memory if you run the this script on the same GPU
as the training script. You can set the environment variable
CUDA_VISIBLE_DEVICES=""
to force the script to run on CPU. If it runs too
slowly on CPU, you can decrease the value of --num_eval_examples
.
DATA_DIR="${HOME}/skip_thoughts/data"
MODEL_DIR="${HOME}/skip_thoughts/model"
# Ignore GPU devices (only necessary if your GPU is currently memory
# constrained, for example, by running the training script).
export CUDA_VISIBLE_DEVICES=""
# Run the evaluation script. This will run in a loop, periodically loading the
# latest model checkpoint file and computing evaluation metrics.
bazel-bin/skip_thoughts/track_perplexity \
--input_file_pattern="${DATA_DIR}/validation-?????-of-00001" \
--checkpoint_dir="${MODEL_DIR}/train" \
--eval_dir="${MODEL_DIR}/val" \
--num_eval_examples=50000
If you started the track_perplexity
script, run a
TensorBoard
server in a separate process for real-time monitoring of training summaries and
validation perplexity.
MODEL_DIR="${HOME}/skip_thoughts/model"
# Run a TensorBoard server.
tensorboard --logdir="${MODEL_DIR}"
The vocabulary generated by the preprocessing script contains only 20,000 words which is insufficient for many tasks. For example, a sentence from Wikipedia might contain nouns that do not appear in this vocabulary.
A solution to this problem described in the Skip-Thought Vectors paper is to learn a mapping that transfers word representations from one model to another. This idea is based on the "Translation Matrix" method from the paper Exploiting Similarities Among Languages for Machine Translation.
Specifically, we will load the word embeddings from a trained Skip-Thoughts model and from a trained word2vec model (which has a much larger vocabulary). We will train a linear regression model without regularization to learn a linear mapping from the word2vec embedding space to the Skip-Thoughts embedding space. We will then apply the linear model to all words in the word2vec vocabulary, yielding vectors in the Skip- Thoughts word embedding space for the union of the two vocabularies.
The linear regression task is to learn a parameter matrix W to minimize
|| X - Y * W ||2, where X is a matrix of Skip-Thoughts
embeddings of shape [num_words, dim1]
, Y is a matrix of word2vec embeddings
of shape [num_words, dim2]
, and W is a matrix of shape [dim2, dim1]
.
First you will need to download and unpack a pretrained word2vec model from this website (direct download link). This model was trained on the Google News dataset (about 100 billion words).
Also ensure that you have already installed gensim.
# Path to checkpoint file or a directory containing checkpoint files (the script
# will select the most recent).
CHECKPOINT_PATH="${HOME}/skip_thoughts/model/train"
# Vocabulary file generated by the preprocessing script.
SKIP_THOUGHTS_VOCAB="${HOME}/skip_thoughts/data/vocab.txt"
# Path to downloaded word2vec model.
WORD2VEC_MODEL="${HOME}/skip_thoughts/googlenews/GoogleNews-vectors-negative300.bin"
# Output directory.
EXP_VOCAB_DIR="${HOME}/skip_thoughts/exp_vocab"
# Build the vocabulary expansion script.
cd tensorflow-models/skip_thoughts
bazel build -c opt //skip_thoughts:vocabulary_expansion
# Run the vocabulary expansion script.
bazel-bin/skip_thoughts/vocabulary_expansion \
--skip_thoughts_model=${CHECKPOINT_PATH} \
--skip_thoughts_vocab=${SKIP_THOUGHTS_VOCAB} \
--word2vec_model=${WORD2VEC_MODEL} \
--output_dir=${EXP_VOCAB_DIR}
The model can be evaluated using the benchmark tasks described in the Skip-Thought Vectors paper. The following tasks are supported (refer to the paper for full details):
- SICK semantic relatedness task.
- MSRP (Microsoft Research Paraphrase Corpus) paraphrase detection task.
- Binary classification tasks:
- MR movie review sentiment task.
- CR customer product review task.
- SUBJ subjectivity/objectivity task.
- MPQA opinion polarity task.
- TREC question-type classification task.
You will need to clone or download the skip-thoughts GitHub repository by ryankiros (the first author of the Skip-Thoughts paper):
# Folder to clone the repository to.
ST_KIROS_DIR="${HOME}/skip_thoughts/skipthoughts_kiros"
# Clone the repository.
git clone [email protected]:ryankiros/skip-thoughts.git "${ST_KIROS_DIR}/skipthoughts"
# Make the package importable.
export PYTHONPATH="${ST_KIROS_DIR}/:${PYTHONPATH}"
You will also need to download the data needed for each evaluation task. See the instructions here.
For example, the CR (customer review) dataset is found here. For this task we want the
files custrev.pos
and custrev.neg
.
In the following example we will evaluate a unidirectional model ("uni-skip" in
the paper) on the CR task. To use a bidirectional model ("bi-skip" in the
paper), simply pass the flags --bi_vocab_file
, --bi_embeddings_file
and
--bi_checkpoint_path
instead. To use the "combine-skip" model described in the
paper you will need to pass both the unidirectional and bidirectional flags.
# Path to checkpoint file or a directory containing checkpoint files (the script
# will select the most recent).
CHECKPOINT_PATH="${HOME}/skip_thoughts/model/train"
# Vocabulary file generated by the vocabulary expansion script.
VOCAB_FILE="${HOME}/skip_thoughts/exp_vocab/vocab.txt"
# Embeddings file generated by the vocabulary expansion script.
EMBEDDINGS_FILE="${HOME}/skip_thoughts/exp_vocab/embeddings.npy"
# Directory containing files custrev.pos and custrev.neg.
EVAL_DATA_DIR="${HOME}/skip_thoughts/eval_data"
# Build the evaluation script.
cd tensorflow-models/skip_thoughts
bazel build -c opt //skip_thoughts:evaluate
# Run the evaluation script.
bazel-bin/skip_thoughts/evaluate \
--eval_task=CR \
--data_dir=${EVAL_DATA_DIR} \
--uni_vocab_file=${VOCAB_FILE} \
--uni_embeddings_file=${EMBEDDINGS_FILE} \
--uni_checkpoint_path=${CHECKPOINT_PATH}
Output:
[0.82539682539682535, 0.84084880636604775, 0.83023872679045096,
0.86206896551724133, 0.83554376657824936, 0.85676392572944293,
0.84084880636604775, 0.83023872679045096, 0.85145888594164454,
0.82758620689655171]
The output is a list of accuracies of 10 cross-validation classification models. To get a single number, simply take the average:
ipython # Launch iPython.
In [0]:
import numpy as np
np.mean([0.82539682539682535, 0.84084880636604775, 0.83023872679045096,
0.86206896551724133, 0.83554376657824936, 0.85676392572944293,
0.84084880636604775, 0.83023872679045096, 0.85145888594164454,
0.82758620689655171])
Out [0]: 0.84009936423729525
In this example we will encode data from the movie review dataset (specifically the sentence polarity dataset v1.0).
ipython # Launch iPython.
In [0]:
# Imports.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
import numpy as np
import os.path
import scipy.spatial.distance as sd
from skip_thoughts import configuration
from skip_thoughts import encoder_manager
In [1]:
# Set paths to the model.
VOCAB_FILE = "/path/to/vocab.txt"
EMBEDDING_MATRIX_FILE = "/path/to/embeddings.npy"
CHECKPOINT_PATH = "/path/to/model.ckpt-9999"
# The following directory should contain files rt-polarity.neg and
# rt-polarity.pos.
MR_DATA_DIR = "/dir/containing/mr/data"
In [2]:
# Set up the encoder. Here we are using a single unidirectional model.
# To use a bidirectional model as well, call load_model() again with
# configuration.model_config(bidirectional_encoder=True) and paths to the
# bidirectional model's files. The encoder will use the concatenation of
# all loaded models.
encoder = encoder_manager.EncoderManager()
encoder.load_model(configuration.model_config(),
vocabulary_file=VOCAB_FILE,
embedding_matrix_file=EMBEDDING_MATRIX_FILE,
checkpoint_path=CHECKPOINT_PATH)
In [3]:
# Load the movie review dataset.
data = []
with open(os.path.join(MR_DATA_DIR, 'rt-polarity.neg'), 'rb') as f:
data.extend([line.decode('latin-1').strip() for line in f])
with open(os.path.join(MR_DATA_DIR, 'rt-polarity.pos'), 'rb') as f:
data.extend([line.decode('latin-1').strip() for line in f])
In [4]:
# Generate Skip-Thought Vectors for each sentence in the dataset.
encodings = encoder.encode(data)
In [5]:
# Define a helper function to generate nearest neighbors.
def get_nn(ind, num=10):
encoding = encodings[ind]
scores = sd.cdist([encoding], encodings, "cosine")[0]
sorted_ids = np.argsort(scores)
print("Sentence:")
print("", data[ind])
print("\nNearest neighbors:")
for i in range(1, num + 1):
print(" %d. %s (%.3f)" %
(i, data[sorted_ids[i]], scores[sorted_ids[i]]))
In [6]:
# Compute nearest neighbors of the first sentence in the dataset.
get_nn(0)
Output:
Sentence:
simplistic , silly and tedious .
Nearest neighbors:
1. trite , banal , cliched , mostly inoffensive . (0.247)
2. banal and predictable . (0.253)
3. witless , pointless , tasteless and idiotic . (0.272)
4. loud , silly , stupid and pointless . (0.295)
5. grating and tedious . (0.299)
6. idiotic and ugly . (0.330)
7. black-and-white and unrealistic . (0.335)
8. hopelessly inane , humorless and under-inspired . (0.335)
9. shallow , noisy and pretentious . (0.340)
10. . . . unlikable , uninteresting , unfunny , and completely , utterly inept . (0.346)