notebooks/Flair Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Flair Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2018-2024 by [Damir Cavar](https://damir.cavar.me/)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Version:** 0.5, January 2024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Download:** This and various other Jupyter notebooks are available from my [GitHub repo](https://github.com/dcavar/python-tutorial-for-ipython)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**License:** [Creative Commons Attribution-ShareAlike 4.0 International License](https://creativecommons.org/licenses/by-sa/4.0/) ([CA BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Prerequisites:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U flair"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This material was used in my Advanced Topics in AI class, introduction to Deep Learning environments in Spring 2019 at Indiana University."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two types of central objects:\n",
    "\n",
    "- Sentence\n",
    "- Token\n",
    "\n",
    "A Sentence holds a textual sentence and is essentially a list of Token objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For creating a Sentence object we first import the Sentence class from the *flair.data* module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.data import Sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now define a sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence: \"The grass is green .\"   [− Tokens: 5]\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('The grass is green .')\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the tokens of a sentence via their token id or with their index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token: 4 green\n",
      "Token: 4 green\n"
     ]
    }
   ],
   "source": [
    "print(sentence.get_token(4))\n",
    "print(sentence[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also iterate over all tokens in a sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token: 1 The\n",
      "Token: 2 grass\n",
      "Token: 3 is\n",
      "Token: 4 green\n",
      "Token: 5 .\n"
     ]
    }
   ],
   "source": [
    "for token in sentence:\n",
    "    print(token)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a simple tokenizer included in the library using the lightweight segtok library to tokenize your text for such a Sentence definition. In the Sentence constructor use the flag *use_tokenize* to tokenize the input string before instantiating a Sentence object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence: \"The grass is green .\"   [− Tokens: 5]\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('The grass is green.', use_tokenizer=True)\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tags on Tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Token has fields for linguistic annotation:\n",
    "\n",
    "- lemma\n",
    "- part-of-speech tag\n",
    "- named entity tag\n",
    "\n",
    "We can add a tag by specifying the tag type and the tag value.\n",
    "\n",
    "We will adding an NER tag of type 'color' to the word 'green'. This means that we've tagged this word as an entity of type color:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The grass is green <color> .\n"
     ]
    }
   ],
   "source": [
    "sentence[3].add_tag('ner', 'color')\n",
    "print(sentence.to_tagged_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each tag is of class Label. An associated score indicates confidence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"Token: 4 green\" is tagged as \"color\" with confidence score \"1.0\"\n"
     ]
    }
   ],
   "source": [
    "from flair.data import Label\n",
    "\n",
    "tag: Label = sentence[3].get_tag('ner')\n",
    "print(f'\"{sentence[3]}\" is tagged as \"{tag.value}\" with confidence score \"{tag.score}\"')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The manually added color tag has a score of 1.0. A tag predicted by a sequence labeler will have a score value that indicates the classifier confidence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Sentence can have one or multiple labels that can for example be used in text classification tasks. For instance, the example below shows how we add the label 'sports' to a sentence, thereby labeling it as belonging to the sports category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence: \"France is the current world cup winner .\"   [− Tokens: 8  − Sentence-Labels: {'topic': [sports (1.0)], 'language': [English (1.0)]}]\n",
      "Sentence: \"France is the current world cup winner .\"   [− Tokens: 8  − Sentence-Labels: {'topic': [sports (1.0)]}]\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('France is the current world cup winner.')\n",
    "sentence.add_label('topic', 'sports')\n",
    "sentence.add_label('language', 'English')\n",
    "print(sentence)\n",
    "# sentence.add_labels(1, ['sports', 'world cup'])\n",
    "sentence = Sentence('France is the current world cup winner.').add_label('topic', 'sports')\n",
    "print(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Labels are also of the Label class. So, you can print a sentence's labels like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentence: \"France is the current world cup winner .\"   [− Tokens: 8  − Sentence-Labels: {'sports': [world cup (1.0)]}]\n",
      "world cup (1.0)\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('France is the current world cup winner.').add_label('sports', 'world cup')\n",
    "\n",
    "print(sentence)\n",
    "for label in sentence.labels:\n",
    "    print(label)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tagging Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Pre-Trained Sequence Tagging Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flair has numerous pre-trained models. For the named entity recognition (NER) task there is a model that was trained on the English CoNLL-03 task and can recognize 4 different entity types. Import it using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-18 16:15:08,786 loading file /home/damir/.flair/models/en-ner-conll03-v0.4.pt\n"
     ]
    }
   ],
   "source": [
    "from flair.models import SequenceTagger\n",
    "\n",
    "tagger = SequenceTagger.load('ner')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the predict() method of the tagger on a sentence to add predicted tags to the tokens in the sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tim <B-PER> Cook <E-PER> went to New <B-LOC> York <I-LOC> City <E-LOC> .\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('Tim Cook went to New York City .')\n",
    "\n",
    "tagger.predict(sentence)\n",
    "\n",
    "print(sentence.to_tagged_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting annotated spans for multi-word expressions can be achieved using the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Span [1,2]: \"Tim Cook\"   [− Labels: PER (0.9997)]\n",
      "Span [5,6,7]: \"New York City\"   [− Labels: LOC (0.9595)]\n"
     ]
    }
   ],
   "source": [
    "for entity in sentence.get_spans('ner'):\n",
    "    print(entity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Which indicates that \"George Washington\" is a person (PER) and \"Washington\" is a location (LOC). Each such Span has a text, a tag value, its position in the sentence and \"score\" that indicates how confident the tagger is that the prediction is correct. You can also get additional information, such as the position offsets of each entity in the sentence by calling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': 'Tim Cook went to New York City .', 'labels': [], 'entities': [{'text': 'Tim Cook', 'start_pos': 0, 'end_pos': 8, 'labels': [PER (0.9997)]}, {'text': 'New York City', 'start_pos': 17, 'end_pos': 30, 'labels': [LOC (0.9595)]}]}\n"
     ]
    }
   ],
   "source": [
    "print(sentence.to_dict(tag_type='ner'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': 'Tim Cook went to New York City .', 'labels': [], 'entities': []}\n"
     ]
    }
   ],
   "source": [
    "print(sentence.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flair contains various sequence tagger models. You choose which pre-trained model you load by passing the appropriate string to the load() method of the SequenceTagger class. Currently, the following pre-trained models are provided:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As indicated in the list above, we also provide pre-trained models for languages other than English. Currently, we support German, French, and Dutch other languages are forthcoming. To tag a German sentence, just load the appropriate model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-18 16:16:52,772 loading file /home/damir/.flair/models/de-ner-conll03-v0.4.pt\n",
      "George <B-PER> Washington <E-PER> ging nach Washington <S-LOC> .\n"
     ]
    }
   ],
   "source": [
    "tagger = SequenceTagger.load('de-ner')\n",
    "\n",
    "sentence = Sentence('George Washington ging nach Washington .')\n",
    "\n",
    "tagger.predict(sentence)\n",
    "\n",
    "print(sentence.to_tagged_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flair offers access to multi-lingual models for multi-lingual text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-18 16:17:05,167 loading file /home/damir/.flair/models/pos-multi-v0.1.pt\n",
      "George <PROPN> Washington <PROPN> lebte <VERB> in <ADP> Washington <PROPN> . <PUNCT> Dort <PROPN> kaufte <VERB> er <PRON> a <DET> horse <NOUN> . <PUNCT>\n"
     ]
    }
   ],
   "source": [
    "tagger = SequenceTagger.load('pos-multi')\n",
    "\n",
    "sentence = Sentence('George Washington lebte in Washington . Dort kaufte er a horse .')\n",
    "\n",
    "tagger.predict(sentence)\n",
    "\n",
    "print(sentence.to_tagged_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Semantic Frames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For English, Flair provides a pre-trained model that detects semantic frames in text, trained using Propbank 3.0 frames. This provides some sort of word sense disambiguation for frame evoking words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-18 16:17:43,673 loading file /home/damir/.flair/models/en-frame-ontonotes-v0.4.pt\n",
      "George returned <return.01> to Berlin to return <return.02> his hat .\n",
      "He had <have.03> a look <look.01> at different hats .\n"
     ]
    }
   ],
   "source": [
    "tagger = SequenceTagger.load('frame')\n",
    "\n",
    "sentence_1 = Sentence('George returned to Berlin to return his hat .')\n",
    "sentence_2 = Sentence('He had a look at different hats .')\n",
    "\n",
    "tagger.predict(sentence_1)\n",
    "tagger.predict(sentence_2)\n",
    "\n",
    "print(sentence_1.to_tagged_string())\n",
    "print(sentence_2.to_tagged_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The frame detector makes a distinction in sentence 1 between two different meanings of the word 'return'. 'return.01' means returning to a location, while 'return.02' means giving something back.\n",
    "\n",
    "Similarly, in sentence 2 the frame detector finds a light verb construction in which 'have' is the light verb and 'look' is a frame evoking word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentence Tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To tag an entire text corpus, one needs to split the corpus into sentences and pass a list of Sentence objects to the .predict() method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-18 16:18:30,907 loading file /home/damir/.flair/models/en-ner-conll03-v0.4.pt\n",
      "This is a sentence .\n",
      "John <S-PER> read a book .\n",
      "This is another sentence .\n",
      "I love Berlin <S-LOC> .\n"
     ]
    }
   ],
   "source": [
    "text = \"This is a sentence. John read a book. This is another sentence. I love Berlin.\"\n",
    "\n",
    "from segtok.segmenter import split_single\n",
    "\n",
    "sentences = [Sentence(sent, use_tokenizer=True) for sent in split_single(text)]\n",
    "\n",
    "tagger: SequenceTagger = SequenceTagger.load('ner')\n",
    "tagger.predict(sentences)\n",
    "for i in sentences:\n",
    "    print(i.to_tagged_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the mini_batch_size parameter of the .predict() method, you can set the size of mini batches passed to the tagger. Depending on your resources, you might want to play around with this parameter to optimize speed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-Trained Text Classification Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flair provides a pre-trained model for detecting positive or negative comments. It was trained on the IMDB dataset and it can recognize positive and negative sentiment in English text. The IMDB data set can be downloaded from the [linked site](https://ai.stanford.edu/~amaas/data/sentiment/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-11-18 16:19:13,086 loading file /home/damir/.flair/models/sentiment-en-mix-distillbert_3.1.pt\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Widget Javascript not detected.  It may not be installed or enabled properly.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a60170abc3ca4284bdd2478351cf7d04"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from flair.models import TextClassifier\n",
    "\n",
    "classifier = TextClassifier.load('en-sentiment')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We call the predict() method of the classifier on a sentence. This will add the predicted label to the sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[POSITIVE (0.9983)]\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('This film hurts. It is so bad that I am confused.')\n",
    "sentence = Sentence(\"I love this film.\")\n",
    "classifier.predict(sentence)\n",
    "\n",
    "print(sentence.labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[POSITIVE (0.9977)]\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('This film is fantastic. I love it.')\n",
    "\n",
    "classifier.predict(sentence)\n",
    "\n",
    "print(sentence.labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flair has a pre-trained German and English model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flair provides a set of classes with which we can embed the words in sentences in various ways."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All word embedding classes inherit from the TokenEmbeddings class and implement the embed() method which we need to call to embed our text. This means that for most users of Flair, the complexity of different embeddings remains hidden behind this interface. Simply instantiate the embedding class we require and call embed() to embed our text.\n",
    "\n",
    "All embeddings produced with Flair's methods are Pytorch vectors, so they can be immediately used for training and fine-tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classic word embeddings are static and word-level, meaning that each distinct word gets exactly one pre-computed embedding. Most embeddings fall under this class, including the popular GloVe or Komninos embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We instantiate the WordEmbeddings class and pass a string identifier of the embedding we wish to load. If we want to use GloVe embeddings, we pass the string 'glove' to the constructor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import WordEmbeddings\n",
    "\n",
    "glove_embedding = WordEmbeddings('glove')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create an example sentence and call the embedding's embed() method. We can also pass a list of sentences to this method since some embedding types make use of batching to increase speed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token: 1 The\n",
      "tensor([-0.0382, -0.2449,  0.7281, -0.3996,  0.0832,  0.0440, -0.3914,  0.3344,\n",
      "        -0.5755,  0.0875,  0.2879, -0.0673,  0.3091, -0.2638, -0.1323, -0.2076,\n",
      "         0.3340, -0.3385, -0.3174, -0.4834,  0.1464, -0.3730,  0.3458,  0.0520,\n",
      "         0.4495, -0.4697,  0.0263, -0.5415, -0.1552, -0.1411, -0.0397,  0.2828,\n",
      "         0.1439,  0.2346, -0.3102,  0.0862,  0.2040,  0.5262,  0.1716, -0.0824,\n",
      "        -0.7179, -0.4153,  0.2033, -0.1276,  0.4137,  0.5519,  0.5791, -0.3348,\n",
      "        -0.3656, -0.5486, -0.0629,  0.2658,  0.3020,  0.9977, -0.8048, -3.0243,\n",
      "         0.0125, -0.3694,  2.2167,  0.7220, -0.2498,  0.9214,  0.0345,  0.4674,\n",
      "         1.1079, -0.1936, -0.0746,  0.2335, -0.0521, -0.2204,  0.0572, -0.1581,\n",
      "        -0.3080, -0.4162,  0.3797,  0.1501, -0.5321, -0.2055, -1.2526,  0.0716,\n",
      "         0.7056,  0.4974, -0.4206,  0.2615, -1.5380, -0.3022, -0.0734, -0.2831,\n",
      "         0.3710, -0.2522,  0.0162, -0.0171, -0.3898,  0.8742, -0.7257, -0.5106,\n",
      "        -0.5203, -0.1459,  0.8278,  0.2706], device='cuda:0')\n",
      "Token: 2 grass\n",
      "tensor([-0.8135,  0.9404, -0.2405, -0.1350,  0.0557,  0.3363,  0.0802, -0.1015,\n",
      "        -0.5478, -0.3537,  0.0734,  0.2587,  0.1987, -0.1433,  0.2507,  0.4281,\n",
      "         0.1950,  0.5346,  0.7424,  0.0578, -0.3178,  0.9436,  0.8145, -0.0824,\n",
      "         0.6166,  0.7284, -0.3262, -1.3641,  0.1232,  0.5373, -0.5123,  0.0246,\n",
      "         1.0822, -0.2296,  0.6039,  0.5541, -0.9610,  0.4803,  0.0022,  0.5591,\n",
      "        -0.1637, -0.8468,  0.0741, -0.6216,  0.0260, -0.5162, -0.0525, -0.1418,\n",
      "        -0.0161, -0.4972, -0.5534, -0.4037,  0.5096,  1.0276, -0.0840, -1.1179,\n",
      "         0.3226,  0.4928,  0.9488,  0.2040,  0.5388,  0.8397, -0.0689,  0.3136,\n",
      "         1.0450, -0.2267, -0.0896, -0.6427,  0.6443, -1.1001, -0.0096,  0.2668,\n",
      "        -0.3230, -0.6065,  0.0479, -0.1664,  0.8571,  0.2335,  0.2539,  1.2546,\n",
      "         0.5472, -0.1980, -0.7186,  0.2076, -0.2587, -0.3650,  0.0834,  0.6932,\n",
      "         0.1574,  1.0931,  0.0913, -1.3773, -0.2717,  0.7071,  0.1872, -0.3307,\n",
      "        -0.2836,  0.1030,  1.2228,  0.8374], device='cuda:0')\n",
      "Token: 3 is\n",
      "tensor([-0.5426,  0.4148,  1.0322, -0.4024,  0.4669,  0.2182, -0.0749,  0.4733,\n",
      "         0.0810, -0.2208, -0.1281, -0.1144,  0.5089,  0.1157,  0.0282, -0.3628,\n",
      "         0.4382,  0.0475,  0.2028,  0.4986, -0.1007,  0.1327,  0.1697,  0.1165,\n",
      "         0.3135,  0.2571,  0.0928, -0.5683, -0.5297, -0.0515, -0.6733,  0.9253,\n",
      "         0.2693,  0.2273,  0.6636,  0.2622,  0.1972,  0.2609,  0.1877, -0.3454,\n",
      "        -0.4263,  0.1398,  0.5634, -0.5691,  0.1240, -0.1289,  0.7248, -0.2610,\n",
      "        -0.2631, -0.4360,  0.0789, -0.8415,  0.5160,  1.3997, -0.7646, -3.1453,\n",
      "        -0.2920, -0.3125,  1.5129,  0.5243,  0.2146,  0.4245, -0.0884, -0.1780,\n",
      "         1.1876,  0.1058,  0.7657,  0.2191,  0.3582, -0.1164,  0.0933, -0.6248,\n",
      "        -0.2190,  0.2180,  0.7406, -0.4374,  0.1434,  0.1472, -1.1605, -0.0505,\n",
      "         0.1268, -0.0144, -0.9868, -0.0913, -1.2054, -0.1197,  0.0478, -0.5400,\n",
      "         0.5246, -0.7096, -0.3253, -0.1346, -0.4131,  0.3343, -0.0072,  0.3225,\n",
      "        -0.0442, -1.2969,  0.7622,  0.4635], device='cuda:0')\n",
      "Token: 4 green\n",
      "tensor([-6.7907e-01,  3.4908e-01, -2.3984e-01, -9.9652e-01,  7.3782e-01,\n",
      "        -6.5911e-04,  2.8010e-01,  1.7287e-02, -3.6063e-01,  3.6955e-02,\n",
      "        -4.0395e-01,  2.4092e-02,  2.8958e-01,  4.0497e-01,  6.9992e-01,\n",
      "         2.5269e-01,  8.0350e-01,  4.9370e-02,  1.5562e-01, -6.3286e-03,\n",
      "        -2.9414e-01,  1.4728e-01,  1.8977e-01, -5.1791e-01,  3.6986e-01,\n",
      "         7.4582e-01,  8.2689e-02, -7.2601e-01, -4.0939e-01, -9.7822e-02,\n",
      "        -1.4096e-01,  7.1121e-01,  6.1933e-01, -2.5014e-01,  4.2250e-01,\n",
      "         4.8458e-01, -5.1915e-01,  7.7125e-01,  3.6685e-01,  4.9652e-01,\n",
      "        -4.1298e-02, -1.4683e+00,  2.0038e-01,  1.8591e-01,  4.9860e-02,\n",
      "        -1.7523e-01, -3.5528e-01,  9.4153e-01, -1.1898e-01, -5.1903e-01,\n",
      "        -1.1887e-02, -3.9186e-01, -1.7479e-01,  9.3451e-01, -5.8931e-01,\n",
      "        -2.7701e+00,  3.4522e-01,  8.6533e-01,  1.0808e+00, -1.0291e-01,\n",
      "        -9.1220e-02,  5.5092e-01, -3.9473e-01,  5.3676e-01,  1.0383e+00,\n",
      "        -4.0658e-01,  2.4590e-01, -2.6797e-01, -2.6036e-01, -1.4151e-01,\n",
      "        -1.2022e-01,  1.6234e-01, -7.4320e-01, -6.4728e-01,  4.7133e-02,\n",
      "         5.1642e-01,  1.9898e-01,  2.3919e-01,  1.2550e-01,  2.2471e-01,\n",
      "         8.2613e-01,  7.8328e-02, -5.7020e-01,  2.3934e-02, -1.5410e-01,\n",
      "        -2.5739e-01,  4.1262e-01, -4.6967e-01,  8.7914e-01,  7.2629e-01,\n",
      "         5.3862e-02, -1.1575e+00, -4.7835e-01,  2.0139e-01, -1.0051e+00,\n",
      "         1.1515e-01, -9.6609e-01,  1.2960e-01,  1.8388e-01, -3.0383e-02],\n",
      "       device='cuda:0')\n",
      "Token: 5 .\n",
      "tensor([-0.3398,  0.2094,  0.4635, -0.6479, -0.3838,  0.0380,  0.1713,  0.1598,\n",
      "         0.4662, -0.0192,  0.4148, -0.3435,  0.2687,  0.0446,  0.4213, -0.4103,\n",
      "         0.1546,  0.0222, -0.6465,  0.2526,  0.0431, -0.1945,  0.4652,  0.4565,\n",
      "         0.6859,  0.0913,  0.2188, -0.7035,  0.1679, -0.3508, -0.1263,  0.6638,\n",
      "        -0.2582,  0.0365, -0.1361,  0.4025,  0.1429,  0.3813, -0.1228, -0.4589,\n",
      "        -0.2528, -0.3043, -0.1121, -0.2618, -0.2248, -0.4455,  0.2991, -0.8561,\n",
      "        -0.1450, -0.4909,  0.0083, -0.1749,  0.2752,  1.4401, -0.2124, -2.8435,\n",
      "        -0.2796, -0.4572,  1.6386,  0.7881, -0.5526,  0.6500,  0.0864,  0.3901,\n",
      "         1.0632, -0.3538,  0.4833,  0.3460,  0.8417,  0.0987, -0.2421, -0.2705,\n",
      "         0.0453, -0.4015,  0.1139,  0.0062,  0.0367,  0.0185, -1.0213, -0.2081,\n",
      "         0.6407, -0.0688, -0.5864,  0.3348, -1.1432, -0.1148, -0.2509, -0.4591,\n",
      "        -0.0968, -0.1795, -0.0634, -0.6741, -0.0689,  0.5360, -0.8777,  0.3180,\n",
      "        -0.3924, -0.2339,  0.4730, -0.0288], device='cuda:0')\n",
      "tensor([])\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('The grass is green .')\n",
    "\n",
    "glove_embedding.embed(sentence)\n",
    "\n",
    "for token in sentence:\n",
    "    print(token)\n",
    "    print(token.embedding)\n",
    "print(sentence.embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "GloVe embeddings are Pytorch vectors of dimensionality 100.\n",
    "\n",
    "We choose which pre-trained embeddings we want to load by passing the appropriate id string to the constructor of the WordEmbeddings class. We would use the two-letter language code to init an embedding, so 'en' for English and 'de' for German and so on. By default, this will initialize FastText embeddings trained over Wikipedia. We can also always use FastText embeddings over Web crawls, by instantiating with '-crawl'. The 'de-crawl' option would use embeddings trained over German web crawls.\n",
    "\n",
    "For English, Flair provides a few more options. We can choose between instantiating 'en-glove', 'en-extvec' and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to load German FastText embeddings, instantiate as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "german_embedding = WordEmbeddings('de')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to load German FastText embeddings trained over crawls, we instantiate as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "german_embedding = WordEmbeddings('de-crawl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the models are not locally available, Flair will automatically download them and install them into the local user cache."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is recommended to use the FastText embeddings, or GloVe if we want a smaller model.\n",
    "\n",
    "If we want to use any other embeddings (not listed in the list above), we can load those by calling:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "custom_embedding = WordEmbeddings('path/to/your/custom/embeddings.gensim')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to load custom embeddings, we need to make sure that the custom embeddings are correctly formatted to gensim.\n",
    "\n",
    "We can, for example, convert FastText embeddings to gensim using the following code snippet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "\n",
    "word_vectors = gensim.models.KeyedVectors.load_word2vec_format('/path/to/fasttext/embeddings.txt', binary=False)\n",
    "word_vectors.save('/path/to/converted')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Character Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some embeddings - such as character-features - are not pre-trained but rather trained on the downstream task. Normally this requires an implementation of a hierarchical embedding architecture.\n",
    "\n",
    "With Flair, we don't need to worry about such things. Just choose the appropriate embedding class and the character features will then automatically train during downstream task training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence: \"The grass is green .\"   [− Tokens: 5]]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from flair.embeddings import CharacterEmbeddings\n",
    "\n",
    "embedding = CharacterEmbeddings()\n",
    "\n",
    "sentence = Sentence('The grass is green .')\n",
    "\n",
    "embedding.embed(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sub-Word Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Flair now also includes the byte pair embeddings calulated by @bheinzerling that segment words into subsequences. This can dramatically reduce the model size vis-a-vis using normal word embeddings at nearly the same accuracy. So, if we want to train small models try out the new BytePairEmbeddings class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We initialize with a language code (275 languages supported), a number of 'syllables', and a number of dimensions (one of 50, 100, 200 or 300). The following initializes and uses byte pair embeddings for English:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence: \"The grass is green .\"   [− Tokens: 5]]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from flair.embeddings import BytePairEmbeddings\n",
    "\n",
    "embedding = BytePairEmbeddings('en')\n",
    "\n",
    "sentence = Sentence('The grass is green .')\n",
    "\n",
    "embedding.embed(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Sub-word embeddings](https://nlp.h-its.org/bpemb/) are interesting, since \n",
    "\n",
    "[BPEmb](https://nlp.h-its.org/bpemb/) is a collection of pre-trained subword embeddings in 275 languages, based on Byte-Pair Encoding (BPE) and trained on Wikipedia. Its intended use is as input for neural models in natural language processing.\n",
    "\n",
    "- subwords allow guessing the meaning of unknown / out-of-vocabulary words. E.g., the suffix -shire in Melfordshire indicates a location.\n",
    "- Byte-Pair Encoding gives a subword segmentation that is often good enough, without requiring tokenization or morphological analysis. In this case the BPE segmentation might be something like melf ord shire.\n",
    "- Pre-trained byte-pair embeddings work surprisingly well, while requiring no tokenization and being much smaller than alternatives: an 11 MB BPEmb English model matches the results of the 6 GB FastText model in our evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you are using word embeddings like word2vec or GloVe, you have probably encountered out-of-vocabulary words, i.e., words for which no embedding exists. A makeshift solution is to replace such words with an UNK token and train a generic embedding representing such unknown words.\n",
    "\n",
    "Subword approaches try to solve the unknown word problem differently, by assuming that you can reconstruct a word's meaning from its parts. For example, the suffix -shire lets you guess that Melfordshire is probably a location, or the suffix -osis that Myxomatosis might be a sickness.\n",
    "\n",
    "There are many ways of splitting a word into subwords. A simple method is to split into characters and then learn to transform this character sequence into a vector representation by feeding it to a convolutional neural network (CNN) or a recurrent neural network (RNN), usually a long-short term memory (LSTM). This vector representation can then be used like a word embedding.\n",
    "\n",
    "Another, more linguistically motivated way is a morphological analysis, but this requires tools and training data which might not be available for your language and domain of interest.\n",
    "\n",
    "Enter Byte-Pair Encoding (BPE) [Sennrich et al, 2016], an unsupervised subword segmentation method. BPE starts with a sequence of symbols, for example characters, and iteratively merges the most frequent symbol pair into a new symbol.\n",
    "\n",
    "For example, applying BPE to English might first merge the characters h and e into a new symbol he, then t and h into th, then t and he into the, and so on.\n",
    "\n",
    "Learning these merge operations from a large corpus (e.g. all Wikipedia articles in a given language) often yields reasonable subword segementations. For example, a BPE model trained on English Wikipedia splits melfordshire into mel, ford, and shire."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Stacked Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stacked embeddings are one of the most important concepts of Flair. We can use them to combine different embeddings together, for instance if we want to use both traditional embeddings together with contextual string embeddings (see below). Stacked embeddings allow us to mix and match. We find that a combination of embeddings often gives best results.\n",
    "\n",
    "All we need to do is use the StackedEmbeddings class and instantiate it by passing a list of embeddings that we wish to combine. For instance, lets combine classic GloVe embeddings with character embeddings. This is effectively the architecture proposed in (Lample et al., 2016)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import WordEmbeddings, CharacterEmbeddings\n",
    "\n",
    "glove_embedding = WordEmbeddings('glove')\n",
    "\n",
    "character_embeddings = CharacterEmbeddings()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now instantiate the StackedEmbeddings class and pass it a list containing these two embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import StackedEmbeddings\n",
    "\n",
    "stacked_embeddings = StackedEmbeddings(\n",
    "    embeddings=[glove_embedding, character_embeddings])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use this embedding like all the other embeddings, i.e. call the embed() method over our sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token: 1 The\n",
      "tensor([-3.8194e-02, -2.4487e-01,  7.2812e-01, -3.9961e-01,  8.3172e-02,\n",
      "         4.3953e-02, -3.9141e-01,  3.3440e-01, -5.7545e-01,  8.7459e-02,\n",
      "         2.8787e-01, -6.7310e-02,  3.0906e-01, -2.6384e-01, -1.3231e-01,\n",
      "        -2.0757e-01,  3.3395e-01, -3.3848e-01, -3.1743e-01, -4.8336e-01,\n",
      "         1.4640e-01, -3.7304e-01,  3.4577e-01,  5.2041e-02,  4.4946e-01,\n",
      "        -4.6971e-01,  2.6280e-02, -5.4155e-01, -1.5518e-01, -1.4107e-01,\n",
      "        -3.9722e-02,  2.8277e-01,  1.4393e-01,  2.3464e-01, -3.1021e-01,\n",
      "         8.6173e-02,  2.0397e-01,  5.2624e-01,  1.7164e-01, -8.2378e-02,\n",
      "        -7.1787e-01, -4.1531e-01,  2.0335e-01, -1.2763e-01,  4.1367e-01,\n",
      "         5.5187e-01,  5.7908e-01, -3.3477e-01, -3.6559e-01, -5.4857e-01,\n",
      "        -6.2892e-02,  2.6584e-01,  3.0205e-01,  9.9775e-01, -8.0481e-01,\n",
      "        -3.0243e+00,  1.2540e-02, -3.6942e-01,  2.2167e+00,  7.2201e-01,\n",
      "        -2.4978e-01,  9.2136e-01,  3.4514e-02,  4.6745e-01,  1.1079e+00,\n",
      "        -1.9358e-01, -7.4575e-02,  2.3353e-01, -5.2062e-02, -2.2044e-01,\n",
      "         5.7162e-02, -1.5806e-01, -3.0798e-01, -4.1625e-01,  3.7972e-01,\n",
      "         1.5006e-01, -5.3212e-01, -2.0550e-01, -1.2526e+00,  7.1624e-02,\n",
      "         7.0565e-01,  4.9744e-01, -4.2063e-01,  2.6148e-01, -1.5380e+00,\n",
      "        -3.0223e-01, -7.3438e-02, -2.8312e-01,  3.7104e-01, -2.5217e-01,\n",
      "         1.6215e-02, -1.7099e-02, -3.8984e-01,  8.7424e-01, -7.2569e-01,\n",
      "        -5.1058e-01, -5.2028e-01, -1.4590e-01,  8.2780e-01,  2.7062e-01,\n",
      "        -1.7617e-01, -1.2585e-02, -9.5894e-02,  7.7040e-03,  2.9357e-01,\n",
      "         9.9538e-02, -2.8684e-01,  2.6311e-01, -2.5935e-02, -7.0002e-02,\n",
      "         6.8972e-02,  8.9397e-02, -3.3586e-02, -7.2867e-02, -3.3263e-03,\n",
      "         1.1027e-01, -1.9998e-01, -3.2931e-02, -2.2797e-01,  2.2202e-02,\n",
      "        -1.0233e-01, -2.8771e-01, -9.4372e-02,  1.1899e-01, -4.7857e-02,\n",
      "        -6.6779e-03,  1.7606e-02, -4.0335e-02,  5.7265e-02,  3.8763e-03,\n",
      "         4.8965e-02,  1.0253e-01,  5.2455e-02, -1.1397e-02, -1.1984e-02,\n",
      "         1.1108e-01,  2.4731e-02,  1.6330e-01, -1.0065e-02,  1.1879e-01,\n",
      "        -1.1240e-01,  1.1673e-02,  5.6632e-04, -3.4064e-02, -1.8810e-01,\n",
      "        -3.5050e-02,  1.7466e-01,  4.3025e-02, -1.6146e-04, -5.3217e-03],\n",
      "       grad_fn=<CatBackward>)\n",
      "Token: 2 grass\n",
      "tensor([-0.8135,  0.9404, -0.2405, -0.1350,  0.0557,  0.3363,  0.0802, -0.1015,\n",
      "        -0.5478, -0.3537,  0.0734,  0.2587,  0.1987, -0.1433,  0.2507,  0.4281,\n",
      "         0.1950,  0.5346,  0.7424,  0.0578, -0.3178,  0.9436,  0.8145, -0.0824,\n",
      "         0.6166,  0.7284, -0.3262, -1.3641,  0.1232,  0.5373, -0.5123,  0.0246,\n",
      "         1.0822, -0.2296,  0.6039,  0.5541, -0.9610,  0.4803,  0.0022,  0.5591,\n",
      "        -0.1637, -0.8468,  0.0741, -0.6216,  0.0260, -0.5162, -0.0525, -0.1418,\n",
      "        -0.0161, -0.4972, -0.5534, -0.4037,  0.5096,  1.0276, -0.0840, -1.1179,\n",
      "         0.3226,  0.4928,  0.9488,  0.2040,  0.5388,  0.8397, -0.0689,  0.3136,\n",
      "         1.0450, -0.2267, -0.0896, -0.6427,  0.6443, -1.1001, -0.0096,  0.2668,\n",
      "        -0.3230, -0.6065,  0.0479, -0.1664,  0.8571,  0.2335,  0.2539,  1.2546,\n",
      "         0.5472, -0.1980, -0.7186,  0.2076, -0.2587, -0.3650,  0.0834,  0.6932,\n",
      "         0.1574,  1.0931,  0.0913, -1.3773, -0.2717,  0.7071,  0.1872, -0.3307,\n",
      "        -0.2836,  0.1030,  1.2228,  0.8374, -0.0208, -0.2314,  0.0963,  0.1327,\n",
      "         0.3344,  0.2486,  0.0979,  0.1403,  0.2769,  0.0422, -0.1331,  0.0295,\n",
      "         0.2725, -0.0674,  0.5953, -0.1043, -0.1866, -0.0059,  0.0936,  0.4954,\n",
      "         0.0865, -0.2322,  0.4411, -0.2761, -0.0693,  0.0451,  0.0936, -0.0368,\n",
      "        -0.3555,  0.3346, -0.0378, -0.0294, -0.0820, -0.0938,  0.0535, -0.1861,\n",
      "         0.0722, -0.1207, -0.0187,  0.0612, -0.1020,  0.0145,  0.0715,  0.0698,\n",
      "         0.2575, -0.0369,  0.1113, -0.0123, -0.0997, -0.2311],\n",
      "       grad_fn=<CatBackward>)\n",
      "Token: 3 is\n",
      "tensor([-0.5426,  0.4148,  1.0322, -0.4024,  0.4669,  0.2182, -0.0749,  0.4733,\n",
      "         0.0810, -0.2208, -0.1281, -0.1144,  0.5089,  0.1157,  0.0282, -0.3628,\n",
      "         0.4382,  0.0475,  0.2028,  0.4986, -0.1007,  0.1327,  0.1697,  0.1165,\n",
      "         0.3135,  0.2571,  0.0928, -0.5683, -0.5297, -0.0515, -0.6733,  0.9253,\n",
      "         0.2693,  0.2273,  0.6636,  0.2622,  0.1972,  0.2609,  0.1877, -0.3454,\n",
      "        -0.4263,  0.1398,  0.5634, -0.5691,  0.1240, -0.1289,  0.7248, -0.2610,\n",
      "        -0.2631, -0.4360,  0.0789, -0.8415,  0.5160,  1.3997, -0.7646, -3.1453,\n",
      "        -0.2920, -0.3125,  1.5129,  0.5243,  0.2146,  0.4245, -0.0884, -0.1780,\n",
      "         1.1876,  0.1058,  0.7657,  0.2191,  0.3582, -0.1164,  0.0933, -0.6248,\n",
      "        -0.2190,  0.2180,  0.7406, -0.4374,  0.1434,  0.1472, -1.1605, -0.0505,\n",
      "         0.1268, -0.0144, -0.9868, -0.0913, -1.2054, -0.1197,  0.0478, -0.5400,\n",
      "         0.5246, -0.7096, -0.3253, -0.1346, -0.4131,  0.3343, -0.0072,  0.3225,\n",
      "        -0.0442, -1.2969,  0.7622,  0.4635, -0.0208, -0.0273,  0.1333,  0.1252,\n",
      "         0.2564,  0.0649,  0.0407,  0.1673,  0.1888, -0.0461, -0.1066, -0.0421,\n",
      "         0.1248, -0.0764,  0.5163, -0.0668, -0.0824, -0.0140,  0.1371,  0.2761,\n",
      "         0.0126, -0.2985,  0.4317, -0.3402,  0.5021,  0.0451,  0.0936, -0.0368,\n",
      "        -0.3555,  0.3346, -0.0378, -0.0294, -0.0820, -0.0938,  0.0535, -0.1861,\n",
      "         0.0722, -0.1207, -0.0187,  0.0612, -0.1020,  0.0145,  0.0715,  0.0698,\n",
      "         0.2575, -0.0369,  0.1113, -0.0123, -0.0997, -0.2311],\n",
      "       grad_fn=<CatBackward>)\n",
      "Token: 4 green\n",
      "tensor([-6.7907e-01,  3.4908e-01, -2.3984e-01, -9.9652e-01,  7.3782e-01,\n",
      "        -6.5911e-04,  2.8010e-01,  1.7287e-02, -3.6063e-01,  3.6955e-02,\n",
      "        -4.0395e-01,  2.4092e-02,  2.8958e-01,  4.0497e-01,  6.9992e-01,\n",
      "         2.5269e-01,  8.0350e-01,  4.9370e-02,  1.5562e-01, -6.3286e-03,\n",
      "        -2.9414e-01,  1.4728e-01,  1.8977e-01, -5.1791e-01,  3.6986e-01,\n",
      "         7.4582e-01,  8.2689e-02, -7.2601e-01, -4.0939e-01, -9.7822e-02,\n",
      "        -1.4096e-01,  7.1121e-01,  6.1933e-01, -2.5014e-01,  4.2250e-01,\n",
      "         4.8458e-01, -5.1915e-01,  7.7125e-01,  3.6685e-01,  4.9652e-01,\n",
      "        -4.1298e-02, -1.4683e+00,  2.0038e-01,  1.8591e-01,  4.9860e-02,\n",
      "        -1.7523e-01, -3.5528e-01,  9.4153e-01, -1.1898e-01, -5.1903e-01,\n",
      "        -1.1887e-02, -3.9186e-01, -1.7479e-01,  9.3451e-01, -5.8931e-01,\n",
      "        -2.7701e+00,  3.4522e-01,  8.6533e-01,  1.0808e+00, -1.0291e-01,\n",
      "        -9.1220e-02,  5.5092e-01, -3.9473e-01,  5.3676e-01,  1.0383e+00,\n",
      "        -4.0658e-01,  2.4590e-01, -2.6797e-01, -2.6036e-01, -1.4151e-01,\n",
      "        -1.2022e-01,  1.6234e-01, -7.4320e-01, -6.4728e-01,  4.7133e-02,\n",
      "         5.1642e-01,  1.9898e-01,  2.3919e-01,  1.2550e-01,  2.2471e-01,\n",
      "         8.2613e-01,  7.8328e-02, -5.7020e-01,  2.3934e-02, -1.5410e-01,\n",
      "        -2.5739e-01,  4.1262e-01, -4.6967e-01,  8.7914e-01,  7.2629e-01,\n",
      "         5.3862e-02, -1.1575e+00, -4.7835e-01,  2.0139e-01, -1.0051e+00,\n",
      "         1.1515e-01, -9.6609e-01,  1.2960e-01,  1.8388e-01, -3.0383e-02,\n",
      "         8.6600e-02,  1.7025e-01, -7.9673e-02,  4.2197e-02,  1.5061e-01,\n",
      "         9.9412e-02, -2.0540e-01,  2.7523e-01,  1.1016e-02, -4.6306e-02,\n",
      "        -8.6975e-02, -5.7998e-02, -5.6193e-02, -7.4325e-02, -4.1259e-02,\n",
      "        -1.9225e-02, -1.6871e-01, -6.4165e-02, -2.1887e-01,  1.5801e-01,\n",
      "        -7.8998e-02, -2.7801e-01,  9.3853e-02,  1.0397e-01, -1.8899e-02,\n",
      "        -7.1831e-02, -1.7894e-02,  3.5567e-02, -6.6185e-02,  9.5183e-02,\n",
      "        -5.0360e-02,  1.1167e-01, -4.9596e-02,  5.6578e-02,  8.6343e-02,\n",
      "        -6.8124e-02,  2.5113e-01,  3.0578e-01, -1.0710e-01,  9.7605e-02,\n",
      "        -1.0524e-01, -2.2156e-03,  4.0687e-03, -1.2599e-01, -1.5051e-01,\n",
      "        -3.9721e-02,  6.2582e-02,  3.2761e-02, -1.5059e-01, -2.2314e-01],\n",
      "       grad_fn=<CatBackward>)\n",
      "Token: 5 .\n",
      "tensor([-0.3398,  0.2094,  0.4635, -0.6479, -0.3838,  0.0380,  0.1713,  0.1598,\n",
      "         0.4662, -0.0192,  0.4148, -0.3435,  0.2687,  0.0446,  0.4213, -0.4103,\n",
      "         0.1546,  0.0222, -0.6465,  0.2526,  0.0431, -0.1945,  0.4652,  0.4565,\n",
      "         0.6859,  0.0913,  0.2188, -0.7035,  0.1679, -0.3508, -0.1263,  0.6638,\n",
      "        -0.2582,  0.0365, -0.1361,  0.4025,  0.1429,  0.3813, -0.1228, -0.4589,\n",
      "        -0.2528, -0.3043, -0.1121, -0.2618, -0.2248, -0.4455,  0.2991, -0.8561,\n",
      "        -0.1450, -0.4909,  0.0083, -0.1749,  0.2752,  1.4401, -0.2124, -2.8435,\n",
      "        -0.2796, -0.4572,  1.6386,  0.7881, -0.5526,  0.6500,  0.0864,  0.3901,\n",
      "         1.0632, -0.3538,  0.4833,  0.3460,  0.8417,  0.0987, -0.2421, -0.2705,\n",
      "         0.0453, -0.4015,  0.1139,  0.0062,  0.0367,  0.0185, -1.0213, -0.2081,\n",
      "         0.6407, -0.0688, -0.5864,  0.3348, -1.1432, -0.1148, -0.2509, -0.4591,\n",
      "        -0.0968, -0.1795, -0.0634, -0.6741, -0.0689,  0.5360, -0.8777,  0.3180,\n",
      "        -0.3924, -0.2339,  0.4730, -0.0288, -0.1724,  0.1918,  0.1550, -0.0513,\n",
      "        -0.1722, -0.0642, -0.0541,  0.0492,  0.3848, -0.1525,  0.0064, -0.0609,\n",
      "        -0.1195,  0.0601,  0.0055,  0.1832,  0.0856, -0.1034, -0.0172, -0.0748,\n",
      "         0.0379, -0.1593,  0.0460, -0.0978,  0.1496,  0.0253,  0.0899, -0.0954,\n",
      "         0.0100,  0.0279, -0.1318, -0.1254, -0.0051, -0.0430, -0.1063,  0.1715,\n",
      "        -0.0401, -0.0060, -0.1903,  0.1459, -0.1412, -0.1238, -0.0328, -0.0746,\n",
      "         0.1660, -0.2634,  0.0990,  0.0707,  0.0767,  0.0440],\n",
      "       grad_fn=<CatBackward>)\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('The grass is green .')\n",
    "\n",
    "stacked_embeddings.embed(sentence)\n",
    "\n",
    "for token in sentence:\n",
    "    print(token)\n",
    "    print(token.embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words are now embedded using a concatenation of two different embeddings. This means that the resulting embedding vector is still a single PyTorch vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other Embeddings: BERT, ELMo, Flair"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next to standard WordEmbeddings and CharacterEmbeddings, Flair also provides classes for BERT, ELMo and Flair embeddings. These embeddings enable us to train truly state-of-the-art NLP models.\n",
    "\n",
    "All word embedding classes inherit from the TokenEmbeddings class and implement the embed() method which we need to call to embed our text. This means that for most users of Flair, the complexity of different embeddings remains hidden behind this interface. We instantiate the embedding class we require and call embed() to embed our text.\n",
    "\n",
    "All embeddings produced with Flair's methods are Pytorch vectors, so they can be immediately used for training and fine-tuning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flair Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Contextual string embeddings are powerful embeddings that capture latent syntactic-semantic information that goes beyond standard word embeddings. Key differences are: (1) they are trained without any explicit notion of words and thus fundamentally model words as sequences of characters. And (2) they are contextualized by their surrounding text, meaning that the same word will have different embeddings depending on its contextual use.\n",
    "\n",
    "Recent advances in language modeling using recurrent neural networks have made it viable to model language as distributions over characters. By learning to predict the next character on the basis of previous characters, such models have been shown to automatically internalize linguistic concepts such as words, sentences, subclauses and even sentiment. In Flair the internal states of a trained character language model is leveraged to produce a novel type of word embedding which the authors refer to as contextual string embeddings. The proposed embeddings have the distinct properties that they (a) are trained without any explicit notion of words and thus fundamentally model words as sequences of characters, and (b) are contextualized by their surrounding text, meaning that the same word will have different embeddings depending on its contextual use. The authors conduct a comparative evaluation against previous embeddings and find that their embeddings are highly useful for downstream tasks: across four classic sequence labeling tasks they consistently outperform the previous state-of-the-art. In particular, they significantly outperform previous work on English and German named entity recognition (NER), allowing them to report new state-of-the-art F1-scores on the CONLL03 shared task.\n",
    "\n",
    "With Flair, we can use these embeddings simply by instantiating the appropriate embedding class, same as standard word embeddings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence: \"The grass is green .\"   [− Tokens: 5]]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from flair.embeddings import FlairEmbeddings\n",
    "\n",
    "flair_embedding_forward = FlairEmbeddings('news-forward')\n",
    "\n",
    "sentence = Sentence('The grass is green .')\n",
    "\n",
    "flair_embedding_forward.embed(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can choose which embeddings we load by passing the appropriate string to the constructor of the FlairEmbeddings class. Currently, there are numerous contextual string embeddings provided in models (more coming). See [list](https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_4_ELMO_BERT_FLAIR_EMBEDDING.md)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommendation is to combine both forward and backward Flair embeddings. Depending on the task, it is also recommended to add standard word embeddings into the mix. So, the recommendation is to use StackedEmbedding for most English tasks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import WordEmbeddings, FlairEmbeddings, StackedEmbeddings\n",
    "\n",
    "stacked_embeddings = StackedEmbeddings([\n",
    "                                        WordEmbeddings('glove'), \n",
    "                                        FlairEmbeddings('news-forward'), \n",
    "                                        FlairEmbeddings('news-backward'),\n",
    "                                       ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would use this embedding like all the other embeddings, i.e. call the embed() method over our sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BERT Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BERT embeddings were developed by Devlin et al. (2018) and are a different kind of powerful word embedding based on a bidirectional transformer architecture. Flair is using the implementation of huggingface. The embeddings are wrapped into our simple embedding interface, so that they can be used like any other embedding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Sentence: \"The grass is green .\"   [− Tokens: 5]]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from flair.embeddings import TransformerWordEmbeddings\n",
    "\n",
    "embedding = TransformerWordEmbeddings()\n",
    "\n",
    "sentence = Sentence('The grass is green .')\n",
    "\n",
    "embedding.embed(sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token: 1 The tensor([-0.0323, -0.3904, -1.1946,  ...,  0.1305, -0.1365, -0.4323],\n",
      "       device='cuda:0')\n",
      "Token: 2 grass tensor([-0.3973,  0.2652, -0.1337,  ...,  0.3715,  0.1097, -1.1625],\n",
      "       device='cuda:0')\n",
      "Token: 3 is tensor([ 0.1374, -0.3688, -0.8292,  ...,  0.2533,  0.0294,  0.4293],\n",
      "       device='cuda:0')\n",
      "Token: 4 green tensor([-0.7722, -0.1152,  0.3661,  ...,  0.1575, -0.0682, -0.7661],\n",
      "       device='cuda:0')\n",
      "Token: 5 . tensor([ 0.1441, -0.1772, -0.5911,  ..., -1.4830,  0.1995, -0.0112],\n",
      "       device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "for i in sentence:\n",
    "    print(i, i.embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can load any of the pre-trained BERT models by providing the model string during initialization:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 'bert-base-uncased': English; 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "- 'bert-large-uncased': English; 24-layer, 1024-hidden, 16-heads, 340M parameters\n",
    "- 'bert-base-cased': English; 12-layer, 768-hidden, 12-heads , 110M parameters\n",
    "- 'bert-large-cased': English; 24-layer, 1024-hidden, 16-heads, 340M parameters\n",
    "- 'bert-base-multilingual-cased': 104 languages; 12-layer, 768-hidden, 12-heads, 110M parameters\n",
    "- 'bert-base-chinese': Chinese Simplified and Traditional; 12-layer, 768-hidden, 12-heads, 110M parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ELMo Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ELMo embeddings were presented by Peters et al. in 2018. They are using a bidirectional recurrent neural network to predict the next word in a text. Flair is using the implementation of AllenNLP. As this implementation comes with a lot of sub-dependencies, which Flair authors don't want to include in Flair, you need to first install the library via `pip install allennlp` before we can use it in Flair. Using the embeddings is as simple as using any other embedding type. To specify the correct model, pick one of *small*, *medium*, *portugese*, *original*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import ELMoEmbeddings\n",
    "\n",
    "embedding = ELMoEmbeddings(model=\"small\")\n",
    "\n",
    "sentence = Sentence('The grass is green .')\n",
    "\n",
    "embedding.embed(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "AllenNLP provides the following pre-trained models. To use any of the following models inside Flair simple specify the embedding id when initializing the ELMoEmbeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 'small': English; 1024-hidden, 1 layer, 14.6M parameters\n",
    "- 'medium': English; 2048-hidden, 1 layer, 28.0M parameters\n",
    "- 'original': English; 4096-hidden, 2 layers, 93.6M parameters\n",
    "- 'pt': Portuguese\n",
    "- 'pubmed': English biomedical data; more information"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BERT and Flair Combined"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can very easily mix and match Flair, ELMo, BERT and classic word embeddings. We instantiate each embedding we wish to combine and use them in a StackedEmbedding.\n",
    "\n",
    "For instance, let's say we want to combine the multilingual Flair and BERT embeddings to train a hyper-powerful multilingual downstream task model.\n",
    "\n",
    "First, instantiate the embeddings we wish to combine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-46-df95a10b584c>:6: DeprecationWarning: Call to deprecated method __init__. (Use 'TransformerWordEmbeddings' for all transformer-based word embeddings) -- Deprecated since version 0.4.5.\n",
      "  bert_embedding = BertEmbeddings('bert-base-multilingual-cased')\n"
     ]
    }
   ],
   "source": [
    "from flair.embeddings import FlairEmbeddings, BertEmbeddings\n",
    "\n",
    "flair_forward_embedding = FlairEmbeddings('multi-forward')\n",
    "flair_backward_embedding = FlairEmbeddings('multi-backward')\n",
    "\n",
    "bert_embedding = BertEmbeddings('bert-base-multilingual-cased')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we instantiate the StackedEmbeddings class and pass it a list containing these three embeddings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "from flair.embeddings import StackedEmbeddings\n",
    "\n",
    "stacked_embeddings = StackedEmbeddings(\n",
    "    embeddings=[flair_forward_embedding, flair_backward_embedding, bert_embedding])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use this embedding like all the other embeddings, i.e. call the embed() method over our sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Token: 1 The\n",
      "tensor([0.6800, 0.2429, 0.0012,  ..., 0.3829, 0.4721, 0.2985], device='cuda:0')\n",
      "Token: 2 grass\n",
      "tensor([ 2.9200e-01,  2.2066e-02,  4.5290e-05,  ...,  8.5283e-01,\n",
      "        -5.0724e-02,  3.4476e-01], device='cuda:0')\n",
      "Token: 3 is\n",
      "tensor([-0.5447,  0.0229,  0.0078,  ..., -0.1828,  0.7153,  0.0051],\n",
      "       device='cuda:0')\n",
      "Token: 4 green\n",
      "tensor([1.4772e-01, 1.0973e-01, 8.5618e-04,  ..., 1.0157e+00, 7.5358e-01,\n",
      "        1.1230e-01], device='cuda:0')\n",
      "Token: 5 .\n",
      "tensor([-1.5555e-01,  6.7598e-03,  5.3829e-06,  ..., -6.0930e-01,\n",
      "         9.0591e-01,  1.7857e-01], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "sentence = Sentence('The grass is green .')\n",
    "\n",
    "stacked_embeddings.embed(sentence)\n",
    "\n",
    "for token in sentence:\n",
    "    print(token)\n",
    "    print(token.embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words are now embedded using a concatenation of three different embeddings. This means that the resulting embedding vector is still a single PyTorch vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Document Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Document embeddings are different from word embeddings in that they provide one embedding for an entire text, whereas word embeddings provide embeddings for individual words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2018-2024 by [Damir Cavar](https://damir.cavar.me/)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}