examples/Tutorial 02 - Using Tailor default perturb. func with NLI.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c338ee30",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "is_cuda = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2da417ba",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "This notebook is a more advanced demo on how to create perturbations for the Natural Language Inference task, using [SNLI dataset](https://www.google.com/search?hl=zh-CN&q=huggingface+snli). It describes in detail the running steps in Tailor (i.e., explains the implementation logic to some extent). If this does not look like the level of detail you are looking for, we recommend checking out the Basic Demo notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e9e16d8",
   "metadata": {},
   "source": [
    "## Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "5d47e614",
   "metadata": {},
   "outputs": [],
   "source": [
    "import datasets\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "07b24d32",
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_as_dataframe(list_of_namedtuples, keys=None):\n",
    "    if isinstance(list_of_namedtuples[0], list):\n",
    "        if not keys:\n",
    "            keys = range(len(list_of_namedtuples))\n",
    "        return pd.concat([pd.DataFrame(tup_list) for tup_list in list_of_namedtuples], keys=keys)\n",
    "    return pd.DataFrame(list_of_namedtuples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "981bbb7c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset snli (/home/alexisr/.cache/huggingface/datasets/snli/plain_text/1.0.0/1f60b67533b65ae0275561ff7828aad5ee4282d0e6f844fd148d05d3c6ea251b)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d2639d474af047668cf1aea7f2c349fb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load the data\n",
    "snli = datasets.load_dataset(\"snli\")\n",
    "# since SNLI has sentence pairs of (premise, hypothesis), \n",
    "# here we denote the perturbation target as the hypothesis sentence.\n",
    "key_to_perturb = \"hypothesis\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "025cbecd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['A man in a restaurant is waiting for his meal to arrive.',\n",
       " 'A blond man getting a drink of water from a fountain in the park.',\n",
       " 'A blond man wearing a brown shirt is reading a book on a bench in the park',\n",
       " 'A blond man drinking water from a fountain.',\n",
       " 'The friends scowl at each other over a full dinner table.',\n",
       " 'There are two woman in this picture.',\n",
       " 'The friends have just met for the first time in 20 years, and have had a great time catching up.',\n",
       " 'The two sisters saw each other across the crowded diner and shared a hug, both clutching their doggie bags.',\n",
       " 'Two groups of rival gang members flipped each other off.',\n",
       " 'Two women hug each other.',\n",
       " 'A team is trying to score the games winning out.',\n",
       " 'A team is trying to tag a runner out.',\n",
       " 'A team is playing baseball on Saturn.',\n",
       " 'A school hosts a basketball game.',\n",
       " 'A high school is hosting an event.',\n",
       " 'A school is hosting an event.',\n",
       " 'The women do not care what clothes they wear.',\n",
       " 'Women are waiting by a tram.',\n",
       " 'The women enjoy having a good fashion sense.',\n",
       " 'A child with mom and dad, on summer vacation at the beach.',\n",
       " 'A family of three is at the beach.',\n",
       " 'A family of three is at the mall shopping.',\n",
       " 'The people waiting on the train are sitting.',\n",
       " 'There are people just getting on a train',\n",
       " 'There are people waiting on a train.',\n",
       " 'A couple are playing with a young child outside.',\n",
       " 'A couple are playing frisbee with a young child at the beach.',\n",
       " 'A couple watch a little girl play by herself on the beach.',\n",
       " 'The family is sitting down for dinner.',\n",
       " 'The family is outside.',\n",
       " 'The family is on vacation.',\n",
       " 'The people are standing still on the curb.',\n",
       " 'Near a couple of restaurants, two people walk across the street.',\n",
       " 'The couple are walking across the street together.',\n",
       " 'The woman is nake.',\n",
       " 'The woman is cold.',\n",
       " 'The woman is wearing green.',\n",
       " 'The man with the sign is caucasian.',\n",
       " 'They are protesting outside the capital.',\n",
       " 'A woman in white.',\n",
       " 'A man is advertising for a restaurant.',\n",
       " 'The woman is wearing black.',\n",
       " 'A man and a woman walk down a crowded city street.',\n",
       " 'The woman is wearing white.',\n",
       " \"They are working for John's Pizza.\",\n",
       " 'Olympic swimming.',\n",
       " \"A man and a soman are eating together at John's Pizza and Gyro.\",\n",
       " 'They are walking with a sign.',\n",
       " 'The woman is waiting for a friend.',\n",
       " \"The man is sitting down while he has a sign for John's Pizza and Gyro in his arms.\"]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# take a look as some randome sentences in the training dataset\n",
    "data_dict = snli[\"train\"][20:70]\n",
    "sentences = data_dict[key_to_perturb]\n",
    "sentences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c50a9d7a",
   "metadata": {},
   "source": [
    "## Preprocess the data\n",
    "\n",
    "Tailor handles semantic perturbation by taking advantage of sentences' semantic roles. Thus, we need to do some preprocessing to augment the texts with semantic features. We provide:\n",
    "- built-in SpaCy processor, and \n",
    "- SRL tagger ([default model from AllenNLP](https://demo.allennlp.org/semantic-role-labeling))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "58a5303b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[A man in a restaurant is waiting for his meal to arrive.,\n",
       " A blond man getting a drink of water from a fountain in the park.,\n",
       " A blond man wearing a brown shirt is reading a book on a bench in the park,\n",
       " A blond man drinking water from a fountain.,\n",
       " The friends scowl at each other over a full dinner table.,\n",
       " There are two woman in this picture.,\n",
       " The friends have just met for the first time in 20 years, and have had a great time catching up.,\n",
       " The two sisters saw each other across the crowded diner and shared a hug, both clutching their doggie bags.,\n",
       " Two groups of rival gang members flipped each other off.,\n",
       " Two women hug each other.,\n",
       " A team is trying to score the games winning out.,\n",
       " A team is trying to tag a runner out.,\n",
       " A team is playing baseball on Saturn.,\n",
       " A school hosts a basketball game.,\n",
       " A high school is hosting an event.,\n",
       " A school is hosting an event.,\n",
       " The women do not care what clothes they wear.,\n",
       " Women are waiting by a tram.,\n",
       " The women enjoy having a good fashion sense.,\n",
       " A child with mom and dad, on summer vacation at the beach.,\n",
       " A family of three is at the beach.,\n",
       " A family of three is at the mall shopping.,\n",
       " The people waiting on the train are sitting.,\n",
       " There are people just getting on a train,\n",
       " There are people waiting on a train.,\n",
       " A couple are playing with a young child outside.,\n",
       " A couple are playing frisbee with a young child at the beach.,\n",
       " A couple watch a little girl play by herself on the beach.,\n",
       " The family is sitting down for dinner.,\n",
       " The family is outside.,\n",
       " The family is on vacation.,\n",
       " The people are standing still on the curb.,\n",
       " Near a couple of restaurants, two people walk across the street.,\n",
       " The couple are walking across the street together.,\n",
       " The woman is nake.,\n",
       " The woman is cold.,\n",
       " The woman is wearing green.,\n",
       " The man with the sign is caucasian.,\n",
       " They are protesting outside the capital.,\n",
       " A woman in white.,\n",
       " A man is advertising for a restaurant.,\n",
       " The woman is wearing black.,\n",
       " A man and a woman walk down a crowded city street.,\n",
       " The woman is wearing white.,\n",
       " They are working for John's Pizza.,\n",
       " Olympic swimming.,\n",
       " A man and a soman are eating together at John's Pizza and Gyro.,\n",
       " They are walking with a sign.,\n",
       " The woman is waiting for a friend.,\n",
       " The man is sitting down while he has a sign for John's Pizza and Gyro in his arms.]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from tailor.steps.process_with_spacy import GetSpacyModel, ProcessWithSpacy\n",
    "\n",
    "# there are also two additional default parameters; Can change them if you need.\n",
    "# spacy_model_name=\"en_core_web_sm\"\n",
    "# use_white_space_tokenizer=False\n",
    "spacy_model = GetSpacyModel().run(parse=True)\n",
    "spacy_outputs = ProcessWithSpacy().run(sentences=sentences, spacy_model=spacy_model)\n",
    "spacy_outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b9f5e220",
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "logging.getLogger(\"allennlp\").setLevel(\"ERROR\")\n",
    "\n",
    "from tailor.steps.get_srl_tags import GetSRLTags\n",
    "# the spacy-processed docs can be passed for getting the SRL labels\n",
    "processed_sentences = GetSRLTags().run(spacy_outputs=spacy_outputs)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57a8be3",
   "metadata": {},
   "source": [
    "Take a look at one result. The result has the following structure:\n",
    "\n",
    "```\n",
    "- sentence [str]: The original text.\n",
    "- spacy_doc [Spacy Doc]: The processed SpaCy doc.\n",
    "- verbs: A list of the predicates and their corresponding arguments\n",
    "    {\n",
    "        verb [str]: the verb\n",
    "            'training'\n",
    "        description: [str]: sentence with augmented semtnaic roles.\n",
    "            '[ARG0: A person] is [V: training] [ARG2: his horse] [ARG1: for a competition] .'\n",
    "        tags [List of str]:\n",
    "            Semantic roles in BIO format.\n",
    "    }[]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6c32193f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ProcessedSentence(sentence='A man in a restaurant is waiting for his meal to arrive .', spacy_doc=A man in a restaurant is waiting for his meal to arrive., verbs=[{'verb': 'is', 'description': 'A man in a restaurant [V: is] waiting for his meal to arrive .', 'tags': ['O', 'O', 'O', 'O', 'O', 'B-V', 'O', 'O', 'O', 'O', 'O', 'O', 'O']}, {'verb': 'waiting', 'description': '[ARG1: A man in a restaurant] is [V: waiting] [ARG2: for his meal to arrive] .', 'tags': ['B-ARG1', 'I-ARG1', 'I-ARG1', 'I-ARG1', 'I-ARG1', 'O', 'B-V', 'B-ARG2', 'I-ARG2', 'I-ARG2', 'I-ARG2', 'I-ARG2', 'O']}, {'verb': 'arrive', 'description': 'A man in a restaurant is waiting for [ARG1: his meal] to [V: arrive] .', 'tags': ['O', 'O', 'O', 'O', 'O', 'O', 'O', 'O', 'B-ARG1', 'I-ARG1', 'O', 'B-V', 'O']}])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "processed_sentences[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "ccc28560",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>spacy_doc</th>\n",
       "      <th>verbs</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A man in a restaurant is waiting for his meal ...</td>\n",
       "      <td>(A, man, in, a, restaurant, is, waiting, for, ...</td>\n",
       "      <td>[{'verb': 'is', 'description': 'A man in a res...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A blond man getting a drink of water from a fo...</td>\n",
       "      <td>(A, blond, man, getting, a, drink, of, water, ...</td>\n",
       "      <td>[{'verb': 'getting', 'description': '[ARG0: A ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A blond man wearing a brown shirt is reading a...</td>\n",
       "      <td>(A, blond, man, wearing, a, brown, shirt, is, ...</td>\n",
       "      <td>[{'verb': 'wearing', 'description': '[ARG0: A ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A blond man drinking water from a fountain .</td>\n",
       "      <td>(A, blond, man, drinking, water, from, a, foun...</td>\n",
       "      <td>[{'verb': 'drinking', 'description': '[ARG0: A...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>The friends scowl at each other over a full di...</td>\n",
       "      <td>(The, friends, scowl, at, each, other, over, a...</td>\n",
       "      <td>[{'verb': 'scowl', 'description': '[ARG0: The ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>There are two woman in this picture .</td>\n",
       "      <td>(There, are, two, woman, in, this, picture, .)</td>\n",
       "      <td>[{'verb': 'are', 'description': 'There [V: are...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>The friends have just met for the first time i...</td>\n",
       "      <td>(The, friends, have, just, met, for, the, firs...</td>\n",
       "      <td>[{'verb': 'have', 'description': 'The friends ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>The two sisters saw each other across the crow...</td>\n",
       "      <td>(The, two, sisters, saw, each, other, across, ...</td>\n",
       "      <td>[{'verb': 'saw', 'description': '[ARG0: The tw...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Two groups of rival gang members flipped each ...</td>\n",
       "      <td>(Two, groups, of, rival, gang, members, flippe...</td>\n",
       "      <td>[{'verb': 'flipped', 'description': '[ARG0: Tw...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Two women hug each other .</td>\n",
       "      <td>(Two, women, hug, each, other, .)</td>\n",
       "      <td>[{'verb': 'hug', 'description': '[ARG0: Two wo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>A team is trying to score the games winning out .</td>\n",
       "      <td>(A, team, is, trying, to, score, the, games, w...</td>\n",
       "      <td>[{'verb': 'is', 'description': 'A team [V: is]...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>A team is trying to tag a runner out .</td>\n",
       "      <td>(A, team, is, trying, to, tag, a, runner, out, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'A team [V: is]...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>A team is playing baseball on Saturn .</td>\n",
       "      <td>(A, team, is, playing, baseball, on, Saturn, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'A team [V: is]...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>A school hosts a basketball game .</td>\n",
       "      <td>(A, school, hosts, a, basketball, game, .)</td>\n",
       "      <td>[{'verb': 'hosts', 'description': '[ARG0: A sc...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>A high school is hosting an event .</td>\n",
       "      <td>(A, high, school, is, hosting, an, event, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'A high school ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>A school is hosting an event .</td>\n",
       "      <td>(A, school, is, hosting, an, event, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'A school [V: i...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>The women do not care what clothes they wear .</td>\n",
       "      <td>(The, women, do, not, care, what, clothes, the...</td>\n",
       "      <td>[{'verb': 'do', 'description': 'The women [V: ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Women are waiting by a tram .</td>\n",
       "      <td>(Women, are, waiting, by, a, tram, .)</td>\n",
       "      <td>[{'verb': 'are', 'description': 'Women [V: are...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>The women enjoy having a good fashion sense .</td>\n",
       "      <td>(The, women, enjoy, having, a, good, fashion, ...</td>\n",
       "      <td>[{'verb': 'enjoy', 'description': '[ARG0: The ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>A child with mom and dad , on summer vacation ...</td>\n",
       "      <td>(A, child, with, mom, and, dad, ,, on, summer,...</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>A family of three is at the beach .</td>\n",
       "      <td>(A, family, of, three, is, at, the, beach, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': '[ARG1: A famil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>A family of three is at the mall shopping .</td>\n",
       "      <td>(A, family, of, three, is, at, the, mall, shop...</td>\n",
       "      <td>[{'verb': 'is', 'description': '[ARG1: A famil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>The people waiting on the train are sitting .</td>\n",
       "      <td>(The, people, waiting, on, the, train, are, si...</td>\n",
       "      <td>[{'verb': 'waiting', 'description': '[ARG1: Th...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>There are people just getting on a train</td>\n",
       "      <td>(There, are, people, just, getting, on, a, train)</td>\n",
       "      <td>[{'verb': 'are', 'description': 'There [V: are...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>There are people waiting on a train .</td>\n",
       "      <td>(There, are, people, waiting, on, a, train, .)</td>\n",
       "      <td>[{'verb': 'are', 'description': 'There [V: are...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>A couple are playing with a young child outside .</td>\n",
       "      <td>(A, couple, are, playing, with, a, young, chil...</td>\n",
       "      <td>[{'verb': 'are', 'description': 'A couple [V: ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>A couple are playing frisbee with a young chil...</td>\n",
       "      <td>(A, couple, are, playing, frisbee, with, a, yo...</td>\n",
       "      <td>[{'verb': 'are', 'description': 'A couple [V: ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>A couple watch a little girl play by herself o...</td>\n",
       "      <td>(A, couple, watch, a, little, girl, play, by, ...</td>\n",
       "      <td>[{'verb': 'play', 'description': 'A couple wat...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>The family is sitting down for dinner .</td>\n",
       "      <td>(The, family, is, sitting, down, for, dinner, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'The family [V:...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>The family is outside .</td>\n",
       "      <td>(The, family, is, outside, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': '[ARG1: The fam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>The family is on vacation .</td>\n",
       "      <td>(The, family, is, on, vacation, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': '[ARG1: The fam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>The people are standing still on the curb .</td>\n",
       "      <td>(The, people, are, standing, still, on, the, c...</td>\n",
       "      <td>[{'verb': 'are', 'description': 'The people [V...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>Near a couple of restaurants , two people walk...</td>\n",
       "      <td>(Near, a, couple, of, restaurants, ,, two, peo...</td>\n",
       "      <td>[{'verb': 'walk', 'description': '[ARGM-LOC: N...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>The couple are walking across the street toget...</td>\n",
       "      <td>(The, couple, are, walking, across, the, stree...</td>\n",
       "      <td>[{'verb': 'are', 'description': 'The couple [V...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>The woman is nake .</td>\n",
       "      <td>(The, woman, is, nake, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': '[ARG1: The wom...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>The woman is cold .</td>\n",
       "      <td>(The, woman, is, cold, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': '[ARG1: The wom...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>The woman is wearing green .</td>\n",
       "      <td>(The, woman, is, wearing, green, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'The woman [V: ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>The man with the sign is caucasian .</td>\n",
       "      <td>(The, man, with, the, sign, is, caucasian, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': '[ARG1: The man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>They are protesting outside the capital .</td>\n",
       "      <td>(They, are, protesting, outside, the, capital, .)</td>\n",
       "      <td>[{'verb': 'are', 'description': 'They [V: are]...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>A woman in white .</td>\n",
       "      <td>(A, woman, in, white, .)</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>A man is advertising for a restaurant .</td>\n",
       "      <td>(A, man, is, advertising, for, a, restaurant, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'A man [V: is] ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>The woman is wearing black .</td>\n",
       "      <td>(The, woman, is, wearing, black, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'The woman [V: ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>A man and a woman walk down a crowded city str...</td>\n",
       "      <td>(A, man, and, a, woman, walk, down, a, crowded...</td>\n",
       "      <td>[{'verb': 'walk', 'description': '[ARG0: A man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>The woman is wearing white .</td>\n",
       "      <td>(The, woman, is, wearing, white, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'The woman [V: ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>They are working for John 's Pizza .</td>\n",
       "      <td>(They, are, working, for, John, 's, Pizza, .)</td>\n",
       "      <td>[{'verb': 'are', 'description': 'They [V: are]...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>Olympic swimming .</td>\n",
       "      <td>(Olympic, swimming, .)</td>\n",
       "      <td>[]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>A man and a soman are eating together at John ...</td>\n",
       "      <td>(A, man, and, a, soman, are, eating, together,...</td>\n",
       "      <td>[{'verb': 'are', 'description': 'A man and a s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>They are walking with a sign .</td>\n",
       "      <td>(They, are, walking, with, a, sign, .)</td>\n",
       "      <td>[{'verb': 'are', 'description': 'They [V: are]...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>The woman is waiting for a friend .</td>\n",
       "      <td>(The, woman, is, waiting, for, a, friend, .)</td>\n",
       "      <td>[{'verb': 'is', 'description': 'The woman [V: ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>The man is sitting down while he has a sign fo...</td>\n",
       "      <td>(The, man, is, sitting, down, while, he, has, ...</td>\n",
       "      <td>[{'verb': 'is', 'description': 'The man [V: is...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             sentence  \\\n",
       "0   A man in a restaurant is waiting for his meal ...   \n",
       "1   A blond man getting a drink of water from a fo...   \n",
       "2   A blond man wearing a brown shirt is reading a...   \n",
       "3        A blond man drinking water from a fountain .   \n",
       "4   The friends scowl at each other over a full di...   \n",
       "5               There are two woman in this picture .   \n",
       "6   The friends have just met for the first time i...   \n",
       "7   The two sisters saw each other across the crow...   \n",
       "8   Two groups of rival gang members flipped each ...   \n",
       "9                          Two women hug each other .   \n",
       "10  A team is trying to score the games winning out .   \n",
       "11             A team is trying to tag a runner out .   \n",
       "12             A team is playing baseball on Saturn .   \n",
       "13                 A school hosts a basketball game .   \n",
       "14                A high school is hosting an event .   \n",
       "15                     A school is hosting an event .   \n",
       "16     The women do not care what clothes they wear .   \n",
       "17                      Women are waiting by a tram .   \n",
       "18      The women enjoy having a good fashion sense .   \n",
       "19  A child with mom and dad , on summer vacation ...   \n",
       "20                A family of three is at the beach .   \n",
       "21        A family of three is at the mall shopping .   \n",
       "22      The people waiting on the train are sitting .   \n",
       "23           There are people just getting on a train   \n",
       "24              There are people waiting on a train .   \n",
       "25  A couple are playing with a young child outside .   \n",
       "26  A couple are playing frisbee with a young chil...   \n",
       "27  A couple watch a little girl play by herself o...   \n",
       "28            The family is sitting down for dinner .   \n",
       "29                            The family is outside .   \n",
       "30                        The family is on vacation .   \n",
       "31        The people are standing still on the curb .   \n",
       "32  Near a couple of restaurants , two people walk...   \n",
       "33  The couple are walking across the street toget...   \n",
       "34                                The woman is nake .   \n",
       "35                                The woman is cold .   \n",
       "36                       The woman is wearing green .   \n",
       "37               The man with the sign is caucasian .   \n",
       "38          They are protesting outside the capital .   \n",
       "39                                 A woman in white .   \n",
       "40            A man is advertising for a restaurant .   \n",
       "41                       The woman is wearing black .   \n",
       "42  A man and a woman walk down a crowded city str...   \n",
       "43                       The woman is wearing white .   \n",
       "44               They are working for John 's Pizza .   \n",
       "45                                 Olympic swimming .   \n",
       "46  A man and a soman are eating together at John ...   \n",
       "47                     They are walking with a sign .   \n",
       "48                The woman is waiting for a friend .   \n",
       "49  The man is sitting down while he has a sign fo...   \n",
       "\n",
       "                                            spacy_doc  \\\n",
       "0   (A, man, in, a, restaurant, is, waiting, for, ...   \n",
       "1   (A, blond, man, getting, a, drink, of, water, ...   \n",
       "2   (A, blond, man, wearing, a, brown, shirt, is, ...   \n",
       "3   (A, blond, man, drinking, water, from, a, foun...   \n",
       "4   (The, friends, scowl, at, each, other, over, a...   \n",
       "5      (There, are, two, woman, in, this, picture, .)   \n",
       "6   (The, friends, have, just, met, for, the, firs...   \n",
       "7   (The, two, sisters, saw, each, other, across, ...   \n",
       "8   (Two, groups, of, rival, gang, members, flippe...   \n",
       "9                   (Two, women, hug, each, other, .)   \n",
       "10  (A, team, is, trying, to, score, the, games, w...   \n",
       "11  (A, team, is, trying, to, tag, a, runner, out, .)   \n",
       "12    (A, team, is, playing, baseball, on, Saturn, .)   \n",
       "13         (A, school, hosts, a, basketball, game, .)   \n",
       "14       (A, high, school, is, hosting, an, event, .)   \n",
       "15             (A, school, is, hosting, an, event, .)   \n",
       "16  (The, women, do, not, care, what, clothes, the...   \n",
       "17              (Women, are, waiting, by, a, tram, .)   \n",
       "18  (The, women, enjoy, having, a, good, fashion, ...   \n",
       "19  (A, child, with, mom, and, dad, ,, on, summer,...   \n",
       "20      (A, family, of, three, is, at, the, beach, .)   \n",
       "21  (A, family, of, three, is, at, the, mall, shop...   \n",
       "22  (The, people, waiting, on, the, train, are, si...   \n",
       "23  (There, are, people, just, getting, on, a, train)   \n",
       "24     (There, are, people, waiting, on, a, train, .)   \n",
       "25  (A, couple, are, playing, with, a, young, chil...   \n",
       "26  (A, couple, are, playing, frisbee, with, a, yo...   \n",
       "27  (A, couple, watch, a, little, girl, play, by, ...   \n",
       "28   (The, family, is, sitting, down, for, dinner, .)   \n",
       "29                      (The, family, is, outside, .)   \n",
       "30                 (The, family, is, on, vacation, .)   \n",
       "31  (The, people, are, standing, still, on, the, c...   \n",
       "32  (Near, a, couple, of, restaurants, ,, two, peo...   \n",
       "33  (The, couple, are, walking, across, the, stree...   \n",
       "34                          (The, woman, is, nake, .)   \n",
       "35                          (The, woman, is, cold, .)   \n",
       "36                (The, woman, is, wearing, green, .)   \n",
       "37      (The, man, with, the, sign, is, caucasian, .)   \n",
       "38  (They, are, protesting, outside, the, capital, .)   \n",
       "39                           (A, woman, in, white, .)   \n",
       "40   (A, man, is, advertising, for, a, restaurant, .)   \n",
       "41                (The, woman, is, wearing, black, .)   \n",
       "42  (A, man, and, a, woman, walk, down, a, crowded...   \n",
       "43                (The, woman, is, wearing, white, .)   \n",
       "44      (They, are, working, for, John, 's, Pizza, .)   \n",
       "45                             (Olympic, swimming, .)   \n",
       "46  (A, man, and, a, soman, are, eating, together,...   \n",
       "47             (They, are, walking, with, a, sign, .)   \n",
       "48       (The, woman, is, waiting, for, a, friend, .)   \n",
       "49  (The, man, is, sitting, down, while, he, has, ...   \n",
       "\n",
       "                                                verbs  \n",
       "0   [{'verb': 'is', 'description': 'A man in a res...  \n",
       "1   [{'verb': 'getting', 'description': '[ARG0: A ...  \n",
       "2   [{'verb': 'wearing', 'description': '[ARG0: A ...  \n",
       "3   [{'verb': 'drinking', 'description': '[ARG0: A...  \n",
       "4   [{'verb': 'scowl', 'description': '[ARG0: The ...  \n",
       "5   [{'verb': 'are', 'description': 'There [V: are...  \n",
       "6   [{'verb': 'have', 'description': 'The friends ...  \n",
       "7   [{'verb': 'saw', 'description': '[ARG0: The tw...  \n",
       "8   [{'verb': 'flipped', 'description': '[ARG0: Tw...  \n",
       "9   [{'verb': 'hug', 'description': '[ARG0: Two wo...  \n",
       "10  [{'verb': 'is', 'description': 'A team [V: is]...  \n",
       "11  [{'verb': 'is', 'description': 'A team [V: is]...  \n",
       "12  [{'verb': 'is', 'description': 'A team [V: is]...  \n",
       "13  [{'verb': 'hosts', 'description': '[ARG0: A sc...  \n",
       "14  [{'verb': 'is', 'description': 'A high school ...  \n",
       "15  [{'verb': 'is', 'description': 'A school [V: i...  \n",
       "16  [{'verb': 'do', 'description': 'The women [V: ...  \n",
       "17  [{'verb': 'are', 'description': 'Women [V: are...  \n",
       "18  [{'verb': 'enjoy', 'description': '[ARG0: The ...  \n",
       "19                                                 []  \n",
       "20  [{'verb': 'is', 'description': '[ARG1: A famil...  \n",
       "21  [{'verb': 'is', 'description': '[ARG1: A famil...  \n",
       "22  [{'verb': 'waiting', 'description': '[ARG1: Th...  \n",
       "23  [{'verb': 'are', 'description': 'There [V: are...  \n",
       "24  [{'verb': 'are', 'description': 'There [V: are...  \n",
       "25  [{'verb': 'are', 'description': 'A couple [V: ...  \n",
       "26  [{'verb': 'are', 'description': 'A couple [V: ...  \n",
       "27  [{'verb': 'play', 'description': 'A couple wat...  \n",
       "28  [{'verb': 'is', 'description': 'The family [V:...  \n",
       "29  [{'verb': 'is', 'description': '[ARG1: The fam...  \n",
       "30  [{'verb': 'is', 'description': '[ARG1: The fam...  \n",
       "31  [{'verb': 'are', 'description': 'The people [V...  \n",
       "32  [{'verb': 'walk', 'description': '[ARGM-LOC: N...  \n",
       "33  [{'verb': 'are', 'description': 'The couple [V...  \n",
       "34  [{'verb': 'is', 'description': '[ARG1: The wom...  \n",
       "35  [{'verb': 'is', 'description': '[ARG1: The wom...  \n",
       "36  [{'verb': 'is', 'description': 'The woman [V: ...  \n",
       "37  [{'verb': 'is', 'description': '[ARG1: The man...  \n",
       "38  [{'verb': 'are', 'description': 'They [V: are]...  \n",
       "39                                                 []  \n",
       "40  [{'verb': 'is', 'description': 'A man [V: is] ...  \n",
       "41  [{'verb': 'is', 'description': 'The woman [V: ...  \n",
       "42  [{'verb': 'walk', 'description': '[ARG0: A man...  \n",
       "43  [{'verb': 'is', 'description': 'The woman [V: ...  \n",
       "44  [{'verb': 'are', 'description': 'They [V: are]...  \n",
       "45                                                 []  \n",
       "46  [{'verb': 'are', 'description': 'A man and a s...  \n",
       "47  [{'verb': 'are', 'description': 'They [V: are]...  \n",
       "48  [{'verb': 'is', 'description': 'The woman [V: ...  \n",
       "49  [{'verb': 'is', 'description': 'The man [V: is...  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# See the result in a more structured formate\n",
    "show_as_dataframe(processed_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f953d0c2",
   "metadata": {},
   "source": [
    "## Tailor perturbations on sentences\n",
    "\n",
    "Now we start to do perturbations on the processed sentences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fb55120d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tailor.common.perturb_function import ChangeVoice, SwapCoreWithContext, SwapCoreWithoutContext, ShortenCoreArgument, ChangeTense\n",
    "from tailor.steps.perturb_prompt import PerturbPromptWithString, PerturbPromptWithFunction\n",
    "from tailor.steps.combine_all_prompts import CombineAllPrompts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b15d0d05",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0%|                                                                                 | 0/50 [00:00<?, ?it/s]No prompts created; unique_tlemmas: set()\n",
      "100%|███████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 102.00it/s]\n",
      "  0%|                                                                                 | 0/50 [00:00<?, ?it/s]Warning. Perturbed prompt is same as base prompt. This should not be happening.\n",
      "100%|███████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 122.79it/s]\n",
      "100%|███████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 112.87it/s]\n",
      "100%|████████████████████████████████████████████████████████████████████████| 50/50 [00:29<00:00,  1.70it/s]\n"
     ]
    }
   ],
   "source": [
    "sentence_prompts = []\n",
    "perturbs = {\n",
    "    ChangeVoice: \"preserves_meaning\",\n",
    "    ShortenCoreArgument: \"preserves_meaning\",\n",
    "    SwapCoreWithContext: \"changes_meaning\",\n",
    "    SwapCoreWithoutContext: \"changes_meaning\",\n",
    "}\n",
    "for perturb_fn in [ChangeVoice, SwapCoreWithContext, SwapCoreWithoutContext, ShortenCoreArgument]:\n",
    "    perturbations = PerturbPromptWithString().run(\n",
    "        # the procssed sentence\n",
    "        processed_sentences=processed_sentences, \n",
    "        # we have pre-implemented some functions you can call\n",
    "        perturb_str_func=perturb_fn(), \n",
    "        # you can add arbitrary descriptions\n",
    "        description=perturbs[perturb_fn])\n",
    "    sentence_prompts.append(perturbations)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ab66cdc7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['change_voice',\n",
       " 'change_tense',\n",
       " 'change_lemma',\n",
       " 'delete_text',\n",
       " 'delete_punctuation',\n",
       " 'swap_core_with_context',\n",
       " 'swap_core_without_context',\n",
       " 'shorten_core_argument']"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the available pre-defined functions: \n",
    "from tailor.common.perturb_function import PerturbStringFunction\n",
    "PerturbStringFunction.list_available()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f226a5b5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# additionally, Tailor allows defining perturbation functions. \n",
    "# We will not go into details, but please feel free to check the implementaiton.\n",
    "\n",
    "from tailor.tasks.nli.perturbations import ReplaceCoreWithSubsequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "2ac9232c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████| 50/50 [00:37<00:00,  1.34it/s]\n"
     ]
    }
   ],
   "source": [
    "replace_core_with_subs = PerturbPromptWithFunction().run(\n",
    "    processed_sentences=processed_sentences, \n",
    "    perturb_fn=ReplaceCoreWithSubsequence(), \n",
    "    description=\"changes_meaning\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0f4999b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_prompts.append(replace_core_with_subs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "0e58b085",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence_prompts = CombineAllPrompts().run(sentence_prompts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "440d051d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>prompt</th>\n",
       "      <th>answer</th>\n",
       "      <th>meta</th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">0</th>\n",
       "      <th>0</th>\n",
       "      <td>[VERB+passive+present: be]  &lt;extra_id_0&gt; &lt;extr...</td>\n",
       "      <td>A man in a restaurant [VERB: is] waiting for h...</td>\n",
       "      <td>{'noncore_args': [], 'core_args': [], 'blank_i...</td>\n",
       "      <td>change_voice</td>\n",
       "      <td>preserves_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[VERB+passive+present: wait | PATIENT+complete...</td>\n",
       "      <td>[PATIENT: A man in a restaurant] is [VERB: wai...</td>\n",
       "      <td>{'noncore_args': [{'tlemma': 'for his meal to ...</td>\n",
       "      <td>change_voice</td>\n",
       "      <td>preserves_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[VERB+passive+present: arrive | PATIENT+comple...</td>\n",
       "      <td>A man in a restaurant is waiting for [PATIENT:...</td>\n",
       "      <td>{'noncore_args': [], 'core_args': [{'tlemma': ...</td>\n",
       "      <td>change_voice</td>\n",
       "      <td>preserves_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>0</th>\n",
       "      <td>[VERB+passive+present: get | AGENT+complete: b...</td>\n",
       "      <td>[AGENT: A blond man] [VERB: getting] [PATIENT:...</td>\n",
       "      <td>{'noncore_args': [{'tlemma': 'from a fountain ...</td>\n",
       "      <td>change_voice</td>\n",
       "      <td>preserves_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[VERB+active+present: get | AGENT+complete: A ...</td>\n",
       "      <td>[PATIENT: A blond man] [VERB: getting] [AGENT:...</td>\n",
       "      <td>{'noncore_args': [{'tlemma': 'from a fountain ...</td>\n",
       "      <td>swap_core_with_context</td>\n",
       "      <td>changes_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">49</th>\n",
       "      <th>1</th>\n",
       "      <td>[VERB+passive+present: sit | PATIENT+complete:...</td>\n",
       "      <td>[PATIENT: The man] is [VERB: sitting] down [TE...</td>\n",
       "      <td>{'noncore_args': [{'tlemma': 'while he has a s...</td>\n",
       "      <td>change_voice</td>\n",
       "      <td>preserves_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[VERB+passive+present: have | AGENT+complete: ...</td>\n",
       "      <td>The man is sitting down while [AGENT: he] [VER...</td>\n",
       "      <td>{'noncore_args': [], 'core_args': [{'tlemma': ...</td>\n",
       "      <td>change_voice</td>\n",
       "      <td>preserves_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[VERB+active+present: have | AGENT+partial: A ...</td>\n",
       "      <td>The man is sitting down while [PATIENT: he] [V...</td>\n",
       "      <td>{'noncore_args': [], 'core_args': [{'tlemma': ...</td>\n",
       "      <td>swap_core_with_context</td>\n",
       "      <td>changes_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[VERB+active+present: have | AGENT+partial: A ...</td>\n",
       "      <td>The man is sitting down while [PATIENT: he] [V...</td>\n",
       "      <td>{'noncore_args': [], 'core_args': [{'tlemma': ...</td>\n",
       "      <td>swap_core_without_context</td>\n",
       "      <td>changes_meaning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[VERB+active+present: have | AGENT+complete: H...</td>\n",
       "      <td>The man is sitting down while [AGENT: he] [VER...</td>\n",
       "      <td>{'noncore_args': [], 'core_args': [{'tlemma': ...</td>\n",
       "      <td>shorten_core_argument</td>\n",
       "      <td>preserves_meaning</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 prompt  \\\n",
       "0  0  [VERB+passive+present: be]  <extra_id_0> <extr...   \n",
       "   1  [VERB+passive+present: wait | PATIENT+complete...   \n",
       "   2  [VERB+passive+present: arrive | PATIENT+comple...   \n",
       "1  0  [VERB+passive+present: get | AGENT+complete: b...   \n",
       "   1  [VERB+active+present: get | AGENT+complete: A ...   \n",
       "...                                                 ...   \n",
       "49 1  [VERB+passive+present: sit | PATIENT+complete:...   \n",
       "   2  [VERB+passive+present: have | AGENT+complete: ...   \n",
       "   3  [VERB+active+present: have | AGENT+partial: A ...   \n",
       "   4  [VERB+active+present: have | AGENT+partial: A ...   \n",
       "   5  [VERB+active+present: have | AGENT+complete: H...   \n",
       "\n",
       "                                                 answer  \\\n",
       "0  0  A man in a restaurant [VERB: is] waiting for h...   \n",
       "   1  [PATIENT: A man in a restaurant] is [VERB: wai...   \n",
       "   2  A man in a restaurant is waiting for [PATIENT:...   \n",
       "1  0  [AGENT: A blond man] [VERB: getting] [PATIENT:...   \n",
       "   1  [PATIENT: A blond man] [VERB: getting] [AGENT:...   \n",
       "...                                                 ...   \n",
       "49 1  [PATIENT: The man] is [VERB: sitting] down [TE...   \n",
       "   2  The man is sitting down while [AGENT: he] [VER...   \n",
       "   3  The man is sitting down while [PATIENT: he] [V...   \n",
       "   4  The man is sitting down while [PATIENT: he] [V...   \n",
       "   5  The man is sitting down while [AGENT: he] [VER...   \n",
       "\n",
       "                                                   meta  \\\n",
       "0  0  {'noncore_args': [], 'core_args': [], 'blank_i...   \n",
       "   1  {'noncore_args': [{'tlemma': 'for his meal to ...   \n",
       "   2  {'noncore_args': [], 'core_args': [{'tlemma': ...   \n",
       "1  0  {'noncore_args': [{'tlemma': 'from a fountain ...   \n",
       "   1  {'noncore_args': [{'tlemma': 'from a fountain ...   \n",
       "...                                                 ...   \n",
       "49 1  {'noncore_args': [{'tlemma': 'while he has a s...   \n",
       "   2  {'noncore_args': [], 'core_args': [{'tlemma': ...   \n",
       "   3  {'noncore_args': [], 'core_args': [{'tlemma': ...   \n",
       "   4  {'noncore_args': [], 'core_args': [{'tlemma': ...   \n",
       "   5  {'noncore_args': [], 'core_args': [{'tlemma': ...   \n",
       "\n",
       "                           name        description  \n",
       "0  0               change_voice  preserves_meaning  \n",
       "   1               change_voice  preserves_meaning  \n",
       "   2               change_voice  preserves_meaning  \n",
       "1  0               change_voice  preserves_meaning  \n",
       "   1     swap_core_with_context    changes_meaning  \n",
       "...                         ...                ...  \n",
       "49 1               change_voice  preserves_meaning  \n",
       "   2               change_voice  preserves_meaning  \n",
       "   3     swap_core_with_context    changes_meaning  \n",
       "   4  swap_core_without_context    changes_meaning  \n",
       "   5      shorten_core_argument  preserves_meaning  \n",
       "\n",
       "[200 rows x 5 columns]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Here we generate the actual inputs to Tailor (the prompt column). \n",
    "# Note that the name and description are preserved for future reference.\n",
    "show_as_dataframe(sentence_prompts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "213dab4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from tailor.common.utils.model_utils import load_generator\n",
    "\n",
    "generator = load_generator()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e50ee808",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████████████████████████████| 2/2 [00:09<00:00,  4.60s/it]\n",
      "100%|████████████████████████████████████████████████████████████████████| 200/200 [00:00<00:00, 7740.92it/s]\n"
     ]
    }
   ],
   "source": [
    "# finally we can make the actual generation using the prompts.\n",
    "\n",
    "from tailor.steps.generate_from_prompts import GenerateFromPrompts\n",
    "\n",
    "generations = GenerateFromPrompts().run(\n",
    "    processed_sentences=processed_sentences,\n",
    "    prompts=sentence_prompts,\n",
    "    spacy_model=spacy_model,\n",
    "    compute_perplexity=True,\n",
    "    generator=generator,\n",
    "    max_length=75,\n",
    "    num_perturbations=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb8cbe23",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "867717bc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>prompt_no_header</th>\n",
       "      <th>sentence</th>\n",
       "      <th>clean_sentence</th>\n",
       "      <th>meta</th>\n",
       "      <th>annotations</th>\n",
       "      <th>words</th>\n",
       "      <th>vidx</th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>is_valid</th>\n",
       "      <th>perplexities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>I'm [VERB: getting] [PATIENT: a drink of water...</td>\n",
       "      <td>I 'm getting a drink of water - by blond man |...</td>\n",
       "      <td>I'm getting a drink of water-by blond man | fr...</td>\n",
       "      <td>{'match': '&lt;re.Match object; span=(0, 27), mat...</td>\n",
       "      <td>[{'tag': 'VERB', 'start': 2, 'end': 3, 'pred':...</td>\n",
       "      <td>[I, 'm, getting, a, drink, of, water, -, by, b...</td>\n",
       "      <td>3</td>\n",
       "      <td>change_voice</td>\n",
       "      <td>preserves_meaning</td>\n",
       "      <td>None</td>\n",
       "      <td>{'pr_sent': 56.27299499511719, 'pr_phrase': -3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[AGENT: A drink of water] [VERB: gets] - [PATI...</td>\n",
       "      <td>A drink of water gets - a blond man -- from ''...</td>\n",
       "      <td>A drink of water gets-a blond man--from'' spit...</td>\n",
       "      <td>{'match': '&lt;re.Match object; span=(0, 26), mat...</td>\n",
       "      <td>[{'tag': 'AGENT', 'start': 0, 'end': 4, 'pred'...</td>\n",
       "      <td>[A, drink, of, water, gets, -, a, blond, man, ...</td>\n",
       "      <td>4</td>\n",
       "      <td>swap_core_with_context</td>\n",
       "      <td>changes_meaning</td>\n",
       "      <td>None</td>\n",
       "      <td>{'pr_sent': 83.90037155151367, 'pr_phrase': 0.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[AGENT: A drink of water] [VERB: gets]'[PATIEN...</td>\n",
       "      <td>A drink of water gets ' a blond man -- from , ...</td>\n",
       "      <td>A drink of water gets' a blond man--from, foun...</td>\n",
       "      <td>{'match': '&lt;re.Match object; span=(0, 26), mat...</td>\n",
       "      <td>[{'tag': 'AGENT', 'start': 0, 'end': 4, 'pred'...</td>\n",
       "      <td>[A, drink, of, water, gets, ', a, blond, man, ...</td>\n",
       "      <td>4</td>\n",
       "      <td>swap_core_without_context</td>\n",
       "      <td>changes_meaning</td>\n",
       "      <td>None</td>\n",
       "      <td>{'pr_sent': 58.69119644165039, 'pr_phrase': 0.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>- [AGENT: a blond man] [VERB: gets]'[PATIENT- ...</td>\n",
       "      <td>- a blond man gets ' ee-</td>\n",
       "      <td>a blond man gets' ee-</td>\n",
       "      <td>{'match': '&lt;re.Match object; span=(0, 26), mat...</td>\n",
       "      <td>[{'tag': 'AGENT', 'start': 1, 'end': 4, 'pred'...</td>\n",
       "      <td>[-, a, blond, man, gets, ', ee-]</td>\n",
       "      <td>4</td>\n",
       "      <td>shorten_core_argument</td>\n",
       "      <td>preserves_meaning</td>\n",
       "      <td>None</td>\n",
       "      <td>{'pr_sent': 7.019298553466797, 'pr_phrase': 7....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[AGENT: A blond man] [VERB: gets] a drink of w...</td>\n",
       "      <td>A blond man gets a drink of water ] \" ooh . ''</td>\n",
       "      <td>A blond man gets a drink of water ]\" ooh.''</td>\n",
       "      <td>{'match': '&lt;re.Match object; span=(0, 26), mat...</td>\n",
       "      <td>[{'tag': 'AGENT', 'start': 0, 'end': 3, 'pred'...</td>\n",
       "      <td>[A, blond, man, gets, a, drink, of, water, ], ...</td>\n",
       "      <td>3</td>\n",
       "      <td>shorten_core_argument</td>\n",
       "      <td>preserves_meaning</td>\n",
       "      <td>None</td>\n",
       "      <td>{'pr_sent': 33.38823318481445, 'pr_phrase': 0.0}</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Almost] a blond man] [VERB: gets] to [PATIENT:...</td>\n",
       "      <td>Almost ] a blond man ] gets to spout of water</td>\n",
       "      <td>Almost ] a blond man ] gets to spout of water</td>\n",
       "      <td>{'match': '&lt;re.Match object; span=(0, 26), mat...</td>\n",
       "      <td>[{'tag': 'VERB', 'start': 6, 'end': 7, 'pred':...</td>\n",
       "      <td>[Almost, ], a, blond, man, ], gets, to, spout,...</td>\n",
       "      <td>6</td>\n",
       "      <td>replace_core_with_subsequence</td>\n",
       "      <td>changes_meaning</td>\n",
       "      <td>None</td>\n",
       "      <td>{'pr_sent': 37.33604049682617, 'pr_phrase': 24...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    prompt_no_header  \\\n",
       "0  I'm [VERB: getting] [PATIENT: a drink of water...   \n",
       "1  [AGENT: A drink of water] [VERB: gets] - [PATI...   \n",
       "2  [AGENT: A drink of water] [VERB: gets]'[PATIEN...   \n",
       "3  - [AGENT: a blond man] [VERB: gets]'[PATIENT- ...   \n",
       "4  [AGENT: A blond man] [VERB: gets] a drink of w...   \n",
       "5  Almost] a blond man] [VERB: gets] to [PATIENT:...   \n",
       "\n",
       "                                            sentence  \\\n",
       "0  I 'm getting a drink of water - by blond man |...   \n",
       "1  A drink of water gets - a blond man -- from ''...   \n",
       "2  A drink of water gets ' a blond man -- from , ...   \n",
       "3                           - a blond man gets ' ee-   \n",
       "4     A blond man gets a drink of water ] \" ooh . ''   \n",
       "5      Almost ] a blond man ] gets to spout of water   \n",
       "\n",
       "                                      clean_sentence  \\\n",
       "0  I'm getting a drink of water-by blond man | fr...   \n",
       "1  A drink of water gets-a blond man--from'' spit...   \n",
       "2  A drink of water gets' a blond man--from, foun...   \n",
       "3                              a blond man gets' ee-   \n",
       "4        A blond man gets a drink of water ]\" ooh.''   \n",
       "5      Almost ] a blond man ] gets to spout of water   \n",
       "\n",
       "                                                meta  \\\n",
       "0  {'match': '<re.Match object; span=(0, 27), mat...   \n",
       "1  {'match': '<re.Match object; span=(0, 26), mat...   \n",
       "2  {'match': '<re.Match object; span=(0, 26), mat...   \n",
       "3  {'match': '<re.Match object; span=(0, 26), mat...   \n",
       "4  {'match': '<re.Match object; span=(0, 26), mat...   \n",
       "5  {'match': '<re.Match object; span=(0, 26), mat...   \n",
       "\n",
       "                                         annotations  \\\n",
       "0  [{'tag': 'VERB', 'start': 2, 'end': 3, 'pred':...   \n",
       "1  [{'tag': 'AGENT', 'start': 0, 'end': 4, 'pred'...   \n",
       "2  [{'tag': 'AGENT', 'start': 0, 'end': 4, 'pred'...   \n",
       "3  [{'tag': 'AGENT', 'start': 1, 'end': 4, 'pred'...   \n",
       "4  [{'tag': 'AGENT', 'start': 0, 'end': 3, 'pred'...   \n",
       "5  [{'tag': 'VERB', 'start': 6, 'end': 7, 'pred':...   \n",
       "\n",
       "                                               words  vidx  \\\n",
       "0  [I, 'm, getting, a, drink, of, water, -, by, b...     3   \n",
       "1  [A, drink, of, water, gets, -, a, blond, man, ...     4   \n",
       "2  [A, drink, of, water, gets, ', a, blond, man, ...     4   \n",
       "3                   [-, a, blond, man, gets, ', ee-]     4   \n",
       "4  [A, blond, man, gets, a, drink, of, water, ], ...     3   \n",
       "5  [Almost, ], a, blond, man, ], gets, to, spout,...     6   \n",
       "\n",
       "                            name        description is_valid  \\\n",
       "0                   change_voice  preserves_meaning     None   \n",
       "1         swap_core_with_context    changes_meaning     None   \n",
       "2      swap_core_without_context    changes_meaning     None   \n",
       "3          shorten_core_argument  preserves_meaning     None   \n",
       "4          shorten_core_argument  preserves_meaning     None   \n",
       "5  replace_core_with_subsequence    changes_meaning     None   \n",
       "\n",
       "                                        perplexities  \n",
       "0  {'pr_sent': 56.27299499511719, 'pr_phrase': -3...  \n",
       "1   {'pr_sent': 83.90037155151367, 'pr_phrase': 0.0}  \n",
       "2   {'pr_sent': 58.69119644165039, 'pr_phrase': 0.0}  \n",
       "3  {'pr_sent': 7.019298553466797, 'pr_phrase': 7....  \n",
       "4   {'pr_sent': 33.38823318481445, 'pr_phrase': 0.0}  \n",
       "5  {'pr_sent': 37.33604049682617, 'pr_phrase': 24...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "show_as_dataframe(generations[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0c87ecf",
   "metadata": {},
   "source": [
    "## Data augmentation with Tailor\n",
    "\n",
    "Tailor also has additional steps that can support data augmentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b0b05894",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "50it [00:00, 50051.36it/s]\n"
     ]
    }
   ],
   "source": [
    "from tailor.tasks.nli.augment import AugmentNLI\n",
    "\n",
    "new_data = AugmentNLI().run(\n",
    "    # these are perturbation settings we used at the beginning.\n",
    "    dataset=data_dict, \n",
    "    perturbed_field=key_to_perturb, \n",
    "    # with the keywords of \"changed_meaning\" and \"preserved_meaning\", we can auto-label the new examples.\n",
    "    generated_prompt_dicts=generations, \n",
    "    max_augment_per_instance=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4c0acd46",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['premise', 'hypothesis', 'label', 'perturbation_strategy'],\n",
       "    num_rows: 72\n",
       "})"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can conver them into Huggingface dataset format.\n",
    "\n",
    "from tailor.steps.convert_dataset_to_dict import ConvertDictToDataset\n",
    "new_dataset = ConvertDictToDataset().run(new_data)\n",
    "\n",
    "new_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "041ff70f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'premise': ['A blond man getting a drink of water from a fountain in the park.',\n",
       "  'A blond man getting a drink of water from a fountain in the park.',\n",
       "  'A blond man getting a drink of water from a fountain in the park.',\n",
       "  'A blond man getting a drink of water from a fountain in the park.',\n",
       "  'A blond man wearing a brown shirt is reading a book on a bench in the park',\n",
       "  'A blond man wearing a brown shirt is reading a book on a bench in the park',\n",
       "  'A blond man wearing a brown shirt is reading a book on a bench in the park',\n",
       "  'A blond man wearing a brown shirt is reading a book on a bench in the park',\n",
       "  'A blond man drinking water from a fountain.',\n",
       "  'A blond man drinking water from a fountain.',\n",
       "  'A blond man drinking water from a fountain.',\n",
       "  'A blond man drinking water from a fountain.',\n",
       "  'The friends scowl at each other over a full dinner table.',\n",
       "  'The friends scowl at each other over a full dinner table.',\n",
       "  'The friends scowl at each other over a full dinner table.',\n",
       "  'The friends scowl at each other over a full dinner table.',\n",
       "  'The friends have just met for the first time in 20 years, and have had a great time catching up.',\n",
       "  'The friends have just met for the first time in 20 years, and have had a great time catching up.',\n",
       "  'The friends have just met for the first time in 20 years, and have had a great time catching up.',\n",
       "  'The friends have just met for the first time in 20 years, and have had a great time catching up.',\n",
       "  'The two sisters saw each other across the crowded diner and shared a hug, both clutching their doggie bags.',\n",
       "  'The two sisters saw each other across the crowded diner and shared a hug, both clutching their doggie bags.',\n",
       "  'The two sisters saw each other across the crowded diner and shared a hug, both clutching their doggie bags.',\n",
       "  'The two sisters saw each other across the crowded diner and shared a hug, both clutching their doggie bags.',\n",
       "  'Two groups of rival gang members flipped each other off.',\n",
       "  'Two groups of rival gang members flipped each other off.',\n",
       "  'Two groups of rival gang members flipped each other off.',\n",
       "  'Two groups of rival gang members flipped each other off.',\n",
       "  'Two women hug each other.',\n",
       "  'Two women hug each other.',\n",
       "  'Two women hug each other.',\n",
       "  'Two women hug each other.',\n",
       "  'A team is trying to score the games winning out.',\n",
       "  'A team is trying to score the games winning out.',\n",
       "  'A team is trying to score the games winning out.',\n",
       "  'A team is trying to tag a runner out.',\n",
       "  'A team is trying to tag a runner out.',\n",
       "  'A team is trying to tag a runner out.',\n",
       "  'A team is trying to tag a runner out.',\n",
       "  'A team is playing baseball on Saturn.',\n",
       "  'A team is playing baseball on Saturn.',\n",
       "  'A team is playing baseball on Saturn.',\n",
       "  'A school hosts a basketball game.',\n",
       "  'A school hosts a basketball game.',\n",
       "  'A high school is hosting an event.',\n",
       "  'A school is hosting an event.',\n",
       "  'A school is hosting an event.',\n",
       "  'The women do not care what clothes they wear.',\n",
       "  'The women do not care what clothes they wear.',\n",
       "  'The women do not care what clothes they wear.',\n",
       "  'The women do not care what clothes they wear.',\n",
       "  'The women enjoy having a good fashion sense.',\n",
       "  'The women enjoy having a good fashion sense.',\n",
       "  'A couple are playing with a young child outside.',\n",
       "  'A couple are playing frisbee with a young child at the beach.',\n",
       "  'A couple watch a little girl play by herself on the beach.',\n",
       "  'The people are standing still on the curb.',\n",
       "  'Near a couple of restaurants, two people walk across the street.',\n",
       "  'The woman is wearing green.',\n",
       "  'They are protesting outside the capital.',\n",
       "  'A man is advertising for a restaurant.',\n",
       "  'The woman is wearing black.',\n",
       "  'The woman is wearing black.',\n",
       "  'A man and a woman walk down a crowded city street.',\n",
       "  'A man and a woman walk down a crowded city street.',\n",
       "  'The woman is wearing white.',\n",
       "  \"They are working for John's Pizza.\",\n",
       "  \"They are working for John's Pizza.\",\n",
       "  \"A man and a soman are eating together at John's Pizza and Gyro.\",\n",
       "  'They are walking with a sign.',\n",
       "  \"The man is sitting down while he has a sign for John's Pizza and Gyro in his arms.\",\n",
       "  \"The man is sitting down while he has a sign for John's Pizza and Gyro in his arms.\"],\n",
       " 'hypothesis': [\"A drink of water gets' a blond man--from, fountain in the park.\",\n",
       "  \"a blond man gets' ee-\",\n",
       "  \"I'm getting a drink of water-by blond man | from s fountain in the park\",\n",
       "  'A blond man gets a drink of water ]\" ooh.\\'\\'',\n",
       "  \"A book on a bench in the park'' reading ta blond man wearing eel shirt\",\n",
       "  \"A book on a bench in the park'' reads s blond man wearing d. m\",\n",
       "  'A blond man wearing a c-',\n",
       "  \"''-a book on bbc in the park, read by blond man wearing cbs\",\n",
       "  \"' A blond man drinks hydr\",\n",
       "  'a blond man drinking-water',\n",
       "  'a blond man drinking, water',\n",
       "  'water, drunk afs and steamed by a blond man from heartstastique.',\n",
       "  'the friends scowl-at each other',\n",
       "  'gotten GGGGRRg: by the friends ah in en-mishy in the mouth oaths of her ear jar]. Hs ianderata to the foodew',\n",
       "  \"The friends' are scowling at each other--\",\n",
       "  \"' At each other, scowling the friends * over a full dinner table\",\n",
       "  \"The friends' had a great time catching up\",\n",
       "  \"A great time catching up have just met for the first time in 20 years, and've had a the friends\",\n",
       "  'Having just gotten met by the friends in for the first time in 20 years',\n",
       "  'Having the friends, met',\n",
       "  'The two sisters saw each other across the crowded diner and shared a hug, Their doggie bags clutching-both',\n",
       "  'their doggie bags are clutched by both.',\n",
       "  'Having seen each other across the crowded diner',\n",
       "  'Having seen each other, the two sisters',\n",
       "  \"Two groups of rival gang members' flipped each other off.\",\n",
       "  \"each other's flipped by two groups of rival gang members\",\n",
       "  'Each other flipped-off members.',\n",
       "  \"' Each other flipped-two groups of rival gang members\",\n",
       "  \"Each other's hugging two women.\",\n",
       "  \"Hugging each other'' by two women\",\n",
       "  'Each other hugging, woman',\n",
       "  'Two women hug, each other',\n",
       "  'The is been ] won a>]',\n",
       "  'A team, trying to-do a job ; attempting sexist aims and practices based on race and ethnicity: vs. team size / style versus team ratio to score the games winning out',\n",
       "  'A team scores, the games winning out',\n",
       "  \"' A team, trying to tag a runner out\",\n",
       "  'A runner tags a team',\n",
       "  \"Lonelier f a team by a teams ] s to tag next year,-LRB-n't enough... by an team\",\n",
       "  \"A team's tagging a runner out.\",\n",
       "  '\\'\\' Baseball, playing a team--on saturn\".',\n",
       "  'Baseball plays a d the m r]] at l.',\n",
       "  \"A team's playing baseball\",\n",
       "  \"a basketball game'' is hosted by-school\",\n",
       "  \"A school's hosting game--\",\n",
       "  \"A high school's hosting event\",\n",
       "  'an event is hosted by a school.',\n",
       "  \"A school's hosting an event\",\n",
       "  'What clothes they wear does not care-the women',\n",
       "  'They wear-what clothes',\n",
       "  'The women care, what clothes they wear',\n",
       "  \"what clothes they wear's not cared for] at all.\",\n",
       "  'The women have a s good fashion sense',\n",
       "  \"The women's enjoy having a good fashion sense\",\n",
       "  'Played-outside ; by a couple with a young child',\n",
       "  'Frisbee\\'s VERB: playing--with, uh.-\" at the beach\"',\n",
       "  'a little girl ] playing.',\n",
       "  \"These Those few have heard their ji are themselves ] they're still d y. on the curb standing is a y U, m n t fucked up in the concrete slab\",\n",
       "  \"two people're walking\",\n",
       "  \"The woman's wearing green\",\n",
       "  \"they're protesting.\",\n",
       "  'A man advertises, for a restaurant.',\n",
       "  \"The woman's wearing black\",\n",
       "  \"woman's black by her\",\n",
       "  'Getting walked by a man and a woman ] the would [ver liqf[/ huhnds djesses',\n",
       "  'a man and / woman walk.',\n",
       "  'The woman wearing a white',\n",
       "  \"they're working\",\n",
       "  \"They are working be by them.-LRW-] for john's pizza\",\n",
       "  'a man and soman, eat [Cafes] and',\n",
       "  'they walk.',\n",
       "  \"s wife and son are the man'd been sitting while he has a sign for john'' has pizza and gyro in his arms and hands\",\n",
       "  \"He has a sign for john's pizza and gyro in his arms ]--LRB-he'll be right back\"],\n",
       " 'label': ['neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'neutral',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'neutral',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment',\n",
       "  'entailment'],\n",
       " 'perturbation_strategy': ['swap_core_without_context',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'swap_core_with_context',\n",
       "  'swap_core_without_context',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'replace_core_with_subsequence',\n",
       "  'change_voice',\n",
       "  'replace_core_with_subsequence',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'swap_core_with_context',\n",
       "  'shorten_core_argument',\n",
       "  'swap_core_with_context',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'swap_core_with_context',\n",
       "  'change_voice',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'swap_core_with_context',\n",
       "  'swap_core_without_context',\n",
       "  'swap_core_with_context',\n",
       "  'change_voice',\n",
       "  'swap_core_without_context',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'swap_core_without_context',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'swap_core_with_context',\n",
       "  'swap_core_without_context',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'swap_core_with_context',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'swap_core_without_context',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument',\n",
       "  'shorten_core_argument',\n",
       "  'change_voice',\n",
       "  'shorten_core_argument']}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_dataset[:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2685d356",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "4\n",
      "5\n",
      "10\n",
      "12\n",
      "15\n",
      "17\n",
      "20\n",
      "26\n",
      "27\n",
      "28\n",
      "30\n",
      "36\n",
      "39\n",
      "40\n",
      "47\n",
      "54\n"
     ]
    }
   ],
   "source": [
    "for idx, row in enumerate(new_dataset):\n",
    "    if row['label'] == 'neutral':\n",
    "        print(idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "6fc350fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'premise': 'Two groups of rival gang members flipped each other off.',\n",
       " 'hypothesis': \"each other's flipped by two groups of rival gang members\",\n",
       " 'label': 'entailment',\n",
       " 'perturbation_strategy': 'change_voice'}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_dataset[25]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "f4e25a50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'premise': 'Two groups of rival gang members flipped each other off.',\n",
       " 'hypothesis': 'Each other flipped-off members.',\n",
       " 'label': 'neutral',\n",
       " 'perturbation_strategy': 'swap_core_with_context'}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_dataset[26]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "7fd3e1bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'premise': 'A school is hosting an event.',\n",
       " 'hypothesis': 'an event is hosted by a school.',\n",
       " 'label': 'entailment',\n",
       " 'perturbation_strategy': 'change_voice'}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_dataset[45]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}