03_Sentiment_analysis_limited_merged.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Sentiment analysis\n",
        "\n",
        "**Author**: Andrea Cass"
      ],
      "metadata": {
        "id": "MZ_c3jCfN4wv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 1. About this notebook\n",
        "\n",
        "The purpose of this Google Colab notebook is to run a sentiment analysis using a cross-lingual language model called XLM-T (twitter-XLM-roBERTa-base-sentiment) on all tweets pre-processed in the Notebook titled, 02_Pre-processing_limited_merged:\n",
        "> *02_Pre-processed_limited_merged.csv*\n",
        "\n",
        "The model was developed and described by Barbieri et al. (2022). Code for employing the model can be accessed here: https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment?text=%F0%9F%A4%97\n",
        "\n",
        "**NOTE**: The model output will be a set of 3 probability scores for each tweet. \n",
        "1. The first represents the probability that the tweet carries **negative** sentiment.\n",
        "2. The second represents the probability that the tweet carries **neutral** sentiment.\n",
        "3. The third represents the probaability that the tweet carries **positive** sentiment\n",
        "\n",
        "Goals:\n",
        "* Preprocess text to remove all usernames and replace them with \"@user\" as well as remove links\n",
        "* Run model to predict probability scores\n",
        "\n",
        "After scores are predicted, the dataset will be saved as a csv titled,\n",
        "> *03_Sentiment-analysis_limited_merged.csv*\n",
        "\n",
        "**NOTE**: This notebook was carried out on Google Colab due to issues using the transformers library on Jupyter notebook. It is my recommendation that you run this notebook on Google Colab. If you run this notebook on Jupyter notebook or another platform, several pieces of code may need to be altered.\n"
      ],
      "metadata": {
        "id": "ntuAfv_IOCtO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 2. Imports"
      ],
      "metadata": {
        "id": "qluSSl8SRNLU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers"
      ],
      "metadata": {
        "id": "bHsGP4pkRR5R"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install datasets evaluate "
      ],
      "metadata": {
        "id": "qhuDAB__RUxG"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install sentencepiece"
      ],
      "metadata": {
        "id": "ey2Y2vK1eZmA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import pipeline\n",
        "import torch\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from transformers import AutoTokenizer, AutoModelForMaskedLM\n",
        "from transformers import AutoTokenizer, AutoConfig\n",
        "from transformers import TFAutoModelForSequenceClassification\n",
        "from transformers import AutoModelForSequenceClassification\n",
        "import pandas as pd\n",
        "import seaborn as sns\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import confusion_matrix, classification_report\n",
        "from collections import defaultdict\n",
        "from textwrap import wrap\n",
        "from torch import nn, optim\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "import torch.nn.functional as F\n",
        "from scipy.special import softmax\n",
        "from google.colab import files\n",
        "import io\n",
        "import sentencepiece"
      ],
      "metadata": {
        "id": "0JNGCb4pRW0S"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3. Loading the data\n",
        "Running code below will prompt you to choose a file from your local drive. The desired file is the data derived from the previous Notebook:\n",
        "\n",
        "> 02_Pre-processed_limited_merged.csv\n",
        "\n",
        "The file should be saved in your folder called \"CASS_thesis\". Select and upload the file. Then, continue running the code as usual."
      ],
      "metadata": {
        "id": "1pNvFmATTHGT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "uploaded = files.upload()"
      ],
      "metadata": {
        "id": "Po-dUhAoTHWy",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 74
        },
        "outputId": "d3fb17de-e92c-4dac-d79f-2a59961c27fb"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ],
            "text/html": [
              "\n",
              "     <input type=\"file\" id=\"files-34d3f6b9-6e5d-43d4-a14f-81c9ada470f7\" name=\"files[]\" multiple disabled\n",
              "        style=\"border:none\" />\n",
              "     <output id=\"result-34d3f6b9-6e5d-43d4-a14f-81c9ada470f7\">\n",
              "      Upload widget is only available when the cell has been executed in the\n",
              "      current browser session. Please rerun this cell to enable.\n",
              "      </output>\n",
              "      <script>// Copyright 2017 Google LLC\n",
              "//\n",
              "// Licensed under the Apache License, Version 2.0 (the \"License\");\n",
              "// you may not use this file except in compliance with the License.\n",
              "// You may obtain a copy of the License at\n",
              "//\n",
              "//      http:https://www.apache.org/licenses/LICENSE-2.0\n",
              "//\n",
              "// Unless required by applicable law or agreed to in writing, software\n",
              "// distributed under the License is distributed on an \"AS IS\" BASIS,\n",
              "// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
              "// See the License for the specific language governing permissions and\n",
              "// limitations under the License.\n",
              "\n",
              "/**\n",
              " * @fileoverview Helpers for google.colab Python module.\n",
              " */\n",
              "(function(scope) {\n",
              "function span(text, styleAttributes = {}) {\n",
              "  const element = document.createElement('span');\n",
              "  element.textContent = text;\n",
              "  for (const key of Object.keys(styleAttributes)) {\n",
              "    element.style[key] = styleAttributes[key];\n",
              "  }\n",
              "  return element;\n",
              "}\n",
              "\n",
              "// Max number of bytes which will be uploaded at a time.\n",
              "const MAX_PAYLOAD_SIZE = 100 * 1024;\n",
              "\n",
              "function _uploadFiles(inputId, outputId) {\n",
              "  const steps = uploadFilesStep(inputId, outputId);\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  // Cache steps on the outputElement to make it available for the next call\n",
              "  // to uploadFilesContinue from Python.\n",
              "  outputElement.steps = steps;\n",
              "\n",
              "  return _uploadFilesContinue(outputId);\n",
              "}\n",
              "\n",
              "// This is roughly an async generator (not supported in the browser yet),\n",
              "// where there are multiple asynchronous steps and the Python side is going\n",
              "// to poll for completion of each step.\n",
              "// This uses a Promise to block the python side on completion of each step,\n",
              "// then passes the result of the previous step as the input to the next step.\n",
              "function _uploadFilesContinue(outputId) {\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  const steps = outputElement.steps;\n",
              "\n",
              "  const next = steps.next(outputElement.lastPromiseValue);\n",
              "  return Promise.resolve(next.value.promise).then((value) => {\n",
              "    // Cache the last promise value to make it available to the next\n",
              "    // step of the generator.\n",
              "    outputElement.lastPromiseValue = value;\n",
              "    return next.value.response;\n",
              "  });\n",
              "}\n",
              "\n",
              "/**\n",
              " * Generator function which is called between each async step of the upload\n",
              " * process.\n",
              " * @param {string} inputId Element ID of the input file picker element.\n",
              " * @param {string} outputId Element ID of the output display.\n",
              " * @return {!Iterable<!Object>} Iterable of next steps.\n",
              " */\n",
              "function* uploadFilesStep(inputId, outputId) {\n",
              "  const inputElement = document.getElementById(inputId);\n",
              "  inputElement.disabled = false;\n",
              "\n",
              "  const outputElement = document.getElementById(outputId);\n",
              "  outputElement.innerHTML = '';\n",
              "\n",
              "  const pickedPromise = new Promise((resolve) => {\n",
              "    inputElement.addEventListener('change', (e) => {\n",
              "      resolve(e.target.files);\n",
              "    });\n",
              "  });\n",
              "\n",
              "  const cancel = document.createElement('button');\n",
              "  inputElement.parentElement.appendChild(cancel);\n",
              "  cancel.textContent = 'Cancel upload';\n",
              "  const cancelPromise = new Promise((resolve) => {\n",
              "    cancel.onclick = () => {\n",
              "      resolve(null);\n",
              "    };\n",
              "  });\n",
              "\n",
              "  // Wait for the user to pick the files.\n",
              "  const files = yield {\n",
              "    promise: Promise.race([pickedPromise, cancelPromise]),\n",
              "    response: {\n",
              "      action: 'starting',\n",
              "    }\n",
              "  };\n",
              "\n",
              "  cancel.remove();\n",
              "\n",
              "  // Disable the input element since further picks are not allowed.\n",
              "  inputElement.disabled = true;\n",
              "\n",
              "  if (!files) {\n",
              "    return {\n",
              "      response: {\n",
              "        action: 'complete',\n",
              "      }\n",
              "    };\n",
              "  }\n",
              "\n",
              "  for (const file of files) {\n",
              "    const li = document.createElement('li');\n",
              "    li.append(span(file.name, {fontWeight: 'bold'}));\n",
              "    li.append(span(\n",
              "        `(${file.type || 'n/a'}) - ${file.size} bytes, ` +\n",
              "        `last modified: ${\n",
              "            file.lastModifiedDate ? file.lastModifiedDate.toLocaleDateString() :\n",
              "                                    'n/a'} - `));\n",
              "    const percent = span('0% done');\n",
              "    li.appendChild(percent);\n",
              "\n",
              "    outputElement.appendChild(li);\n",
              "\n",
              "    const fileDataPromise = new Promise((resolve) => {\n",
              "      const reader = new FileReader();\n",
              "      reader.onload = (e) => {\n",
              "        resolve(e.target.result);\n",
              "      };\n",
              "      reader.readAsArrayBuffer(file);\n",
              "    });\n",
              "    // Wait for the data to be ready.\n",
              "    let fileData = yield {\n",
              "      promise: fileDataPromise,\n",
              "      response: {\n",
              "        action: 'continue',\n",
              "      }\n",
              "    };\n",
              "\n",
              "    // Use a chunked sending to avoid message size limits. See b/62115660.\n",
              "    let position = 0;\n",
              "    do {\n",
              "      const length = Math.min(fileData.byteLength - position, MAX_PAYLOAD_SIZE);\n",
              "      const chunk = new Uint8Array(fileData, position, length);\n",
              "      position += length;\n",
              "\n",
              "      const base64 = btoa(String.fromCharCode.apply(null, chunk));\n",
              "      yield {\n",
              "        response: {\n",
              "          action: 'append',\n",
              "          file: file.name,\n",
              "          data: base64,\n",
              "        },\n",
              "      };\n",
              "\n",
              "      let percentDone = fileData.byteLength === 0 ?\n",
              "          100 :\n",
              "          Math.round((position / fileData.byteLength) * 100);\n",
              "      percent.textContent = `${percentDone}% done`;\n",
              "\n",
              "    } while (position < fileData.byteLength);\n",
              "  }\n",
              "\n",
              "  // All done.\n",
              "  yield {\n",
              "    response: {\n",
              "      action: 'complete',\n",
              "    }\n",
              "  };\n",
              "}\n",
              "\n",
              "scope.google = scope.google || {};\n",
              "scope.google.colab = scope.google.colab || {};\n",
              "scope.google.colab._files = {\n",
              "  _uploadFiles,\n",
              "  _uploadFilesContinue,\n",
              "};\n",
              "})(self);\n",
              "</script> "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Saving 02_Pre-processed_limited_merged.csv to 02_Pre-processed_limited_merged.csv\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# importing the dataset as dataframe\n",
        "\n",
        "df = pd.read_csv(io.BytesIO(uploaded['02_Pre-processed_limited_merged.csv']), index_col=[0])"
      ],
      "metadata": {
        "id": "frFXloK0UiiI"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "id": "M4j6MT7w5KId"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 4. Pre-processing text\n",
        "The code below to pre-process tweets was provided by the creators of XLM-T. For more details, please refer to the link provided in the first section of this Notebook."
      ],
      "metadata": {
        "id": "LHWX8su3VoV7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Defining function to preprocess text (username placeholders and link removal)\n",
        "\n",
        "def preprocess(text):\n",
        "    new_text = []\n",
        "    for t in text.split(\" \"):\n",
        "        t = '@user' if t.startswith('@') and len(t) > 1 else t\n",
        "        t = 'http' if t.startswith('http') else t\n",
        "        new_text.append(t)\n",
        "    return \" \".join(new_text)"
      ],
      "metadata": {
        "id": "J2r9ccl-V1A-"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Converting text to string\n",
        "\n",
        "df['text'] = df['text'].astype(str)"
      ],
      "metadata": {
        "id": "ul7NSrNbV_nw"
      },
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Applying function to preprocess text and create column for the new text\n",
        "\n",
        "df['new text'] = df['text'].apply(preprocess)"
      ],
      "metadata": {
        "id": "crzeIAOVWXyR"
      },
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4.1. Viewing the dataframe"
      ],
      "metadata": {
        "id": "RVG-AI1vWeRb"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "SKqTViTv2lhR",
        "outputId": "35fc90ca-254b-4e90-ccc5-8806e88fac24"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                    text     author_id  \\\n",
              "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4.122038e+09   \n",
              "1      Habe schon lang nicht gehört, daß Flüchtling G...  1.179544e+09   \n",
              "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  2.246076e+08   \n",
              "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2.480764e+09   \n",
              "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  6.062653e+08   \n",
              "...                                                  ...           ...   \n",
              "66260  For day 1 of week 2, @AnnaMariaKonsta discusse...  1.104025e+08   \n",
              "66261  @ariadneconill Europe is racist, but in a diff...  2.521809e+09   \n",
              "66262  A labour of love, inspired by Middle-earth.\\n\\...  5.633818e+08   \n",
              "66263  @simongerman600 I must have missed the great f...  2.591892e+09   \n",
              "66264  #Wuerzburg today an African immigrant or refug...  1.344017e+08   \n",
              "\n",
              "                      created_at lang      geo.place_id  \\\n",
              "0      2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
              "1      2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
              "2      2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
              "3      2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
              "4      2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
              "...                          ...  ...               ...   \n",
              "66260  2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
              "66261  2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
              "66262  2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
              "66263  2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
              "66264  2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
              "\n",
              "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
              "0                               0.0                         0.0   \n",
              "1                               0.0                         0.0   \n",
              "2                               0.0                         0.0   \n",
              "3                               0.0                         0.0   \n",
              "4                               0.0                         1.0   \n",
              "...                             ...                         ...   \n",
              "66260                           0.0                         1.0   \n",
              "66261                           0.0                         1.0   \n",
              "66262                           0.0                         1.0   \n",
              "66263                           0.0                         0.0   \n",
              "66264                           8.0                         6.0   \n",
              "\n",
              "       public_metrics.like_count  public_metrics.quote_count  \\\n",
              "0                            0.0                         0.0   \n",
              "1                            0.0                         0.0   \n",
              "2                            0.0                         0.0   \n",
              "3                            4.0                         0.0   \n",
              "4                            0.0                         0.0   \n",
              "...                          ...                         ...   \n",
              "66260                        2.0                         0.0   \n",
              "66261                        6.0                         0.0   \n",
              "66262                       17.0                         0.0   \n",
              "66263                        1.0                         0.0   \n",
              "66264                       38.0                         6.0   \n",
              "\n",
              "       public_metrics.impression_count  ...  \\\n",
              "0                                  0.0  ...   \n",
              "1                                  0.0  ...   \n",
              "2                                  0.0  ...   \n",
              "3                                  0.0  ...   \n",
              "4                                  0.0  ...   \n",
              "...                                ...  ...   \n",
              "66260                              0.0  ...   \n",
              "66261                              0.0  ...   \n",
              "66262                              0.0  ...   \n",
              "66263                              0.0  ...   \n",
              "66264                              0.0  ...   \n",
              "\n",
              "                                       entities.hashtags  week month    year  \\\n",
              "0                                                    NaN  16.0   4.0  2016.0   \n",
              "1                                                    NaN  16.0   4.0  2016.0   \n",
              "2                                                    NaN  16.0   4.0  2016.0   \n",
              "3       [{'start': 23, 'end': 35, 'tag': 'Flüchtlinge'}]  16.0   4.0  2016.0   \n",
              "4      [{'start': 9, 'end': 21, 'tag': 'Flüchtlinge'}...  16.0   4.0  2016.0   \n",
              "...                                                  ...   ...   ...     ...   \n",
              "66260  [{'start': 95, 'end': 108, 'tag': 'SocialRight...  26.0   6.0  2021.0   \n",
              "66261                                                NaN  25.0   6.0  2021.0   \n",
              "66262                                                NaN  25.0   6.0  2021.0   \n",
              "66263                                                NaN  25.0   6.0  2021.0   \n",
              "66264      [{'start': 0, 'end': 10, 'tag': 'Wuerzburg'}]  25.0   6.0  2021.0   \n",
              "\n",
              "       year-week  year-month Language                       date      inflow  \\\n",
              "0        2016-16     2016-04   German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "1        2016-16     2016-04   German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "2        2016-16     2016-04   German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "3        2016-16     2016-04   German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "4        2016-16     2016-04   German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "...          ...         ...      ...                        ...         ...   \n",
              "66260    2021-26     2021-06  English  2021-06-28 00:00:00+00:00  Ukrainians   \n",
              "66261    2021-26     2021-06  English  2021-06-27 00:00:00+00:00  Ukrainians   \n",
              "66262    2021-26     2021-06  English  2021-06-27 00:00:00+00:00  Ukrainians   \n",
              "66263    2021-25     2021-06  English  2021-06-26 00:00:00+00:00  Ukrainians   \n",
              "66264    2021-25     2021-06  English  2021-06-25 00:00:00+00:00  Ukrainians   \n",
              "\n",
              "                                                new text  \n",
              "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  \n",
              "1      Habe schon lang nicht gehört, daß Flüchtling G...  \n",
              "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  \n",
              "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  \n",
              "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  \n",
              "...                                                  ...  \n",
              "66260  For day 1 of week 2, @user discusses the Socia...  \n",
              "66261  @user Europe is racist, but in a different way...  \n",
              "66262  A labour of love, inspired by Middle-earth.\\n\\...  \n",
              "66263  @user I must have missed the great flight of t...  \n",
              "66264  #Wuerzburg today an African immigrant or refug...  \n",
              "\n",
              "[66265 rows x 22 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-7ef2bf58-9b1a-4f82-bc7a-a84aec1aaddc\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>author_id</th>\n",
              "      <th>created_at</th>\n",
              "      <th>lang</th>\n",
              "      <th>geo.place_id</th>\n",
              "      <th>public_metrics.retweet_count</th>\n",
              "      <th>public_metrics.reply_count</th>\n",
              "      <th>public_metrics.like_count</th>\n",
              "      <th>public_metrics.quote_count</th>\n",
              "      <th>public_metrics.impression_count</th>\n",
              "      <th>...</th>\n",
              "      <th>entities.hashtags</th>\n",
              "      <th>week</th>\n",
              "      <th>month</th>\n",
              "      <th>year</th>\n",
              "      <th>year-week</th>\n",
              "      <th>year-month</th>\n",
              "      <th>Language</th>\n",
              "      <th>date</th>\n",
              "      <th>inflow</th>\n",
              "      <th>new text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
              "      <td>4.122038e+09</td>\n",
              "      <td>2016-04-20 22:55:08+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>06d9a7c249c59bcd</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
              "      <td>1.179544e+09</td>\n",
              "      <td>2016-04-20 21:27:37+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>e99b714fe65be4fb</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
              "      <td>2.246076e+08</td>\n",
              "      <td>2016-04-20 21:18:58+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>3078869807f9dd36</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
              "      <td>2.480764e+09</td>\n",
              "      <td>2016-04-20 18:25:11+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>8abc99434d4f5d28</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>[{'start': 23, 'end': 35, 'tag': 'Flüchtlinge'}]</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
              "      <td>6.062653e+08</td>\n",
              "      <td>2016-04-20 16:27:28+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>e11a8b8e3771f9fa</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>[{'start': 9, 'end': 21, 'tag': 'Flüchtlinge'}...</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66260</th>\n",
              "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
              "      <td>1.104025e+08</td>\n",
              "      <td>2021-06-28 14:43:31+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>fcbb3c6e0a7eba22</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>[{'start': 95, 'end': 108, 'tag': 'SocialRight...</td>\n",
              "      <td>26.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-26</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-28 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>For day 1 of week 2, @user discusses the Socia...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66261</th>\n",
              "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
              "      <td>2.521809e+09</td>\n",
              "      <td>2021-06-27 13:03:29+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>e385d4d639c6a423</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>25.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-26</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-27 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>@user Europe is racist, but in a different way...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66262</th>\n",
              "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
              "      <td>5.633818e+08</td>\n",
              "      <td>2021-06-27 08:37:21+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>257640324f249a73</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>17.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>25.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-26</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-27 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66263</th>\n",
              "      <td>@simongerman600 I must have missed the great f...</td>\n",
              "      <td>2.591892e+09</td>\n",
              "      <td>2021-06-26 08:03:22+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>000b71538f35fe46</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>NaN</td>\n",
              "      <td>25.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-25</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-26 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>@user I must have missed the great flight of t...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66264</th>\n",
              "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
              "      <td>1.344017e+08</td>\n",
              "      <td>2021-06-25 21:02:17+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>08f94128283f7bc6</td>\n",
              "      <td>8.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>38.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>[{'start': 0, 'end': 10, 'tag': 'Wuerzburg'}]</td>\n",
              "      <td>25.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-25</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-25 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>66265 rows × 22 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7ef2bf58-9b1a-4f82-bc7a-a84aec1aaddc')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http:https://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-7ef2bf58-9b1a-4f82-bc7a-a84aec1aaddc button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-7ef2bf58-9b1a-4f82-bc7a-a84aec1aaddc');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 5. Sentiment analysis with XLMT\n",
        "The code below to load and run the model was largely provided by the creators of XLM-T. For more details, please refer to the link provided in the first section of this Notebook.\n",
        "### 5.1. Loading in the model"
      ],
      "metadata": {
        "id": "kTbctDSdWj-_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "MODEL = f\"cardiffnlp/twitter-xlm-roberta-base-sentiment\""
      ],
      "metadata": {
        "id": "DzulOjufW-YP"
      },
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "tokenizer = AutoTokenizer.from_pretrained(MODEL)"
      ],
      "metadata": {
        "id": "BM5behymesuo"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "config = AutoConfig.from_pretrained(MODEL)\n",
        "model = AutoModelForSequenceClassification.from_pretrained(MODEL)"
      ],
      "metadata": {
        "id": "elkU6EBqezK3"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.2 Running the model\n",
        "**NOTE**: When running the model, a waiting period of up to multiple hours may be needed. It is my recommendation to not let your computer sleep during this time."
      ],
      "metadata": {
        "id": "wX6CeEfBXOcZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Defining a function to predict scores\n",
        "\n",
        "def predict(new_text):\n",
        "  encoded_input = tokenizer(new_text, return_tensors='pt', padding = True, truncation = True)\n",
        "  output = model(**encoded_input)\n",
        "  scores = output[0][0].detach().numpy()\n",
        "  scores = softmax(scores, axis=-1)\n",
        "  return scores"
      ],
      "metadata": {
        "id": "vz2Lko3MXSZu"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Applying predict function to find scores\n",
        "\n",
        "df['scores'] = df['new text'].apply(predict)"
      ],
      "metadata": {
        "id": "j2nAgJF-Xonh",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "1a0a2a7e-ff4d-4366-ee73-5861d9136d7b"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 5.3 Viewing the dataframe"
      ],
      "metadata": {
        "id": "ezB3_YIpYYgX"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "FdooyK4SYcWf",
        "outputId": "24acf9f3-33be-4fae-852a-c0f1773997e6"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                                    text     author_id  \\\n",
              "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4.122038e+09   \n",
              "1      Habe schon lang nicht gehört, daß Flüchtling G...  1.179544e+09   \n",
              "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  2.246076e+08   \n",
              "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2.480764e+09   \n",
              "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  6.062653e+08   \n",
              "...                                                  ...           ...   \n",
              "66260  For day 1 of week 2, @AnnaMariaKonsta discusse...  1.104025e+08   \n",
              "66261  @ariadneconill Europe is racist, but in a diff...  2.521809e+09   \n",
              "66262  A labour of love, inspired by Middle-earth.\\n\\...  5.633818e+08   \n",
              "66263  @simongerman600 I must have missed the great f...  2.591892e+09   \n",
              "66264  #Wuerzburg today an African immigrant or refug...  1.344017e+08   \n",
              "\n",
              "                      created_at lang      geo.place_id  \\\n",
              "0      2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
              "1      2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
              "2      2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
              "3      2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
              "4      2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
              "...                          ...  ...               ...   \n",
              "66260  2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
              "66261  2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
              "66262  2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
              "66263  2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
              "66264  2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
              "\n",
              "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
              "0                               0.0                         0.0   \n",
              "1                               0.0                         0.0   \n",
              "2                               0.0                         0.0   \n",
              "3                               0.0                         0.0   \n",
              "4                               0.0                         1.0   \n",
              "...                             ...                         ...   \n",
              "66260                           0.0                         1.0   \n",
              "66261                           0.0                         1.0   \n",
              "66262                           0.0                         1.0   \n",
              "66263                           0.0                         0.0   \n",
              "66264                           8.0                         6.0   \n",
              "\n",
              "       public_metrics.like_count  public_metrics.quote_count  \\\n",
              "0                            0.0                         0.0   \n",
              "1                            0.0                         0.0   \n",
              "2                            0.0                         0.0   \n",
              "3                            4.0                         0.0   \n",
              "4                            0.0                         0.0   \n",
              "...                          ...                         ...   \n",
              "66260                        2.0                         0.0   \n",
              "66261                        6.0                         0.0   \n",
              "66262                       17.0                         0.0   \n",
              "66263                        1.0                         0.0   \n",
              "66264                       38.0                         6.0   \n",
              "\n",
              "       public_metrics.impression_count  ...  week  month    year  year-week  \\\n",
              "0                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
              "1                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
              "2                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
              "3                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
              "4                                  0.0  ...  16.0    4.0  2016.0    2016-16   \n",
              "...                                ...  ...   ...    ...     ...        ...   \n",
              "66260                              0.0  ...  26.0    6.0  2021.0    2021-26   \n",
              "66261                              0.0  ...  25.0    6.0  2021.0    2021-26   \n",
              "66262                              0.0  ...  25.0    6.0  2021.0    2021-26   \n",
              "66263                              0.0  ...  25.0    6.0  2021.0    2021-25   \n",
              "66264                              0.0  ...  25.0    6.0  2021.0    2021-25   \n",
              "\n",
              "       year-month  Language                       date      inflow  \\\n",
              "0         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "1         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "2         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "3         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "4         2016-04    German  2016-04-20 00:00:00+00:00     Syrians   \n",
              "...           ...       ...                        ...         ...   \n",
              "66260     2021-06   English  2021-06-28 00:00:00+00:00  Ukrainians   \n",
              "66261     2021-06   English  2021-06-27 00:00:00+00:00  Ukrainians   \n",
              "66262     2021-06   English  2021-06-27 00:00:00+00:00  Ukrainians   \n",
              "66263     2021-06   English  2021-06-26 00:00:00+00:00  Ukrainians   \n",
              "66264     2021-06   English  2021-06-25 00:00:00+00:00  Ukrainians   \n",
              "\n",
              "                                                new text  \\\n",
              "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...   \n",
              "1      Habe schon lang nicht gehört, daß Flüchtling G...   \n",
              "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   \n",
              "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...   \n",
              "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   \n",
              "...                                                  ...   \n",
              "66260  For day 1 of week 2, @user discusses the Socia...   \n",
              "66261  @user Europe is racist, but in a different way...   \n",
              "66262  A labour of love, inspired by Middle-earth.\\n\\...   \n",
              "66263  @user I must have missed the great flight of t...   \n",
              "66264  #Wuerzburg today an African immigrant or refug...   \n",
              "\n",
              "                                      scores  \n",
              "0       [0.0679844, 0.88380396, 0.048211697]  \n",
              "1        [0.3660313, 0.5833739, 0.050594788]  \n",
              "2        [0.30297568, 0.4430802, 0.25394407]  \n",
              "3      [0.022602528, 0.076046735, 0.9013506]  \n",
              "4       [0.6982864, 0.27545568, 0.026257832]  \n",
              "...                                      ...  \n",
              "66260  [0.035507657, 0.82208896, 0.14240335]  \n",
              "66261   [0.8931986, 0.09380516, 0.012996121]  \n",
              "66262    [0.15655968, 0.5201203, 0.32332003]  \n",
              "66263  [0.91320664, 0.06375945, 0.023033775]  \n",
              "66264     [0.92373, 0.06414817, 0.012121829]  \n",
              "\n",
              "[66265 rows x 23 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-09200179-1d94-46e1-9c39-8972807e9132\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>author_id</th>\n",
              "      <th>created_at</th>\n",
              "      <th>lang</th>\n",
              "      <th>geo.place_id</th>\n",
              "      <th>public_metrics.retweet_count</th>\n",
              "      <th>public_metrics.reply_count</th>\n",
              "      <th>public_metrics.like_count</th>\n",
              "      <th>public_metrics.quote_count</th>\n",
              "      <th>public_metrics.impression_count</th>\n",
              "      <th>...</th>\n",
              "      <th>week</th>\n",
              "      <th>month</th>\n",
              "      <th>year</th>\n",
              "      <th>year-week</th>\n",
              "      <th>year-month</th>\n",
              "      <th>Language</th>\n",
              "      <th>date</th>\n",
              "      <th>inflow</th>\n",
              "      <th>new text</th>\n",
              "      <th>scores</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
              "      <td>4.122038e+09</td>\n",
              "      <td>2016-04-20 22:55:08+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>06d9a7c249c59bcd</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
              "      <td>[0.0679844, 0.88380396, 0.048211697]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
              "      <td>1.179544e+09</td>\n",
              "      <td>2016-04-20 21:27:37+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>e99b714fe65be4fb</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
              "      <td>[0.3660313, 0.5833739, 0.050594788]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
              "      <td>2.246076e+08</td>\n",
              "      <td>2016-04-20 21:18:58+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>3078869807f9dd36</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
              "      <td>[0.30297568, 0.4430802, 0.25394407]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
              "      <td>2.480764e+09</td>\n",
              "      <td>2016-04-20 18:25:11+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>8abc99434d4f5d28</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
              "      <td>[0.022602528, 0.076046735, 0.9013506]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
              "      <td>6.062653e+08</td>\n",
              "      <td>2016-04-20 16:27:28+00:00</td>\n",
              "      <td>de</td>\n",
              "      <td>e11a8b8e3771f9fa</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>16.0</td>\n",
              "      <td>4.0</td>\n",
              "      <td>2016.0</td>\n",
              "      <td>2016-16</td>\n",
              "      <td>2016-04</td>\n",
              "      <td>German</td>\n",
              "      <td>2016-04-20 00:00:00+00:00</td>\n",
              "      <td>Syrians</td>\n",
              "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
              "      <td>[0.6982864, 0.27545568, 0.026257832]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66260</th>\n",
              "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
              "      <td>1.104025e+08</td>\n",
              "      <td>2021-06-28 14:43:31+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>fcbb3c6e0a7eba22</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>26.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-26</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-28 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>For day 1 of week 2, @user discusses the Socia...</td>\n",
              "      <td>[0.035507657, 0.82208896, 0.14240335]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66261</th>\n",
              "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
              "      <td>2.521809e+09</td>\n",
              "      <td>2021-06-27 13:03:29+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>e385d4d639c6a423</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>25.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-26</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-27 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>@user Europe is racist, but in a different way...</td>\n",
              "      <td>[0.8931986, 0.09380516, 0.012996121]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66262</th>\n",
              "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
              "      <td>5.633818e+08</td>\n",
              "      <td>2021-06-27 08:37:21+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>257640324f249a73</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>17.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>25.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-26</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-27 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
              "      <td>[0.15655968, 0.5201203, 0.32332003]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66263</th>\n",
              "      <td>@simongerman600 I must have missed the great f...</td>\n",
              "      <td>2.591892e+09</td>\n",
              "      <td>2021-06-26 08:03:22+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>000b71538f35fe46</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>25.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-25</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-26 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>@user I must have missed the great flight of t...</td>\n",
              "      <td>[0.91320664, 0.06375945, 0.023033775]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>66264</th>\n",
              "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
              "      <td>1.344017e+08</td>\n",
              "      <td>2021-06-25 21:02:17+00:00</td>\n",
              "      <td>en</td>\n",
              "      <td>08f94128283f7bc6</td>\n",
              "      <td>8.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>38.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>...</td>\n",
              "      <td>25.0</td>\n",
              "      <td>6.0</td>\n",
              "      <td>2021.0</td>\n",
              "      <td>2021-25</td>\n",
              "      <td>2021-06</td>\n",
              "      <td>English</td>\n",
              "      <td>2021-06-25 00:00:00+00:00</td>\n",
              "      <td>Ukrainians</td>\n",
              "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
              "      <td>[0.92373, 0.06414817, 0.012121829]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>66265 rows × 23 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-09200179-1d94-46e1-9c39-8972807e9132')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http:https://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-09200179-1d94-46e1-9c39-8972807e9132 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-09200179-1d94-46e1-9c39-8972807e9132');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 6. Downloading dataframe\n",
        "The code below will download the data as a csv file with the title:\n",
        "> 3_Sentiment-analysis_limited_merged.csv\n",
        "\n",
        "Once downloaded, please ensure that you manually move this file to your folder called \"CASS_thesis\", as the code will not automatically do so."
      ],
      "metadata": {
        "id": "cGeZmTA0Yg58"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df.to_csv('03_Sentiment-analysis_limited_merged.csv') \n",
        "files.download('03_Sentiment-analysis_limited_merged.csv')"
      ],
      "metadata": {
        "id": "QqK7UjqWY0zt",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "outputId": "046d6426-fafc-4267-c7c3-83032c6d0d1f"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_bfd53e86-999d-4798-9097-cb306a9a0f63\", \"03_Sentiment-analysis_limited_merged.csv\", 26891660)"
            ]
          },
          "metadata": {}
        }
      ]
    }
  ]
}