notebooks/Document Classification Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "njeqYj-FiSZW"
   },
   "source": [
    "# Document Classification Tutorial 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2019-2024 by [Damir Cavar](https://damir.cavar.me/)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Prerequisites:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WvqjkRurFSk5"
   },
   "source": [
    "## Amazon Reviews"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WcCEhwVCI0V7"
   },
   "source": [
    "See for more details the source of this tutorial: [https://www.analyticsvidhya.com/blog/2018/04/a-comprehensive-guide-to-understand-and-implement-text-classification-in-python/](https://www.analyticsvidhya.com/blog/2018/04/a-comprehensive-guide-to-understand-and-implement-text-classification-in-python/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1lGpzc1UIB-h"
   },
   "source": [
    "We will use the data provided at [this site](https://gist.github.com/kunalj101/ad1d9c58d338e20d09ff26bcc06c4235). This is a collection of 3.6 mil. Amazon text reviews and labels. The data is formated using the [FastText](https://fasttext.cc/docs/en/supervised-tutorial.html) corpus format, that is, each file contains lines with a label followed by the text.\n",
    "\n",
    "  `__label__2 Stuning even for the non-gamer: This sound track was beautiful! It paints the senery in your mind so well I would recomend it even to people who hate vid. game music! I have played the game Chrono Cross but out of all of the games I have ever played it has the best music! It backs away from crude keyboarding and takes a fresher step with grate guitars and soulful orchestras. It would impress anyone who cares to listen! ^_^`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nQnLM2g-JZLG"
   },
   "source": [
    "We load the data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 237
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1629,
     "status": "error",
     "timestamp": 1554864735451,
     "user": {
      "displayName": "Damir Cavar",
      "photoUrl": "https://lh6.googleusercontent.com/-9-d7Q_7Zrko/AAAAAAAAAAI/AAAAAAAABfk/wbJLz30h4Xk/s64/photo.jpg",
      "userId": "10576263158268115293"
     },
     "user_tz": 240
    },
    "id": "i3AQ6pzkJerG",
    "outputId": "ebe23b6f-d023-4ebb-e417-2d74a9e0d6f1"
   },
   "outputs": [],
   "source": [
    "data = open('data/corpus').read()\n",
    "labels, texts = [], []\n",
    "\n",
    "for line in data.split(\"\\n\"):\n",
    "    content = line.split(' ', 1)\n",
    "    labels.append(content[0])\n",
    "    texts.append(content[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "a57G2gLkh6Jn"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Stuning even for the non-gamer: This sound track was beautiful! It paints the senery in your mind so well I would recomend it even to people who hate vid. game music! I have played the game Chrono Cross but out of all of the games I have ever played it has the best music! It backs away from crude keyboarding and takes a fresher step with grate guitars and soulful orchestras. It would impress anyone who cares to listen! ^_^', \"The best soundtrack ever to anything.: I'm reading a lot of reviews saying that this is the best 'game soundtrack' and I figured that I'd write a review to disagree a bit. This in my opinino is Yasunori Mitsuda's ultimate masterpiece. The music is timeless and I'm been listening to it for years now and its beauty simply refuses to fade.The price tag on this is pretty staggering I must say, but if you are going to buy any cd for this much money, this is the only one that I feel would be worth every penny.\", 'Amazing!: This soundtrack is my favorite music of all time, hands down. The intense sadness of \"Prisoners of Fate\" (which means all the more if you\\'ve played the game) and the hope in \"A Distant Promise\" and \"Girl who Stole the Star\" have been an important inspiration to me personally throughout my teen years. The higher energy tracks like \"Chrono Cross ~ Time\\'s Scar~\", \"Time of the Dreamwatch\", and \"Chronomantique\" (indefinably remeniscent of Chrono Trigger) are all absolutely superb as well.This soundtrack is amazing music, probably the best of this composer\\'s work (I haven\\'t heard the Xenogears soundtrack, so I can\\'t say for sure), and even if you\\'ve never played the game, it would be worth twice the price to buy it.I wish I could give it 6 stars.']\n"
     ]
    }
   ],
   "source": [
    "print(texts[:3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use Pandas to store the labels and texts in a DataFrame. We import Pandas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Packing the data into a Pandas DataFrame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = pandas.DataFrame()\n",
    "corpus['text'] = texts\n",
    "corpus['label'] = labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From *scikit_learn* we will import *model_selection*. This module contains a function *train_test_split* that splits arrays or matrices into random train and test subsets. See for more details the [documentation page](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import model_selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will select a third of the data set for testing. The *random_state* in the default will use *np.random* in this function call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_text, test_text, train_label, test_label = model_selection.train_test_split(corpus['text'],\n",
    "                                                                                  corpus['label'],\n",
    "                                                                                  test_size=0.33)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7985    Brilliant album but not the best of Celine: Ex...\n",
      "7760    Stargate Continuum: So glad I purchased the mo...\n",
      "Name: text, dtype: object\n",
      "9898    THIS SUCKS: THIS IS THE STUPIDIEST MOVIE EVER ...\n",
      "2748    Great produt easy to install: great product to...\n",
      "Name: text, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(train_text[:2])\n",
    "print(test_text[:2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the *scikit_learn* module for *preprocessing*. We will use the *LabelEncoder* in the *preprocessing* module to normalize the labels such that they contain only values between 0 and n_classes-1. See for more details the [documentation page](https://scikit-learn.org/stable/modules/preprocessing_targets.html#preprocessing-targets)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import preprocessing\n",
    "\n",
    "encoder = preprocessing.LabelEncoder()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We encode the labels for the training and test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9898    __label__1\n",
      "2748    __label__2\n",
      "5025    __label__2\n",
      "5453    __label__1\n",
      "3782    __label__1\n",
      "8702    __label__2\n",
      "8124    __label__2\n",
      "7988    __label__1\n",
      "9393    __label__2\n",
      "6324    __label__2\n",
      "Name: label, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(test_label[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_label = encoder.fit_transform(train_label)\n",
    "test_label = encoder.fit_transform(test_label)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 1 0 0 1 1 0 1 1]\n"
     ]
    }
   ],
   "source": [
    "print(test_label[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To engineer a classifier, we will select different types of features. We will start using the count vectors as features. In count vectors, each row represents a document from the corpus and each column represents a word from the corpus. The scalar in each vector contains the frequency of a particular token (column) in the document (row). We will import the *CountVectorizer* from the *scikit-learn* module and its *feature_extraction.text* collection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *CountVectorizer* should make features of word n-grams, as specified in *analyzer='word'*. The *token_pattern* parameter is a regular expression denoting what constitutes a token and it is only used if *analyzer == 'word'*. The regular expression here selects words to be tokens of one or more characters. See for more details the [documentation page](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorizer = CountVectorizer(analyzer='word', token_pattern=r'\\w{1,}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *fit* method applied to the *vectorizer* object learns a vocabulary dictionary of all tokens in the raw texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip_accents=None, token_pattern='\\\\w{1,}', tokenizer=None,\n",
       "        vocabulary=None)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vectorizer.fit(corpus['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now transform the training and test data using the *vectorizer* object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_text_count = vectorizer.transform(train_text)\n",
    "test_text_count = vectorizer.transform(test_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the *scikit_learn* module for *linear models*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import linear_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now apply logistic regression on the transformed data and print the resulting accuracy. We create an instance of a Logistic Regression classifier using the *liblinear* algorithm as a solver for optimization. We train the model and generate the predictions for the test data. See for more details the [documentation page](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifier = linear_model.LogisticRegression(solver='liblinear')\n",
    "classifier.fit(train_text_count, train_label)\n",
    "predictions = classifier.predict(test_text_count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use the *metrics* module in *scikit_learn* to compute the accuracy score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the accuracy score, we provide the *accuracy_score* function in the *metrics* module with the predicted labels for the test data set and the real labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR, Count Vectors:  0.8512121212121212\n"
     ]
    }
   ],
   "source": [
    "accuracy = metrics.accuracy_score(predictions, test_label)\n",
    "print(\"LR, Count Vectors: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case logistic regression as a classifier on the word count vectors results in more than 84% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**(C) 2021-2024 by [Damir Cavar](https://damir.cavar.me/) <<dcavar@iu.edu>>**"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Untitled0.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.1"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": false,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}