notebooks/Sequence embedding with pyspark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EMBEDDINGS "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different users interact with website in different ways, sequence embedding use traditional machine learning models to discoer the relation, even amongst unique users"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Techniques like count vectorizer, TF-IDF, and hashing vectorization does not consider the semantic meaning of the word, embedding does<br>\n",
    "embeddings can be used in one of the two ways\n",
    "1. Skip Gram\n",
    "2. Continuous Bag of Words (CBOW)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "both methods gives embedding values that are nothing but weights for neural networks\n",
    "1. the word2vec gives the embedding value for each words\n",
    "2. the doc2vec gives embedding for the entire sentence\n",
    "\n",
    "sequence embeddings are similar to doc2vec and results are weighted mean of individual embeddings of the words appearning in the sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.context import SparkContext\n",
    "from pyspark.sql import SparkSession\n",
    "sc = SparkContext('local', 'seq_embedding')\n",
    "spark = SparkSession(sc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### below is alternate dataset, not being used for the project, download the data by turning and running this cell as code, \n",
    "! wget https://media.githubusercontent.com/media/djentleman/emoji_semantics/c5f0f175d41f50556edf85d3ec9159d58fb5f020/data/preprocessed/embedding_dataset.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "axisdata.csv\t\t       Log_Reg_dataset.csv  sample_data.csv\r\n",
      "embedding_dataset.csv\t       movie_reviews.csv\r\n",
      "Linear_regression_dataset.csv  Movie_reviews.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls ../datasets/nlp_with_pyspark/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.csv('../datasets/nlp_with_pyspark/embedding_dataset.csv',\n",
    "              header=True, inferSchema=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10879500"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- word_a: integer (nullable = true)\n",
      " |-- word_b: integer (nullable = true)\n",
      " |-- has_context: integer (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}