coursera_ml/how_to_use_floorsensordata2604.csv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is designed to run in a IBM Watson Studio default runtime (NOT the Watson Studio Apache Spark Runtime as the default runtime with 1 vCPU is free of charge). Therefore, we install Apache Spark in local mode for test purposes only. Please don't use it in production.\n",
    "\n",
    "In case you are facing issues, please read the following two documents first:\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/Environment-Setup\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/FAQ\n",
    "\n",
    "Then, please feel free to ask:\n",
    "\n",
    "https://coursera.org/learn/machine-learning-big-data-apache-spark/discussions/all\n",
    "\n",
    "Please make sure to follow the guidelines before asking a question:\n",
    "\n",
    "https://github.com/IBM/skillsnetwork/wiki/FAQ#im-feeling-lost-and-confused-please-help-me\n",
    "\n",
    "\n",
    "If running outside Watson Studio, this should work as well. In case you are running in an Apache Spark context outside Watson Studio, please remove the Apache Spark setup in the first notebook cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Markdown, display\n",
    "def printmd(string):\n",
    "    display(Markdown('# <span style=\"color:red\">'+string+'</span>'))\n",
    "\n",
    "\n",
    "if ('sc' in locals() or 'sc' in globals()):\n",
    "    printmd('<<<<<!!!!! It seems that you are running in a IBM Watson Studio Apache Spark Notebook. Please run it in an IBM Watson Studio Default Runtime (without Apache Spark) !!!!!>>>>>')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyspark==2.4.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    from pyspark import SparkContext, SparkConf\n",
    "    from pyspark.sql import SparkSession\n",
    "except ImportError as e:\n",
    "    printmd('<<<<<!!!!! Please restart your kernel after installing Apache Spark !!!!!>>>>>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SparkContext.getOrCreate(SparkConf().setMaster(\"local[*]\"))\n",
    "\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Waiting for a Spark session to start...\n",
      "Spark Initialization Done! ApplicationId = app-20190426153814-0000\n",
      "KERNEL_ID = 40797199-0eef-4565-ba9c-f973863e89f6\n",
      "--2019-04-26 15:38:16--  https://github.com/IBM/coursera/blob/master/coursera_ml/floorsensordata2604.csv.zip?raw=true\n",
      "Resolving github.com (github.com)... 192.30.253.113, 192.30.253.112\n",
      "Connecting to github.com (github.com)|192.30.253.113|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://github.com/IBM/coursera/raw/master/coursera_ml/floorsensordata2604.csv.zip [following]\n",
      "--2019-04-26 15:38:16--  https://github.com/IBM/coursera/raw/master/coursera_ml/floorsensordata2604.csv.zip\n",
      "Reusing existing connection to github.com:443.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/IBM/coursera/master/coursera_ml/floorsensordata2604.csv.zip [following]\n",
      "--2019-04-26 15:38:17--  https://raw.githubusercontent.com/IBM/coursera/master/coursera_ml/floorsensordata2604.csv.zip\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 199.232.8.133\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|199.232.8.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 7960237 (7.6M) [application/zip]\n",
      "Saving to: 'floorsensordata2604.csv.zip?raw=true'\n",
      "\n",
      "floorsensordata2604 100%[===================>]   7.59M  --.-KB/s    in 0.1s    \n",
      "\n",
      "2019-04-26 15:38:17 (69.1 MB/s) - 'floorsensordata2604.csv.zip?raw=true' saved [7960237/7960237]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://github.com/IBM/coursera/blob/master/coursera_ml/floorsensordata2604.csv.zip?raw=true\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mv floorsensordata2604.csv.zip?raw=true floorsensordata2604.csv.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  floorsensordata2604.csv.zip\n",
      "  inflating: floorsensordata2604.csv  \n"
     ]
    }
   ],
   "source": [
    "!unzip floorsensordata2604.csv.zip floorsensordata2604.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = spark.read.option('header',True).csv('floorsensordata2604.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+--------------------+----------+--------------------+------+\n",
      "|_c0|            itemname|sensorname|                time| value|\n",
      "+---+--------------------+----------+--------------------+------+\n",
      "|  0|52760_5_temperatu...|  item0001|2018-02-19T07:05:...|  20.5|\n",
      "|  1|47406_3_temperatu...|  item0002|2018-02-19T07:05:...| 20.61|\n",
      "|  2|452B5_3_temperatu...|  item0003|2018-02-19T07:05:...| 19.89|\n",
      "|  3|452F9_3_temperatu...|  item0004|2018-02-19T07:05:...|  19.7|\n",
      "|  4|4521C_3_temperatu...|  item0005|2018-02-19T07:05:...| 18.89|\n",
      "|  5|51A51_5_temperatu...|  item0006|2018-02-19T07:05:...|21.812|\n",
      "|  6|512F5_5_temperatu...|  item0007|2018-02-19T07:05:...|    21|\n",
      "|  7|45974_3_temperatu...|  item0008|2018-02-19T07:05:...| 21.06|\n",
      "|  8|52686_5_temperatu...|  item0009|2018-02-19T07:05:...|    21|\n",
      "|  9|45152_3_temperatu...|  item0010|2018-02-19T07:05:...|  20.2|\n",
      "| 10|51BEB_5_temperatu...|  item0011|2018-02-19T07:05:...|    21|\n",
      "| 11|45A95_3_temperatu...|  item0012|2018-02-19T07:05:...| 19.26|\n",
      "| 12|516AF_5_temperatu...|  item0013|2018-02-19T07:05:...|  21.5|\n",
      "| 13|51711_5_temperatu...|  item0014|2018-02-19T07:05:...|  20.5|\n",
      "| 14|51B62_5_temperatu...|  item0015|2018-02-19T07:05:...|  21.5|\n",
      "| 15|476CA_3_temperatu...|  item0016|2018-02-19T07:05:...| 20.72|\n",
      "| 16|51174_5_temperatu...|  item0017|2018-02-19T07:05:...|  21.5|\n",
      "| 17|511D3_5_temperatu...|  item0018|2018-02-19T07:05:...|  21.5|\n",
      "| 18|45A94_3_temperatu...|  item0019|2018-02-19T07:05:...| 19.86|\n",
      "| 19|5161B_5_temperatu...|  item0020|2018-02-19T07:05:...|    21|\n",
      "+---+--------------------+----------+--------------------+------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}