Spark Playground

Project to learn and experiment with the Spark cluster computing system.

Setup

spark installed at ~/dev/spark-2.2.0-bin-hadoop2.7 and is part of PATH in .bash_profile

export SPARK_HOME=~/dev/spark-2.2.0-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH

$ spark-submit dataframes-and-sql-examples.py

$ docker run -i -t -P --name spark -v /Users/pfeilbr/Dropbox/mac01/Users/brianpfeil/projects/spark-playground:/src sequenceiq/spark:latest bash

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
.vscode		.vscode
data		data
README.md		README.md
dataframes-and-sql-examples.py		dataframes-and-sql-examples.py
main.scala		main.scala
run.sh		run.sh
simple-app.py		simple-app.py