Sample Spark app to to demonstrate Python packaging and deployment in Spark cluster (Databricks, HDInsight, Local)

./scripts/run_local.sh

make build

First setup Databricks CLI

• Copy egg package and main file (wordcount.py) & files to Databricks filesystem:

dbfs cp dist/spk_sample-1.0.0-py3.6.egg dbfs:/spk_sample/dist/spk_sample-1.0.0-py3.6.egg

dbfs cp spk_sample/wordcount.py dbfs:/spk_sample/wordcount.py

dbfs cp sample.txt dbfs:/spk_sample/sample.txt

• Run Spark submit from Jobs UI with following parameters:

["--py-files","dbfs:/spk_sample/dist/spk_sample-1.0.0-py3.6.egg","dbfs:/spk_sample/wordcount.py", "dbfs:/spk_sample/sample.txt"]

• With spark-submit, Databricks always creates new cluster. You cannot run this job on already running cluster.