Sample Spark app to to demonstrate Python packaging and deployment in Spark cluster (Databricks, HDInsight, Local)
./scripts/run_local.sh
make build
First setup Databricks CLI
- Copy egg package and main file (wordcount.py) & files to Databricks filesystem:
dbfs cp dist/spk_sample-1.0.0-py3.6.egg dbfs:/spk_sample/dist/spk_sample-1.0.0-py3.6.egg
dbfs cp spk_sample/wordcount.py dbfs:/spk_sample/wordcount.py
dbfs cp sample.txt dbfs:/spk_sample/sample.txt
- Run Spark submit from Jobs UI with following parameters:
["--py-files","dbfs:/spk_sample/dist/spk_sample-1.0.0-py3.6.egg","dbfs:/spk_sample/wordcount.py", "dbfs:/spk_sample/sample.txt"]
- With
spark-submit
, Databricks always creates new cluster. You cannot run this job on already running cluster.