[PIO-20] Merge ActionML fork

thinkingtime · Aug 15, 2016 · eeae40b · eeae40b
1 parent 8565a73
commit eeae40b
Show file tree

Hide file tree

Showing 24 changed files with 719 additions and 79 deletions.
diff --git a/README.md b/README.md
@@ -3,9 +3,8 @@
 [![Build
 Status](https://api.travis-ci.org/apache/incubator-predictionio.svg?branch=develop)](https://travis-ci.org/apache/incubator-predictionio)
 
-PredictionIO is an open source machine learning framework for developers and
-data scientists. It supports event collection, deployment of algorithms,
-evaluation, querying predictive results via REST APIs.
+PredictionIO is an open source machine learning framework for developers,
+data scientists, and end users. It supports event collection, deployment of algorithms, evaluation, querying predictive results via REST APIs. It is based on scalable open source services like Hadoop, Hbase (and other DBs), Elasticsearch, Spark and implements what is called a Lambda Architecture.
 
 To get started, check out http:https://predictionio.incubator.apache.org!
 

diff --git a/RELEASE.md b/RELEASE.md
@@ -1,26 +1,58 @@
-##Release Notes and News
+#Release Notes and News
 
 **Note:** For upgrade instructions please refer to [this page](http:https://predictionio.incubator.apache.org/resources/upgrade/).
 
-###v0.9.6
+## Version History
 
-November, 2015 | [Release Notes](https://github.com/apache/incubator-predictionio/blob/master/RELEASE.md) have been moved to Github and you are reading them. For a detailed list of commits check [this page](https://github.com/apache/incubator-predictionio/commits/master)
+### 0.10.0
+
+Aug ??, 2016
+
+ - Make SSL optional
+ - Merge ActionML fork
+ - First Apache PredictionIO (incubating) release
+
+### 0.9.7-aml (ActionML fork)
+
+Aug 5, 2016
+
+ - changed version id so artifacts don't conflict with naming in the Salesforce sponsored project.
+ - bug fix in memory use during moving window event trim and compaction EventStore data.
+ - update [install.sh](https://github.com/actionml/PredictionIO/blob/master/bin/install.sh) script for single line installs with options that support various combinations required by some templates.
+
+### v0.9.6
+
+April 11, 2015 
+
+For a detailed list of commits check [this page](https://github.com/apache/incubator-predictionio/commits/master)
 
 - Upgrade components for install/runtime to Hbase 1, Spark 1.5.2 PIO still runs on older HBase and Spark back to 1.3.1, upgrading install of Elaticsearch to 1.5.2 since pio run well on it but also runs on older versions.
 - Support for maintaining a moving window of events by discarding old events from the EventStore
 - Support for doing a deploy without creating a Spark Context
 
+### v0.9.6 (ActionML fork)
+
+March 26, 2016
+
+- Upgrade components for install/runtime to Hbase 1.X, Spark 1.5.2 PIO still runs on older HBase and Spark back to 1.3.1, upgrading install of Elasticsearch to 1.5.2 since pio run well on it but also runs on older versions.
+- Support for maintaining a moving window of events by discarding old events from the EventStore
+- Support for doing a deploy without creating a Spark Context
+
 ###v0.9.5 
 
-October 14th, 2015 | [Release Notes](https://github.com/apache/incubator-predictionio/blob/master/RELEASE.md) have been moved to Github and you are reading them. For a detailed list of commits check [this page](https://github.com/apache/incubator-predictionio/commits/v0.9.5)
+October 14th, 2015 
+
+[Release Notes](https://github.com/apache/incubator-predictionio/blob/master/RELEASE.md) have been moved to Github and you are reading them. For a detailed list of commits check [this page](https://github.com/apache/incubator-predictionio/commits/v0.9.5)
 
 - Support batches of events sent to the EventServer as json arrays
 - Support creating an Elasticsearch StorageClient created for an Elasticsearch cluster from variables in pio-env.sh
 - Fixed some errors installing PredictionIO through install.sh when SBT was not correctly downloaded
 
 ###v0.9.4
 
-July 16th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13700)
+July 16th, 2015
+
+[Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13700)
 
 - Support event permissions with different access keys at the Event Server interface
 - Support detection of 3rd party Apache Spark distributions
@@ -29,7 +61,10 @@ July 16th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secure
 
 ###v0.9.3
 
-May 20th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13600)
+May 20th, 2015
+
+
+[Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13600)
 
 - Add support of developing prediction engines in Java
 - Add support of PostgreSQL and MySQL
@@ -39,7 +74,9 @@ May 20th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secure/
 
 ###v0.9.2
 
-April 14th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13500)
+April 14th, 2015
+
+[Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13500)
 
 - Channels in the Event Server
 - Spark 1.3+ support (upgrade to Spark 1.3+ required)
@@ -49,14 +86,18 @@ April 14th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secur
 
 ###v0.9.1
 
-March 17th, 2015 | [Releaes Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13401)
+March 17th, 2015
+
+[Releaes Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13401)
 
 - Improved `pio-start-all`
 - Fixed a bug where `pio build` failed to set PredictionIO dependency version for engine templates
 
 ###v0.9.0
 
-March 4th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13400)
+March 4th, 2015
+
+[Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13400)
 
 - [E-Commerce Recommendation Template](http:https://templates.prediction.io/PredictionIO/template-scala-parallel-ecommercerecommendation) which includes 1) out-of-stock items support 2) new user recommendation 3) unseen items only
 - [Complementary Purchase Template](http:https://templates.prediction.io/PredictionIO/template-scala-parallel-complementarypurchase) for shopping cart recommendation
@@ -65,7 +106,9 @@ March 4th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secure
 
 ###v0.8.6
 
-Feb 10th, 2015 | [Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13300)
+Feb 10th, 2015
+
+[Release Notes](https://predictionio.atlassian.net/jira/secure/ReleaseNote.jspa?projectId=10000&version=13300)
 
 - New engine template - [Product Ranking](/templates/productranking/quickstart/) for personalized product listing
 - [CloudFormation deployment](/system/deploy-cloudformation/) available
diff --git a/bin/install.sh b/bin/install.sh
@@ -18,11 +18,11 @@
 #
 
 OS=`uname`
-PIO_VERSION=0.9.6
-SPARK_VERSION=1.6.0
+PIO_VERSION=0.10.0-SNAPSHOT
+SPARK_VERSION=1.6.2
 # Looks like support for Elasticsearch 2.0 will require 2.0 so deferring
-ELASTICSEARCH_VERSION=1.7.3
-HBASE_VERSION=1.1.2
+ELASTICSEARCH_VERSION=1.7.5
+HBASE_VERSION=1.2.2
 POSTGRES_VERSION=9.4-1204.jdbc41
 MYSQL_VERSION=5.1.37
 PIO_DIR=$HOME/PredictionIO
@@ -35,6 +35,7 @@ DISTRO_OTHER="Other"
 
 PGSQL="PostgreSQL"
 MYSQL="MySQL"
+ES_PGSQL="Elasticsearch + PostgreSQL"
 ES_HB="Elasticsearch + HBase"
 
 # Ask a yes/no question, with a default of "yes".
@@ -102,7 +103,6 @@ if [[ "$OS" = "Linux" && $(cat /proc/1/cgroup) == *cpu:/docker/* ]]; then
  # Java Install
  echo -e "\033[1;36mStarting Java install...\033[0m"
 
- sudo add-apt-repository -y ppa:openjdk-r/ppa
  sudo apt-get update
  sudo apt-get install openjdk-8-jdk libgfortran3 -y
 
@@ -136,9 +136,8 @@ elif [[ "$1" == "-y" ]]; then
  echo -e "\033[1;36mStarting Java install...\033[0m"
 
  # todo: make java installation platform independent
- sudo add-apt-repository -y ppa:openjdk-r/ppa
  sudo apt-get update
- sudo apt-get install openjdk-8-jdk libgfortran3 python-pip -y
+ sudo apt-get install openjdk-7-jdk libgfortran3 python-pip -y
  sudo pip install predictionio
 
  echo -e "\033[1;32mJava install done!\033[0m"
@@ -154,15 +153,18 @@ else
  read -e -p "Vendor path ($pio_dir/vendors): " vendors_dir
  vendors_dir=${vendors_dir:-$pio_dir/vendors}
 
- echo -e "\033[1mPlease choose between the following sources (1, 2 or 3):\033[0m"
- select source_setup in "$PGSQL" "$MYSQL" "$ES_HB"; do
+ echo -e "\033[1mPlease choose between the following sources (1, 2, 3 or 4):\033[0m"
+ select source_setup in "$PGSQL" "$MYSQL" "$ES_PGSQL" "$ES_HB"; do
  case ${source_setup} in
  "$PGSQL")
  break
  ;;
  "$MYSQL")
  break
  ;;
+ "$ES_PGSQL")
+ break
+ ;;
  "$ES_HB")
  break
  ;;
@@ -208,6 +210,11 @@ else
  # MySQL installed by apt-get so no path is printed beforehand
  break
  ;;
+ "$ES_PGSQL")
+ # PostgreSQL installed by apt-get so no path is printed beforehand
+ echo "Elasticsearch: $elasticsearch_dir"
+ break
+ ;;
  "$ES_HB")
  echo "Elasticsearch: $elasticsearch_dir"
  echo "HBase: $hbase_dir"
@@ -244,9 +251,8 @@ else
  echo -e "\033[33mThis script requires superuser access!\033[0m"
  echo -e "\033[33mYou will be prompted for your password by sudo:\033[0m"
 
- sudo add-apt-repository -y ppa:openjdk-r/ppa
  sudo apt-get update
- sudo apt-get install openjdk-8-jdk libgfortran3 python-pip -y
+ sudo apt-get install openjdk-7-jdk libgfortran3 python-pip -y
  sudo pip install predictionio
 
  echo -e "\033[1;32mJava install done!\033[0m"
@@ -287,11 +293,22 @@ echo "JAVA_HOME is now set to: $JAVA_HOME"
 
 # PredictionIO
 echo -e "\033[1;36mStarting PredictionIO setup in:\033[0m $pio_dir"
+
 cd ${TEMP_DIR}
+
 if [[ ! -e ${PIO_FILE} ]]; then
  echo "Downloading PredictionIO..."
- curl -OL https://github.com/PredictionIO/PredictionIO/releases/download/v${PIO_VERSION}/${PIO_FILE}
+ curl -OL https://codeload.github.com/apache/incubator-predictionio/tar.gz/develop
+
+ tar zxf incubator-predictionio-develop.tar.gz 
+
+ mv incubator-predictionio-develop PredictionIO-${PIO_VERSION}
+
+ sh PredictionIO-${PIO_VERSION}/make-distribution.sh
+ cp PredictionIO-${PIO_VERSION}/${PIO_FILE} ${TEMP_DIR}
+ rm -r PredictionIO-${PIO_VERSION}
 fi
+
 tar zxf ${PIO_FILE}
 rm -rf ${pio_dir}
 mv PredictionIO-${PIO_VERSION} ${pio_dir}
@@ -312,7 +329,7 @@ mkdir ${vendors_dir}
 echo -e "\033[1;36mStarting Spark setup in:\033[0m $spark_dir"
 if [[ ! -e spark-${SPARK_VERSION}-bin-hadoop2.6.tgz ]]; then
  echo "Downloading Spark..."
- curl -O http:https://d3kbcqa49mib13.cloudfront.net/spark-${SPARK_VERSION}-bin-hadoop2.6.tgz
+ curl -O http:https://www-us.apache.org/dist/spark/spark-${SPARK_VERSION}/spark-${SPARK_VERSION}-bin-hadoop2.6.tgz
 fi
 tar xf spark-${SPARK_VERSION}-bin-hadoop2.6.tgz
 rm -rf ${spark_dir}
@@ -323,11 +340,10 @@ ${SED_CMD} "s|SPARK_HOME=.*|SPARK_HOME=$spark_dir|g" ${pio_dir}/conf/pio-env.sh
 
 echo -e "\033[1;32mSpark setup done!\033[0m"
 
-case $source_setup in
- "$PGSQL")
- if [[ ${distribution} = "$DISTRO_DEBIAN" ]]; then
+installPGSQL () {
+ if [[ ${distribution} = "$DISTRO_DEBIAN" ]]; then
  echo -e "\033[1;36mInstalling PostgreSQL...\033[0m"
- sudo apt-get install postgresql -y
+ sudo apt-get install postgresql-9.4 -y
  echo -e "\033[1;36mPlease use the default password 'pio' when prompted to enter one\033[0m"
  sudo -u postgres createdb pio
  sudo -u postgres createuser -P pio
@@ -339,6 +355,34 @@ case $source_setup in
  fi
  curl -O https://jdbc.postgresql.org/download/postgresql-${POSTGRES_VERSION}.jar
  mv postgresql-${POSTGRES_VERSION}.jar ${PIO_DIR}/lib/
+
+ echo "Updating: $pio_dir/conf/pio-env.sh"
+ ${SED_CMD} "s|PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL|PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH|" ${pio_dir}/conf/pio-env.sh
+ echo -e "\033[1;32mPGSQL setup done!\033[0m"
+}
+
+installES() {
+ echo -e "\033[1;36mStarting Elasticsearch setup in:\033[0m $elasticsearch_dir"
+ if [[ -e elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz ]]; then
+ if confirm "Delete existing elasticsearch-$ELASTICSEARCH_VERSION.tar.gz?"; then
+ rm elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz
+ fi
+ fi
+ if [[ ! -e elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz ]]; then
+ echo "Downloading Elasticsearch..."
+ curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz
+ fi
+ tar zxf elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz
+ rm -rf ${elasticsearch_dir}
+ mv elasticsearch-${ELASTICSEARCH_VERSION} ${elasticsearch_dir}
+
+ echo "Updating: $elasticsearch_dir/config/elasticsearch.yml"
+ echo 'network.host: 127.0.0.1' >> ${elasticsearch_dir}/config/elasticsearch.yml
+}
+
+case $source_setup in
+ "$PGSQL")
+ installPGSQL
  ;;
  "$MYSQL")
  if [[ ${distribution} = "$DISTRO_DEBIAN" ]]; then
@@ -362,18 +406,7 @@ case $source_setup in
  ;;
  "$ES_HB")
  # Elasticsearch
- echo -e "\033[1;36mStarting Elasticsearch setup in:\033[0m $elasticsearch_dir"
- if [[ ! -e elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz ]]; then
- echo "Downloading Elasticsearch..."
- curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz
- fi
- tar zxf elasticsearch-${ELASTICSEARCH_VERSION}.tar.gz
- rm -rf ${elasticsearch_dir}
- mv elasticsearch-${ELASTICSEARCH_VERSION} ${elasticsearch_dir}
-
- echo "Updating: $elasticsearch_dir/config/elasticsearch.yml"
- echo 'network.host: 127.0.0.1' >> ${elasticsearch_dir}/config/elasticsearch.yml
-
+ installES
  echo "Updating: $pio_dir/conf/pio-env.sh"
  ${SED_CMD} "s|PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=PGSQL|PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH|" ${pio_dir}/conf/pio-env.sh
  ${SED_CMD} "s|PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=PGSQL|PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS|" ${pio_dir}/conf/pio-env.sh
@@ -382,7 +415,6 @@ case $source_setup in
  ${SED_CMD} "s|# PIO_STORAGE_SOURCES_LOCALFS|PIO_STORAGE_SOURCES_LOCALFS|" ${pio_dir}/conf/pio-env.sh
  ${SED_CMD} "s|# PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE|PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE|" ${pio_dir}/conf/pio-env.sh
  ${SED_CMD} "s|# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=.*|PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$elasticsearch_dir|" ${pio_dir}/conf/pio-env.sh
-
  echo -e "\033[1;32mElasticsearch setup done!\033[0m"
 
  # HBase

diff --git a/bin/pio-start-all b/bin/pio-start-all
@@ -60,6 +60,26 @@ else
  exit 1
 fi
 
+#PGSQL
+pgsqlStatus="$(ps auxwww | grep postgres | wc -l)"
+if [[ "$pgsqlStatus" < 5 ]]; then
+ # Detect OS
+ OS=`uname`
+ if [[ "$OS" = "Darwin" ]]; then
+ pg_cmd=`which pg_ctl`
+ if [[ "$pg_cmd" != "" ]]; then
+ pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
+ fi
+ elif [[ "$OS" = "Linux" ]]; then
+ sudo service postgresql start
+ else
+ echo -e "\033[1;31mYour OS $OS is not yet supported for automatic postgresql startup:(\033[0m"
+ echo -e "\033[1;31mPlease do a manual startup!\033[0m"
+ ${PIO_HOME}/bin/pio-stop-all
+ exit 1
+ fi
+fi
+
 # PredictionIO Event Server
 echo "Waiting 10 seconds for HBase to fully initialize..."
 sleep 10

diff --git a/bin/pio-stop-all b/bin/pio-stop-all
@@ -46,3 +46,18 @@ if [ -e ${PIDFILE} ]; then
  cat ${PIDFILE} | xargs kill
  rm ${PIDFILE}
 fi
+
+#PGSQL
+OS=`uname`
+if [[ "$OS" = "Darwin" ]]; then
+ pg_cmd=`which pg_ctl`
+ if [[ "$pg_cmd" != "" ]]; then
+ pg_ctl -D /usr/local/var/postgres stop -s -m fast
+ fi
+elif [[ "$OS" = "Linux" ]]; then
+ sudo service postgresql stop
+else
+ echo -e "\033[1;31mYour OS $OS is not yet supported for automatic postgresql startup:(\033[0m"
+ echo -e "\033[1;31mPlease do a manual shutdown!\033[0m"
+ exit 1
+fi
diff --git a/build.sbt b/build.sbt
@@ -19,7 +19,7 @@ import UnidocKeys._
 
 name := "pio"
 
-version in ThisBuild := "0.9.7-SNAPSHOT"
+version in ThisBuild := "0.10.0-SNAPSHOT"
 
 organization in ThisBuild := "org.apache.predictionio"
 
@@ -190,3 +190,8 @@ concurrentRestrictions in Global := Seq(
  Tags.limitAll( 1 )
 )
 
+parallelExecution := false
+
+parallelExecution in Global := false
+
+testOptions in Test += Tests.Argument("-oDF")