Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.7.0 release #481

Merged
merged 70 commits into from
Jun 11, 2020
Merged
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
77e2229
Revert "Revert back to Spark 2.3 (#399)"
tovbinm Sep 4, 2019
485fcd5
Update to Spark 2.4.3 and XGBoost 0.90
tovbinm May 30, 2019
12f5333
special double serializer fix
tovbinm May 30, 2019
a4ee986
fix serialization
tovbinm May 30, 2019
47703fa
fix serialization
tovbinm May 30, 2019
ea4b11f
docs
tovbinm May 30, 2019
ab81e31
fixed missng value for test
wsuchy May 30, 2019
c017869
meta fix
tovbinm May 30, 2019
99ea7e1
Updated DecisionTreeNumericMapBucketizer test to deal with the change…
Jauntbox May 31, 2019
daa2672
fix params meta test
tovbinm May 31, 2019
7d3ebb7
FIxed failing xgboost test
wsuchy May 31, 2019
5661640
ident
tovbinm May 31, 2019
8852d69
cleanup
tovbinm May 31, 2019
2ab8924
added dataframe reader and writer extensions
tovbinm Jun 3, 2019
7c8b988
added const
tovbinm Jun 3, 2019
d98c8a9
cherrypick fixes
nicodv Oct 14, 2019
cce0d8f
added xgboost params + update models to use public predict method
tovbinm Jun 21, 2019
8afbae7
blarg
tovbinm Jun 21, 2019
e8770f6
double ser test
tovbinm Jun 21, 2019
afffc56
update mleap and spark testing base
tovbinm Jun 24, 2019
ed43719
Update README.md
tovbinm Jul 16, 2019
8804acd
type fix
nicodv Oct 14, 2019
a1461c7
Merge branch 'master' into ndv/spark2.4
nicodv Oct 15, 2019
b54d0f5
bump minor version
nicodv Oct 15, 2019
861f862
Merge branch 'master' into ndv/spark2.4
tovbinm Oct 18, 2019
41f9dc4
Merge branch 'master' into revert-399-mt/revert-spark-2.4
tovbinm Oct 18, 2019
cb4cb7b
Update Spark version in the README
tovbinm Oct 18, 2019
abca58b
bump version
nicodv Oct 18, 2019
ab8daf8
Merge remote-tracking branch 'origin/ndv/spark2.4' into ndv/spark2.4
nicodv Oct 18, 2019
e58da6c
Update build.gradle
tovbinm Oct 19, 2019
37614d7
Update pom.xml
tovbinm Oct 19, 2019
e8a27ea
Merge branch 'master' into revert-399-mt/revert-spark-2.4
tovbinm Oct 19, 2019
e205856
Merge branch 'ndv/spark2.4' into revert-399-mt/revert-spark-2.4
nicodv Oct 21, 2019
f93db0b
Merge branch 'master' into revert-399-mt/revert-spark-2.4
nicodv Oct 21, 2019
505a881
set correct json4s version
nicodv Nov 5, 2019
e21d06c
Merge branch 'master' into revert-399-mt/revert-spark-2.4
tovbinm Nov 7, 2019
84bb70a
Merge branch 'master' into revert-399-mt/revert-spark-2.4
nicodv Nov 9, 2019
08db1db
Merge remote-tracking branch 'origin/revert-399-mt/revert-spark-2.4' …
nicodv Nov 9, 2019
3ba95b6
upgrade helloworld deps
nicodv Nov 15, 2019
d77515c
upgrade notebook deps on TMog and Spark
nicodv Nov 15, 2019
d3b53be
Merge branch 'master' into revert-399-mt/revert-spark-2.4
tovbinm Nov 26, 2019
1634927
Merge branch 'master' into revert-399-mt/revert-spark-2.4
nicodv Dec 5, 2019
b8a43b5
bump to version 0.7.0 for Spark update
nicodv Dec 5, 2019
094be05
align helloworld dependencies
nicodv Dec 5, 2019
cc7de97
Merge remote-tracking branch 'origin/revert-399-mt/revert-spark-2.4' …
nicodv Dec 5, 2019
4bd5977
align helloworld dependencies
nicodv Dec 5, 2019
46fc60a
get -> getOrElse with exception
nicodv Dec 6, 2019
0211fde
Merge branch 'master' into revert-399-mt/revert-spark-2.4
nicodv Dec 6, 2019
7147b72
Merge branch 'master' into revert-399-mt/revert-spark-2.4
nicodv Jan 7, 2020
4d686c7
fix helloworld compilation
nicodv Jan 7, 2020
9fcc3ae
style
nicodv Jan 7, 2020
2bf8151
WIP release notes
nicodv Jan 8, 2020
1992d0d
TMog version bump
nicodv Jan 8, 2020
3d2a67b
Merge branch 'master' into 0.7.0-release
nicodv Jan 21, 2020
c3067ca
update release notes
nicodv Jan 21, 2020
74d4ee4
update release notes
nicodv Jan 27, 2020
2569bd8
Merge branch 'master' into 0.7.0-release
nicodv Apr 1, 2020
496837c
updates to changelog
nicodv Apr 1, 2020
7fd49db
Merge branch 'master' into 0.7.0-release
nicodv May 13, 2020
323fbed
updates to changelog
nicodv May 13, 2020
2e9d226
updates to changelog
nicodv May 13, 2020
2b56541
updates to changelog
nicodv May 14, 2020
63cbc83
Merge branch 'master' into 0.7.0-release
nicodv May 20, 2020
d284b74
Merge branch 'master' into 0.7.0-release
nicodv Jun 2, 2020
76cb0e6
updates to changelog
nicodv Jun 2, 2020
ff021ee
Merge branch 'master' into 0.7.0-release
nicodv Jun 11, 2020
59db865
updates to changelog
nicodv Jun 11, 2020
4480745
fix changelog
nicodv Jun 11, 2020
be6ad78
fix changelog
nicodv Jun 11, 2020
e292501
keep helloworld on 0.6.1 until release
nicodv Jun 11, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
FIxed failing xgboost test
  • Loading branch information
wsuchy authored and nicodv committed Oct 14, 2019
commit 7d3ebb7344e5c5dc791770eecc6bed89b095a23a
Original file line number Diff line number Diff line change
Expand Up @@ -134,27 +134,22 @@ class OpClassifierModelTest extends FlatSpec with TestSparkContext with OpXGBoos
.setLabelCol(labelF.name)
val spk = cl.fit(rawDF)
val op = toOP(spk, spk.uid).setInput(labelF, featureV)
compareOutputs(spk.transform(rawDF), op.transform(rawDF), false)
compareOutputs(spk.transform(rawDF), op.transform(rawDF))
}

def compareOutputs
(
df1: DataFrame,
df2: DataFrame,
fullRawPred: Boolean = true
)(implicit arrayEquality: Equality[Array[Double]]): Unit = {
def compareOutputs(df1: DataFrame, df2: DataFrame)(implicit arrayEquality: Equality[Array[Double]]): Unit = {
def keysStartsWith(name: String, value: Map[String, Double]): Array[Double] = {
val names = value.keys.filter(_.startsWith(name)).toArray.sorted
names.map(value)
}

val sorted1 = df1.collect().sortBy(_.getAs[Double](4))
val sorted2 = df2.collect().sortBy(_.getAs[Map[String, Double]](2)(Prediction.Keys.PredictionName))
sorted1.zip(sorted2).foreach{ case (r1, r2) =>
sorted1.zip(sorted2).foreach { case (r1, r2) =>
val map = r2.getAs[Map[String, Double]](2)
r1.getAs[Double](4) shouldEqual map(Prediction.Keys.PredictionName)
r1.getAs[Vector](3).toArray shouldEqual keysStartsWith(Prediction.Keys.ProbabilityName, map)
if (fullRawPred) r1.getAs[Vector](2).toArray shouldEqual keysStartsWith(Prediction.Keys.RawPredictionName, map)
else r1.getAs[Vector](2).toArray shouldEqual keysStartsWith(Prediction.Keys.RawPredictionName, map).tail
r1.getAs[Vector](2).toArray shouldEqual keysStartsWith(Prediction.Keys.RawPredictionName, map)
}
}

Expand Down