Revert back to Spark 2.3 #399
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… made to decision tree pruning in Spark 2.4. If nodes are split, but both child nodes lead to the same prediction then the split is pruned away. This updates the test so this doesn't happen for feature 'b'
tovbinm
requested review from
gerashegalov,
Jauntbox,
leahmcguire and
wsuchy
as code owners
Codecov Report
@@ Coverage Diff @@
## master #399 +/- ##
===========================================
- Coverage 86.89% 74.87% -12.03%
===========================================
Files 337 337
Lines 11076 11054 -22
Branches 351 590 +239
===========================================
- Hits 9625 8277 -1348
- Misses 1451 2777 +1326
Continue to review full report at Codecov.
|
leahmcguire
approved these changes
Sep 3, 2019
Merged
gerashegalov
added a commit
that referenced
this pull request
Sep 11, 2019
Bug fixes: - Ensure correct metrics despite model failures on some CV folds [#404](#404) - Fix flaky `ModelInsight` tests [#395](#395) - Avoid creating `SparseVector`s for LOCO [#377](#377) New features / updates: - Model combiner [#385](#399) - Added new sample for HousingPrices [#365](#365) - Test to verify that custom metrics appear in model insight metrics [#387](#387) - Add `FeatureDistribution` to `SerializationFormat`s [#383](#383) - Add metadata to `OpStandadrdScaler` to allow for descaling [#378](#378) - Improve json serde error in `evalMetFromJson` [#380](#380) - Track mean & standard deviation as metrics for numeric features and for text length of text features [#354](#354) - Making model selectors robust to failing models [#372](#372) - Use compact and compressed model json by default [#375](#375) - Descale feature contribution for Linear Regression & Logistic Regression [#345](#345) Dependency updates: - Update tika version [#382](#382)
curious to know why we are not ready for spark 2.4? i didnt observe any issues |
|
The main suite of products that use TransmogrifAI @ Salesforce requires Spark 2.3. Once they are ready to get upgrade we will move to 2.4. |
nicodv
added a commit
that referenced
this pull request
Feb 19, 2020
* Revert "Revert back to Spark 2.3 (#399)" This reverts commit 95a77b1. * Update to Spark 2.4.3 and XGBoost 0.90 * special double serializer fix * fix serialization * fix serialization * docs * fixed missng value for test * meta fix * Updated DecisionTreeNumericMapBucketizer test to deal with the change made to decision tree pruning in Spark 2.4. If nodes are split, but both child nodes lead to the same prediction then the split is pruned away. This updates the test so this doesn't happen for feature 'b' * fix params meta test * FIxed failing xgboost test * ident * cleanup * added dataframe reader and writer extensions * added const * cherrypick fixes * added xgboost params + update models to use public predict method * blarg * double ser test * update mleap and spark testing base * Update README.md * type fix * bump minor version * Update Spark version in the README * bump version * Update build.gradle * Update pom.xml * set correct json4s version * upgrade helloworld deps * upgrade notebook deps on TMog and Spark * bump to version 0.7.0 for Spark update * align helloworld dependencies * align helloworld dependencies * get -> getOrElse with exception * fix helloworld compilation * Spark 2.4.5 * Spark 2.4.5 * Spark 2.4.5 * Update OpTitanicSimple.ipynb * Update OpIris.ipynb * Revert "Spark 2.4.5" This reverts commit b3c0a74. * Revert "Spark 2.4.5" This reverts commit f4ab3fd. * Revert "Spark 2.4.5" This reverts commit 50d9dfb. * Revert "Update OpTitanicSimple.ipynb" This reverts commit 3417972. * Revert "Update OpIris.ipynb" This reverts commit df38bcc. Co-authored-by: Christopher Suchanek <[email protected]> Co-authored-by: Kevin Moore <[email protected]> Co-authored-by: Nico de Vos <[email protected]>
nicodv
added a commit
that referenced
this pull request
Jun 11, 2020
* Revert "Revert back to Spark 2.3 (#399)" This reverts commit 95a77b1. * Update to Spark 2.4.3 and XGBoost 0.90 * special double serializer fix * fix serialization * fix serialization * docs * fixed missng value for test * meta fix * Updated DecisionTreeNumericMapBucketizer test to deal with the change made to decision tree pruning in Spark 2.4. If nodes are split, but both child nodes lead to the same prediction then the split is pruned away. This updates the test so this doesn't happen for feature 'b' * fix params meta test * FIxed failing xgboost test * ident * cleanup * added dataframe reader and writer extensions * added const * cherrypick fixes * added xgboost params + update models to use public predict method * blarg * double ser test * update mleap and spark testing base * Update README.md * type fix * bump minor version * Update Spark version in the README * bump version * Update build.gradle * Update pom.xml * set correct json4s version * upgrade helloworld deps * upgrade notebook deps on TMog and Spark * bump to version 0.7.0 for Spark update * align helloworld dependencies * align helloworld dependencies * get -> getOrElse with exception * fix helloworld compilation * style * WIP release notes * TMog version bump * update release notes * update release notes * updates to changelog * updates to changelog * updates to changelog * updates to changelog * updates to changelog * updates to changelog * fix changelog * fix changelog * keep helloworld on 0.6.1 until release Co-authored-by: Matthew Tovbin <[email protected]> Co-authored-by: Matthew Tovbin <[email protected]> Co-authored-by: Christopher Suchanek <[email protected]> Co-authored-by: Kevin Moore <[email protected]> Co-authored-by: Matthew Tovbin <[email protected]>
|
Thanks for the contribution! Before we can merge this, we need @wsuchy to sign the Salesforce.com Contributor License Agreement. |
|
Thanks for the contribution! It looks like @Jauntbox is an internal user so signing the CLA is not required. However, we need to confirm this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related issues
We are not ready for Spark 2.4 (#327)
Describe the proposed solution
Reverting to Spark 2.3 for now.
I will raise another PR with the 2.4 so we can have it ready to go once needed.
Describe alternatives you've considered
N/A