Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

updated string indexer enum to match spark #93

Merged
merged 5 commits into from
Aug 27, 2018
Merged

Conversation

leahmcguire
Copy link
Collaborator

Related issues
OpStringIndexer fails when NoFilter is set for handle invalid because this value doesnt exist in spark

Describe the proposed solution
Update enum to match spark supported valuese

@RunWith(classOf[JUnitRunner])
class OpStringIndexerTest extends FlatSpec with TestSparkContext{

Spec[OpStringIndexer[_]] should "correctly set the wrapped spark stage params" in {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please move OpStringIndexer test portion from OpStringIndexerNoFilterTest to here as well

@@ -256,7 +256,7 @@ trait RichTextFeature {
*/
def indexed(
unseenName: String = OpStringIndexerNoFilter.UnseenNameDefault,
handleInvalid: StringIndexerHandleInvalid = StringIndexerHandleInvalid.NoFilter
handleInvalid: StringIndexerHandleInvalid = StringIndexerHandleInvalid.Keep
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why change the default?

@@ -32,6 +32,7 @@ package com.salesforce.op.stages.impl.feature

import com.salesforce.op.UID
import com.salesforce.op.features.types._
import com.salesforce.op.stages.impl.feature.StringIndexerHandleInvalid.{Keep, Skip}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can do import com.salesforce.op.stages.impl.feature.StringIndexerHandleInvalid._
then assert(Seq(Skip, Error, Keep).contains(value), ...)

Copy link
Collaborator

@tovbinm tovbinm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some minor comments, otherwise lgtm!

@tovbinm
Copy link
Collaborator

tovbinm commented Aug 27, 2018

error file=/home/circleci/repo/core/src/main/scala/com/salesforce/op/stages/impl/feature/OpStringIndexer.scala message=File line length exceeds 120 characters line=66

@codecov
Copy link

codecov bot commented Aug 27, 2018

Codecov Report

Merging #93 into master will increase coverage by 0.51%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #93      +/-   ##
==========================================
+ Coverage   85.61%   86.13%   +0.51%     
==========================================
  Files         292      292              
  Lines        8761     9519     +758     
  Branches      308      333      +25     
==========================================
+ Hits         7501     8199     +698     
- Misses       1260     1320      +60
Impacted Files Coverage Δ
...force/op/stages/impl/feature/OpStringIndexer.scala 100% <100%> (ø) ⬆️
...es/src/main/scala/com/salesforce/op/OpParams.scala 89.58% <0%> (+4.16%) ⬆️
...om/salesforce/op/testkit/FeatureFactoryOwner.scala 100% <0%> (+100%) ⬆️
...a/com/salesforce/op/testkit/PartiallyDefined.scala 100% <0%> (+100%) ⬆️
...in/scala/com/salesforce/op/testkit/RandomSet.scala 66.66% <0%> (+66.66%) ⬆️
...com/salesforce/op/testkit/StandardRandomData.scala 100% <0%> (+100%) ⬆️
...ala/com/salesforce/op/testkit/InfiniteStream.scala 100% <0%> (+100%) ⬆️
...n/scala/com/salesforce/op/testkit/RandomData.scala 100% <0%> (+100%) ⬆️
...ala/com/salesforce/op/testkit/RandomIntegral.scala 100% <0%> (+100%) ⬆️
...com/salesforce/op/testkit/ProbabilityOfEmpty.scala 100% <0%> (+100%) ⬆️
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b0e1480...9ece16b. Read the comment docs.

@tovbinm tovbinm merged commit cba4ac6 into master Aug 27, 2018
@tovbinm tovbinm deleted the lm/stringIndex branch August 27, 2018 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants