Multi-class classification training limit #414
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… that inherit from splitter can use them
…ogrifAI into LocoTestRefactor merge master
…AdamChit/TransmogrifAI into achit/regression-training-limit
…plitterTest.scala Co-Authored-By: Christopher Rupley <[email protected]>
…egressionModelSelectorTest.scala Co-Authored-By: Christopher Rupley <[email protected]>
…AdamChit/TransmogrifAI into achit/regression-training-limit
…params to DataCutterSummary
…et/get fuctions moved to SplitterParams
AdamChit
requested review from
Jauntbox,
leahmcguire,
tovbinm and
wsuchy
as code owners
Codecov Report
@@ Coverage Diff @@
## master #414 +/- ##
==========================================
- Coverage 86.96% 86.92% -0.04%
==========================================
Files 337 337
Lines 11083 11099 +16
Branches 356 593 +237
==========================================
+ Hits 9638 9648 +10
- Misses 1445 1451 +6
Continue to review full report at Codecov.
|
leahmcguire
reviewed
Oct 11, 2019
| summary = Option(DataCutterSummary( | ||
| preSplitterDataCount = dataSetSize, | ||
| downSamplingFraction = getDownSampleFraction, | ||
| labelsKept = getLabelsToKeep, | ||
| labelsDropped = getLabelsToDrop, | ||
| labelsDroppedTotal = getLabelsDroppedTotal | ||
| )) | ||
| PrevalidationVal(summary, Option(dataPrep)) |
There was a problem hiding this comment.
I think it makes more sense to do the downsampling in the pre-validationPrepare than in the validation prepare - the difference being that the validation prepare is called within the CV folds. So since you upsample for binary it needs to be here to prevent label leakage but since there is only downsampling here it can go earlier
There was a problem hiding this comment.
The 2 main reasons behind doing it this way:
- Keeping it consistent with dataSplitter/dataBalancer
- Easier to implement stratified sampling (or other data balancing techniques) in the future which would upsample the minority classes
| @@ -203,7 +233,11 @@ class DataCutter(uid: String = UID[DataCutter]) extends Splitter(uid = uid) with | |||
| s" minLabelFraction = $minLabelFract, maxLabelCategories = $maxLabels. \n" + | |||
| s"Label counts were: ${labelCounts.collect().toSeq}") | |||
| } | |||
| DataCutterSummary(labelsKept.toSeq, labelsDropped.toSeq, labelsDroppedTotal.toLong) | |||
| DataCutterSummary( | |||
| labelsKept = labelsKept.toSeq, | |||
There was a problem hiding this comment.
do you want to add the downsample fraction?
tovbinm
approved these changes
Oct 19, 2019
| val dataPrep = super.validationPrepare(data) | ||
|
|
||
| // check if down sampling is needed | ||
| val balanced: DataFrame = if (getDownSampleFraction < 1) { |
| @@ -273,6 +307,8 @@ private[impl] trait DataCutterParams extends SplitterParams { | |||
| */ | |||
| case class DataCutterSummary | |||
| ( | |||
| preSplitterDataCount: Long = 0, | |||
| @@ -129,6 +129,23 @@ trait SplitterParams extends Params { | |||
| def setReserveTestFraction(value: Double): this.type = set(reserveTestFraction, value) | |||
| def getReserveTestFraction: Double = $(reserveTestFraction) | |||
|
|
|||
| /** | |||
| * Fraction to sample majority data | |||
| * Value should be in ]0.0, 1.0] | |||
| @@ -65,6 +68,8 @@ class DataCutterTest extends FlatSpec with TestSparkContext with SplitterSummary | |||
| s.labelsKept.length shouldBe 1000 | |||
| s.labelsDropped.length shouldBe 0 | |||
| s shouldBe DataCutterSummary( | |||
| preSplitterDataCount = dataSize, | |||
| downSamplingFraction = 1, | |||
There was a problem hiding this comment.
Use 1.0 instead of 1 here and everywhere with Double values
…/AdamChit/TransmogrifAI into achit/multi-class-training-limit
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related issues
DataBalancer for binary classification has a parameter that controls the max data passed into modeling - multiclass classification should allow similar limits
Describe the proposed solution
The solution is to downsample once we reach the training set limit
Describe alternatives you've considered
N/A
Additional context
similar to #413