Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adjust bin values for text features in RFF #99

Merged
merged 27 commits into from
Sep 4, 2018
Merged

Adjust bin values for text features in RFF #99

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
da5ba89
adjust bin value based on number of token
sxd929 Aug 28, 2018
1d90e8f
cleanup
sxd929 Aug 28, 2018
3b6c45f
address comments
sxd929 Aug 28, 2018
eea1cf2
merge
sxd929 Aug 28, 2018
b958887
add sum and count to summary
sxd929 Aug 29, 2018
a54131b
change formula
sxd929 Aug 29, 2018
dae829d
add todo
sxd929 Aug 29, 2018
d6ad74f
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
d9dee94
use bins as default
sxd929 Aug 30, 2018
1bb28c6
Merge branch 'xs/adjustBinValueForText' of https://github.com/salesfo…
sxd929 Aug 30, 2018
441ab06
Merge branch 'master' into xs/adjustBinValueForText
sxd929 Aug 30, 2018
546fcfc
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
13cb288
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
c47e9bc
address comments
sxd929 Aug 31, 2018
e8d616b
cleanup
sxd929 Aug 31, 2018
e133bb4
Merge branch 'xs/adjustBinValueForText' of https://github.com/salesfo…
sxd929 Aug 31, 2018
20a9c40
cleanup
sxd929 Aug 31, 2018
e39dad6
Merge branch 'master' into xs/adjustBinValueForText
sxd929 Aug 31, 2018
1c6d359
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 31, 2018
5e2857c
Added textBinsFormula to RFF
tovbinm Aug 31, 2018
a2ffd3a
make scalastyle happy
tovbinm Aug 31, 2018
3c35a13
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 31, 2018
2f4f77a
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 1, 2018
6187e55
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 2, 2018
7ff99fe
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 3, 2018
bdc0608
update docs
tovbinm Sep 4, 2018
dfc1a40
Added docs
tovbinm Sep 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cleanup
  • Loading branch information
@sxd929
sxd929 committed Aug 31, 2018
commit 20a9c40332c13ae7ff1bccbe285f9b941702c8dc
9 changes: 0 additions & 9 deletions core/src/main/scala/com/salesforce/op/filters/FeatureDistribution.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,15 +143,6 @@ case class FeatureDistribution
private[op] object FeatureDistribution {

val MaxBins = 100000
val AvgBinValue = 5000
val MaxTokenLowerLimit = 10
val getBins = (sum: Summary, bins: Int) => bins
// Todo: find out the right formula example:
// {
// To catch categoricals
// if (sum.max < MaxTokenLowerLimit) bins
// else math.min(math.max(bins, sum.sum / AvgBinValue), MaxBins).intValue()
// }

implicit val semigroup: Semigroup[FeatureDistribution] = new Semigroup[FeatureDistribution] {
override def plus(l: FeatureDistribution, r: FeatureDistribution) = l.reduce(r)
Expand Down
2 changes: 2 additions & 0 deletions core/src/main/scala/com/salesforce/op/filters/Summary.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ private[op] case class Summary(min: Double, max: Double, sum: Double, count: Dou
*/
def getBinsText(bins: Int): Int = bins
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The user should be able to pass the formula for how to compute the text feature bin size into RawFeatureFilter as a parameter. We should provide a default which for now can be no formula but later can be updated

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

// Todo: find out the right formula example:
// val AvgBinValue = 5000
// val MaxTokenLowerLimit = 10
// {
// To catch categoricals
// if (max < MaxTokenLowerLimit) bins
Expand Down