Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adjust bin values for text features in RFF #99

Merged
merged 27 commits into from
Sep 4, 2018
Merged

Adjust bin values for text features in RFF #99

Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
da5ba89
adjust bin value based on number of token
sxd929 Aug 28, 2018
1d90e8f
cleanup
sxd929 Aug 28, 2018
3b6c45f
address comments
sxd929 Aug 28, 2018
eea1cf2
merge
sxd929 Aug 28, 2018
b958887
add sum and count to summary
sxd929 Aug 29, 2018
a54131b
change formula
sxd929 Aug 29, 2018
dae829d
add todo
sxd929 Aug 29, 2018
d6ad74f
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
d9dee94
use bins as default
sxd929 Aug 30, 2018
1bb28c6
Merge branch 'xs/adjustBinValueForText' of https://github.com/salesfo…
sxd929 Aug 30, 2018
441ab06
Merge branch 'master' into xs/adjustBinValueForText
sxd929 Aug 30, 2018
546fcfc
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
13cb288
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
c47e9bc
address comments
sxd929 Aug 31, 2018
e8d616b
cleanup
sxd929 Aug 31, 2018
e133bb4
Merge branch 'xs/adjustBinValueForText' of https://github.com/salesfo…
sxd929 Aug 31, 2018
20a9c40
cleanup
sxd929 Aug 31, 2018
e39dad6
Merge branch 'master' into xs/adjustBinValueForText
sxd929 Aug 31, 2018
1c6d359
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 31, 2018
5e2857c
Added textBinsFormula to RFF
tovbinm Aug 31, 2018
a2ffd3a
make scalastyle happy
tovbinm Aug 31, 2018
3c35a13
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 31, 2018
2f4f77a
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 1, 2018
6187e55
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 2, 2018
7ff99fe
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 3, 2018
bdc0608
update docs
tovbinm Sep 4, 2018
dfc1a40
Added docs
tovbinm Sep 4, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add todo
  • Loading branch information
@sxd929
sxd929 committed Aug 29, 2018
commit dae829d3eb1c3081c2068161215bf9fbb2f52849
1 change: 1 addition & 0 deletions core/src/main/scala/com/salesforce/op/filters/FeatureDistribution.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,7 @@ private[op] object FeatureDistribution {
case Left(seq) => {
val numBins = getBins(sum, bins)

// Todo: creating too many hashers may cause problem, efficiency, garbage collection etc
val hasher: HashingTF = new HashingTF(numFeatures = numBins)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have to create hasher every time or perhaps we can create it once?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the hashing dimension can be different for different features, the minimum number would be bins
but we can have a shared one with numFeatures = bins, and use that for every case if there is not too many tokens; OR we can create a couple shared hashers with different scales, and choose one based on the scale of token numbers
What do you think? I was assuming creating a hasher for every feature will not be very resource consuming

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let see if we can reuse the hashing function without creating HashingTF everytime.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and I think we can. See HashingTF.transform and HashingTF object

.setBinary(false)
.setHashAlgorithm(HashAlgorithm.MurMur3.toString.toLowerCase)
Expand Down