Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust bin values for text features in RFF #99

Merged
merged 27 commits into from
Sep 4, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
da5ba89
adjust bin value based on number of token
sxd929 Aug 28, 2018
1d90e8f
cleanup
sxd929 Aug 28, 2018
3b6c45f
address comments
sxd929 Aug 28, 2018
eea1cf2
merge
sxd929 Aug 28, 2018
b958887
add sum and count to summary
sxd929 Aug 29, 2018
a54131b
change formula
sxd929 Aug 29, 2018
dae829d
add todo
sxd929 Aug 29, 2018
d6ad74f
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
d9dee94
use bins as default
sxd929 Aug 30, 2018
1bb28c6
Merge branch 'xs/adjustBinValueForText' of https://github.com/salesfo…
sxd929 Aug 30, 2018
441ab06
Merge branch 'master' into xs/adjustBinValueForText
sxd929 Aug 30, 2018
546fcfc
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
13cb288
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 30, 2018
c47e9bc
address comments
sxd929 Aug 31, 2018
e8d616b
cleanup
sxd929 Aug 31, 2018
e133bb4
Merge branch 'xs/adjustBinValueForText' of https://github.com/salesfo…
sxd929 Aug 31, 2018
20a9c40
cleanup
sxd929 Aug 31, 2018
e39dad6
Merge branch 'master' into xs/adjustBinValueForText
sxd929 Aug 31, 2018
1c6d359
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 31, 2018
5e2857c
Added textBinsFormula to RFF
tovbinm Aug 31, 2018
a2ffd3a
make scalastyle happy
tovbinm Aug 31, 2018
3c35a13
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Aug 31, 2018
2f4f77a
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 1, 2018
6187e55
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 2, 2018
7ff99fe
Merge branch 'master' into xs/adjustBinValueForText
tovbinm Sep 3, 2018
bdc0608
update docs
tovbinm Sep 4, 2018
dfc1a40
Added docs
tovbinm Sep 4, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Added docs
  • Loading branch information
tovbinm committed Sep 4, 2018
commit dfc1a407f1fb75f6d999a188edca7e284daeadc5
5 changes: 3 additions & 2 deletions core/src/main/scala/com/salesforce/op/OpWorkflow.scala
Original file line number Diff line number Diff line change
Expand Up @@ -503,8 +503,9 @@ class OpWorkflow(val uid: String = UID[OpWorkflow]) extends OpWorkflowCore {
* @param protectedFeatures list of features that should never be removed (features that are used to create them will
* also be protected)
* @param textBinsFormula formula to compute the text features bin size.
* Input arguments are [[Summary]] and number of bins to use in computing feature distributions
* (histograms for numerics, hashes for strings). Output is the bins for the text features.
* Input arguments are [[Summary]] and number of bins to use in computing
* feature distributions (histograms for numerics, hashes for strings).
* Output is the bins for the text features.
* @tparam T Type of the data read in
*/
@Experimental
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,10 @@ private[op] object FeatureDistribution {
* @param summary feature summary
* @param value optional processed sequence
* @param bins number of histogram bins
* @param textBinsFormula formula to compute the text features bin size
* @param textBinsFormula formula to compute the text features bin size.
* Input arguments are [[Summary]] and number of bins to use in computing feature distributions
* (histograms for numerics, hashes for strings). Output is the bins for the text features.
* @return a pair consisting of response and predictor feature distributions (in this order)
* @return feature distribution given the provided information
*/
def apply(
Expand Down Expand Up @@ -183,7 +186,10 @@ private[op] object FeatureDistribution {
* @param values values to bin
* @param summary summary info for feature (max, min, etc)
* @param bins number of bins to produce
* @param textBinsFormula formula to compute the text features bin size
* @param textBinsFormula formula to compute the text features bin size.
* Input arguments are [[Summary]] and number of bins to use in computing feature distributions
* (histograms for numerics, hashes for strings). Output is the bins for the text features.
* @return a pair consisting of response and predictor feature distributions (in this order)
* @return the bin information and the binned counts
*/
private def histValues(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -75,8 +75,9 @@ import scala.util.Failure
* @param jsDivergenceProtectedFeatures features that are protected from removal by JS divergence check
* @param protectedFeatures features that are protected from removal
* @param textBinsFormula formula to compute the text features bin size.
* Input arguments are [[Summary]] and number of bins to use in computing feature distributions
* (histograms for numerics, hashes for strings). Output is the bins for the text features.
* Input arguments are [[Summary]] and number of bins to use in computing feature
* distributions (histograms for numerics, hashes for strings).
* Output is the bins for the text features.
* @tparam T datatype of the reader
*/
class RawFeatureFilter[T]
Expand Down