Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metadata to OpStandardScaler to allow for descaling #378

Merged
merged 11 commits into from
Aug 5, 2019
Merged

Add metadata to OpStandardScaler to allow for descaling #378

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e5eb137
Add metadata to OpScalarStandardScaler
Aug 1, 2019
804a7e7
Add test for descaling to OpScalarStandardScaler tests
Aug 1, 2019
7459f31
Fix scalastyle
Aug 1, 2019
007467e
Merge branch 'master' into ec/standardMetadata
Aug 1, 2019
8c22ace
Merge branch 'master' into ec/standardMetadata
Aug 1, 2019
c6deab5
Improve tests
Aug 2, 2019
560e9c1
Merge branch 'ec/standardMetadata' of https://github.com/salesforce/T…
Aug 2, 2019
2b02ec8
Merge branch 'master' into ec/standardMetadata
Aug 2, 2019
36edde1
Add explanation to selection of std and mean
Aug 5, 2019
f1a184e
Merge branch 'ec/standardMetadata' of https://github.com/salesforce/T…
Aug 5, 2019
4674762
Add clarification
Aug 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Add metadata to OpScalarStandardScaler
  • Loading branch information
Erica Chiu committed Aug 1, 2019
commit e5eb1376a2a34da44540eda05172a438082a35a9
12 changes: 10 additions & 2 deletions core/src/main/scala/com/salesforce/op/stages/impl/feature/OpScalarStandardScaler.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,17 @@ class OpScalarStandardScaler
val internalScaler = new MLLibStandardScaler(withMean = estimator.getWithMean, withStd = estimator.getWithStd)
val scalerModel = internalScaler.fit(vecData)

val std = scalerModel.std.toArray
erica-chiu marked this conversation as resolved.
Show resolved Hide resolved
val mean = scalerModel.mean.toArray
val stdMean = std.sum / std.length
erica-chiu marked this conversation as resolved.
Show resolved Hide resolved
val meanMean = mean.sum / mean.length
val scalingArgs = LinearScalerArgs(1 / stdMean, meanMean / stdMean)
val meta = ScalerMetadata(ScalingType.Linear, scalingArgs).toMetadata()
setMetadata(meta)

new OpScalarStandardScalerModel(
std = scalerModel.std.toArray,
mean = scalerModel.mean.toArray,
std = std,
mean = mean,
withStd = scalerModel.withStd,
withMean = scalerModel.withMean,
operationName = operationName,
Expand Down