Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add metadata to OpStandardScaler to allow for descaling #378

Merged
merged 11 commits into from
Aug 5, 2019
Next Next commit
Add metadata to OpScalarStandardScaler
  • Loading branch information
Erica Chiu committed Aug 1, 2019
commit e5eb1376a2a34da44540eda05172a438082a35a9
Original file line number Diff line number Diff line change
Expand Up @@ -60,9 +60,17 @@ class OpScalarStandardScaler
val internalScaler = new MLLibStandardScaler(withMean = estimator.getWithMean, withStd = estimator.getWithStd)
val scalerModel = internalScaler.fit(vecData)

val std = scalerModel.std.toArray
erica-chiu marked this conversation as resolved.
Show resolved Hide resolved
val mean = scalerModel.mean.toArray
val stdMean = std.sum / std.length
erica-chiu marked this conversation as resolved.
Show resolved Hide resolved
val meanMean = mean.sum / mean.length
val scalingArgs = LinearScalerArgs(1 / stdMean, meanMean / stdMean)
val meta = ScalerMetadata(ScalingType.Linear, scalingArgs).toMetadata()
setMetadata(meta)

new OpScalarStandardScalerModel(
std = scalerModel.std.toArray,
mean = scalerModel.mean.toArray,
std = std,
mean = mean,
withStd = scalerModel.withStd,
withMean = scalerModel.withMean,
operationName = operationName,
Expand Down