Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Specify categorical variables in metadata #120

Merged
merged 16 commits into from
Sep 11, 2018
Merged

Specify categorical variables in metadata #120

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
12ce677
Adding Attributes when converting to Metadata
mweilsalesforce Sep 5, 2018
1ac45eb
Treated Text types as special uses cases
mweilsalesforce Sep 5, 2018
9c7d9ff
FirstTest
mweilsalesforce Sep 5, 2018
f7546db
Tested on DateVectorizers
mweilsalesforce Sep 6, 2018
96a1d8c
Up to GeoLocationTests
mweilsalesforce Sep 6, 2018
0531c68
Up to NumericVectorizerTest
mweilsalesforce Sep 6, 2018
8c64851
Up to OpMapVectorizer
mweilsalesforce Sep 7, 2018
96427de
Up to SmartTextVectorizer
mweilsalesforce Sep 7, 2018
d52131a
Up to URLVectorizerTests
mweilsalesforce Sep 7, 2018
e9e692c
fix scalastyle
mweilsalesforce Sep 7, 2018
3c1ceeb
Merge branch 'master' into mw/categorical-metadata
mweilsalesforce Sep 7, 2018
95d43cc
with AttributeAsserts
mweilsalesforce Sep 7, 2018
186d81f
import com.salesforce.op.features.types._
mweilsalesforce Sep 7, 2018
78edeca
FreatureType.typeName
mweilsalesforce Sep 9, 2018
34f516f
Avoiding `.get`
mweilsalesforce Sep 9, 2018
23e805d
Addressing remaining PR comments
mweilsalesforce Sep 10, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Next Next commit
FreatureType.typeName
  • Loading branch information
@mweilsalesforce
mweilsalesforce committed Sep 9, 2018
commit 78edecaed67ddabe964aab041270ac77c71208a6
10 changes: 6 additions & 4 deletions features/src/main/scala/com/salesforce/op/utils/spark/OpVectorMetadata.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
package com.salesforce.op.utils.spark

import com.salesforce.op.FeatureHistory
import com.salesforce.op.features.types._
import com.salesforce.op.features.types.{FeatureType, _}
import org.apache.spark.ml.attribute.{AttributeGroup, BinaryAttribute, NumericAttribute}
import org.apache.spark.ml.linalg.SQLDataTypes._
import org.apache.spark.sql.types.{Metadata, MetadataBuilder, StructField}
Expand Down Expand Up @@ -75,8 +75,10 @@ class OpVectorMetadata private
newColumns: Array[OpVectorColumnMetadata]
): OpVectorMetadata = OpVectorMetadata(name, newColumns, history)

val textTypes = Seq(MultiPickList, MultiPickListMap, Text, TextArea, TextAreaMap, TextMap, Binary, BinaryMap,
TextList).map(_.getClass.getName.dropRight(1))
val categoricalTypes = Seq(FeatureType.typeName[MultiPickList], FeatureType.typeName[MultiPickListMap],
FeatureType.typeName[Text], FeatureType.typeName[TextArea], FeatureType.typeName[TextAreaMap],
FeatureType.typeName[TextMap], FeatureType.typeName[Binary], FeatureType.typeName[BinaryMap],
FeatureType.typeName[TextList])
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

picklist? Combo box? country, state, city, id

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah If it is only hashing + count, let's remove all these Text types. Do we only do hashing to Combo box, country, state, city, id?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we do pivot - so they should be picked up automatically. I think we also do pivot on multiPickList. So you may want to remove the categorical types check completely and only rely on the indicatorValue


/**
* Serialize to spark metadata
Expand All @@ -96,7 +98,7 @@ class OpVectorMetadata private
.putMetadata(OpVectorMetadata.HistoryKey, FeatureHistory.toMetadata(history))
.build()
val attributes = columns.map { c =>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

val attributes = columns.map {
    case c if c.indicatorValue.isDefined || categoricalTypes.exists(c.parentFeatureType.contains) =>
        BinaryAttribute.defaultAttr.withName(c.makeColName()).withIndex(c.index)
    case c =>
        NumericAttribute.defaultAttr.withName(c.makeColName()).withIndex(c.index)
}

if (c.indicatorValue.isDefined || textTypes.exists(c.parentFeatureType.contains)) {
if (c.indicatorValue.isDefined || categoricalTypes.exists(c.parentFeatureType.contains)) {
BinaryAttribute.defaultAttr.withName(c.makeColName()).withIndex(c.index)
} else {
NumericAttribute.defaultAttr.withName(c.makeColName()).withIndex(c.index)
Expand Down