feat(ingest): salesforce - add connector (#5104)

Co-authored-by: Shirshanka Das <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>

metadata-ingestion/docs/sources/salesforce/salesforce.md

@@ -0,0 +1,30 @@
+### Prerequisites
+
+In order to ingest metadata from Salesforce, you will need:
+
+- Salesforce username, password, [security token](https://developer.Salesforce.com/docs/atlas.en-us.api.meta/api/sforce_api_concepts_security.htm) OR 
+- Salesforce instance url and access token/session id (suitable for one-shot ingestion only, as access token typically expires after 2 hours of inactivity)
+
+## Integration Details
+This plugin extracts Salesforce Standard and Custom Objects and their details (fields, record count, etc) from a Salesforce instance.
+Python library [simple-salesforce](https://pypi.org/project/simple-salesforce/) is used for authenticating and calling [Salesforce REST API](https://developer.Salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/intro_what_is_rest_api.htm) to retrive details from Salesforce instance.
+
+### REST API Resources used in this integration
+- [Versions](https://developer.Salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/resources_versions.htm)
+- [Tooling API Query](https://developer.salesforce.com/docs/atlas.en-us.api_tooling.meta/api_tooling/intro_rest_resources.htm) on objects EntityDefinition, EntityParticle, CustomObject, CustomField
+- [Record Count](https://developer.Salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/resources_record_count.htm)
+
+### Concept Mapping
+
+This ingestion source maps the following Source System Concepts to DataHub Concepts:
+
+| Source Concept | DataHub Concept | Notes |
+| -- | -- | -- |
+| `Salesforce` | [Data Platform](../../metamodel/entities/dataPlatform.md) | |
+|Standard Object | [Dataset](../../metamodel/entities/dataset.md) | subtype "Standard Object" |
+|Custom Object | [Dataset](../../metamodel/entities/dataset.md) | subtype "Custom Object" |
+
+### Caveats
+- This connector has only been tested with Salesforce Developer Edition.
+- This connector only supports table level profiling (Row and Column counts) as of now. Row counts are approximate as returned by [Salesforce RecordCount REST API](https://developer.Salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/resources_record_count.htm).
+- This integration does not support ingesting Salesforce [External Objects](https://developer.Salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_external_objects.htm)

metadata-ingestion/docs/sources/salesforce/salesforce_recipe.yml

@@ -0,0 +1,25 @@
+pipeline_name: my_salesforce_pipeline
+source:
+ type: "salesforce"
+ config:
+ instance_url: "https://mydomain.my.salesforce.com/"
+ username: user@company
+ password: password_for_user
+ security_token: security_token_for_user
+ platform_instance: mydomain-dev-ed
+ domain:
+ sales:
+ allow:
+ - "Opportunity$"
+ - "Lead$"
+
+ object_pattern:
+ allow:
+ - "Account$"
+ - "Opportunity$"
+ - "Lead$"
+
+sink:
+ type: "datahub-rest"
+ config:
+ server: "http:https://localhost:8080"

metadata-ingestion/setup.py

@@ -262,6 +262,7 @@ def get_long_description():
  "redshift": sql_common | redshift_common,
  "redshift-usage": sql_common | usage_common | redshift_common,
  "sagemaker": aws_common,
+ "salesforce":{"simple-salesforce"},
  "snowflake": snowflake_common,
  "snowflake-usage": snowflake_common
  | usage_common
@@ -366,6 +367,7 @@ def get_long_description():
  "starburst-trino-usage",
  "powerbi",
  "vertica",
+ "salesforce"
  # airflow is added below
  ]
  for dependency in plugins[plugin]
@@ -509,6 +511,7 @@ def get_long_description():
  "vertica = datahub.ingestion.source.sql.vertica:VerticaSource",
  "presto-on-hive = datahub.ingestion.source.sql.presto_on_hive:PrestoOnHiveSource",
  "pulsar = datahub.ingestion.source.pulsar:PulsarSource",
+ "salesforce = datahub.ingestion.source.salesforce:SalesforceSource",
  ],
  "datahub.ingestion.sink.plugins": [
  "file = datahub.ingestion.sink.file:FileSink",