forked from datahub-project/datahub
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
doc(ingestion): looker & lookml ingestion guide (datahub-project#8006)
Co-authored-by: MohdSiddiqueBagwan <[email protected]> Co-authored-by: Hyejin Yoon <[email protected]>
- Loading branch information
3 people
committed
Sep 26, 2023
1 parent
54ec12a
commit 2e1afaf
Showing
4 changed files
with
427 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,212 @@ | ||
| --- | ||
| title: Configuration | ||
| --- | ||
| # Configuring Looker & LookML Connector | ||
|
|
||
| Now that you have created a DataHub-specific API key with the relevant access in [the prior step](setup.md), it's time to set up a connection via the DataHub UI. | ||
|
|
||
| ## Configure Secrets | ||
|
|
||
| You must create two secrets to configure a connection with Looker or LookerML. | ||
|
|
||
| * `LOOKER_CLIENT_ID` | ||
| * `LOOKER_CLIENT_SECRET` | ||
|
|
||
| On your DataHub instance, navigate to the **Ingestion** tab in your screen's top right corner. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="Navigate to the "Ingestion Tab"" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/common/common_ingestion_ingestion_button.png"/> | ||
| </p> | ||
|
|
||
| :::note | ||
| If you do not see the Ingestion tab, please get in touch with your DataHub admin to grant you the correct permissions. | ||
| ::: | ||
|
|
||
| Navigate to the **Secrets** tab and click **Create new secret**. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="Secrets Tab" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/common/common_ingestion_secrets_tab.png"/> | ||
| </p> | ||
|
|
||
| First, create a secret for the **Client Id**. The value should be the **Client Id** of the API key created in the [prior step](http:https://localhost:3000/docs/next/quick-ingestion-guides/looker/setup#create-an-api-key). | ||
|
|
||
| <p align="center"> | ||
| <img width="70%" alt="API Key Client ID" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/looker-client-id-secret.png"/> | ||
| </p> | ||
|
|
||
| Then, create a secret for the **Client Secret**. The value should be the **Client Secret** of the API key created in the [prior step](http:https://localhost:3000/docs/next/quick-ingestion-guides/looker/setup#create-an-api-key). | ||
|
|
||
| <p align="center"> | ||
| <img width="70%" alt="API Key client secret" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/looker-client-secret.png"/> | ||
| </p> | ||
|
|
||
|
|
||
| ## Configure Looker Ingestion | ||
|
|
||
| ### Configure Recipe | ||
|
|
||
| Navigate to the **Sources** tab and click **Create new source**. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="Click "Create new source"" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/common/common_ingestion_click_create_new_source_button.png"/> | ||
| </p> | ||
|
|
||
| Choose `Looker`. | ||
|
|
||
| <p align="center"> | ||
| <img width="70%" alt="Select Looker from the options" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/looker-choose-looker.png"/> | ||
| </p> | ||
|
|
||
| Enter the details into the Looker Recipe. | ||
|
|
||
| * **Base URL:** This is your looker instance URL. (i.e. `https://<your-looker-instance>.cloud.looker.com`) | ||
| * **Client ID:** Use the secret LOOKER_CLIENT_ID with the format `${LOOKER_CLIENT_ID}`. | ||
| * **Client Secret:** Use the secret LOOKER_CLIENT_SECRET with the format `${LOOKER_CLIENT_SECRET}`. | ||
|
|
||
|
|
||
| Optionally, use the `dashboard_pattern` and `chart_pattern` fields to filter for specific dashboard and chart. | ||
|
|
||
| config: | ||
| ... | ||
| dashboard_pattern: | ||
| allow: | ||
| - "2" | ||
| chart_pattern: | ||
| allow: | ||
| - "258829b1-82b1-4bdb-b9fb-6722c718bbd3" | ||
|
|
||
| Your recipe should look something like this: | ||
|
|
||
| <p align="center"> | ||
| <img width="70%" alt="Looker Recipe" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/looker-ingestion-source-recipe.png"/> | ||
| </p> | ||
|
|
||
| After completing the recipe, click **Next**. | ||
|
|
||
| ### Schedule Execution | ||
|
|
||
| Now, it's time to schedule a recurring ingestion pipeline to extract metadata from your Looker instance regularly. | ||
|
|
||
| Decide how regularly you want this ingestion to run-- day, month, year, hour, minute, etc. Select from the dropdown. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="schedule selector" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/common/common_ingestion_set_execution_schedule.png"/> | ||
| </p> | ||
|
|
||
| Ensure you've configured your correct timezone. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="timezone_selector" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/common/common_ingestion_set_execution_timezone.png"/> | ||
| </p> | ||
|
|
||
| Finally, click **Next** when you are done. | ||
|
|
||
| ### Finish Up | ||
|
|
||
| Name your ingestion source, then click **Save and Run**. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="Name your ingestion" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/looker-ingestion-source-window.png"/> | ||
| </p> | ||
|
|
||
| You will now find your new ingestion source running. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="ingestion_running" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/looker-ingestion-running.png"/> | ||
| </p> | ||
|
|
||
| ## Configure LookML Connector | ||
|
|
||
| Now that you have created a DataHub-specific API key and Deploy Key with the relevant access in [the prior step](setup.md), it's time to set up a connection via the DataHub UI. | ||
|
|
||
| ### Configure Recipe | ||
|
|
||
| Navigate to the **Sources** tab and click **Create new source**. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="Click "Create new source"" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/common/common_ingestion_click_create_new_source_button.png"/> | ||
| </p> | ||
|
|
||
| Choose `LooML`. | ||
|
|
||
| <p align="center"> | ||
| <img width="70%" alt="Select Looker from the options" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/lookml-choose-lookml.png"/> | ||
| </p> | ||
|
|
||
| Enter the details into the Looker Recipe. You need to set a minimum 5 fields in the recipe for this quick ingestion guide: | ||
|
|
||
| * **GitHub Repository:** This is your GitHub repository where LookML models are stored. You can provide the full URL (example: https://gitlab.com/gitlab-org/gitlab) or organization/repo; in this case, the connector assume it is a GitHub repo | ||
| * **GitHub Deploy Key:** Copy the content of `looker_datahub_deploy_key` and paste into this filed. | ||
| * **Looker Base URL:** This is your looker instance URL. (i.e. https://abc.cloud.looker.com) | ||
| * **Looker Client ID:** Use the secret LOOKER_CLIENT_ID with the format `${LOOKER_CLIENT_ID}`. | ||
| * **Looker Client Secret:** Use the secret LOOKER_CLIENT_SECRET with the format `${LOOKER_CLIENT_SECRET}`. | ||
|
|
||
| Your recipe should look something like this: | ||
|
|
||
| <p align="center"> | ||
| <img width="70%" alt="LookML Recipe" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/lookml-ingestion-source-recipe.png"/> | ||
| </p> | ||
|
|
||
|
|
||
| After completing the recipe, click **Next**. | ||
|
|
||
| ### Schedule Execution | ||
|
|
||
| Now, it's time to schedule a recurring ingestion pipeline to extract metadata from your Looker instance regularly. | ||
|
|
||
| Decide how regularly you want this ingestion to run-- day, month, year, hour, minute, etc. Select from the dropdown. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="schedule selector" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/common/common_ingestion_set_execution_schedule.png"/> | ||
| </p> | ||
|
|
||
| Ensure you've configured your correct timezone. | ||
| <p align="center"> | ||
| <img width="75%" alt="timezone_selector" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/common/common_ingestion_set_execution_timezone.png"/> | ||
| </p> | ||
|
|
||
| Click **Next** when you are done. | ||
|
|
||
| ### Finish Up | ||
|
|
||
| Name your ingestion source, then click **Save and Run**. | ||
| <p align="center"> | ||
| <img width="75%" alt="Name your ingestion" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/lookml-ingestion-source-window.png"/> | ||
| </p> | ||
|
|
||
| You will now find your new ingestion source running. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="ingestion_running" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/lookml-ingestion-running.png"/> | ||
| </p> | ||
|
|
||
| ## Validate Ingestion Runs | ||
|
|
||
| View the latest status of ingestion runs on the Ingestion page. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="ingestion succeeded" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/lookml-ingestion-succeeded.png"/> | ||
| </p> | ||
|
|
||
| Click the `+` sign to expand the complete list of historical runs and outcomes; click **Details** to see the results of a specific run. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="ingestion_details" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/lookml-ingestion-history.png"/> | ||
| </p> | ||
|
|
||
| From the Ingestion Run Details page, pick **View All** to see which entities were ingested. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="ingestion_details_view_all" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/lookml-ingestion-detail.png"/> | ||
| </p> | ||
|
|
||
| Pick an entity from the list to manually validate if it contains the detail you expected. | ||
|
|
||
| <p align="center"> | ||
| <img width="75%" alt="ingestion_details_view_all" src="https://raw.githubusercontent.com/datahub-project/static-assets/main/imgs/guides/looker/lookml-ingestion-assets.png"/> | ||
| </p> | ||
|
|
||
|
|
||
| **Congratulations!** You've successfully set up Looker & LookML as an ingestion source for DataHub! | ||
|
|
||
| *Need more help? Join the conversation in [Slack](http:https://slack.datahubproject.io)!* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| --- | ||
| title: Overview | ||
| --- | ||
| # Looker & LookML Ingestion Guide: Overview | ||
|
|
||
| ## What You Will Get Out of This Guide | ||
|
|
||
| This guide will help you set up the Looker & LookML connectors to begin ingesting metadata into DataHub. | ||
| Upon completing this guide, you will have a recurring ingestion pipeline to extract metadata from Looker & LookML and load it into DataHub. | ||
|
|
||
| ### Looker | ||
|
|
||
| Looker connector will ingest Looker asset types: | ||
|
|
||
| * [Dashboards](https://cloud.google.com/looker/docs/dashboards) | ||
| * [Charts](https://cloud.google.com/looker/docs/creating-visualizations) | ||
| * [Explores](https://cloud.google.com/looker/docs/reference/param-explore-explore) | ||
| * [Schemas](https://developers.looker.com/api/explorer/4.0/methods/Metadata/connection_schemas) | ||
| * [Owners of Dashboards](https://cloud.google.com/looker/docs/creating-user-defined-dashboards) | ||
|
|
||
| :::note | ||
|
|
||
| To get complete Looker metadata integration (including Looker views and lineage to the underlying warehouse tables), you must also use the [lookml](https://datahubproject.io/docs/generated/ingestion/sources/looker#module-lookml) connector. | ||
|
|
||
| ::: | ||
|
|
||
|
|
||
| ### LookML | ||
|
|
||
| LookMl connector will include the following LookML asset types: | ||
|
|
||
| * [LookML views from model files in a project](https://cloud.google.com/looker/docs/reference/param-view-view) | ||
| * [Metadata for dimensions](https://cloud.google.com/looker/docs/reference/param-field-dimension) | ||
| * [Metadata for measures](https://cloud.google.com/looker/docs/reference/param-measure-types) | ||
| * [Dimension Groups as tag](https://cloud.google.com/looker/docs/reference/param-field-dimension-group) | ||
|
|
||
| :::note | ||
|
|
||
| To get complete Looker metadata integration (including Looker views and lineage to the underlying warehouse tables), you must also use the [looker](https://datahubproject.io/docs/generated/ingestion/sources/looker#module-looker) connector. | ||
|
|
||
| ::: | ||
|
|
||
| ## Next Steps | ||
| Please continue to the [setup guide](setup.md), where we'll describe the prerequisites. | ||
|
|
||
| ### Reference | ||
|
|
||
| If you want to ingest metadata from Looker using the DataHub CLI, check out the following resources: | ||
| * Learn about CLI Ingestion in the [Introduction to Metadata Ingestion](../../../metadata-ingestion/README.md) | ||
| * [Looker Ingestion Source](https://datahubproject.io/docs/generated/ingestion/sources/Looker) | ||
|
|
||
| *Need more help? Join the conversation in [Slack](http:https://slack.datahubproject.io)!* |
Oops, something went wrong.