V0.8.43 (#84)

* feat(ingest): working with multiple bigquery projects (datahub-project#5240)

* fix(build): missing libs (datahub-project#5254)

* fix(build): use correct creds (datahub-project#5261)

* feat(ingest): Option to define path spec for Redshift lineage generation (datahub-project#5256)

* fix(ui): Enable previews properly when browsing for DataJob (datahub-project#5250)

* fix(docs): Fix acronym on mxe docs (datahub-project#5249)

* fix(ui): Support deleting references to glossary terms / nodes, users, assertions, and groups (datahub-project#5248)

* Adding referential integrity to deletes API

* Updating comments

* Fix build

* fix checkstyle

* Fixing Delete Entity utils Test

* feat(docs) add links in quickstart for adding users (datahub-project#5267)

* fix(siblings) Display sibling assertions in Validations tab (datahub-project#5268)

* fix(siblings) Display sibling assertions in Validations tab

* query changes

Co-authored-by: Chris Collins <[email protected]>

* feat(domain) Add ability to edit a Domain name from the UI (datahub-project#5266)

* feat(ingest): delta-lake: adding support for delta lake (datahub-project#5259)

Co-authored-by: Shirshanka Das <[email protected]>

* fix(siblings) Update the names of siblings utils args for readability (datahub-project#5269)

Co-authored-by: Chris Collins <[email protected]>

* docs(adopters): add showroomprive and n26 as DataHub adopters (datahub-project#5271)

* feat(glossary) Add Source section to sidebar for Glossary Terms (datahub-project#5262)

* fix(ingest): delta-lake - fix dependency issue for snowflake due to s3_util (datahub-project#5274)

* fix(ingest): s3 - Remove unneeded methods from s3_util (datahub-project#5276)

* feat(ui): Selector recommendations in Owner, Tag and Domain Modal (datahub-project#5197)

* fix(security) Sanitize rich text before sending to backend or rendering on frontend (datahub-project#5278)

* feat(GraphQL): Support for Deleting Domains, Tags via GraphQL API (datahub-project#5272)

* feat(build): reduce build time for ingestion image (datahub-project#5225)

* fix(ingestion): profiling - Fixing partitioned table profiling in BQ (datahub-project#5283)

* fix(ingest) redshift: Adding missing dependencies and relaxing sqlalchemy dependency (datahub-project#5284)

Relaxing sqlalchemy deps to make our plugins work with Airflow 2.3

* fix(ingestion): Reverting sqlalchemy upgrade because it caused issues with mssql and redshift-usage (datahub-project#5289)

* fix(Siblings): Have sibling hook use entity client (datahub-project#5279)

* fixing dbt platform issues

* have sibling hook use entity client over entity service

* switching search service as well

* lint

* more lint

* more specific exceptions

* refactor(ui): Show message when related glossary terms are empty. (datahub-project#5285)

* docs(adopter): add Digital Turbine as DataHub adopter (datahub-project#5290)

* docs(docker): Update schema-registry  docker.env (datahub-project#5231)

* feat(siblings): index sibling aspects for historical dbt metadata (datahub-project#5291)

* fixing dbt platform issues

* starting sibling restore index job work

* finish restore indices

* migrating to list urns

* rename constant

* disaster recovery

* feat(ui) Adding support for deleting Tags and Domains via the UI (datahub-project#5280)

* Adding support for deleting tags and domains via the UI

* Fixing tests

* fix(test): add cleanup in tests, make urls configurable (datahub-project#5287)

* fix(docs,quickstart): release related changes for 0.8.40 (datahub-project#5299)

* fix(doc): config typo on confluent cloud doc (datahub-project#5293)

* fix(cli): suppress secrets in stacktraces (datahub-project#5302)

* Minor UI bug fuix (datahub-project#5292)

* fix(cli): timeline - category should be owner not ownership (datahub-project#5304)

* perf(ui): reduce data fetched by siblings in lineage (datahub-project#5308)

* fix(ingest): bigquery - Fix for bigquery error when there was no bigquery catalog specified (datahub-project#5303)

* fix(ui) Fix entity profile sidebar width issues (datahub-project#5305)

Co-authored-by: Chris Collins <[email protected]>

* perf(search): Improve search default performance  (datahub-project#5311)

* perf(ui): Performance improvements and misc refactorings in the UI (datahub-project#5310)

* feat(ui): Modified the drop down of Menu Items (datahub-project#5301)

* fix(validation) Fail validation error silently instead of crashing (datahub-project#5314)

* feat(docs) Add documentation on authorization & authentication (datahub-project#5265)

* fix(ui) Make profile icon clickable to expand header menu (datahub-project#5317)

* refactor(ui): Extract searchable page into its own component (perf + ux)  (datahub-project#5318)

* fix(gms) Remove auto-creating status aspects if not present when ingesting metadata (datahub-project#5315)

* fix(ui): Add missing SearchRoutes component (datahub-project#5321)

* feat(ingest): looker - ingest dashboard create/update/delete timestamps (datahub-project#5312)

* fix(ui): Fix pipeline tasks list loading (datahub-project#5332)

* feat(ingest): lookml - adding support for only emitting reachable views from explores (datahub-project#5333)

* fix(ingest): tableau - omit schema fields when name is absent (datahub-project#5275)

* fix(siblings) Combine siblings data but remove duplicate data (datahub-project#5337)

* fix(docs): Fix typo in metadata-ingestion.md (datahub-project#5338)

* fix(me) Cache the me query for performance reasons (datahub-project#5316)

* fix(tokens) Adds non-admin tests for access tokens (datahub-project#5174)

* feat(bigquery): support size, rowcount, lastmodified based table selection for profiling (datahub-project#5329)

Co-authored-by: Shirshanka Das <[email protected]>

* chore: Refactor Python Codebase (datahub-project#5113)

* docs(bigquery): profiling report enhancement (datahub-project#5342)

* feat(ingest): update CSV source to support description and ownership type (datahub-project#5346)

* fix(ui): fixed the ui issue (datahub-project#5341)

* feat(ingest): salesforce - add connector (datahub-project#5104)

Co-authored-by: Shirshanka Das <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>

* feat(bootstrap): create abstract class UpgradeStep to abstract away upgrade logic (datahub-project#5349)

* fix(ingest): bigquery-usage - dataset name for sharded tables (datahub-project#5347)

Co-authored-by: Shirshanka Das <[email protected]>

* docs(features): update grammar (datahub-project#5350)

* fix(ci): fix mysql test and attempt kafka-connect ingestion (datahub-project#5352)

* feat(ui): add copy function for stats table sample value (datahub-project#5331)

* fix(ui) Correct show/hide tabs in Settings based on privileges (datahub-project#5355)

Co-authored-by: Chris Collins <[email protected]>

* fix(siblings): add useMutationUrn to domain section (datahub-project#5270)

* fixing dbt platform issues

* useMutationUrn for domains modal

* feat(schema) Show last observed timestamp in the schema tab (datahub-project#5348)

* fix(glossary) Fixes a bug for yaml ingested terms without source_url (datahub-project#5356)

* feat(lineage) Add Lineage tab to Chart and Dashboard entity profiles (datahub-project#5357)

* fix(cassandra): fix Cassandra queries used by IngestDataPlatformInstancesStep (datahub-project#5199)

* refactor(ui): Use createTag mutation for creating new tags from the UI (datahub-project#5359)

* feat(ui): Added recommendation on group modal (datahub-project#5362)

* refactor(ui): Remove unnecessary fields in GraphQL (datahub-project#5358)

* feat(ingest) - add audit actor urn to auditStamp (datahub-project#5264)

* feat(ingest): improve domain ingestion usability (datahub-project#5366)

* fix(config): fixes config key in DataHubAuthorizerFactory (datahub-project#5371)

* fix(ingest): domains - check whether urn based domain exists during resolution (datahub-project#5373)

* feat(quickstart): Adding env variables and cli options for customizing mapped ports in  quickstart (datahub-project#5353)

Co-authored-by: Shirshanka Das <[email protected]>

* fix(build): tweak ingestion build (datahub-project#5374)

* feat(sdk): python - add get_aspects_for_entity (datahub-project#5255)

Co-authored-by: Shirshanka Das <[email protected]>

* fix(airflow): fix for failing serialisation when Param was specified + support for external task sensor (datahub-project#5368)

fixes datahub-project#4546

* fix(users): fix to not get invite token unless the invite token modal is visible (datahub-project#5380)

* fix(gms) Propagate cache exception upstream (datahub-project#5381)

* fix(bootstrap): skip ingesting data platforms that already exist (datahub-project#5382)

* fix(cli): respect server telemetry settings correctly (datahub-project#5384)

Co-authored-by: Shirshanka Das <[email protected]>

* fix(ingest): bigquery - Graceful bq partition id date parsing failure (datahub-project#5386)

* feat(airflow): Circuit breaker and python api for Assertion and Operation (datahub-project#5196)

* feat(kafka-setup): add options for sasl_plaintext (datahub-project#5385)

allow sasl_plaintext options using environment variables

* fix(bigquery): multi-project GCP setup run query through correct project (datahub-project#5393)

* fix(bigquery): add storage project name (datahub-project#5395)

* Add Changes to support smoke test on Datahub deployed on kubernetes Cluster (datahub-project#5334)

Co-authored-by: Aseem Bansal <[email protected]>

* fix(PlayCookie) PLAY_TOKEN cookie rejected because userprofile exceeds 4096 chars (datahub-project#5114)

* feat(dashboards): add datasets field to DashboardInfo aspect (datahub-project#5188)

Co-authored-by: John Joyce <[email protected]>

* feat(siblings): allow viewing siblings separately (datahub-project#5390)

* allow pulling back curtain for siblings

* sibling pullback working for lineage + property merge

* propagating provinence to ui

* fixups from merge & some renames

* fix styling & add tooltip

* adding cypress tests

* fix lint

* updating mocks

* updating smoke test

* fixing domains smoke test

* responding to comments

* refactor(ui): Added Cursor pointer to tags (datahub-project#5389)

* feat(GMS): Adding Dashboard Usage Models (datahub-project#5399)

* fix(quickstart): use platform agnostic way to get folder (datahub-project#5400)

* Adds support for Domains in CSV source (datahub-project#5372)

* feat(ingestion) Build out UI form for Snowflake Managed Ingestion (datahub-project#5391)

* fix(kafka): add missing configs (datahub-project#5394)

* feat(model): dashboard usage model, is_null condition added (datahub-project#5397)

* fix(datahub-client): Fix kafka config issue (datahub-project#5403)

* build: improve comprehensiveness of gradle clean (datahub-project#5003)

* fix(gms): Change MessageDigest to be thread safe (datahub-project#5405)

* fix(metadata-ingestion) Fix broken csv enricher test (datahub-project#5406)

* fix(tests): Removes duplicate policies tests & makes DataHub user configurable (datahub-project#5365)

* feat(quickstart,docs): updates for v0.8.41 (datahub-project#5409)

* fix(ingest): ensure upgrade checks run async (datahub-project#5383)

* fix(ingest): looker - pass transport options to all api calls (datahub-project#5417)

* feat(quickstart): moving to official confluent images for m1 (datahub-project#5416)

* fix(documentation) Fix erratic cursor in documentation editor bug (datahub-project#5411)

Co-authored-by: Chris Collins <[email protected]>

* feat(ui): Supporting enriched search preview + misc improvements  (datahub-project#5419)

* chore: remove unnecessary modules from codebase (datahub-project#5420)

* fix(ingest): looker - extract usage for dashboards allowed by pattern (datahub-project#5424)

* fix(docker): fix kafka-setup command to support same capabilities as previous (datahub-project#5428)

* fix(protobuf) Set undeprecated ownership type & fix case sentitive urn corpGroup (datahub-project#5425)

* fix(ui): add dataset qualifiedName parameter to lineage query (datahub-project#5427)

* fix(glossary) Fix dropdown where disabled buttons are still clickable (datahub-project#5430)

Co-authored-by: Chris Collins <[email protected]>

* docs(bigquery): add changelog and unittest for profiling limits (datahub-project#5407)

* fix(siblings): fixing lineage fetching for siblings & sources (datahub-project#5415)

* fix(ui): Fixing unreleased search preview bugs  (datahub-project#5432)

* feat(ui): Adding Statistics Summary to Dataset + Dashboard Profiles  (datahub-project#5440)

* feat(ingest): add test source connection feature, structured report file (datahub-project#5442)

* fix(ingest/glue): handle error when generating s3 tags for virtual view tables (datahub-project#5398)

Co-authored-by: Tim Costa <[email protected]>
Co-authored-by: Shirshanka Das <[email protected]>

* feat(ingest): model - adding a small extension to support communicating structured responses (datahub-project#5429)

* fix(ingest): bigquery-usage - fix dataset name for sharded table (datahub-project#5412)

* feat(ingestion) Add new endpoint to test an ingestion connection (datahub-project#5438)

* feat(cli,build): remove deprecated variables GMS_HOST/_PORT (datahub-project#5451)

* fix(search): make filters by default an empty list if null (datahub-project#5454)

* fix(ingest): hive - add column comment as a column description (datahub-project#5449)

* feat(groups): add native groups concept to DataHub (datahub-project#5443)

* fix(ingest): fix serialization of report to handle nesting (datahub-project#5455)

* fix(ingest): tableau - fix tableau db error, add more logs (datahub-project#5423)

* build(deps): bump terser from 5.9.0 to 5.14.2 in /docs-website (datahub-project#5448)

Bumps [terser](https://github.com/terser/terser) from 5.9.0 to 5.14.2.
- [Release notes](https://github.com/terser/terser/releases)
- [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md)
- [Commits](https://github.com/terser/terser/commits)

---
updated-dependencies:
- dependency-name: terser
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* docs: spark-lineage - configuration details for Amazon EMR (datahub-project#5459)

* feat(app): schema-history - remove blame language for the schema history feature (datahub-project#5457)

* Worked on the alignment of menu icon in search header (datahub-project#5458)

* build(deps): bump terser from 4.8.0 to 4.8.1 in /datahub-web-react (datahub-project#5446)

Bumps [terser](https://github.com/terser/terser) from 4.8.0 to 4.8.1.
- [Release notes](https://github.com/terser/terser/releases)
- [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md)
- [Commits](https://github.com/terser/terser/commits)

---
updated-dependencies:
- dependency-name: terser
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feat(ingest): snowflake - basic test connection capability (datahub-project#5464)

* fix(ingest/trino): Avoid exception if $properties table empty or not readable (datahub-project#5447)

Under some configuration of access rules in Trino, the user may not have
read access to the content of the table, which will result in an exception
(`fetchone()` returns `None`)

This commit ensures no exception are raised and the ingestion can proceed.

* feat(ingest): preflight - Add way to check/upgrade brew package version in preflight if needed (datahub-project#5435)

* fix(build): add base image with gradle wrapper cached (datahub-project#5467)

* doc(bigquery): groups grants by requirements (datahub-project#5468)

* fix(docs,build): remove base image not needed, cleanup docs (datahub-project#5469)

* feat(ui): Partial support for Chart usage (datahub-project#5473)

* fix(ingest): bigquery: multiproject profiling fix (datahub-project#5474)

* fix(ingest): kafka - revert deps back to < 1.9.0 (datahub-project#5476)

* feat(ci): datahub-upgrade - support multiplatform image (datahub-project#5477)

* feat(cli): quickstart - experimental support for backup restore (datahub-project#5418)

* feat(ingest): dbt - updating source lineage logic (datahub-project#5414)

Co-authored-by: Shirshanka Das <[email protected]>

* Ingestion: Added form in Big Query type to edit the queries. (datahub-project#5431)

* docs(reindex): fix docsearch config (datahub-project#5479)

* refactor(ui): Adding checkbox option to select multiple results at once. (datahub-project#5422)

* feat(cli): delete - hard delete deletes soft deleted entities (datahub-project#5478)

* fix(docs): add missing closing marker for note section (datahub-project#5480)

* fix(ci): intermittent failure in github actions (datahub-project#5452)

* feat(model, ingest): add user email in dashboard user usage counts (datahub-project#5471)

* feat(ingest): snowflake - test_connection add support for capability report (datahub-project#5472)

Co-authored-by: Shirshanka Das <[email protected]>

* feat(build): automatically mark issues as stale to close inactive issues (datahub-project#5482)

* fix(ingest): loosen confluent-kafka dep requirement (datahub-project#5489)

* refactor(ingest): cleanup importlib.import_module calls (datahub-project#5490)

* build(ingest): make gradle build less chatty (datahub-project#5491)

* fix(ingest): dbt - add support for trino datatypes (datahub-project#5379)

* refactor(ci): use custom action for checking codegen status (datahub-project#5493)

* feat(spark-lineage, java-emitter): Support ssl cert disable verification functionality (datahub-project#5488)

Co-authored-by: Shirshanka Das <[email protected]>

* docs(auth): fix link to point to new doc (datahub-project#5501)

* docs(updating-datahub): add note for breaking change in looker usage ingestion (datahub-project#5499)

* fix(ingest): cleanup unused flake8 noqa statements (datahub-project#5492)

* fix(ingest): cleanup unused flake8 noqa statements

In the future, we can discover these using `flake8-noqa`.

* add back c901

* refactor(ci): refactor Docker build-and-push workflows (datahub-project#5494)

* docs(slack): update to Slack guidelines (datahub-project#5504)

* feat(cli): delete - add --only-soft-deleted option, perf improvements (datahub-project#5485)

* fix(ingest): use temp dir for file generated during test (datahub-project#5505)

* feat(ui) Show Glossary and Domains header links to everyone (datahub-project#5506)

Co-authored-by: Chris Collins <[email protected]>

* fix(ui): Fix Flickering Issue on search input field (datahub-project#5503)

* fix(ingest): respect rest emitter timeout setting (datahub-project#5508)

* fix(ui): Flickering Issue on search input field (datahub-project#5515)

* feat(ui): Added form to Looker and Tableau (datahub-project#5487)

* feat(identity): update azure and okta connectors to emit Origin aspects (datahub-project#5495)

* feat(ui): Adding Search Select feature(frontend only)  (datahub-project#5507)

* test(ingest): limit GMS retries in test (datahub-project#5509)

* fix(ingest): airflow: update subdag check for compatibility with older Airflow versions (datahub-project#5523)

* use getattr to default None if no subdag

* add None check

* add other None check

* Apply suggestions from code review- double quotes

Co-authored-by: Harshal Sheth <[email protected]>

* minor tweak to fix lint

Co-authored-by: Harshal Sheth <[email protected]>

* fix(ingest): fix unbound variable bug in cli ingest list-runs (datahub-project#5527)

* fix(ui) Display Term Group name properly in Recently Viewed (datahub-project#5528)

* feat(ingestion) Add frontend connection test for Snowflake (datahub-project#5520)

* fix(glossary) Fix Glossary success messages and sort Glossary (datahub-project#5533)

* show error and success messages in glossary properly

* sort glossary nodes and terms alphabetically

Co-authored-by: Chris Collins <[email protected]>

* feat(apache-ranger): Apache Ranger Authorizer support in datahub-gms (datahub-project#4999)

* feat(ingest): add deprecation warning for Python 3.6 (datahub-project#5519)

* docs(townhall) add past townhall agendas (datahub-project#5536)

* feat(ingestion): add groups to ldap users (datahub-project#5470)

* chore(issues): reduce time for issues to be marked stale and then closed (datahub-project#5537)

* fix(ingestion) Set pipeline_name on UI recipes with forms (datahub-project#5535)

* Fixing OIDC logout issues (datahub-project#5538)

* fix(analytics-tab) - fix analytics tab config variable for gms (datahub-project#5529)

* feat(ui): Support batch adding / remove tags from search lists. (Batch Actions part 2/7)  (datahub-project#5534)

* fix(ingestionSource): improve error experience when ingestion source is in an inconsistent state (datahub-project#5522)

* fix(docs): Fixed typo in schema history markdown! (datahub-project#5545)

* fix(docker): Fixing dev docker and quickstart  (datahub-project#5550)

* feat(ui): Support Batch adding and removing Glossary Terms (Batch Actions 3/7) (datahub-project#5544)

* feat(ci): test quickstart works (datahub-project#5518)

* feat(ci): test quickstart works

* do not fail fast

* remove macos

* add some debug information

* tweak triggers

* fix workflow file

* remove running on every PR

* Update .github/workflows/check-quickstart.yml

Co-authored-by: Harshal Sheth <[email protected]>

* Update .github/workflows/check-quickstart.yml

Co-authored-by: Harshal Sheth <[email protected]>

Co-authored-by: Harshal Sheth <[email protected]>

* test(ingest): mark trino/hana tests as xfail due to flakes (datahub-project#5549)

* feat(ingestion): superset - add display_uri to config (datahub-project#5408)

* fix(quickstart): failure on a path not being present (datahub-project#5554)

* fix(dbt): fix issue of assertion error when stateful ingestion is used with dbt tests (datahub-project#5540)

* fix(dbt): fix issue of dbt stateful ingestion with tests

Co-authored-by: MugdhaHardikar-GSLab <[email protected]>
Co-authored-by: MohdSiddique Bagwan <[email protected]>
Co-authored-by: Ravindra Lanka <[email protected]>

* feat(ui): Batch add & remove Owners to assets via the UI (datahub-project#5552)

* feat(ingestion) Update managed ingestion scheduler to be easier to use (datahub-project#5559)

* fix(ingestion): correct trino datatype handling (datahub-project#5541)

Co-authored-by: Ravindra Lanka <[email protected]>

* feat(ingest) Allow ingestion of Elasticsearch index template (datahub-project#5444)


Co-authored-by: Ravindra Lanka <[email protected]>

* fix(ingest): fix some typos and logging issues (datahub-project#5564)

* feat(transformers): Add domain transformer for dataset (datahub-project#5456)

Co-authored-by: MohdSiddique Bagwan <[email protected]>

* chore(0.8.42): update breaking changes doc (datahub-project#5563)

* fix(ingest): activate mypy support for ParamSpec typing annotation (datahub-project#5551)

* (chore): upgrading ingestion to 0.8.42 (datahub-project#5562)

* fix(gms): ensure directory is present (datahub-project#5568)

* fix(ci): flaky smoke test fix (datahub-project#5569)

* fix(gms): missing directory for gms (datahub-project#5570)

* chore(build): tweak stale issue timing (datahub-project#5571)

* feat(ui): Batch set & unset Domain for assets via the UI (datahub-project#5560)

* extending assertion std model (datahub-project#5575)

* feat(ui): Support batch deprecation from the UI (Batch actions part 6/7) (datahub-project#5572)

* feat(graphql): add MutableTypeBatchResolver (datahub-project#4976)

* feat(ingestion) Implement secrets in new managed ingestion form (datahub-project#5574)

* fix(ui): Fixing batch set domains bug (datahub-project#5580)

* chore(gradle): update node version for docs site (datahub-project#5581)

* feat(test): add read-only smoke tests (datahub-project#5558)

* feat(ingestion) Add Save & Run button to managed ingestion builder (datahub-project#5579)

* fix(ingest): handle when current server version is unavailable (datahub-project#5547)

* feat(ingest): dbt - control over emitting test_results, test_definitions, etc. (datahub-project#5328)

Co-authored-by: Piotr Sierkin <[email protected]>
Co-authored-by: Shirshanka Das <[email protected]>

* feat(datahub-client): add java file emitter (datahub-project#5578)

Co-authored-by: Shirshanka Das <[email protected]>

* feat(ingest): infer aspectName from aspect type in MCP (datahub-project#5566)

* fix(ingest): sql-common - db2, snowflake bug fixes to extract table descriptions (datahub-project#5526)

Co-authored-by: Harshal Sheth <[email protected]>
Co-authored-by: Shirshanka Das <[email protected]>

* fix(ingest): moving delta-lake connector to be 3.7+ only (datahub-project#5584)

* feat(ingest): delta-lake - extract table history into operation aspect (datahub-project#5277)

Co-authored-by: Shirshanka Das <[email protected]>

* fix apache ranger plugin readme file rendering (datahub-project#5585)

* feat(ui): make container description searchable and have description show up in results (datahub-project#5586)

* fix(groups): fix user, search, and preview group membership to be fetched for both external and native group memberships (datahub-project#5587)

* feat(ingest): power-bi - make ownership ingestion optional (datahub-project#5335)


Co-authored-by: MohdSiddique Bagwan <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>

* Expose catalog_name in athena.py (datahub-project#5548)

* expose catalog_name to the sql alchemy uri that is passed into pyathena

Co-authored-by: Ravindra Lanka <[email protected]>
Co-authored-by: Shirshanka Das <[email protected]>

* Fix profiling when using {table}. (datahub-project#5531)

* profiling fix for when using {table}

Co-authored-by: Shirshanka Das <[email protected]>
Co-authored-by: Ravindra Lanka <[email protected]>

* feat(ui): Support batch deleting from ui (datahub-project#5582)

* feat(ingest): clickhouse - add metadata modification time and data size (datahub-project#5330)

Co-authored-by: Ravindra Lanka <[email protected]>

* feat(ui): Add rich UI ingestion run summary (datahub-project#5577)

* fix(ci): smoke test less flaky, add src, dev dep in smoke image (datahub-project#5594)

* updated mock custom to pass the test suite

* added env for mysql-setup for smoketest to pass

* added env for mysql-setup for smoketest to pass

* added env for mysql-setup for smoketest to pass

* push to heruko repo instead of linkedin

Co-authored-by: Aseem Bansal <[email protected]>
Co-authored-by: Tamas Nemeth <[email protected]>
Co-authored-by: Michael A. Schlosser <[email protected]>
Co-authored-by: John Joyce <[email protected]>
Co-authored-by: Pedro Silva <[email protected]>
Co-authored-by: Chris Collins <[email protected]>
Co-authored-by: Chris Collins <[email protected]>
Co-authored-by: Mugdha Hardikar <[email protected]>
Co-authored-by: Shirshanka Das <[email protected]>
Co-authored-by: Chris Collins <[email protected]>
Co-authored-by: Maggie Hays <[email protected]>
Co-authored-by: Ankit keshari <[email protected]>
Co-authored-by: Gabe Lyons <[email protected]>
Co-authored-by: liyuhui666 <[email protected]>
Co-authored-by: Tengis Batsaikhan <[email protected]>
Co-authored-by: Chris Collins <[email protected]>
Co-authored-by: Pedro Silva <[email protected]>
Co-authored-by: Mayuri Nehate <[email protected]>
Co-authored-by: dougpm <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
Co-authored-by: Aditya Radhakrishnan <[email protected]>
Co-authored-by: Amanda Ng <[email protected]>
Co-authored-by: Chris Collins <[email protected]>
Co-authored-by: Justin Marozas <[email protected]>
Co-authored-by: Sergio Gómez Villamor <[email protected]>
Co-authored-by: Navin Sharma <[email protected]>
Co-authored-by: Aezo <[email protected]>
Co-authored-by: abiwill <[email protected]>
Co-authored-by: Felix Lüdin <[email protected]>
Co-authored-by: Harshal Sheth <[email protected]>
Co-authored-by: Chris Collins <[email protected]>
Co-authored-by: leifker <[email protected]>
Co-authored-by: Alexey Kravtsov <[email protected]>
Co-authored-by: Tim Costa <[email protected]>
Co-authored-by: Tim Costa <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Guillaume Gardey <[email protected]>
Co-authored-by: Vishal Shah <[email protected]>
Co-authored-by: mohdsiddique <[email protected]>
Co-authored-by: Salih Can <[email protected]>
Co-authored-by: RyanHolstien <[email protected]>
Co-authored-by: Skyler Sinclair <[email protected]>
Co-authored-by: Dan Andreescu <[email protected]>
Co-authored-by: MohdSiddique Bagwan <[email protected]>
Co-authored-by: Ravindra Lanka <[email protected]>
Co-authored-by: Marcin Szymański <[email protected]>
Co-authored-by: xiphl <[email protected]>
Co-authored-by: NoahFournier <[email protected]>
Co-authored-by: Piotr Sierkin <[email protected]>
Co-authored-by: Piotr Sierkin <[email protected]>
Co-authored-by: Jordan Wolinsky <[email protected]>

.github/actions/docker-custom-build-and-push/action.yml

@@ -0,0 +1,92 @@
+name: Custom Docker build and push
+description: "Build and push a Docker image to Docker Hub"
+
+inputs:
+ username:
+ description: "Docker Hub username"
+ password:
+ description: "Docker Hub password"
+ publish:
+ description: "Set to true to actually publish the image to Docker Hub"
+
+ context:
+ description: "Same as docker/build-push-action"
+ required: false
+ file:
+ description: "Same as docker/build-push-action"
+ required: false
+ platforms:
+ description: "Same as docker/build-push-action"
+ required: false
+
+ images:
+ # e.g. linkedin/datahub-gms
+ description: "List of Docker images to use as base name for tags"
+ required: true
+ tags:
+ # e.g. latest,head,sha12345
+ description: "List of tags to use for the Docker image"
+ required: true
+outputs:
+ image_tag:
+ description: "Docker image tags"
+ value: ${{ steps.docker_meta.outputs.tags }}
+ # image_name: ${{ env.DATAHUB_GMS_IMAGE }}
+
+runs:
+ using: "composite"
+
+ steps:
+ - name: Docker meta
+ id: docker_meta
+ uses: crazy-max/ghaction-docker-meta@v1
+ with:
+ # list of Docker images to use as base name for tags
+ images: ${{ inputs.images }}
+ # add git short SHA as Docker tag
+ tag-custom: ${{ inputs.tags }}
+ tag-custom-only: true
+
+ # Code for testing the build when not pushing to Docker Hub.
+ - name: Build and Load image for testing (if not publishing)
+ uses: docker/build-push-action@v2
+ if: ${{ inputs.publish != 'true' }}
+ with:
+ context: ${{ inputs.context }}
+ file: ${{ inputs.file }}
+ # TODO this only does single-platform builds in testing?
+ # leaving it for now since it matches the previous behavior
+ platforms: linux/amd64
+ tags: ${{ steps.docker_meta.outputs.tags }}
+ load: true
+ push: false
+ - name: Upload image locally for testing (if not publishing)
+ uses: ishworkh/docker-image-artifact-upload@v1
+ if: ${{ inputs.publish != 'true' }}
+ with:
+ image: ${{ steps.docker_meta.outputs.tags }}
+
+ # Code for building multi-platform images and pushing to Docker Hub.
+ - name: Set up QEMU
+ uses: docker/setup-qemu-action@v1
+ if: ${{ inputs.publish == 'true' }}
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v1
+ if: ${{ inputs.publish == 'true' }}
+ - name: Login to DockerHub
+ uses: docker/login-action@v1
+ if: ${{ inputs.publish == 'true' }}
+ with:
+ username: ${{ inputs.username }}
+ password: ${{ inputs.password }}
+ - name: Build and Push Multi-Platform image
+ uses: docker/build-push-action@v2
+ if: ${{ inputs.publish == 'true' }}
+ with:
+ context: ${{ inputs.context }}
+ file: ${{ inputs.file }}
+ platforms: ${{ inputs.platforms }}
+ tags: ${{ steps.docker_meta.outputs.tags }}
+ push: true
+
+ # TODO add code for vuln scanning?

.github/actions/ensure-codegen-updated/action.yml

@@ -0,0 +1,16 @@
+name: 'Ensure codegen is updated'
+description: 'Will check the local filesystem against git, and abort if there are uncommitted changes.'
+
+runs:
+ using: "composite"
+ steps:
+ - shell: bash
+ run: |
+ if output=$(git status --porcelain) && [ ! -z "$output" ]; then
+ # See https://unix.stackexchange.com/a/155077/378179.
+ echo 'There are uncommitted changes:'
+ echo $output
+ exit 1
+ else
+ echo 'All good!'
+ fi

.github/workflows/build-and-test.yml

@@ -10,12 +10,15 @@ on:
  branches:
  - master
  paths-ignore:
- - "docker/**"
  - "docs/**"
  - "**.md"
  release:
  types: [published, edited]
 
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
 jobs:
  build:
  runs-on: ubuntu-latest
@@ -45,15 +48,7 @@ jobs:
  **/build/test-results/test/**
  **/junit.*.xml
  - name: Ensure codegen is updated
- run: |
- if output=$(git status --porcelain) && [ ! -z "$output" ]; then
- # See https://unix.stackexchange.com/a/155077/378179.
- echo 'There are uncommitted changes:'
- echo $output
- exit 1
- else
- echo 'All good!'
- fi
+ uses: ./.github/actions/ensure-codegen-updated
  - name: Slack failure notification
  if: failure() && github.event_name == 'push'
  uses: kpritam/slack-job-status-action@v1

.github/workflows/check-datahub-jars.yml

@@ -5,6 +5,7 @@ on:
  branches:
  - master
  paths-ignore:
+ - "docker/**"
  - "docs/**"
  - "**.md"
  pull_request:
@@ -17,6 +18,10 @@ on:
  release:
  types: [published, edited]
 
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
 jobs:
 
  check_jars:

.github/workflows/check-quickstart.yml

@@ -0,0 +1,47 @@
+name: check quickstart
+on:
+ push:
+ branches:
+ - master
+ workflow_dispatch:
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
+
+jobs:
+ test-quickstart:
+ strategy:
+ fail-fast: false
+ matrix:
+ os: [ubuntu-20.04]
+ runs-on: ${{ matrix.os }}
+ steps:
+ - uses: actions/setup-python@v2
+ with:
+ python-version: "3.9.9"
+ - name: Install acryl-datahub
+ run: |
+ pip install --upgrade acryl-datahub
+ datahub version
+ python -c "import platform; print(platform.platform())"
+ - name: Run quickstart
+ run: |
+ datahub docker quickstart
+ - name: Ingest sample data
+ run: |
+ datahub docker ingest-sample-data
+ - name: See status
+ run: |
+ docker ps -a && datahub docker check
+ - name: store logs
+ if: failure()
+ run: |
+ docker logs datahub-gms >& quickstart-gms.log
+ - name: Upload logs
+ uses: actions/upload-artifact@v2
+ if: failure()
+ with:
+ name: docker-quickstart-logs-${{ matrix.os }}
+ path: "*.log"

.github/workflows/close-stale-issues.yml

@@ -0,0 +1,22 @@
+name: Close inactive issues
+on:
+ schedule:
+ - cron: "30 1 * * *"
+
+jobs:
+ close-issues:
+ runs-on: ubuntu-latest
+ permissions:
+ issues: write
+ pull-requests: write
+ steps:
+ - uses: actions/stale@v5
+ with:
+ days-before-issue-stale: 30
+ days-before-issue-close: 30
+ stale-issue-label: "stale"
+ stale-issue-message: "This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io"
+ close-issue-message: "This issue was closed because it has been inactive for 30 days since being marked as stale."
+ days-before-pr-stale: -1
+ days-before-pr-close: -1
+ repo-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/docker-feast-source.yml

@@ -3,21 +3,22 @@ on:
  push:
  branches:
  - master
- paths-ignore:
- - 'docs/**'
- - '**.md'
+ paths:
+ - 'metadata-ingestion/src/datahub/ingestion/source/feast_image/**'
+ - '.github/workflows/docker-feast-source.yml'
  pull_request:
  branches:
  - master
  paths:
  - 'metadata-ingestion/src/datahub/ingestion/source/feast_image/**'
  - '.github/workflows/docker-feast-source.yml'
- paths_ignore:
- - '**.md'
- - '**.env'
  release:
  types: [published, edited]
 
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
 jobs:
  setup:
  runs-on: ubuntu-latest

.github/workflows/docker-ingestion-base.yml

@@ -0,0 +1,43 @@
+name: ingestion base
+on:
+ release:
+ types: [published, edited]
+ push:
+ branches:
+ - master
+ paths:
+ - "docker/datahub-ingestion/**"
+ - "gradle*"
+ workflow_dispatch:
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
+jobs:
+
+ build-base:
+ name: Build and Push Docker Image to Docker Hub
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Set up QEMU
+ uses: docker/setup-qemu-action@v1
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v1
+ - name: Login to DockerHub
+ uses: docker/login-action@v1
+ with:
+ username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
+ password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
+ - name: Build and Push image
+ uses: docker/build-push-action@v2
+ with:
+ context: ./docker/datahub-ingestion
+ file: ./docker/datahub-ingestion/base.Dockerfile
+ platforms: linux/amd64,linux/arm64
+ tags: acryldata/datahub-ingestion-base:latest
+ push: true

.github/workflows/docker-ingestion-smoke.yml

@@ -0,0 +1,42 @@
+name: ingestion smoke
+on:
+ release:
+ types: [published, edited]
+ push:
+ branches:
+ - master
+ paths:
+ - "docker/datahub-ingestion/**"
+ workflow_dispatch:
+
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
+jobs:
+
+ build-smoke:
+ name: Build and Push Docker Image to Docker Hub
+ runs-on: ubuntu-latest
+ steps:
+ - name: Check out the repo
+ uses: actions/checkout@v2
+ with:
+ fetch-depth: 0
+ - name: Set up QEMU
+ uses: docker/setup-qemu-action@v1
+ - name: Set up Docker Buildx
+ uses: docker/setup-buildx-action@v1
+ - name: Login to DockerHub
+ uses: docker/login-action@v1
+ with:
+ username: ${{ secrets.ACRYL_DOCKER_USERNAME }}
+ password: ${{ secrets.ACRYL_DOCKER_PASSWORD }}
+ - name: Build and Push image
+ uses: docker/build-push-action@v2
+ with:
+ context: .
+ file: ./docker/datahub-ingestion/smoke.Dockerfile
+ platforms: linux/amd64,linux/arm64
+ tags: acryldata/datahub-ingestion-base:smoke
+ push: true

.github/workflows/docker-ingestion.yml

@@ -10,14 +10,17 @@ on:
  branches:
  - master
  paths:
- - "docker/**"
+ - "metadata-ingestion/**"
+ - "metadata-models/**"
+ - "docker/datahub-ingestion/**"
  - ".github/workflows/docker-ingestion.yml"
- paths_ignore:
- - "**.md"
- - "**.env"
  release:
  types: [published, edited]
 
+concurrency:
+ group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+ cancel-in-progress: true
+
 jobs:
  setup:
  runs-on: ubuntu-latest
@@ -66,7 +69,7 @@ jobs:
  with:
  # list of Docker images to use as base name for tags
  images: |
- linkedin/datahub-ingestion
+ heruko/datahub-ingestion
  # add git short SHA as Docker tag
  tag-custom: ${{ needs.setup.outputs.tag }}
  tag-custom-only: true