From 224b948bba99714b78dc5895a8b7103ecef64a60 Mon Sep 17 00:00:00 2001 From: Willy Lulciuc Date: Sat, 11 Jan 2020 18:45:47 -0800 Subject: [PATCH] Rename datasets.last_modified (#700) * Rename datasets.last_modified Signed-off-by: wslulciuc * continued: Rename datasets.last_modified Signed-off-by: wslulciuc --- docs/openapi.html | 12 ++++++------ docs/openapi.yml | 12 ++++++------ src/main/java/marquez/api/mappers/Mapper.java | 4 ++-- .../java/marquez/api/models/DatasetResponse.java | 10 +++++----- .../java/marquez/api/models/DbTableResponse.java | 4 ++-- src/main/java/marquez/api/models/StreamResponse.java | 6 +++--- src/main/java/marquez/db/Columns.java | 2 +- src/main/java/marquez/db/DatasetDao.java | 6 +++--- src/main/java/marquez/db/RunStateDao.java | 4 ++-- .../java/marquez/db/mappers/DatasetRowMapper.java | 2 +- .../marquez/db/mappers/ExtendedDatasetRowMapper.java | 2 +- src/main/java/marquez/db/models/DatasetRow.java | 6 +++--- src/main/java/marquez/service/mappers/Mapper.java | 4 ++-- src/main/java/marquez/service/models/Dataset.java | 10 +++++----- src/main/java/marquez/service/models/DbTable.java | 4 ++-- src/main/java/marquez/service/models/Stream.java | 4 ++-- .../V11__alter_datasets_to_rename_last_modified.sql | 1 + src/test/java/marquez/MarquezAppIntegrationTest.java | 4 ++-- src/test/java/marquez/db/DatasetDaoTest.java | 6 +++--- 19 files changed, 52 insertions(+), 51 deletions(-) create mode 100644 src/main/resources/db/migration/V11__alter_datasets_to_rename_last_modified.sql diff --git a/docs/openapi.html b/docs/openapi.html index a2c56acc05..85c572570f 100644 --- a/docs/openapi.html +++ b/docs/openapi.html @@ -330,32 +330,32 @@ runId
string

The ID associated with the run modifying the table.

Responses

200

OK

put /namespaces/{namespace}/datasets/{dataset}

Local API server

-
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}

Request samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "physicalName": "wedata.room_bookings",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "description": "All room booking occupancy data."
}

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "name": "wedata.room_bookings",
  • "physicalName": "wedata.room_bookings",
  • "createdAt": "2019-05-09T19:49:24.201Z",
  • "updatedAt": "2019-05-09T19:49:24.201Z",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "lastModified": null,
  • "description": "All room booking occupancy data."
}

Get a dataset

Returns a dataset.

+
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}

Request samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "physicalName": "wedata.room_bookings",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "description": "All room booking occupancy data."
}

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "name": "wedata.room_bookings",
  • "physicalName": "wedata.room_bookings",
  • "createdAt": "2019-05-09T19:49:24.201Z",
  • "updatedAt": "2019-05-09T19:49:24.201Z",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "lastModifiedAt": null,
  • "description": "All room booking occupancy data."
}

Get a dataset

Returns a dataset.

path Parameters
namespace
required
string <= 1024 characters
Example: "wework"

The name of the namespace.

dataset
required
string <= 1024 characters
Example: "wedata.room_bookings"

The name of the dataset.

Responses

200

OK

get /namespaces/{namespace}/datasets/{dataset}

Local API server

-
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "name": "wedata.room_bookings",
  • "physicalName": "wedata.room_bookings",
  • "createdAt": "2019-05-09T19:49:24.201Z",
  • "updatedAt": "2019-05-09T19:49:24.201Z",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "lastModified": null,
  • "description": "All room booking occupancy data."
}

List all datasets

Returns a list of datasets.

+
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "name": "wedata.room_bookings",
  • "physicalName": "wedata.room_bookings",
  • "createdAt": "2019-05-09T19:49:24.201Z",
  • "updatedAt": "2019-05-09T19:49:24.201Z",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "lastModifiedAt": null,
  • "description": "All room booking occupancy data."
}

List all datasets

Returns a list of datasets.

path Parameters
namespace
required
string <= 1024 characters
Example: "wework"

The name of the namespace.

dataset
required
string <= 1024 characters
Example: "wedata.room_bookings"

The name of the dataset.

query Parameters
limit
integer

The number of results to return from offset

offset
integer

The initial position from which to return results

Responses

200

OK

get /namespaces/{namespace}/datasets

Local API server

-
http://localhost:5000/api/v1/namespaces/{namespace}/datasets

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "datasets":
    [
    ]
}

Tag a dataset

Tag an existing dataset.

+
http://localhost:5000/api/v1/namespaces/{namespace}/datasets

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "datasets":
    [
    ]
}

Tag a dataset

Tag an existing dataset.

path Parameters
namespace
required
string <= 1024 characters
Example: "wework"

The name of the namespace.

dataset
required
string <= 1024 characters
Example: "wedata.room_bookings"

The name of the dataset.

tag
required
string
Example: "SENSITIVE"

The name of the tag.

Responses

200

OK

post /namespaces/{namespace}/datasets/{dataset}/tags/{tag}

Local API server

-
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}/tags/{tag}

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "name": "wedata.room_bookings",
  • "physicalName": "wedata.room_bookings",
  • "createdAt": "2019-05-09T19:49:24.201Z",
  • "updatedAt": "2019-05-09T19:49:24.201Z",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "lastModified": null,
  • "description": "All room booking occupancy data."
}

Tag a field

Tag an existing field of a dataset.

+
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}/tags/{tag}

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "name": "wedata.room_bookings",
  • "physicalName": "wedata.room_bookings",
  • "createdAt": "2019-05-09T19:49:24.201Z",
  • "updatedAt": "2019-05-09T19:49:24.201Z",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "lastModifiedAt": null,
  • "description": "All room booking occupancy data."
}

Tag a field

Tag an existing field of a dataset.

path Parameters
namespace
required
string <= 1024 characters
Example: "wework"

The name of the namespace.

dataset
required
string <= 1024 characters
Example: "wedata.room_bookings"

The name of the dataset.

field
required
string
Example: "member_id"

The name of the field.

tag
required
string
Example: "SENSITIVE"

The name of the tag.

Responses

200

OK

post /namespaces/{namespace}/datasets/{dataset}/fields/{field}/tags/{tag}

Local API server

-
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}/fields/{field}/tags/{tag}

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "name": "wedata.room_bookings",
  • "physicalName": "wedata.room_bookings",
  • "createdAt": "2019-05-09T19:49:24.201Z",
  • "updatedAt": "2019-05-09T19:49:24.201Z",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "lastModified": null,
  • "description": "All room booking occupancy data."
}

Jobs

Create a job

Creates a new job object. All job objects are immutable and are uniquely identified by a generated ID. Marquez will create a version of a job each time the contents of the object is modified. For example, the location of a job may change over time resulting in new versions. The accumulated versions can be listed, used to rerun a specific job version or possibly help debug a failed job run.

+
http://localhost:5000/api/v1/namespaces/{namespace}/datasets/{dataset}/fields/{field}/tags/{tag}

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "type": "DB_TABLE",
  • "name": "wedata.room_bookings",
  • "physicalName": "wedata.room_bookings",
  • "createdAt": "2019-05-09T19:49:24.201Z",
  • "updatedAt": "2019-05-09T19:49:24.201Z",
  • "sourceName": "analytics_db",
  • "fields":
    [
    ],
  • "tags":
    [
    ],
  • "lastModifiedAt": null,
  • "description": "All room booking occupancy data."
}

Jobs

Create a job

Creates a new job object. All job objects are immutable and are uniquely identified by a generated ID. Marquez will create a version of a job each time the contents of the object is modified. For example, the location of a job may change over time resulting in new versions. The accumulated versions can be listed, used to rerun a specific job version or possibly help debug a failed job run.

path Parameters
namespace
required
string <= 1024 characters
Example: "wework"

The name of the namespace.

job
required
string <= 1024 characters
Example: "room_bookings_7_days"

The name of the job.

Request Body schema: application/json
type
required
string
Enum:"BATCH" "STREAM" "SERVICE"

The type of the job.

@@ -419,7 +419,7 @@
get /tags

Local API server

http://localhost:5000/api/v1/tags

Response samples

application/json
Copy
 Expand all Collapse all
{
  • "tags":
    [
    ]
}