Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perf/improve jobdao query #2609

Merged
merged 2 commits into from
Sep 7, 2023
Merged

Perf/improve jobdao query #2609

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6891fb3
perf/ dao query updated
algorithmy1 Sep 6, 2023
0db03bc
./gradlew spotlessJavaCheck
algorithmy1 Sep 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 53 additions & 17 deletions api/src/main/java/marquez/db/JobDao.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,23 +125,59 @@ default Optional<Job> findWithRun(String namespaceName, String jobName) {

@SqlQuery(
"""
SELECT j.*, f.facets
FROM jobs_view AS j
LEFT OUTER JOIN job_versions AS jv ON jv.uuid = j.current_version_uuid
LEFT OUTER JOIN (
SELECT run_uuid, JSON_AGG(e.facet) AS facets
FROM (
SELECT jf.run_uuid, jf.facet
FROM job_facets_view AS jf
INNER JOIN job_versions jv2 ON jv2.latest_run_uuid=jf.run_uuid
INNER JOIN jobs_view j2 ON j2.current_version_uuid=jv2.uuid
WHERE j2.namespace_name=:namespaceName
ORDER BY lineage_event_time ASC
) e
GROUP BY e.run_uuid
) f ON f.run_uuid=jv.latest_run_uuid
WHERE j.namespace_name = :namespaceName
ORDER BY j.name LIMIT :limit OFFSET :offset
WITH jobs_view_page
AS (
SELECT
*
FROM
jobs_view AS j
WHERE
j.namespace_name = :namespaceName
ORDER BY
j.name
LIMIT
:limit
OFFSET
:offset
),
job_versions_temp AS (
SELECT
*
FROM
job_versions AS j
WHERE
j.namespace_name = :namespaceName
),
facets_temp AS (
SELECT
run_uuid,
JSON_AGG(e.facet) AS facets
FROM (
SELECT
jf.run_uuid,
jf.facet
FROM
job_facets_view AS jf
INNER JOIN job_versions_temp jv2
ON jv2.latest_run_uuid = jf.run_uuid
INNER JOIN jobs_view_page j2
ON j2.current_version_uuid = jv2.uuid
ORDER BY
lineage_event_time ASC
) e
GROUP BY e.run_uuid
)
SELECT
j.*,
f.facets
FROM
jobs_view_page AS j
LEFT OUTER JOIN job_versions_temp AS jv
ON jv.uuid = j.current_version_uuid
LEFT OUTER JOIN facets_temp AS f
ON f.run_uuid = jv.latest_run_uuid
ORDER BY
j.name
""")
List<Job> findAll(String namespaceName, int limit, int offset);

Expand Down
Loading