Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Performance issue parsing saved queries #10374

Closed
gshank opened this issue Jun 27, 2024 · 1 comment
Closed

[Bug] Performance issue parsing saved queries #10374

gshank opened this issue Jun 27, 2024 · 1 comment

Comments

@gshank
Copy link
Contributor

gshank commented Jun 27, 2024

Current Behavior

Running parse on a project with a fairly large number of saved queries shows (in perf_info.json) a large "load_all_elapsed" time that's not captured in the parse times for the original projects. This probably means that a large amount of time is being spent in the "post-parse" steps, most likely one related to saved queries.

A project that ran in about 25 seconds on 1.7 was taking 207 seconds on 1.8.

Expected Behavior

The performance should be about the same.

Steps to Reproduce

Presumably a project with a large number of saved queries. A project on which this was reported had 625.

Relevant log output

Top section of 1.7 perf_info.json:

    "path_count": 2367,
    "parsed_path_count": 2367,
    "static_analysis_path_count": 768,
    "static_analysis_parsed_path_count": 405,
    "is_partial_parse_enabled": false,
    "is_static_analysis_enabled": true,
    "read_files_elapsed": 0.5725193090038374,
    "load_macros_elapsed": 4.053906535031274,
    "parse_project_elapsed": 14.454514728975482,
    "patch_sources_elapsed": 2.8166886790422723,
    "process_manifest_elapsed": 2.2025995389558375,
    "load_all_elapsed": 24.982897768029943,

Top section of 1.8 perf_info.json:

    "path_count": 2375,
    "parsed_path_count": 2375,
    "static_analysis_path_count": 768,
    "static_analysis_parsed_path_count": 405,
    "is_partial_parse_enabled": false,
    "is_static_analysis_enabled": true,
    "read_files_elapsed": 0.7103234779788181,
    "load_macros_elapsed": 2.831442817929201,
    "parse_project_elapsed": 8.928063377039507,
    "patch_sources_elapsed": 2.408103325054981,
    "process_manifest_elapsed": 192.18702581699472,
    "load_all_elapsed": 207.356563839945,

Additional Context

@gshank
Copy link
Contributor Author

gshank commented Jul 1, 2024

This issue doesn't apply to dbt core, so closing this.

@gshank gshank closed this as completed Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants