Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix some invalidations during Julia's bootstrap #36427

Merged
merged 2 commits into from
Jun 29, 2020

Conversation

timholy
Copy link
Sponsor Member

@timholy timholy commented Jun 25, 2020

This cuts the total number of invalidations that occur during Julia's build process approximately in half. It may knock about 2% off Julia's build time, though that's about in the noise so I am not sure. Here was one run on master:

Sysimage built. Summary:
Total ───────  52.165692 seconds 
Base: ───────  21.758738 seconds 41.7108%
Stdlibs: ────  30.406945 seconds 58.2892%
    JULIA usr/lib/julia/sys-o.a
Generating precompile statements... 30/30
Executing precompile statements... 1870/1877
Precompilation complete. Summary:
Total ───────  84.620999 seconds
Generation ──  58.195211 seconds 68.7716%
Execution ───  26.425789 seconds 31.2284%

and here were two runs on this(*caveat: also using #35877) branch:

Sysimage built. Summary:
Total ───────  50.963781 seconds 
Base: ───────  20.885574 seconds 40.9812%
Stdlibs: ────  30.078193 seconds 59.0188%
    JULIA usr/lib/julia/sys-o.a
Generating precompile statements... 30/30
Executing precompile statements... 1858/1865
Precompilation complete. Summary:
Total ───────  85.445948 seconds
Generation ──  59.581762 seconds 69.7304%
Execution ───  25.864185 seconds 30.2696%

Sysimage built. Summary:
Total ───────  49.995934 seconds 
Base: ───────  20.612026 seconds 41.2274%
Stdlibs: ────  29.383900 seconds 58.7726%
    JULIA usr/lib/julia/sys-o.a
Generating precompile statements... 30/30
Executing precompile statements... 1858/1865
Precompilation complete. Summary:
Total ───────  85.556325 seconds
Generation ──  59.087194 seconds 69.0623%
Execution ───  26.469131 seconds 30.9377%

Interestingly, both were about a second longer on generating the precompile statements, reducing or eliminating the total advantage. I don't know whether that's systematic. (With fewer to build, you might expect it to be faster, maybe?)

The remaining invalidations fit a couple of patterns:

  • lots of invalidations from loading reducedim.jl, which specialize methods already in active use from reduce.jl
  • lots of promote_rule invalidations due to specializing the fallbacks. None of these have large cascades of consequences, however.
  • a couple of invalidations, one with a consequence of quite a few MethodInstances (~80? I don't really remember), from code that might call a function before it has any methods defined (result: they mt-invalidate themselves):
    • with_output_color gets used by the show method in cmd.jl which gets used by Base.FileRedirect via the logging framework
    • fill! isn't defined until well after its first potential use by show(io::IO, x::Type), via the vcat call

While trying to understand the with_output_color phenomenon I added a @nospecialize, this is a bonus but it seems reasonable.

I'm happy to back out some of these if folks prefer, just let me know. I was basically going down the list, but none of these are game-changing so we can afford to drop some.

@timholy
Copy link
Sponsor Member Author

timholy commented Jun 25, 2020

Related: #36411 (that was not tested along with this, and the combination will get rid of a few more besides being more correct)

@@ -266,6 +266,9 @@ function active_project(search_load_path::Bool=true)
project == "@" && continue
project = load_path_expand(project)
project === nothing && continue
# while this seems well-inferred, nevertheless without the type annotation below
# there are backedges here from abspath(::AbstractString, ::String)
Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems very strange --- AFAICT we already know that project is a String here, so I don't know why this would be. We should investigate.

Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it --- it's due to recursion via load_path_expand. The types get narrowed later, but the edges added before the recursion resolved are still there.

base/shell.jl Outdated
@@ -33,23 +33,25 @@ function shell_parse(str::AbstractString, interpolate::Bool=true;
st = Iterators.Stateful(pairs(s))

function update_arg(x)
if !isa(x,AbstractString) || !isempty(x)
push!(arg, x)
let arg = arg
Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't expect this to do anything --- arg has a declared type, plus this function already only reads the variable.

Copy link
Sponsor Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And it doesn't help, it's still boxed. I meant to back that part out, will do it now. s and i are now unboxed.

@timholy timholy force-pushed the teh/bootstrap_invalidations branch from aeed06f to 0b55425 Compare June 25, 2020 17:54
@timholy
Copy link
Sponsor Member Author

timholy commented Jun 25, 2020

I should have also given absolute numbers: after this there are 332 non-mt_cache invalidations left, conditional on #35877. With #36441 merged should be about 322.

@timholy timholy added the compiler:latency Compiler latency label Jun 27, 2020
@JeffBezanson JeffBezanson merged commit cd13404 into master Jun 29, 2020
@JeffBezanson JeffBezanson deleted the teh/bootstrap_invalidations branch June 29, 2020 22:03
simeonschaub pushed a commit to simeonschaub/julia that referenced this pull request Aug 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler:latency Compiler latency
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants