more predictable global binding resolution #22984

vtjnash · 2017-07-27T19:39:57Z

~~This takes any syntactic assignment to a global inside a function and ensure we create the binding immediately.~~

EDIT: The design of this PR has changed, although the net effect is generally much less breaking (since now the divergence of the new vs. existing behavior was previously typically an error case):

This changes the meaning of `global` from being a directive when used at toplevel,
which forces the introduction of a new global when used in certain contexts,
to always being just an scope annotation that there should exist a global binding for the given name.

fix #18933
fix #17387

vtjnash · 2017-07-27T20:39:17Z

Of course, CI is failing because a couple of deprecations were written to depend on the old behavior of #17387 and will need an minor update to Compat first to disable this fix in one place in the code.

JeffBezanson · 2017-07-28T03:38:57Z

test/core.jl

+# issue 18933
+module GlobalDef18933
+ using Base.Test
+ f() = (global sin; nothing)


Hm, this is kind of a weird case. Normally global sin at the toplevel would create a binding in the current module. But I see that since f doesn't assign to sin, maybe it shouldn't here. I guess this behavior is probably ok.

Yes, it is rather awkward that we use global to mean different operations depending on the scope.

even worse, before we weren't even consistent about what this syntax did. for example:

if true global sin nothing end

did not create a new global binding, but merely annotated that if a variable named sin was used, that its scope is global.

Ow, good point.

Should add a test for exactly that case.

It's a minor breaking change (but I suspect it will break a handful of packages anyways), but this PR means that:

let global function f(); return 1; end end

Now expands to

global f f() = 1

And thus will not add to an existing binding from import, but will instead give an overwrite notice.

It's possible we'll require explicit qualification for extension anyway, so this might be moot. However, in the meantime it's possible that global x should not overwrite "hard-imported" bindings. That will be less breaking, since currently it gives a warning.

Sure. That operation gives a warning now, so presumably it would be less breaking for it to be ignored (have global x mean more consistently the statement "x is a global" rather than the being sometimes the command "make x global").

JeffBezanson · 2017-07-28T03:47:53Z

src/julia-syntax.scm

+ ((ssavalue? var)
+ `(= ,var ,rhs0))
+ (else
+ (error (string "invalid assignment location \"" (deparse var) "\"")))))


This looks like a good refactoring. Am I right that there's no functional change except for this error?

yes. I was thinking of using this code-path to test for globals, but ended up deciding to put it elsewhere.

maybe put this in a separate PR, especially if it's a completely independent change?

It's largely just a re-indentation to accommodate printing the error message. It seems that lisp just usually makes trivial edits require re-indenting the entire function :(

JeffBezanson · 2017-07-28T03:49:54Z

src/julia-syntax.scm

+ (if (and (pair? e) (eq? (car e) 'block))
+ e
+ `(block ,e))
+ glob-decl)))))


I'm not totally convinced it's worth adding this code instead of just looking for global assignments in jl_resolve_globals.

This moves the global decls from local scope (where the syntax does not inherently create new bindings), to global scope (where it does). I'm generally inclined to do all scope and syntax lowering here, rather than implementing any part of it in post-processing code.

Ok, I'm on board.

Also, while having jl_resolve_globals do this will mostly work, it's still broken if the global code has any other side-effects, since the current scope rules can't preserve the happens-before relationship the user may have intended. Ignoring the trivial cases like code that uses eval or include, another example of where we must give the wrong behavior is if import is in the same block as global. (so import Foo.bar; global bar() = 3 will execute in the wrong order, for example)

vtjnash · 2017-07-28T17:45:50Z

Make sure that @nanosoldier runbenchmarks(ALL, vs = ":master") isn't unhappy about the syntax chnage.

tkelman · 2017-07-28T19:33:54Z

NEWS.md

+ * The `global` keyword now always introduces a new binding when appearing in toplevel (global) scope.
+ Whereas, previously, embedding it in a block (such as `begin; global sin; nothing; end`) would change its meaning.
+ Additionally, the new bindings are now created before the statement is executed.
+ For example, `f() = (global sin = rand(); nothing)` will now reliably shadow the `Base.sin` function,


no comma at the end of this line

The comma makes this a dependent clause, which I think is preferable here (the additional statements are present for additional clarity, but aren't as uniformly true as the first statement).

"with a new, undefined sin variable" is an incomplete fragment with no clear subject of its own - it is directly describing the shadowing and doesn't stand on its own

the comma here does not make sense grammatically and should be removed.

this is not important

these are the release notes, which people will read, they should read clearly

https://en.wikipedia.org/wiki/Relative_clause#Restrictive_and_non-restrictive

If it was a complete fragment, this would be a comma splice. The rest of the sentence, on the other hand, is complete and stands on its own.

tkelman · 2017-07-28T19:34:56Z

doc/make.jl

@@ -1,152 +0,0 @@
-# Install dependencies needed to build the documentation.


put this back

fortunately, it looks like AppVeyor won't give this PR a passing mark without this (but FreeBSD will). Need to merge a couple PRs to the doc packages first.

nanosoldier · 2017-07-28T21:02:34Z

Your benchmark job has completed - possible performance regressions were detected. A full report can be found here. cc @ararslan

this ensures that we only introduce package-local bindings for values that we define in this package ref JuliaLang/julia#22984

StefanKarpinski · 2017-08-01T12:40:24Z

test/core.jl

+ f() = (global sin; nothing)
+ g() = (global cos; cos = 2; nothing)
+ @test @isdefined sin
+ @test !@isdefined cos


It would be handy here to have some way to test what a binding refers to without dereferencing it, as in resolvebinding(:sin) == :(Base.sin) and resolvebinding(:cos) == :cos, but that may be beyond the scope of this PR.

If it has a value, @which. I could add some tests here with it for additional clarity / verification, but I think it's basically already covered.

StefanKarpinski · 2017-08-01T12:45:06Z

test/core.jl

+ nothing
+ end
+ @test !@isdefined sincos
+ @test isdefined(Base, :sincos)


I find this last one slightly surprising, but I suppose it fall naturally out of if not introducing scope. Should this also test the same things with assignment (without the let)?

Yes, I think it's a natural outcome of scope rules (one way to "fix" this might be to remove global scope, ala #19324).

But I'm going to change the way this works in an upcoming commit, which may reduce its impact on typical code.

vtjnash

Jeff and I have discussed some of the implications of this fix for the global keyword, and have decided we should change the meaning of the keyword slightly, affecting the error case.

Currently, global means "make a new global" when used as a toplevel statement, but means "there is a global" when used anywhere else. Typically these two statements actually end up doing the same action, and thus differ only in the warning / error case.

This PR had been written to fix the keyword to mean "make a new global" when used in global scope or when there is an assignment to a global in local scope. But to otherwise mean "there is a global".

We have instead decided it should always mean "there is a global". This mainly just impacts cases where we currently give a warning ("WARNING: imported binding for sin overwritten in module Main"), and instead makes that operation a no-op. Because that means bindings can no longer be deleted (which was invalid anyways for various reasons), assignment to an existing global imported from another module will now emit an error ("cannot assign variables in other modules") rather than emitting a warning.

vtjnash · 2017-08-01T15:53:18Z

test/core.jl

+ f() = (global sin; nothing)
+ g() = (global cos; cos = 2; nothing)
+ @test @isdefined sin
+ @test !@isdefined cos


If it has a value, @which. I could add some tests here with it for additional clarity / verification, but I think it's basically already covered.

vtjnash · 2017-08-01T15:59:35Z

test/core.jl

+ nothing
+ end
+ @test !@isdefined sincos
+ @test isdefined(Base, :sincos)


Yes, I think it's a natural outcome of scope rules (one way to "fix" this might be to remove global scope, ala #19324).

But I'm going to change the way this works in an upcoming commit, which may reduce its impact on typical code.

tkelman · 2017-08-03T03:56:20Z

src/module.c

+ else {
+ jl_binding_t *b2 = jl_get_binding(b->owner, var);
+ if (b2 == NULL || b2->value == NULL)
+ jl_errorf("invalid method definition: imported function %s.%s does not exist", jl_symbol_name(b->owner->name), jl_symbol_name(var));


wrap long line

JeffBezanson · 2017-08-04T18:14:17Z

I've been kicking the tires on this a bit and it seems good. 👍

This changes the meaning of `global` from being a directive when used at toplevel, which forces the introduction of a new global when used in certain contexts, to always being just an scope annotation that there should exist a global binding for the given name. fix #18933 fix #17387 (for the syntactic case)

fixed

tkelman added the needs news A NEWS entry is required for this change label Jul 27, 2017

vtjnash force-pushed the jn/18933 branch from 9869fa3 to 455f08f Compare July 27, 2017 20:38

vtjnash mentioned this pull request Jul 27, 2017

only expand the side of the if statement that will be executed JuliaLang/Compat.jl#388

Merged

JeffBezanson reviewed Jul 28, 2017

View reviewed changes

vtjnash force-pushed the jn/18933 branch 2 times, most recently from 9a0073d to 8a71761 Compare July 28, 2017 18:30

tkelman reviewed Jul 28, 2017

View reviewed changes

tkelman previously requested changes Jul 28, 2017

View reviewed changes

vtjnash added a commit to vtjnash/Documenter.jl that referenced this pull request Jul 31, 2017

avoid introducing binding for String

80b055c

this ensures that we only introduce package-local bindings for values that we define in this package ref JuliaLang/julia#22984

vtjnash mentioned this pull request Jul 31, 2017

avoid introducing binding for String JuliaDocs/Documenter.jl#546

Closed

vtjnash added a commit to vtjnash/Documenter.jl that referenced this pull request Jul 31, 2017

avoid introducing binding for String

7dda0f5

this ensures that we only introduce package-local bindings for values that we define in this package ref JuliaLang/julia#22984

StefanKarpinski reviewed Aug 1, 2017

View reviewed changes

vtjnash commented Aug 1, 2017

View reviewed changes

vtjnash removed the needs news A NEWS entry is required for this change label Aug 3, 2017

vtjnash force-pushed the jn/18933 branch from 8a71761 to 5c8e679 Compare August 3, 2017 01:16

tkelman reviewed Aug 3, 2017

View reviewed changes

vtjnash force-pushed the jn/18933 branch from 5c8e679 to d3e52ba Compare August 3, 2017 16:09

tkelman mentioned this pull request Aug 3, 2017

fix codegen bugs with llvm-gc-root pass #23114

Merged

JeffBezanson approved these changes Aug 4, 2017

View reviewed changes

vtjnash force-pushed the jn/18933 branch from d3e52ba to 31904b0 Compare August 5, 2017 02:04

vtjnash force-pushed the jn/18933 branch from 31904b0 to 9e76be1 Compare August 7, 2017 15:57

vtjnash merged commit 4fea203 into master Aug 7, 2017

vtjnash deleted the jn/18933 branch August 7, 2017 22:41

JeffBezanson mentioned this pull request Nov 11, 2017

issues with the semantics of top-level expressions #24569

Open

		@@ -1,152 +0,0 @@
		# Install dependencies needed to build the documentation.

more predictable global binding resolution #22984

more predictable global binding resolution #22984

Conversation

vtjnash commented Jul 27, 2017 • edited Loading

vtjnash commented Jul 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vtjnash commented Jul 28, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tkelman Jul 28, 2017 • edited Loading

Choose a reason for hiding this comment

tkelman Aug 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nanosoldier commented Jul 28, 2017

StefanKarpinski Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StefanKarpinski Aug 1, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vtjnash left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JeffBezanson commented Aug 4, 2017

vtjnash commented Jul 27, 2017 •

edited

Loading

tkelman Jul 28, 2017 •

edited

Loading

tkelman Aug 3, 2017 •

edited

Loading

StefanKarpinski Aug 1, 2017 •

edited

Loading

StefanKarpinski Aug 1, 2017 •

edited

Loading