Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

detect errors in generated function declarations early #18605

Merged
merged 1 commit into from
Sep 20, 2016
Merged

detect errors in generated function declarations early #18605

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
detect errors in generated function declarations early 
this would have detected #18577 before it was merged
  • Loading branch information
@vtjnash
vtjnash committed Sep 20, 2016
commit b4fc653b72d1d011e91023acb8028a39e6305f79
3 changes: 2 additions & 1 deletion base/multidimensional.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1047,9 +1047,10 @@ the order that the first of each set of equivalent elements originally appears.
If `dim` is specified, returns unique regions of the array `itr` along `dim`.
"""
@generated function unique{T,N}(A::AbstractArray{T,N}, dim::Int)
inds = inds -> zeros(UInt, inds)
quote
1 <= dim <= $N || return copy(A)
hashes = similar(inds->zeros(UInt, inds), indices(A, dim))
hashes = similar($inds, indices(A, dim))

# Compute hash for each row
k = 0
Expand Down
18 changes: 15 additions & 3 deletions src/alloc.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -485,10 +485,16 @@ JL_DLLEXPORT jl_code_info_t *jl_code_for_staged(jl_method_instance_t *linfo)
jl_code_info_t *func = NULL;
JL_GC_PUSH4(&ex, &linenum, &sparam_vals, &func);
jl_ptls_t ptls = jl_get_ptls_states();
int last_lineno = jl_lineno;
int last_in = ptls->in_pure_callback;
jl_module_t *last_m = ptls->current_module;
jl_module_t *task_last_m = ptls->current_task->current_module;
assert(jl_svec_len(linfo->def->sparam_syms) == jl_svec_len(sparam_vals));
JL_TRY {
ptls->in_pure_callback = 1;
// need to eval macros in the right module
ptls->current_task->current_module = ptls->current_module = linfo->def->module;

ex = jl_exprn(lambda_sym, 2);

int nargs = linfo->def->nargs;
Expand Down Expand Up @@ -522,18 +528,24 @@ JL_DLLEXPORT jl_code_info_t *jl_code_for_staged(jl_method_instance_t *linfo)
ex = newast;
}

// need to eval macros in the right module, but not give a warning for the `eval` call unless that results in a call to `eval`
func = (jl_code_info_t*)jl_toplevel_eval_in_warn(linfo->def->module, (jl_value_t*)ex, 1);
assert(jl_is_code_info(func));
func = (jl_code_info_t*)jl_expand((jl_value_t*)ex);
if (!jl_is_code_info(func))
jl_error("generated function body is not pure. this likely means it contains a closure or comprehension.");

jl_array_t *stmts = (jl_array_t*)func->code;
for (i = 0, l = jl_array_len(stmts); i < l; i++) {
jl_array_ptr_set(stmts, i, jl_resolve_globals(jl_array_ptr_ref(stmts, i), linfo->def->module));
}
ptls->in_pure_callback = last_in;
jl_lineno = last_lineno;
ptls->current_module = last_m;
ptls->current_task->current_module = task_last_m;
}
JL_CATCH {
ptls->in_pure_callback = last_in;
jl_lineno = last_lineno;
ptls->current_module = last_m;
ptls->current_task->current_module = task_last_m;
jl_rethrow();
}
JL_GC_POP();
Expand Down
23 changes: 4 additions & 19 deletions src/builtins.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -552,51 +552,36 @@ JL_CALLABLE(jl_f__apply)
// eval -----------------------------------------------------------------------

JL_DLLEXPORT jl_value_t *jl_toplevel_eval_in(jl_module_t *m, jl_value_t *ex)
{
return jl_toplevel_eval_in_warn(m, ex, 0);
}

jl_value_t *jl_toplevel_eval_in_warn(jl_module_t *m, jl_value_t *ex, int delay_warn)
{
jl_ptls_t ptls = jl_get_ptls_states();
static int jl_warn_on_eval = 0;
int last_delay_warn = jl_warn_on_eval;
if (m == NULL)
m = jl_main_module;
if (jl_is_symbol(ex))
return jl_eval_global_var(m, (jl_sym_t*)ex);
jl_value_t *v=NULL;
if (ptls->in_pure_callback)
jl_error("eval cannot be used in a generated function");
jl_value_t *v = NULL;
int last_lineno = jl_lineno;
jl_module_t *last_m = ptls->current_module;
jl_module_t *task_last_m = ptls->current_task->current_module;
if (!delay_warn && jl_options.incremental && jl_generating_output()) {
if (jl_options.incremental && jl_generating_output()) {
if (m != last_m) {
jl_printf(JL_STDERR, "WARNING: eval from module %s to %s: \n",
jl_symbol_name(m->name), jl_symbol_name(last_m->name));
jl_static_show(JL_STDERR, ex);
jl_printf(JL_STDERR, "\n ** incremental compilation may be broken for this module **\n\n");
}
else if (jl_warn_on_eval) {
jl_printf(JL_STDERR, "WARNING: eval from staged function in module %s: \n", jl_symbol_name(m->name));
jl_static_show(JL_STDERR, ex);
jl_printf(JL_STDERR, "\n ** incremental compilation may be broken for these modules **\n\n");
}
}
if (ptls->in_pure_callback && !delay_warn)
jl_error("eval cannot be used in a generated function");
JL_TRY {
jl_warn_on_eval = delay_warn && (jl_warn_on_eval || m != last_m); // compute whether a warning was suppressed
ptls->current_task->current_module = ptls->current_module = m;
v = jl_toplevel_eval(ex);
}
JL_CATCH {
jl_warn_on_eval = last_delay_warn;
jl_lineno = last_lineno;
ptls->current_module = last_m;
ptls->current_task->current_module = task_last_m;
jl_rethrow();
}
jl_warn_on_eval = last_delay_warn;
jl_lineno = last_lineno;
ptls->current_module = last_m;
ptls->current_task->current_module = task_last_m;
Expand Down
2 changes: 0 additions & 2 deletions src/julia_internal.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,8 +324,6 @@ jl_function_t *jl_module_call_func(jl_module_t *m);
int jl_is_submodule(jl_module_t *child, jl_module_t *parent);

jl_value_t *jl_toplevel_eval_flex(jl_value_t *e, int fast, int expanded);
jl_value_t *jl_toplevel_eval_in_warn(jl_module_t *m, jl_value_t *ex,
int delay_warn);

jl_code_info_t *jl_wrap_expr(jl_value_t *expr);
jl_value_t *jl_eval_global_var(jl_module_t *m, jl_sym_t *e);
Expand Down
9 changes: 9 additions & 0 deletions test/core.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4578,3 +4578,12 @@ end
fVararg(x) = Vararg{x}
gVararg(a::fVararg(Int)) = length(a)
@test gVararg(1,2,3,4,5) == 5

# issue #18577
@generated f18577() = quote ()->1 end
@test try
f18577()
false
catch e
(e::ErrorException).msg
end == "generated function body is not pure. this likely means it contains a closure or comprehension."
2 changes: 1 addition & 1 deletion test/staged.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ end
@generated function _g_f_with_inner(x)
:(y->y)
end
@test (_g_f_with_inner(1))(8) == 8
@test_throws ErrorException _g_f_with_inner(1)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added by @JeffBezanson in b43c5e0 - presumably this was needed for something?

I hate to backport new restrictions, even if it's just an earlier error on code that may be invalid. Will have to see how many packages are actually doing this. What's allowed in generated functions is starting to become a pretty restrictive subset of the language, and I don't think people have been aware of that or writing them that way.

Copy link
Sponsor Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point, we likely may want to reintroduce this, but right now it gets handled incorrectly by the runtime system and will likely require not-backportable changes to fix.

What's allowed in generated functions is starting to become a pretty restrictive subset of the language

It's not "starting to become" because it's really not a new restriction, it's just better at enforcing correctness up-front now. On the other hand, the fix for #265 will add significant new restrictions to them, although it shouldn't affect anyone who's been fully observant of their definition.

and I don't think people have been aware of that or writing them that way.

That's too bad, but the manual does repeat several times that "you should /never/ write a generated function with side effects" [emphasis original] https://docs.julialang.org/en/latest/manual/metaprogramming/#generated-functions (unfortunately interspersed with various buggy examples). I can repeat it a few more times, but I consider it better just to enforce the restrictions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we please fix and elaborate on the docs while we're at it then? Most people likely don't think of using closures or comprehensions as having side effects in this way.

Copy link
Sponsor Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mixed into the same PR or on its own?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either, as long as it gets done promptly. considering this is newly enforced, it should probably be documented here though.

Copy link
Sponsor Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's not reasonable to consider a generated function with a closure to have side effects --- the side effects there are in our implementation, not in the user's code.

For backporting, it doesn't matter how "new restriction" is defined. All that matters is whether code can break.

Copy link
Sponsor Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I wouldn't have made this an error if that code wasn't already broken, so that this was just adding a clearer error than "segfault", corruption of precompiled files, or other errors in which method gets called.

I look forward to having anonymous functions again, so that they are uniquely defined by content rather than by name.

# @generated functions errors
global gf_err_ref = Ref{Int}()
Expand Down