keyword arguments

[multinormal]

Unless I've missed this in the manual, Julia does not currently have named parameters. I have found this language feature to dramatically improve the readability, correctness, and maintainability of complex mathematical software.

I would like the following features (mostly common to R, and hacked onto Matlab using their string-value pairing convention):

• The ability, when defining a function, to specify mandatory and optional parameters.
• The ability, when defining a function, to specify default values for optional parameters (including the ability to specify that a certain function be called to obtain the default value when the function is called).
• The ability, when calling a function, to use either parameter names (verbose, but clear) or parameter position (terse, but unclear).
• The ability, when calling a function, to place named parameters in any order.

If I had to call it, I'd say Objective-C's requirement that parameter names be used is better than also allowing parameter position, but I appreciate this looks weird to users of some languages (particularly Matlab) and is perhaps too verbose in many cases (and may discourage use of anonymous functions).

I appreciate that Julia encourages defining multiple functions of the same name but of different types to solve the optional-parameters-with-default-values problem, but I suggest that providing syntactical sugar that allows a function to be defined in one place, with default values for parameters, but which is equivalent to defining multiple functions of different type, may be a good solution. Novice programmers may be confused by the ability to define multiple versions of the same function, and poor programmers are going to spread multiple definitions of a given function all over the place.

[StefanKarpinski]

This is actually something we've discussed a lot and wanted from the very beginning. It just hasn't been implemented yet. The basic outline of how keyword parameters are likely to work:

• Each paramater is either positional or given by a keyword, not both.
• No dispatch is done on keyword arguments.
• Keyword arguments always have default values.
• Positional parameters don't have default values, but you can use multiple dispatch to simulate defaults.

I know that kind of disagrees with a lot of your feature requests. The main consideration here is making keyword arguments not completely impossibly complicated to understand in the presence of multiple dispatch. For what it's worth, I initially envisioned this very much like your description — brimming with features and possibilities. However, I think that the above bare bones feature set will ensure clarity and simplicity as much as possible. It's much easier to disentangle scattered method definitions than it is to figure out the interactions between two features like keyword arguments and multiple dispatch, unless their potential for complex interactions is kept at an absolute minimum.

Regarding default values for positional parameters, I was initially for that, but I found that using multiple dispatch to express these things is both clearer and more powerful. For example, Regex constructors are defined like this:

type Regex
    pattern::ByteString
    options::Int32
    regex::Array{Uint8}
    extra::Ptr{Void}

    function Regex(pat::String, opts::Integer, study::Bool)
        pat = cstring(pat); opts = int32(opts)
        if (opts & ~PCRE_OPTIONS_MASK) != 0
            error("invalid regex option(s)")
        end
        re = pcre_compile(pat, opts & PCRE_COMPILE_MASK)
        ex = study ? pcre_study(re) : C_NULL
        new(pat, opts, re, ex)
    end
end
Regex(p::String, s::Bool)    = Regex(p, 0, s)
Regex(p::String, o::Integer) = Regex(p, o, false)
Regex(p::String)             = Regex(p, 0, false)

If you only supply one additional argument, it can either be a boolean, indicating whether to study the regex pattern or not, or it can be an integer, providing regex options. I don't think this could possibly be any clearer with parameter defaults, and it would introduce a lot of new syntax, which we'd like to avoid. In this particular case, use of default values would also obliterate the clean separation between inner and outer constructor methods.

[multinormal]

I'm really pleased to hear that keyword arguments are on the agenda. I'm not suitably qualified to comment on your design, but from the point of view of someone who has used enough languages, written a lot of numerical code in various languages, and had to help PhD students and postdocs fix their code, I can say that keywords can be really useful.

Looking again at Julia's type definition and implicit constructor definition, I worry that seemingly innocuous changes to the type definition (again, by inexperienced programmers) will change the meaning of code elsewhere; I think there's a case to be made for using keywords in constructors. Consider the following:

type Foo
  bar
  baz
end

function f(foo)
  foo.bar / foo.baz
end

f(Foo(1, 2))

Now imagine the code is edited by a new PhD student, who has done little programming before, but needs to write some code to solve a research question in time to meet a tight conference deadline. For whatever reason, they change the definition of Foo to be:

type Foo
  baz
  bar
end

Now, the function f computes baz/bar rather than bar/baz, because they didn't understand the language.

However, if you mandate the use of keyword arguments when calling the implicitly-defined constructors, the code could look like this:

type Foo
  bar
  baz
end

function f(foo)
  foo.bar / foo.baz
end

f(Foo(bar=1, baz=2))

… and the code will be protected from the new PhD student's attempt to “clean up” the definition of Foo.

I acknowledge that in designing a language you are also picking an audience. I'm really excited that you're building a language that looks very much like what I've wanted for over ten years now.

(As an aside, I'd be interested to know the syntax you have in mind for associating keywords with their values. I quite like Mathematica's ->; the PhD students often confuse = and ==, so I'd be against overloading the equals sign further.)

[HarlanH]

Stefan, one concern I have with using multiple dispatch, as you suggest, is that many of the statistical functions I'm familiar with in R, as well as a fair number of functions in Matlab, can have very long lists of options. Like, a dozen or more optional arguments. Requiring the author to write all possible subsets is prohibitive. This said, I understand (sorta) that/how Julia's design makes an R-like implementation of optional named arguments very difficult. What if, instead, there was a standardized convention and data structure to pass things that are options (rather than parameters) in a single generally-last argument? Something like:

function model(x, y, z, o:Options)
  o = Options("foo"=7, "bar"="hi mom", o) # later arguments replace earlier ones
  blahblahblah
end

It'd be better if the options could be bare words instead of strings. Ah, maybe a HashTable? Options({:foo => 7, :bar => "hi mom"})?

[JeffBezanson]

Yes, using multiple dispatch only works for optional positional arguments. You only need n definitions for n optional arguments. But, syntactic sugar for this is entirely possible, such as:

f(x, y=0, z=1) = x+2y+3z

becomes

f(x,y,z) = x+2y+3z
f(x,y) = f(x,y,1)
f(x) = f(x,0,1)

Keyword arguments are a totally different thing. Those will effectively be passed in a separate stack-allocated data structure. I don't recall the details of R's named arguments, but I suspect they don't matter. My thinking is that it will probably be confusing and/or expensive to dispatch based on keyword arguments, so they will be extra "out of band" info that gets passed along. We will eventually be able to do the full deal, including capturing and delegating keywords to other functions, etc.

[HarlanH]

I see. Yeah, the other trick is that if you only want to pass in an optional 14th argument, you either have to specify the first 13, or you have to pass in a bunch of commas, like Matlab: f(a, b, , , , , , , , , , 12). And who can tell what that means?

I really like the idea of "out of band" nature of keyword arguments that don't interact with the dispatch mechanism! Sounds like an elegant solution. Would an Options object implementation work, perhaps with some syntactic sugar, or would you want to make it completely separate from the argument list?

[JeffBezanson]

Of course, you can pass an options object of your own design at any point. We can provide an interface for "splicing" options like we do for positional arguments now: in f(x...), x can be anything iterable and its contents will be used as the arguments. In theory then we could allow, say, a HashTable to supply keyword arguments. The only benefit would be that the system could "destructure" the options into variables for you.

[StefanKarpinski]

There's one other benefit: checking that you only passed valid named parameters. In Ruby, where keyword arguments just become a hash passed into a method, it's a massive pain to check that someone didn't use an invalid keyword.

If you have a function call with 14 non-varargs positional arguments, you're doing it wrong — you should be using keyword arguments. I think the bigest concern I have about keyword arguments is delegation: you have a bunch of methods for the same function and they all end up calling the same core method but all of them can take the same keyword arguments. You don't want to have to list and pass all the keywords every time. So we need a construct that allows delegation of keyword arguments to another method. But it can be more complicated than that: a method that calls a more core method allows an additional keyword argument that it then handles itself. One particularly nasty case would be that a single method takes all the keywords of two different methods that it calls and wants to delegate some to one and some to the other. I think that may be a case where explicit delegation is required. The explicit delegation could happen all in once place, at least and then additional methods that call that core method would just delegate the pooled keywords en masse.

[samueljohn]

I'd appreciate keyword args very much. One of the things really done right in Python :-)
Looking forward to this in Julia.

[tshort]

This topic has been quiet for a while. Here is a proposal for a fairly easy way to add keyword arguments.

Proposal: pass keyword arguments as a tuple with the first element being a symbol representing the left-hand side, and the second element being the evaluated right-hand side. The following:

myfun(a = [1,2], b = "qwerty")

becomes:

myfun((:a, [1,2]), (:b, "qwerty"))

The most common way this would be used is with varargs:

function myfun(args...)
    # Do stuff with args. 
    ...
end

This does leave it up to the function author to manage and convert args to something useful. I think this could be handled much like Tim Holy's options.jl currently does.

function simplefun(x, opts...)
    @defaults opts a=3 b=2 c=2*b
    println(a)
    println(b)
    println(c)
    anotherfun(opts...)
end

Tim's @defaults macro could be adapted to also handle a tuple of tuples. Delegation would be handled by passing "opts..." as shown above.

The function author could handle things differently. For R-style DataFrame creation, this might be more appropriate:

function DataFrame(args::Tuple...)
    d = DataFrame()   # blank DataFrame
    for (k, v) in args
        d[string(k)] = v
    end
    d
end

Then, a DataFrame could be created as:

DataFrame(a = shuffle([1:5]), 
          b = randn(5))

The advantages of this approach are:

• I don't think it's a big change to Julia's parser.
• There's a pretty clear separation of positional and keyword arguments.
• It's easy to delegate keyword arguments.

[JeffBezanson]

That is a clever approach. We could even modify the parser to insert the equivalent of the @defaults expression for you when the function signature contains keyword syntax.

But, there are a few disadvantages. It becomes tricky to have functions with both keyword args and ordinary varargs. This is also a leaky abstraction; one cannot distinguish keyword args from ordinary arguments that happen to be tuples with symbols. There is also less opportunity to optimize calls since the compiler doesn't know what's really going on. It would be better to avoid allocating tuples; we could alternate symbols and values, or put all keyword args in one tuple. Maybe having a special KeywordArg type would help, since then at least you could separate keyword args from other varargs.

[tshort]

Good points. Let's consider the following function:

myfun(x1, x2, args...) = ...
# now call it
myfun(x1, x2, col = "red", y1, (a, b))

The positional arguments are straightforward. It would be more difficult to separate out keyword argument from the non-keyword arguments after it, especially the last one that is a tuple. I think your idea of a KeywordArg type would help separate these and may help with allocation issues. The varargs could come through as:

(KeywordArg(:col, "red"), y1, (a, b))

I like that better than my tuple idea and better than alternating symbols and values (that could be leaky, too).

[samueljohn]

@tshort

myfun(x1, x2, kw1="red", x3)

...is not even allowed in Python.

SyntaxError: non-keyword arg after keyword arg

[tshort]

Interesting. That is allowed in R, but I'm fine with not allowing that.

[samueljohn]

@tshort though what is possible and nice in Python (but I am not saying we need this for julia):

def myfun(a,b):
    print a, b

myfun("hello", "world")
myfun(b="world", a="hello")

"helloworld"
"helloworld"

So by naming the "ordinary" args during a call, we have an extra safty. But at the expense of a dictionary (hash map) lookup, which is not what julia aims for.

[StefanKarpinski]

@samueljohn, that is definitely a non-starter in the presence of multiple dispatch where different methods can have different parameter names.

[tshort]

@StefanKarpinski, something like

myfun(x1, x2, kw1="red", x3)

could work with multiple dispatch if myfun is defined using varargs, and keyword and plain arguments are passed to those:

function myfun(x1, x2, ops...) 
    # do something with ops to capture keyword and plain arguments
    ...
end

[stevengj]

I needed keywords for my PyCall package, in order to call Python functions with keyword arguments. Thanks to Julia's macros, this was fairly painless: you just do func(arg1, arg2, ..., @pykw kw1=val1 kw2=val2 ...), and @pykw converts the keyword arguments into a dictionary that gets passed to Python.

When it comes to syntax, however, this brings up an important use case: the set of keywords may not be known in advance in dynamic settings. If Julia ever supports keyword arguments directly, it would be good to keep this possibility in mind.

Let me propose a possible syntax. A function is declared as

function foo(arg1::Type1, arg2, args...; kw1=default1, kw2::Type2=default2, kws...)
      ....
end

and called as

foo(argval1, argval2, argval3, argval4; kw1=val1, kw3=val3)

Multiple dispatch is done on positional arguments only, but typechecking is performed on keyword arguments if their type was specified. The identifiers arg1, arg2, kw1, and kw2 are defined as local variables inside foo, with the keyword arguments taking on their default values if they were not specified by the caller. If an args... positional argument is declared, then args contains an array of any additional values (here, [argval3, argval4]) that were passed before the ;, as usual for varargs functions. If a kws... argument is declared, then kws is a Dict{Symbol,Any} that contains any unrecognized keywords passed by the caller and their values (here, {:kw3 => val3}). If a kws... argument is not declared, then it is an error for the caller to pass unrecognized keywords (i.e., keywords not specified in the function declaration).

[JeffBezanson]

This formulation of keyword arguments sounds good. I think it is basically identical to what we've been thinking.

[StefanKarpinski]

I'm still not entirely sure about the semicolons, although my experience with the sorting ABI has certainly led me to have a desire for default values to make it easier to express lots of methods more succinctly. That api would also hugely benefit from keywords. E.g. sort(words, by=lowercase) reads rather nicely. Another interesting data point in the design of keywords from the sorting API (and from DataFrames, if I recall correctly), is the fact that dispatch on types starts to look an awful lot like keyword arguments. I'm thinking of sort(words, Sort.By(lowercase)). Keyword arguments could be implemented by having each function-keyword pair have a type and having the appropriate methods to dispatch on that. I.e. f(a, b, k=val) would mean something like f(a, b, Kw_f_k(val)) where Kw_f_k is a type that just wraps a single value and allows f to dispatch appropriately. This is probably not a good actual implementation strategy, but it's worth thinking about how these relate.

[ViralBShah]

We should definitely get keyword arguments as a language feature sooner in the early releases, since they will lead to a lot of API churn in base, as well as in packages. As everyone knows, the later we do this, the more difficult and painful it will be.

[stevengj]

@StefanKarpinski, you need semicolons or something similar to separate keyword arguments from positional arguments, because you have to disambiguate whether x=y is a keyword x argument or simply an expression assigning y to x and hence passing the value of y as a positional argument. Think especially of a function combining varargs with keywords, or keywords that are only known dynamically (as in my case).

[pao]

or simply an expression assigning y to x

#590 made this a syntax error to reserve it for use in keyword arguments.

[nolta]

I've put together a very basic implementation of keyword args on the mn/kwargs branch to toy around with. Probably riddled w/ bugs, but the simple stuff appears to work. See f4a2427 for more details.

[stevengj]

@pao, ah, I didn't realize that. Then there is no syntactic reason why keyword arguments can't just be arbitrarily mixed with positional arguments, with the latter interpreted as if the keyword arguments were not there. (Though would probably be more readable to have the keywords all at the end, maybe the language shouldn't enforce this stylistic preference.)

[stevengj]

@StefanKarpinski, requiring the semicolon in method definitions but omitting it in function calls (and disallowing non-keyword assignments in function-call arguments) seems like a good compromise.

Presumably function delegation with keywords would then work as e.g. f(args...; kw...) = other(args..., kw....).

[johnmyleswhite]

I also like the idea of semicolons going into methods definitions, but not function calls. I also like the idea of disallowing non-keyword assignment occurring inside function call arguments.

[quinnj]

So I'm trying to catch up on the discussion here and I've tried rewriting some of the ODBC package with keywords in mind (which I think would be helpful).

function connect(; dsn::String="",username::String="",password::String="")
    #function body  
end
#method definition for optional arguments
connect(datasource::String) = connect(dsn=datasource,username="",password="")
#call
co = connect(dsn="mydsn")
#would keywords be optional?
co = connect("mydsn")

function query(conn,querystring; output::Union(String,Array{String,1})="DataFrame",delim::Union(Char,Array{Char,1})=',')
    #function body
end
query(querystring::String) = query(conn,querystring,output="DataFrame",delim=',')

So I realize this is a little all over the place, but here are a few questions that don't seem to be answered yet:

1. How would you use a keyword argument without a default value? In my case, I'd like 'dsn' to be keyword argument, but not necessarily supply a default, since there is none really (and "" would probably end up returning an ambiguous error)
2. How would you designate a keyword as optional? Do we still have to rely on additional methods (as in my connect() method definition above)? Not that I'm against using additional methods, I'm actually a big fan and have felt my desire for keywords drastically reduced because they're such a handy way to provide defaults.
3. Would arguments be able to be positional AND keyword? e.g. in my connect() function, could I declare the main function with keywords, and additional methods without? relying on positions?

If someone has some example code somewhere that shows various uses of the proposed design, that'd probably help us all get a better/clearer idea of how the actual implementation would end up.

Sorry if I'm ignorant, but it seems there have been a few discussions on the subject in various locations and I've had a hard time tracking down all the ideas/details.

[StefanKarpinski]

An optional keyword argument would have a default supplied; a non-optional one would have no default supplied. I'm against allowing the same argument to be supplied positionally or by keyword. In my experience it means there are more ways to call a function and two different call sites could be doing the exact same thing but look completely different and the only way I can know that is by knowing the functions signature by heart, which is rather annoying.

[JeffBezanson]

I'm glad people are starting to like the way I'm planning to implement it anyway ;)
Positional and keyword arguments will be disjoint; a given argument cannot be both. And all keyword arguments will be optional. If we really want it will be possible to add required keyword args later by allowing

function f(x; req_kw)
    ...

in definitions (a keyword arg after ; with no =).

I'm working on this now and it's rather complex when you use every feature:

function f(a, b=0, xs...; k=1, ks...)

Just think what we have to go through to call that function...

[JeffBezanson]

Ok, keyword arguments (and optional positional arguments) are now basically feature-complete on my branch. You can even pass keywords from an associative container, the same way any iterable can be used with ....

[StefanKarpinski]

I say merge and let the glorious chaos ensue!

[tshort]

Great news, Jeff!!

On Tue, Apr 2, 2013 at 6:54 PM, Stefan Karpinski
[email protected]:

I say merge and let the glorious chaos ensue!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/485#issuecomment-15807999
.

[andrioni]

Jeff, how keyword arguments will work with the block syntax? I'd love to use them in MPFR to allow things like:

with_mpfr do rounding=RoundToNearest, precision=256, trap_overflow=true
    do_stuff()
end

[JeffBezanson]

This would work fine:

with_mpfr(rounding=RoundToNearest, precision=256, trap_overflow=true) do
    do_stuff()
end

since do is just a call to with_mpfr, plus passing a function (the block) as the first argument.

[andrioni]

Oh, I see! This is really amazing, and I'd definitely love to have it in time for the 0.2 freeze!

[johnmyleswhite]

Having this merged would be great.

[JeffBezanson]

Merged --- in the usual timeframe of slightly before it's totally ready.

Caveats:

• The ability to sort keywords at compile time is not there yet, and will hugely improve performance in some cases.
• The ASTs for these things are not obvious, and that might need to change (i.e. they are not just calls with assignments inside)
• Debug info and stack traces are not quite perfect
• It might be possible to dispatch based on whether any named arguments were passed, but this is not fully supported yet
• Anonymous functions do not yet support optional or named arguments
• Reflective things like which need to be fixed up

[johnmyleswhite]

Such great news. This was the one remaining weakness of the language for me.

[ViralBShah]

This is one of those epic moments!

[mlubin]

Awesome

[StefanKarpinski]

Yes, this is going to be a huge.

[timholy]

Fantastic!

[shabbychef]

Is there a TL;DR summary of how this now works?

[JeffBezanson]

Yes, there is an overview in the manual: http:https://docs.julialang.org/en/latest/manual/functions.html#optional-arguments

[thomasmcoffee]

Is there a TL;DR summary of how this now works?

By trial and error, I have determined some rules for how this works beyond what is described in the docs. The following examples illustrate (as of Julia 0.2.0-2314.r91b975ce):

function f(a, b, c = 2, d = 3, e = 5, xs...; v = 7, w = 8, x = 10, ks...)
    ks = Dict(zip(ks...)...)
    y = ks[:y]
    z = ks[:z]
    tuple(a, b, c, d, e, xs..., v, w, x, y, z)
end

function test1()
    # returns (1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
    f(1, a = 0, b = 0, z = 12, 2, (3, 4, 5, 6, 7)..., w = 9; {:y => 11, :v => 8}...)
end

function test2()
    # returns (1, 2, 3, 4, 5, 8, 9, 10, 11, 12)
    f((1, 2)..., (3, 4)..., z = 12; c = 0, e = 0, {:y => 11, :w => 9, :v => 0}..., w = 0, {:b => 0, :v => 8}..., y = 0)
end

Some rules for function defintion:

• non-default single arguments cannot follow default arguments
• non-default arguments cannot follow positional collections (xs...)
• positional collections (xs...) cannot follow ;

Some rules for function calls:

• positional arguments cannot follow ;
• keyword-argument collections (ks...) must follow ;
• positional argument assignments, including default values in function definition, take precedence over keyword assignments
• later occurrences of keyword assignments take precedence over earlier ones, but all assignments in keyword-argument collections (ks...) are considered later than all single keyword-argument assignments

To me, this suggests a couple of questions:

1. Should the implementation make keyword assignments available inside the function as an Associative type? Right now, the above example uses ks = Dict(zip(ks...)...), which loses the ordering, but allows random access --- I could not find any built-ins to provide random access to the Array that appears by default.
2. Do the above rules about ordering and precedence in argument syntax make sense? They currently allow very confusing code, like the examples above, but also provide some useful features, for instance, overriding a subset of one collection of options by another.

It was also not apparent to me if there's a good reason why tuple(a, xs...) works, but (a, xs...) does not.