Document the lambda nature of function args

[pkoppstein]

I believe there's a bug here, but if not, then hopefully the anomaly can be explained and documented.

In the following, zip(_) is defined like so:

def zip(ary):
  . as $in
  | [ length, ary | length] | min as $min
  | [ range(0; $min) | [ $in[.], ary[.] ] ];

and the input is always:

[1,2]

The following shows pairs of jq expressions and their output:

. as $in | zip( $in )
[[1,1],[2,2]]    # good!

. as $in | $in | zip( $in  )
[[1,1],[2,2]]    # good!

zip( . )
jq: error: Cannot index number with number
jq: error: Cannot index number with number
[[1],[2]]        # not so good

. as $in | $in | zip( .  )
jq: error: Cannot index number with number
jq: error: Cannot index number with number
[[1],[2]]        # not so good

[nicowilliams]

Yes, that's a bug. So far I don't have a simpler way to reproduce it. The key seems to be that if you pass . to zip() then it fails. The differences in the bytecode are minute.

Adding [[ary]] to the array returned by zip shows something really strange!

[[1,[[0]]],[2,[[1]]]]

I'll have to bisect this...

[nicowilliams]

This can't be right, I think my check script is wrong:

983a53a is the first bad commit
commit 983a53a
Author: Stephen Dolan [email protected]
Date: Thu May 23 19:20:19 2013 +0100

'make clean' won't delete jq.1 if it can't be rebuilt.

See #131

:100644 100644 341520566f42a788d3dd72300432e6f64503378e 5508e9a274a3bcc890784d287e1d526be834f52b M Makefile.am
bisect run success

[nicowilliams]

Oh, duh. That means the bug was in 1.3. I may not let this hold up a 1.4 release; we'll see.

[nicowilliams]

Oh, right, silly me. It's not a bug. ary here is a lambda. If the lambda is . then it passes its imput as-is. What is its input? Well, it's the input where it's called! Not the input to zip(). What threw me off is that ary is invoked twice, and it's the second time that ary's . is not what you expected.

Ergo, not a bug. Just surprising.

[pkoppstein]

@nicowilliams wrote:

Ergo, not a bug. Just surprising.

I don't follow your explanation. The following shows that zip(.) should receive "." both from its input and from its argument:

$ jq -n -v 'def id(x): [.,x]; 2 | id(.)'
[ 2,  2 ]

[nicowilliams]

You should write your zip like:

def zip(ary):
  . as $in |
  ary as $ary |
  | [ length, $ary | length] | min as $min
  | [ range(0; $min) | [ $in[.], $ary[.] ] ];

In the original, when ary is the expression ., you can can see that in the first invocation of ary the input to it is the same as the input to zip, but in the second invocation the input to ary is the output of range. Function arguments are closures, and uses of them are calls to them, and the input to them is from the calling context. So when you were passing $in as zip's argument you were passing a closure that captures the $in binding as it was at that moment, and the expression $in ignores its input, outputting the value of $in instead. But when you pass . in instead, that expression doesn't close over any bindings; when it's invoked it will pass its input straight through, and in one of those invocations in zip the value of that input is not at all the one you expected, it's not the input to zip.

It took me a while to get used to this myself, as it bit me a couple of times.

[pkoppstein]

@nicowilliams -- thanks for the explanation. I'm on board now! I see also that the jq manual does give a decent explanation. Unfortunately, I skimmed the important parts of the documentation on user-defined functions, and badly misunderstood the statement:

Arguments to a function work more like callbacks than like value arguments.

I would propose modifying the paragraph in which this statement occurs so that it reads something like the following:

 `jq` functions are filters and can be defined with one or more formal parameters. Each formal parameter is always interpreted as a filter, without being bound to the function's input.  That is, within the body of the function, the input of each occurrence of the filter specified by a formal parameter is determined by context.  For example, given:

def myfilter(a): [ a, ({"a":2} | a)];

the expression {"a":1} | myfilter(.a) would yield [1, 2].

[nicowilliams]

I'll consider some doc improvement for this sort of thing. I've been thinking of a "super-advanced" section for the manual describing this and other things (generators, the comma operator's being a generator, ...) in more detail.