Generalized comprehension syntax

[lindahua]

What about the following?

(f(x) for x in a)   
# this may be some indexable view of a special type that can be 
# passed as arguments, such that we can do

sum(f(x) for x in a) 

In this way, statistics can be easily implemented in an efficient way, for example

var(a) = sum(abs2(x - mean(x)) for x in a) / (length(a) - 1)

This is a one-liner, but it does not create any temporary arrays.

If the Cartesian iteration discussed in #1917 can be implemented, this would become even more powerful:

# weighted mean
wmean(a, ws) = dot(a, ws) / sum(ws)

# weighted variance
wvar(a, ws) = sum(abs2(x - mean(x)) * w for (x, w) in (a, ws)...)

[kmsquire]

A known as a 'generator' in python parlance. I swear this was discussed previously, but I can't find an issue, so perhaps it was only on the mailing list (see here: https://groups.google.com/forum/#!topic/julia-dev/iGVJ-61UmVk).

[lindahua]

I also remembered I saw the discussion somewhere. But I did not actually find an issue here, so I open one.

This, IMHO, is worth our attention.

[JeffBezanson]

Yes this would be a really good feature.

[johnmyleswhite]

+1

[StefanKarpinski]

Yes, I've long wanted this. One nice feature is that although it in principle uses tasks which may require multiple stacks and therefore might be slow, there are probably many cases where one can determine that it isn't necessary to actually have two stacks so it can be optimized away to very efficient code.

[JeffBezanson]

It's a Map iterator, that wraps another iterator and applies some function to each of its elements.

[StefanKarpinski]

Ah, ok. That will work as well.

[quinnj]

+1. Looking forward to having this.

[mbauman]

See my experiments on the mailing list with an implementation on my generators branch. (You have to manually include("base/generator.jl"), which I had forgotten myself when I rebased it today).

(edit: thanks @rfourquet)

[rfourquet]

without an "s" : include("base/generator.jl")

[rfourquet]

On top of @mbauman branch, I tried to prevent expensive allocations, at least up to dim 3.

[quinnj]

As there's quite a bit of interest/work going on here, it'd be great if we could knock out a few other long-standing comprehension issues as well:

• add comprehension support to @threads #209, being able to do @parallel [f(x) for x in itr]
• Support comprehensions with unknown-length iterables #1457 and Support guards/filters in comprehensions #550, supporting comprehensions on unknown length iterables and being able to use a filter/guard in comprehension. These should go hand in hand as they involve push! instead of pre-allocating
• for syntax without a variable #2675, being able to do for loop/comprehensions without a variable, i.e. [f() for 1:10]
• creation of BitArray using comprehension #3166, creating a BitArray instead of Array{Bool,1} when the output is known to be Bool
• type-inference-independent comprehensions #7258, type-inference independent comprehensions (I believe @JeffBezanson has already done some work on this)

[diegozea]

About #1457 and #550... Could the output be preallocated and finally resized to its final length?
Maybe...

T[ f(a) for a in A  if g(a) ]

Can be something like...

N = length(A)
out = Array(T,N)
j = 0
for i in 1:N
  if g( A[i] )
    j += 1
    out[j] = f( A[i] )
  end
end
resize!(out, j)

(push! is slower than this)