julia functions as C callbacks

[vtjnash]

--- update ---
the original issue is pretty much covered by cfunction. the remaining issues are:

• support closures
• adding conversion wrapper in case of inexact matches

---- original text ---
To enable certain function calls to C code, it would be helpful to have a function callback API in Julia (this is partly inspired by the libuv interface, which ended up handling most of these issues by creating a tight coupling between julila function and types and libuv callbacks and types)

The two issues that I currently see would need to be handled are:

1. Since pointers are leaving control of Julia, the garbage collector needs to know not to clean-up anything that may be used in the future (and notified when it can)
2. Since C doesn't natively have type information, it requires some sort of reverse ccall interface

To handle 1, I think it may be safest to create a garbage collector allocation pool. Objects can be pushed to and popped from it from within Julia (using ref counting?). Then possibly add a default behavior that parameters to a ccall containing a Callback object are automatically saved until all callbacks are manually deleted. (since additional and previous calls to C functions may also hold onto object pointers, it may be useful to have this as a separately available function)

To handle 2, it will be necessary to indicate type information to the ccall. One possibility is to make a type that contains the information type Callback{ params <: Tuple } end and is passed to the ccall in the third argument tuple. A generic function name would then be passed in the parameters list. A second possibility is to make Callback a simple type that contains all of that information type Callback; params <: Tuple; f :: Function; end. A third (similar) possibility is to do the exact function lookup in Julia and require something like the value returned by getmethods and printed by whicht. In any case, I envision that ccall would JIT an intermediary function that converts from raw bits types to Julia types. This prompts the last possibility (and possibly my favorite) of having make_callback return a raw pointer to a newly JIT'd intermediary C function:

callback_fcn = make_callback(fcn::Function, arg_types::Tuple)
arg2 = "test"
cpreserve(arg2); cpreserve(fcn) # if args are not automatically preserved
ccall(:cfcn, Void, (Ptr, String), callback_fcn, arg2)
...
crelease(arg2) # if args are not automatically preserved
crelease(fcn) # if args are automatically preserved, arg2 won't be freed till after this point

A final question is whether pointer_to_array is the best way to grab a c array? Or is there some way to indicate this in the argument type?

Thoughts and alternatives? Am I missing any challenging callback argument types?

[dioptre]

Hi mate,

It might be worthwhile looking at:
https://www.erlang.org/doc/apps/erts/driver.html#id82769

That's how we do it in erlang. You may want to look into the argument definitions.

One of the reasons for me checking out Julia today is to see if I can work with it and Erlang. I'd really like to be able to call Julia functions from Erlang and visa-versa. The issue with using callbacks means there will probably end up being a huge marshalling overhead. It would be great if we could produce a "shared nothing" middle man, extending the idea of libuv with common structs/object definitions for direct access and an independent "middle man" garbage tool for synchronizing the shared memory.

I'm looking to do this over clusters of machines, which Erlang is fantastic for - but I don't know if using callbacks will cut the mustard. Ideally, I'd like to directly call Julia, and manage its resources externally (with an api, static link etc.) but potentially getting the two systems to call each other through callbacks via a complex queue and resource monitoring process including garbage collection might be the only way for now. What do you think?

It might be good to do something that would work for everything. Let me know if you are interested.

Cheers
A

[StefanKarpinski]

Having a good interop story (for Python/NumPy in particular, but not exclusively) is very important to us, so this is a helpful design point to consider. Although we can't quite achieve it right now, we will eventually have C-callable Julia functions so that there is no function call overhead if you can do C-style function calls. Do you know if Erlang can do that?

[dioptre]

Yes it can with NIFs and Port Drivers,

Take a look at them here:
https://www.erlang.org/doc/tutorial/nif.html

Erlang (scale) and Julia (speed) would be a really great partnership - it would be great to get these two working together.

If you have anything that you've started on sharing julia as a library, I'd like to see/help.

Cheers
A

[dioptre]

I haven't used the Julia ZMQ implementation, but that may be a good alternative with little overhead in the meantime. What do you think?

Cheers
A

[timholy]

I've got Matlab successfully interoperating with Julia, so in principle it works very well. The main issue is the serializer. I targeted Julia's native serializer, but several folks have pointed out that we might need a more "standardized" serializer for ZMQ interop.

[timholy]

@vtjnash, to me your plans look great. I don't have anything to add, other than my enthusiasm for this development!

I don't know of a way to indicate an array in the arguments to ccall, but you're right that it would be more elegant.

[vtjnash]

I don't see an explanation on those pages for how to pass references to arbitrary Erlang functions and call them from C, which was the original point of this post. If I've missed it, can you help point me in the right direction? An example function of this is calling the system qsort which has declaration: void qsort(void *base, size_t nmemb, size_t size, int(*compar)(const void *, const void *)) (I know Julia has a built-in qsort, this is just an example). Julia already has a great FFI, with the function ccall, that requires no special c code interface for many typical functions. I would like to extend this to allow C code to use Julia functions as native callbacks (with as minimal overhead as possible).

I second the comment that using ZMQ for interop on new programs is probably a really good alternative, with low overhead and more flexible compatibility. I was really excited to see that get added to the Julia library. This proposal is more targeting existing API's that require C function style callbacks.

Using Julia as a shared library is a slightly different issue and would not be covered by this proposal. Although, it could be pretty straightforward to code an interface library that would register callbacks. Although it must be observed that Julia is single-threaded and must not be called from a different thread.

[dioptre]

This is my third day using Julia, so excuse the stab in the dark...

I'm not sure of a way to directly access erlang functions without exporting them (much like c), using callbacks or direct calls (both in the example provided) or using messaging. For the erlang system to be particularly useful, it can use multi-threaded asynchronous callbacks. If we only have a single thread, I'd imagine the call would be blocking? If this is the case why would callbacks be used at all?

If Julia is single threaded, how can we manage a non trivial message queue (or a multithreaded equivalent). I can't see anything in the Julia parrallel computing documentation that describes events, job prioority, or the ability to monitor (and fix) it's health. How does it manage multiple connections if it has only 1 thread? What if it stalls? Is there internal message passing? Can we use this if it exists? How does julia stop blocking? Does it?

I hope your response will be of use to more people interested in integrating with Julia too!

Thanks
A

[pao]

@dioptre, that is a topic well suited for discussion on -dev.

[dioptre]

@pao I've pushed the questions to https://groups.google.com/forum/?hl=en-GB&fromgroups#!topic/julia-dev/Nk-AWe2netA

I discovered this https://github.com/dioptre/julia-nlopt/blob/master/src/nlopt/nlopt_wrapper.c which may be a start.

Anyhow - would still like to hear from anyone how they recommend managing many connections/events/etc.

Thanks
A

[JeffBezanson]

Callbacks are now supported for top-level generic functions whose signatures can be made to match a given C signature. Example:

julia> f(x) = 2x+1

julia> p = cfunction(f, Float64, (Float64,))
Ptr{Void} @0x00007f4df81f8b88

julia> ccall(p, Float64, (Float64,), 2.1)
5.2

Still need to add support for closures, and possibly adding conversions in case of inexact matches.

[StefanKarpinski]

This is such a big deal in and of itself, but also as a step in the direction of being able to generate binaries and shared libraries.

[staticfloat]

This is great, Jeff! For a slightly more worked example:

// lib.c, compile with `gcc -shared -fPIC -o libcallbacktest.dylib lib.c`
#include <stdio.h>

double docallback( double (*callback)(double), double x ) {
    printf( "Input value: %.3lf\n", x );
    sleep(1);
    double y = callback(x);
    printf( "Output value: %.3lf\n", y );
    return y;
}

julia> lib = dlopen( "libcallbacktest" ); 

julia> callback(x) = 2x + 1

julia> cfunc = cfunction(callback, Float64, (Float64,))
Ptr{Void} @0x0000000102b506e8

julia> ccall( dlsym(lib, "docallback"), Float64, (typeof(cfunc), Float64,), cfunc, 1.0 )
Input value: 1.000
Output value: 3.000
3.0

The sleep(1) is just so that julia's output comes out after the printf()'s output.

[vtjnash]

This is great!

qsort{T}(a::Vector{T}, f::Function) = ccall(:qsort, Void, (Ptr{T}, Int, Int, Ptr{Void}), a, length(a), sizeof(T), cfunction(f, Int32, (Ptr{T},Ptr{T})))
sorti(a,b) = (aa=unsafe_ref(a); bb=unsafe_ref(b); print("cmp $aa to $bb\n"); int32(aa - bb))
x = [3,12,5,25,2]
qsort(x,sorti)
println(x)

[StefanKarpinski]

I predict that an awful lot of fun is going to be had with this :-)

[ViralBShah]

Let's also put this in the documentation. Adding the doc tag to this issue.

[ihnorton]

Really nice!

Maybe a small note for the docs - void return doesn't seem to work:

julia> function foo1()
         println("hello")
       end

julia> k = cfunction(foo1, None, ())
function is not yet c-callable
 in cfunction at base.jl:126

julia> k = cfunction(foo1, Void, ())
function is not yet c-callable
 in cfunction at base.jl:126

Is Void a special case?

[JeffBezanson]

Good catch. I will allow a function that returns nothing to be considered a void function.

[tshort]

Just saw this. Super!

My most immediate app uses a closure, so I guess I'll wait some more before trying this (although I've thought of getting rid of the closure for performance reasons).

[stevengj]

A common C pattern for callbacks is to provide a void* argument for callback data. (See e.g. qsort_r). Would be nice to provide a way to use this for closures & re-entrant callbacks.

e.g. for a GNU-libc qsort_r (different from BSD, unfortunately), you have:

void qsort_r(void *base, size_t nmemb, size_t size,
                    int (*compar)(const void *, const void *, void *),
                    void *arg);

Would be nice to be able to call this with something like:

qsort_r{T}(a::Vector{T}, f::Function) 
    fc = cfunction(f, Int32, (Ptr{T},Ptr{T},Closure))
    ccall(:qsort_r, Void, (Ptr{T}, Int, Int, Ptr{Void}, Closure), a, length(a), sizeof(T), fc, cclosure(f))
end

where the Closure argument wouldn't actually be passed to f, but would rather be turned into the proper closure for f(p1,p2). It would be an error for cfunction to be called with more than one Closure in its argument-type tuple.

[JeffBezanson]

Yes, a very good idea, and I'm hoping to implement this at some point.

[stevengj]

Thanks to recent updates, the C idiom of simulating closures by void* pass-through arguments is now pretty simple to exploit in order to pass true closures (see e.g. #2554).

Something like the Closure syntax above might be a nice bit of syntactic sugar, but it seems less essential now, although it could probably be implemented purely in Julia at this point. (Basically, make a macro version of cfunction that creates a top-level wrapper with gensym and munges the arguments appropriately.)

However, it would be good to document in the manual how to pass closures via this idiom.

[StefanKarpinski]

@JeffBezanson, any chance of this happening soon? PyCall and iPython stuff would be simplified by having the ability to correctly cfunction closures, and it turns out that the web framework project also needs this for interacting with the http-parser library.

[stevengj]

@StefanKarpinski, you can correctly cfunction closures now; the only question is whether more syntactic sugar is needed.

[StefanKarpinski]

However, it would be good to document in the manual how to pass closures via this idiom.

Yes, that would be very helpful. I'm not at all clear on how to do it right now.

[StefanKarpinski]

There's also the issue of how to do it in cases where the callback interface doesn't actually use the void* idiom.

[stevengj]

Well, you can always store the closure in a global if it isn't a top-level function. That isn't re-entrant, but in cases where re-entrancy is relevant (and even when it isn't) almost all decent-quality C libraries will already support the void* idiom. (If you really needed to, you could obtain re-entrancy even in this case by eval-ing a new top-level function with a gensym name.)

[vtjnash]

pretty much covered by #7906 now (except the full closure support)

[JeffBezanson]

(except the full closure support)

Um.

[pluskid]

Was about to create a bug report before I found this. Really looking forward to have closure support.

[stevengj]

@pluskid, note that most C callback APIs allow you to pass a void* pointer to arbitrary data that is passed back to the callback. This can be used to handle arbitrary closures as described here.

[pluskid]

@stevengj Thanks for the pointer! That is very helpful and actually solved my problem. I was trying to declare the function to accept Any but was not successful because it seems cfunction also does not support Any. But with the trick using Ptr{Void} and unsafe load, it works! Thanks!

[StefanKarpinski]

This seems like something that can be added post-1.0 without breaking anything, no?

[JeffBezanson]

Yes, I think this is just a missing feature.