Migration towards Julia 1.0: Problem with Schur decomposition

[adamryczkowski]

To fix the following error

ERROR: LoadError: LoadError: LoadError: UndefVarError: select not defined
Stacktrace:
 [1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
 [2] top-level scope at none:0
 [3] include at ./boot.jl:317 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:1038
 [5] include at ./sysimg.jl:29 [inlined]
 [6] include(::String) at /home/adam/tmp/TensorFlow.jl/src/TensorFlow.jl:1
 [7] top-level scope at none:0
 [8] include at ./boot.jl:317 [inlined]
 [9] include_relative(::Module, ::String) at ./loading.jl:1038
 [10] include at ./sysimg.jl:29 [inlined]
 [11] include(::String) at /home/adam/tmp/TensorFlow.jl/src/TensorFlow.jl:1
 [12] top-level scope at none:0
 [13] include at ./boot.jl:317 [inlined]
 [14] include_relative(::Module, ::String) at ./loading.jl:1038
 [15] include(::Module, ::String) at ./sysimg.jl:29
 [16] top-level scope at none:2
 [17] eval at ./boot.jl:319 [inlined]
 [18] eval(::Expr) at ./client.jl:389
 [19] top-level scope at ./none:3
in expression starting at /home/adam/tmp/TensorFlow.jl/src/ops/comparison.jl:25
in expression starting at /home/adam/tmp/TensorFlow.jl/src/ops.jl:196
in expression starting at /home/adam/tmp/TensorFlow.jl/src/TensorFlow.jl:186
ERROR: Failed to precompile TensorFlow [1d978283-2c37-5f34-9a8e-e9c0ece82495] to /home/adam/.julia/compiled/v1.1/TensorFlow/IhIhf.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] macro expansion at ./logging.jl:313 [inlined]
 [3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
 [4] _require(::Base.PkgId) at ./logging.jl:311
 [5] require(::Base.PkgId) at ./loading.jl:852
 [6] macro expansion at ./logging.jl:311 [inlined]
 [7] require(::Module, ::Symbol) at ./loading.jl:834

There is no more select function in the Base (see JuliaLang/julia#28155 and deprecation notices at https://docs.julialang.org/en/v0.7.0/NEWS/)

I believe, that either we rename Base.select into TensorFlow.select or remove the function altogether. Whatever we do, we will break any code that uses Schur decomposition.

I have never used Schur decomposition and I do not know, what part does it play in the TensorFlow, so it is not up to me to decide what to do.

[oxinabox]

Rename it to TensorFlow.select
For now.

Later we might want to remove from public api. And hide behind where
See #380.

But first minimal changes just getting things working

[adamryczkowski]

Another problem - this time with the incompatible library JLD.jl. The library seems unmaintained for the last 2 months and does not compile with Julia 1.0.

Two alternatives:

1. There is a pull request waiting in the queue that fixes many errors on Julia 0.7.0: https://github.com/eford/JLD.jl and I may continue my work with this branch to make the library compatible with Julia 1.0.0. But I do not know, whether my pull request be accepted either.

2. Migrate to JLD2 - reportedly orders magnitude faster than JLD, and already compatible with Jula 1.0.0

What do you think?

[oxinabox]

Rebase your PR.
Master has migrated to JLD2
as of #414

[adamryczkowski]

@oxinabox Thank you.

Now another problem, which I do not know how to correct:

[...]
In expression starting at `TensorFlow.jl/src/TensorFlow.jl:131` - UndefVarError: deallocator not defined
[...]

The piece of code where the error is triggered reads:

function __init__()
    c_deallocator[] = @cfunction(deallocator, Cvoid, (Ptr{Cvoid}, Csize_t, Ptr{Cvoid}))
end

Originally the line was calling cfunction rather than the macro. But according to the 0.7 release notes that state

The function cfunction, has been deprecated in favor of a macro form @cfunction. Most existing uses can be upgraded simply by adding a @. The new syntax now additionally supports allocating closures at runtime, for dealing with C APIs that don't provide a separate void* env-type callback argument. (#26486)

I replaced cfunction => @cfunction

And then I get the quoted error with missing deallocator.

deallocator is defined as an empty function in core.jl:

function deallocator(data, len, arg)

end

So I do not really understand why it is not visible. Prefixing deallocator with TensorFlow. does not work either.

[oxinabox]

I think you might need

c_deallocator[] = @cfunction($deallocator, Cvoid, (Ptr{Cvoid}, Csize_t, Ptr{Cvoid}))

help?> @cfunction
@cfunction(callable, ReturnType, (ArgumentTypes...,)) -> Ptr{Cvoid}
@cfunction($callable, ReturnType, (ArgumentTypes...,)) -> CFunction

Generate a C-callable function pointer from the Julia function closure for the given type signature. To
pass the return value to a ccall, use the argument type Ptr{Cvoid} in the signature.

Note that the argument type tuple must be a literal tuple, and not a tuple-valued variable or expression
(although it can include a splat expression). And that these arguments will be evaluated in global scope
during compile-time (not deferred until runtime). Adding a '$' in front of the function argument changes
this to instead create a runtime closure over the local variable callable.

See manual section on ccall and cfunction usage.

But I am not 100% sure.

Since that function does nothing, you may be able to just remove that entirely.
Though I am not sure something might be going on.
@malmaud why is it like that?

[adamryczkowski]

Prefixing with $ have not helped.

ERROR: InitError: MethodError: Cannot `convert` an object of type Base.CFunction to an object of type Ptr
Closest candidates are:
  convert(::Type{T}, ::T) where T at essentials.jl:154
Stacktrace:
 [1] setproperty!(::Base.RefValue{Ptr}, ::Symbol, ::Base.CFunction) at ./sysimg.jl:19
 [2] setindex!(::Base.RefValue{Ptr}, ::Base.CFunction) at ./refvalue.jl:33
 [3] __init__() at /home/adam/tmp/TensorFlow.jl/src/TensorFlow.jl:132
 [4] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:627
 [5] _require_from_serialized(::String) at ./loading.jl:678
 [6] _require(::Base.PkgId) at ./logging.jl:317
 [7] require(::Base.PkgId) at ./loading.jl:852
 [8] macro expansion at ./logging.jl:311 [inlined]
 [9] require(::Module, ::Symbol) at ./loading.jl:834
during initialization of module TensorFlow

[adamryczkowski]

The only 3 places where c_deallocator[] is used are @tfcall(:TF_NewTensor, [...]) in core.jl, like this one:

@tfcall(:TF_NewTensor, Ptr{Cvoid}, (Cint, Ptr{Cvoid}, Cint, Ptr{Cvoid}, Csize_t, Ptr{Cvoid}, Ptr{Cvoid}),
            Int(dt),
            pointer(dims),
            length(dims),
            pointer(data_boxed),
            sizeof(data_boxed),
            c_deallocator[],
            C_NULL)

[adamryczkowski]

It looks like the code tries to feed the Tensor constructor with a pointer to the no-op function.

I have solved the problem by simply moving the definition of the deallocator from core.jl into the TensorFlow.jl. No need for the $.

[adamryczkowski]

And now TensorFlow fully compiles! Now it is time for unit tests...

[malmaud]

It's a callback passed to TensorFlow to deallocate a tensor when the tensor is deleted on the TensorFlow side. I made id a no-op since we're relying on the Julia GC to free the underlying array.