Dynamically change paths searched by dlopen()

[staticfloat]

It'd be nice to have a way to add libraries to have dlopen search. Something similar to LD_LIBRARY_PATH, but internal to Julia. Possibly even just an array we can pass into dlopen of additional prefixes to search, so that it's not a global thing. This would get rid of the currently kind of ugly try/catch mess that we get in things like ext,jl in Tk.jl.

[staticfloat]

This would also be useful when dealing with things such as non-standard homebrew install paths, etc. I want to try to avoid setting ENV['LD_LIBRARY_PATH'] because I want these dynamic library paths to be local in code, so that a package that needs to search e.g. a custom binary build directory, or a specific Homebrew opt path, etc... doesn't affect other areas of code.

[vtjnash]

I'm am about to start work on adding a DL_LOAD_PATH array global variable that will be searched by dlopen before the system paths. I'm still discussing it with Jeff for name an semantics.

(then I can automatically add ```brew --prefix/lib and such to it)

[StefanKarpinski]

LIBRARY_PATH?

[staticfloat]

Could we add a definition to dlopen() that takes in an array for temporary additions to this global array? For instance, Homebrew has opt-paths that are custom prefixes for each installed package, and these are useful for dynamic linking on OSX (/usr/local/opt/tcl-tk/lib, for instance). I'd like to not pollute a global search path when loading a package.

[vtjnash]

No, we really want to avoid using dlopen. Also, I think it's probably better to have them in the global search path so that each usage of the library doesn't need to be aware of where the library came from.

[staticfloat]

Okay. How do we load libraries if we don't use dlopen()? I'm mostly thinking in reference to things like ext.jl in Tk.jl.

[vtjnash]

I'm working on seeing how that can be simplified.

I'm currently trying implementing a function BinDeps.find_library(libnames::Vector{T}) that searches the system and the packages managers for the first library in the list it can find, adds it (permanently) to the LIBRARY_PATH if necessary, and returns the name that it found that succeeded. A similar function Base.find_library(libnames::Vector{T}) would also be added to do the same search over library name variants, and return the one that it found (but the one in base would not interact directly with the package managers or change the LIBRARY_PATH).

[vtjnash]

@staticfloat @JeffBezanson I have pushed a commit to implement this. Rather than open a pull request, I have just referenced the code here. Thoughts?

Usage has two parts to it:

1. push!(Sys.DL_LOAD_PATH, "/path/to/search/lib") -- for use by the user (or package manager) to manually alter the search path
2. const libtk = Sys.find_library(["libtk8.6", "libtk86", "libtk", "libtk8.5", "libtk85"]) -- for use by the developer to select the first library name found out of a list of alternatives and return it

(edit: those functions are exported from Sys, not Base)

[staticfloat]

This seems like a large improvement over previous methods. If you would rather have only a global namespace for DL_LOAD_PATH, I don't think there's anything to improve on here.