[WIP] Use BinaryBuilder for all available dependencies by default #31441

staticfloat · 2019-03-22T01:19:58Z

The list of BB-enabled build dependencies is now:

LLVM
OpenBLAS
SuiteSparse
OpenLibm
GMP
MbedTLS
LibSSH2
MPFR
curl
LibGit2
PCRE
libunwind
libosxunwind

Furthermore, they default to being used on all systems, as long as it can accurately identify the host system triplet. Use make USE_BINARYBUILDER=0 to disable.

StefanKarpinski · 2019-03-22T02:00:36Z

I’m very excited about this.

staticfloat · 2019-03-22T08:43:04Z

TODO:

We've got weird llvm-config problems on MacOS and FreeBSD where the BB environment's flags are making it out into llvm-config --cflags. I hypothesize this is because on those platforms llvm-config is being built by clang, and it keeps more flags than it otherwise would in that case, which we definitely don't want.
PCRE2 on Windows is calling itself libpcre2-8-0.dll instead of the expected libpcre2-8.dll. Fix that.
openblas_set_num_threads threw a fit on Aarch64
linux64's libgit2 didn't load its certificates properly; this is probably a problem with my libgit2 build, I'm missing a patch or something.

quinnj · 2019-03-22T15:47:28Z

deps/Versions.make

 PATCHELF_VER = 0.9
 MBEDTLS_VER = 2.16.0
-CURL_VER = 7.56.0
+MBEDTLS_BB_REL = v0.17.0


Just a thumbs up here on the right version here for mbedtls 👍 . That's about all I have to contribute in terms of review!

ViralBShah · 2019-03-23T01:39:29Z

I tried make on this branch, but it started building openblas. Perhaps not ready to test just yet?

vtjnash · 2019-03-23T02:18:23Z

The BinaryBuilder files for Win32 SuiteSparse are built incorrectly (JuliaPackaging/Yggdrasil#7)

ViralBShah · 2019-03-23T14:49:43Z

I guess we should get dsfmt into BB as well.

StefanKarpinski · 2019-03-24T01:27:56Z

It doesn’t have to be all or nothing, much easier to get most deps using BB and then add more later.

staticfloat · 2019-03-26T09:15:07Z

Lots of progress; many platforms building properly now; but MacOS has started giving this mysterious error during bootstrap, which is very difficult to track down:

InexactError(:check_top_bit, UInt32, 0xffffffd0)

As I insert println() statements to try and figure out where this is coming from, it seems to move around. My best guess is that it's due to a problem within PCRE; some kind of incompatibility, but I haven't been able to figure out where check_top_bit is being called from, much less why it's throwing an InexactError.

In other news, our fastest platform (linux64, running on nanosoldier) did the full build+test in less than 45 minutes, which makes me pretty happy.

staticfloat · 2019-03-27T06:48:41Z

That's a lot of green. I'm satisfied with this now. I'm going to do a giant rebase to get to a reasonable number of commits, and then I think we're ready to merge.

…ng, and FreeBSD support Also fixes the `--sysroot` problem with `llvm-config --cflags` by manually filtering them out of the LLVM build

but only enable BB if we can parse the triplet

… file as well

Use this to automatically fail out when users try to use a `gcc4` tarball with a system that compiles for `cxx11`. As of the time of this commit, this will apply to LLVM, OpenBLAS, OpenLibm and SuiteSparse, although the only place we truly care aobut this is LLVM, because it's the only one that passes around `std::string`s. Also reduce the amount of work we're forcing `make` to do when constructing BB triplets

staticfloat · 2019-03-27T08:08:27Z

Okay actually it looks like the libuv build is causing a segfault on FreeBSD. I thought this might be because of some missing -fPIC flags, but it doesn't look like it.

staticfloat · 2019-03-30T03:24:11Z

Wow, green flush! I think this is ready to merge.

fredrikekre · 2019-03-30T08:53:25Z

I get

error during bootstrap:
LoadError("sysimg.jl", 3, LoadError("Base.jl", 363, LoadError("version.jl", 50, InexactError(:check_top_bit, UInt32, 0xffffffd0))))
rec_backtrace at /Users/fredrik/julia-master/src/stackwalk.c:94
record_backtrace at /Users/fredrik/julia-master/src/task.c:210
jl_throw at /Users/fredrik/julia-master/src/task.c:417
jl_parse_eval_all at /Users/fredrik/julia-master/src/ast.c:895
jl_load at /Users/fredrik/julia-master/src/toplevel.c:861
exec_program at /Users/fredrik/julia-master/usr/bin/julia (unknown line)
true_main at /Users/fredrik/julia-master/usr/bin/julia (unknown line)
main at /Users/fredrik/julia-master/usr/bin/julia (unknown line)

*** This error might be fixed by running `make clean`. If the error persists, try `make cleanall`. ***
make[1]: *** [/Users/fredrik/julia-master/usr/lib/julia/sys.ji] Error 1
make: *** [julia-sysimg-ji] Error 2

on a fresh clone on macos.

KristofferC · 2019-03-30T09:02:36Z

Works for me on Windows.

ViralBShah · 2019-03-31T21:26:46Z

I now note that on mac, everything except objconv and dsfmt is coming from BB. It seems that yggdrasil does have objconv and JuliaMath has a dsfmt builder.

@staticfloat I assume you have this coming in and a new issue isn't necessary.

Gnimuc · 2019-04-01T12:38:48Z

Will the next official Julia minor release use BB2-built binaries as well?

staticfloat · 2019-04-01T17:57:31Z

That's the way things look, yes.

staticfloat changed the title ~~Use BinaryBuilder for all available dependencies by default~~ [WIP] Use BinaryBuilder for all available dependencies by default Mar 22, 2019

staticfloat closed this Mar 22, 2019

staticfloat reopened this Mar 22, 2019

quinnj reviewed Mar 22, 2019

View reviewed changes

ViralBShah added the domain:building Build system, or building Julia or its dependencies label Mar 23, 2019

staticfloat changed the title ~~[WIP] Use BinaryBuilder for all available dependencies by default~~ Use BinaryBuilder for all available dependencies by default Mar 27, 2019

staticfloat force-pushed the sf/the_great_bb8 branch from f8aa623 to f314d74 Compare March 27, 2019 06:58

staticfloat added 11 commits March 26, 2019 23:58

Bump BinaryBuilder LLVM to include new patches, GCC ABI Multiversioni…

fd86a1e

…ng, and FreeBSD support Also fixes the `--sysroot` problem with `llvm-config --cflags` by manually filtering them out of the LLVM build

Add ability for contrib/refresh_bb_tarballs.sh to refresh a subset

99a8d03

Add GMP BB tarballs

572eabd

Default to BB turned on for the projects we've precompiled so far,

8e907ce

but only enable BB if we can parse the triplet

When distcleaning a BB install target, remove the downloaded source…

1a32a99

… file as well

Move $(COMMA) to Make.inc, add $(SPACE) as well.

4f97b74

Add MbedTLS BB tarball

691d744

Add LibSSH2 BB tarball

9001ea2

Add MPFR v4.0.2 BB tarballs

d89fb51

Update curl version to 7.61.0

c2bddbc

staticfloat force-pushed the sf/the_great_bb8 branch 3 times, most recently from c561478 to 6593c53 Compare March 27, 2019 07:57

staticfloat force-pushed the sf/the_great_bb8 branch from ce1042f to ab1c07e Compare March 28, 2019 00:51

staticfloat added 4 commits March 28, 2019 22:18

Add curl BB tarball

8724e37

Add LibGit2 v0.27.7 BinaryBuilder tarballs

b9ff412

Bump PCRE to 10.32

fe53fde

Add PCRE2 v10.32 BB tarballs

b5fd37e

staticfloat force-pushed the sf/the_great_bb8 branch 3 times, most recently from f23923e to a932d46 Compare March 29, 2019 21:37

Add LibUV BB tarball

bc29074

staticfloat force-pushed the sf/the_great_bb8 branch 2 times, most recently from a4dbf21 to ae21e66 Compare March 30, 2019 00:15

staticfloat added 4 commits March 29, 2019 20:42

Add libunwind and libosxunwind

1ae2707

Update OpenBLAS aarch64 BB tarball hashes

43f160b

Bump SuiteSparse tarballs

058118e

Version location cleanup

09d4e86

staticfloat force-pushed the sf/the_great_bb8 branch from ae21e66 to 09d4e86 Compare March 30, 2019 00:42

staticfloat merged commit 0557467 into master Mar 30, 2019

fredrikekre deleted the sf/the_great_bb8 branch March 30, 2019 08:42

staticfloat mentioned this pull request Mar 30, 2019

Roll back to PCRE2 v10.31 to fix memory problems on older MacOS machines #31556

Merged

vchuravy mentioned this pull request Mar 30, 2019

Build fails on Ubuntu with Nonrepresentable section on output #31555

Closed

ViralBShah mentioned this pull request Mar 31, 2019

Deploy pre-built dependencies onto CI #26640

Closed

16 tasks

Gnimuc mentioned this pull request Apr 12, 2019

Upgrade build system to BB2 -- Part 1 JuliaInterop/Cxx.jl#406

Merged

kalmarek mentioned this pull request Jul 14, 2019

Scs 2.1.1 jump-dev/SCS.jl#149

Merged

anaveragehuman mentioned this pull request Aug 22, 2019

dev-lang/julia: version bump to 1.2.0 gentoo/gentoo#12764

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Use BinaryBuilder for all available dependencies by default #31441

[WIP] Use BinaryBuilder for all available dependencies by default #31441

staticfloat commented Mar 22, 2019 •

edited

Loading

StefanKarpinski commented Mar 22, 2019

staticfloat commented Mar 22, 2019

quinnj Mar 22, 2019

ViralBShah commented Mar 23, 2019

vtjnash commented Mar 23, 2019

ViralBShah commented Mar 23, 2019

StefanKarpinski commented Mar 24, 2019

staticfloat commented Mar 26, 2019

staticfloat commented Mar 27, 2019

staticfloat commented Mar 27, 2019

staticfloat commented Mar 30, 2019

fredrikekre commented Mar 30, 2019 •

edited

Loading

KristofferC commented Mar 30, 2019

ViralBShah commented Mar 31, 2019 •

edited

Loading

Gnimuc commented Apr 1, 2019

staticfloat commented Apr 1, 2019

[WIP] Use BinaryBuilder for all available dependencies by default #31441

[WIP] Use BinaryBuilder for all available dependencies by default #31441

Conversation

staticfloat commented Mar 22, 2019 • edited Loading

StefanKarpinski commented Mar 22, 2019

staticfloat commented Mar 22, 2019

quinnj Mar 22, 2019

Choose a reason for hiding this comment

ViralBShah commented Mar 23, 2019

vtjnash commented Mar 23, 2019

ViralBShah commented Mar 23, 2019

StefanKarpinski commented Mar 24, 2019

staticfloat commented Mar 26, 2019

staticfloat commented Mar 27, 2019

staticfloat commented Mar 27, 2019

staticfloat commented Mar 30, 2019

fredrikekre commented Mar 30, 2019 • edited Loading

KristofferC commented Mar 30, 2019

ViralBShah commented Mar 31, 2019 • edited Loading

Gnimuc commented Apr 1, 2019

staticfloat commented Apr 1, 2019

staticfloat commented Mar 22, 2019 •

edited

Loading

fredrikekre commented Mar 30, 2019 •

edited

Loading

ViralBShah commented Mar 31, 2019 •

edited

Loading