`make O=build` out-of-tree #12463

vtjnash · 2015-08-05T03:18:11Z

this implements out-of-tree builds (without requiring cmake 😄)

inspired by the linux kernel kbuild system, this uses make O= to setup a target directory. it then creates dummy makefiles in the out-of-tree build directories so that the user can more easily do incremental build work. the user can place independent Make.user files in each build tree to override the global Make.user file in the toplevel source directory.

additionally, it moves all source-file related content into deps/srccache for easier tracking (and build sharing).

tkelman · 2015-08-05T11:42:00Z

-1, would rather get further on cmake which just does this for you. If this can be made to work without breaking the CI, then maybe, but right now it's adding even more complication to the makefiles and seems like a really strange thing to be working on right this second.

yuyichao · 2015-08-05T11:47:38Z

base/fs.jl

@@ -47,7 +47,7 @@ export File,
 import Base: uvtype, uvhandle, eventloop, fd, position, stat, close, write, read, read!, readbytes, isopen,
 _sizeof_uv_fs, uv_error

-include("file_constants.jl")
+include(string(length(Core.ARGS)>=2?Core.ARGS[2]:"","file_constants.jl")) # include($BUILDROOT/base/file_constants.jl)


Well, this is a bit that is necessary in the cmake port for proper out-of-tree build as well. I'm currently adding this capability using a fallback path (which can be removed with stringreplace on the binary) in jl_load (the main advantage being not having to change anything in the julia code).

I'm at least interested in which way people prefer.

Why not think about populating these lists of constants in some other way? There's some really godawfully messy perl going on in the makefiles right now to generate these, no one would miss those lines if they went away.

i would rather stick with the minimal change (this) in this PR, but can certainly entertain other ideas for implementation in a separate PR. the most logical change would probably to change this to a ccall to a method that has a big case statement to pick the right answer for the input string.

I still don't get what is going on here, or why some of these need vcat and .data. Let's not rush this with ugly confusing code.

some of these are early enough that strings are defined yet, so they need vcat and .data. it should be possible to make that more opaque in a future PR, but that would take a bit more restructuring and is unrelated to this PR. (this change, or something equivalent also needs to go in to make the cmake build work, so it isn't wasted work)

tkelman · 2015-08-05T15:37:49Z

base/Makefile

-fenv_constants.jl: ../src/fenv_constants.h
- @$(PRINT_PERL) $(CPP_STDOUT) -DJULIA -I../deps/openlibm/include ../src/fenv_constants.h | tail -n 8 | perl -ple 's/\sFE_UN\w+/ 0x10/g; s/\sFE_O\w+/ 0x08/g; s/\sFE_DI\w+/ 0x04/g; s/\sFE_INV\w+/ 0x01/g; s/\sFE_TON\w+/ 0x00/g; s/\sFE_UP\w+/ 0x800/g; s/\sFE_DO\w+/ 0x400/g; s/\sFE_TOW\w+/ 0xc00/g' > $@
+$(BUILDDIR)/fenv_constants.jl: $(SRCDIR)/../src/fenv_constants.h
+ @$(PRINT_PERL) $(CPP_STDOUT) -DJULIA -I$(SRCDIR)/../deps/openlibm/include $< | tail -n 8 | perl -ple 's/\sFE_UN\w+/ 0x10/g; s/\sFE_O\w+/ 0x08/g; s/\sFE_DI\w+/ 0x04/g; s/\sFE_INV\w+/ 0x01/g; s/\sFE_TON\w+/ 0x00/g; s/\sFE_UP\w+/ 0x800/g; s/\sFE_DO\w+/ 0x400/g; s/\sFE_TOW\w+/ 0xc00/g' > $@


the openlibm path strikes me as though it would be broken in a NO_GIT build - do we need it?

only on windows, IIRC

Oh, whoops, I added that in #6230. So only with MSVC I think, which I believe is broken in a NO_GIT build. Possibly for reasons other than this too.

vtjnash · 2015-08-05T17:29:02Z

If this can be made to work without breaking the CI

Of course (also done now)

would rather get further on cmake which just does this for you

sure. the benefit in my mind here is that we can merge this now (since it doesn't change the build system much to add), and keep working on the cmake rewrite

tkelman · 2015-08-05T17:37:17Z

Would rather wait until after branching release-0.4 to merge this. It should theoretically be non-disruptive, but some of these build configurations don't get tested very often.

yuyichao · 2015-08-05T17:45:52Z

FWIW, I do prefer merging the necessary bits for out-of-source build of sysimg first. It has been the part that introduce the most conflict...

vtjnash · 2015-08-05T17:45:57Z

Would rather wait until after branching release-0.4 to merge this. It should theoretically be non-disruptive, but some of these build configurations don't get tested very often.

I'm actually thinking that if we can get this in before v0.4, it'll make it easier to keep a build tree around for v0.4, so that developers can test against many branches more easily.

aside from moving a lot of source files out of deps into deps/srccache, the way i implemented these build options, they are usually set to the empty string, so that this has very minimal impact on the in-tree build.

the src directories have supported this option for quite some time, this just finishes that work by extending support to deps, ui, base, etc.

tkelman · 2015-08-05T17:53:44Z

The ARGS thing feels really hacky, and this breaks Cygwin cross-compile. This is supposed to be minimal impact, but it isn't as minimal as you think it is. It needs to be reviewed and tested, and we're trying to get master into an RC state. I don't think this is high enough priority to worry about right now.

vtjnash · 2015-08-05T18:12:26Z

The ARGS thing feels really hacky, and this breaks Cygwin cross-compile

thanks for point that out. i think i've fixed it (still waiting for my cross-compile build of llvm to finish): 4bba411#diff-b67911656ef5d18c4ae36cb6741b7965R193

tkelman · 2015-08-05T18:14:27Z

Are you cross compiling from cygwin or linux? The line you pointed to isn't the issue, it's the failure to find the stdlibs. You're undoing part of #5965, the difference now is breaking the Cygwin cross-compile means no Windows binaries.

vtjnash · 2015-08-05T18:29:52Z

Are you cross compiling from cygwin or linux? The line you pointed to isn't the issue, it's the failure to find the stdlibs. You're undoing part of #5965, the difference now is breaking the Cygwin cross-compile means no Windows binaries.

my cygwin build is further behind. i didn't realize the mingw64-cygwin compiler was lying about it's paths, i've put that line back.

tkelman · 2015-08-05T22:23:57Z

LLVM patches are still broken on cygwin here

/bin/sh: -c: line 0: syntax error near unexpected token `newline'
/bin/sh: -c: line 0: `cd /home/Tony/julia/deps/srccache/llvm-3.3 && patch -p1 < '
Makefile:669: recipe for target '/home/Tony/julia/deps/srccache/llvm-3.3/llvm-3.3.patch-applied' failed
make[1]: *** [/home/Tony/julia/deps/srccache/llvm-3.3/llvm-3.3.patch-applied] Error 1
Makefile:60: recipe for target 'julia-deps' failed

vtjnash · 2015-08-05T22:38:57Z

ah, that didn't actually work anywhere.

tkelman · 2015-08-06T02:40:25Z

still doesn't?

/bin/sh: line 0: cd: /home/Tony/julia/deps/srccache/llvm-3.3: No such file or directory
Makefile:669: recipe for target '/home/Tony/julia/deps/srccache/llvm-3.3/llvm-3.3.patch-applied' failed
make[1]: *** [/home/Tony/julia/deps/srccache/llvm-3.3/llvm-3.3.patch-applied] Error 1
Makefile:60: recipe for target 'julia-deps' failed
make: *** [julia-deps] Error 2

vtjnash · 2015-08-06T04:56:13Z

oh, right. i guess it would also help if the patch files depended on the source actually existing too.
thanks for checking all of these.

tkelman · 2015-09-14T15:00:57Z

README.md

@@ -77,9 +77,11 @@ When compiled the first time, it will automatically download and build its [exte
 This takes a while, but only has to be done once. If the defaults in the build do not work for you, and you need to set specific make parameters, you can save them in `Make.user`. The build will automatically check for the existence of `Make.user` and use it if it exists.
 Building Julia requires 1.5GiB of disk space and approximately 700MiB of virtual memory.

+For release-0.4 and newer builds of Julia, you can create out-of-tree builds of Julia by specifying `make O=<build-directory> configure` on the command line. This will create a directory mirror, with all of the necessary Makefiles to build Julia, in the specified directory. These builds will share the source files in Julia and `deps/srccache`. Each out-of-tree build directory can have its own `Make.user` file to override the global `Make.user` file in the toplevel folder.


0.5. If we backport this it won't be for a month or so.

tkelman doesn't like it

vtjnash · 2015-09-14T19:28:19Z

I'm waiting for CI on master to be green again before merging this. In the meantime, here's my email draft:

vtjnash · 2015-09-14T19:28:37Z

I'm waiting for CI on master to be green again before merging this. In the meantime, here's my email draft:

I've recently merged #12463 which enables out-of-tree builds. What this means for you is:

deps will rebuild. All external source files are now in deps/srccache, all external build files are now in deps/build.
make O=output will start an out-of-tree build in the folder "output"
dummy Makefiles are setup in the out-of-tree build folders for easier incremental building from any directory
The new makefile target configure will create these dummy Makefiles
by placing a Make.user file at output/Make.user, you can override the behavior of the global Make.user file for that directory, allowing testing for a wider range of systems (windows, 32-bit, llvm-versions, etc), all from the same source tree. Note that the global Make.user file is included in every build folder, but is run prior to the Make.user file to allow overrides.
unlike most other out-of-tree build systems, this system does not care if you also have an in-tree build present

(this is all mentioned in https://github.com/JuliaLang/julia/blob/jn/makefile_o/README.md#source-download-and-compilation also)

tkelman · 2015-09-14T19:35:06Z

Cool, there's a new conflict here but +1 from me on merging once that's resolved.

I'd add that if people plan on switching branches to release-0.4, it would be safer to leave existing build trees in place for release-0.4 (to avoid rebuilding) and doing new clones for master. Hopefully the previous issue with switching branches won't recur but it might.

vtjnash · 2015-09-14T19:39:26Z

as long as you rm -rf julia/usr between switching branches, you should be fine to stay in the same build tree

tkelman · 2015-09-15T00:26:44Z

Good call. The issue that I hit earlier with switching between branches with vs without this change looks fixed now.

The buildbots are having some trouble though:

OSX http:https://buildbot.e.ip.saba.us:8010/builders/package_osx10.9-x64/builds/644/steps/make%20.app/logs/stdio
Linux http:https://buildbot.e.ip.saba.us:8010/builders/package_tarball32/builds/1512/steps/make%20binary-dist/logs/stdio
Windows http:https://buildbot.e.ip.saba.us:8010/builders/package_win6.2-x64/builds/122/steps/make%20binary-dist/logs/stdio (there's also an empty cygpath -w in there causing warnings, RELBUILDROOT should maybe be . instead of empty?)

Keno · 2015-09-15T01:08:59Z

Something seems wrong. On a fresh checkout, I get:

make: *** No rule to make target 'jltypes.o', needed by '/Users/kfischer/Projects/julia-sa/usr/lib/libjulia.dylib'.  Stop.

vtjnash · 2015-09-15T01:41:39Z

that means it was missing one of the dependencies, so it couldn't figure out how to build it. i usually see that if libuv.a or llvm-config is missing

Keno · 2015-09-15T01:43:04Z

Could we get a better error message for that?

vtjnash · 2015-09-15T01:45:47Z

was that the actual issue? i would have to hard-code a test for them, but it is possible.

Keno · 2015-09-15T01:47:07Z

It worked fine after rm -rfing usr. It doesn't have to be a very specific error message, just telling the user to delete usr is enough.

vtjnash · 2015-09-15T01:52:09Z

that should have been overkill. make at the toplevel should have invoked make -C deps, which should have targets to ensure all of the dependencies are installed before it invoked make -C src

vtjnash · 2015-09-15T01:53:50Z

@tkelman: this seems suspicious to me:

make[3]: '/home/Administrator/buildbot/slave/package_win6_2-x64/build/usr/lib/julia/inference.ji' is up to date.

why would that file be up-to-date without having built it? is it possible there is old build state stuck around in the form of a file called usr/.examples that is inhibiting updating of usr/share/docs/julia?

vtjnash · 2015-09-15T01:54:33Z

(there's also an empty cygpath -w in there causing warnings, RELBUILDROOT should maybe be . instead of empty?)

the warning only affects cygwin, and is non-fatal, so i don't care

vtjnash · 2015-09-15T01:55:48Z

Good call. The issue that I hit earlier with switching between branches with vs without this change looks fixed now.

you definitely would still need to rm -rf usr to switch very reliably, but the whole point of moving the build to deps/build was that it makes that transition cleaner and simpler

tkelman · 2015-09-15T02:01:58Z

why would that file be up-to-date without having built it? is it possible there is old build state stuck around in the form of a file called usr/.examples that is inhibiting updating of usr/share/docs/julia?

Potentially. The buildbots generally do incremental builds unless we manually clear them out. I just did so, but it'll be more time-consuming if every time we go back and forth between building release-0.4 and master we need to completely clean them out. Can probably switch them to doing rm -rf usr instead of the make cleanall that they're doing now.

vtjnash · 2015-09-15T02:08:06Z

maybe not a bad idea to do rm -rf usr anyways, but i see the make clean rule that got lost in the rebase attempts.

vtjnash · 2015-09-15T02:13:08Z

fixed in 5b40f2b

(backport tag)

fixes a cygwin warning from #12463 due to `cygpath -w `

mbauman · 2015-09-23T14:12:59Z

Did this change the location of JULIA_HOME for local (non-installed) source builds? I keep editing files in ./usr/share/julia/base instead of ./base. It's not a big deal, but surely others have run into this, too. Any workarounds?

vtjnash · 2015-09-23T15:44:55Z

usr/share/julia/base is a symlink to base, so the exact path shouldn't matter. the former is a more stable location, however, for working with out-of-tree builds.

mbauman · 2015-09-23T15:46:00Z

Should have looked a little closer. That's great. Thanks!

vtjnash force-pushed the jn/makefile_o branch 5 times, most recently from 0e9f22d to e5f062f Compare August 5, 2015 04:54

tkelman added the domain:building Build system, or building Julia or its dependencies label Aug 5, 2015

yuyichao reviewed Aug 5, 2015
View reviewed changes

vtjnash force-pushed the jn/makefile_o branch 2 times, most recently from a4307bb to 2144ea6 Compare August 5, 2015 15:25

tkelman reviewed Aug 5, 2015
View reviewed changes

vtjnash force-pushed the jn/makefile_o branch from 2144ea6 to e861c92 Compare August 5, 2015 15:48

vtjnash mentioned this pull request Aug 5, 2015

remove openlibm include for fenv_constants.jl #12471

Merged

vtjnash force-pushed the jn/makefile_o branch from 7c7d654 to 2c73ade Compare August 5, 2015 16:35

vtjnash force-pushed the jn/makefile_o branch from 2c73ade to 4bba411 Compare August 5, 2015 18:11

vtjnash force-pushed the jn/makefile_o branch from 4bba411 to 998f110 Compare August 5, 2015 18:29

vtjnash force-pushed the jn/makefile_o branch from 81b533e to 70fd792 Compare August 5, 2015 20:34

vtjnash force-pushed the jn/makefile_o branch from d2d4ca6 to ba96d44 Compare August 6, 2015 04:55

tkelman reviewed Sep 14, 2015
View reviewed changes

vtjnash added 2 commits September 14, 2015 14:16

remove changes to PATH variable from WINNT makefile

ea5c2b9

tkelman doesn't like it

makefile output directory PR did not make it into release-0.4

bd22c1e

vtjnash merged commit bd22c1e into master Sep 14, 2015

vtjnash deleted the jn/makefile_o branch September 14, 2015 20:05

tkelman added a commit that referenced this pull request Sep 15, 2015

Modify contrib/relative_path.sh to return ./ instead of empty

cf94c24

fixes a cygwin warning from #12463 due to `cygpath -w `

tkelman mentioned this pull request Sep 15, 2015

RELBUILDROOT does not need to have cygpath_w called on it #13131

Merged

tkelman added a commit that referenced this pull request Sep 15, 2015

RELBUILDROOT does not need to have cygpath_w called on it

c750eb2

fixes a cygwin warning from #12463 due to `cygpath -w `

twadleigh mentioned this pull request Sep 18, 2015

gmp, mpfr, virtualenv build broken in windows (msys2) #13206

Closed

yuyichao mentioned this pull request Sep 20, 2015

set LLVM_VER from LLVM_CONFIG when using system LLVM #13241

Merged

make O=build out-of-tree #12463

make O=build out-of-tree #12463

Conversation

vtjnash commented Aug 5, 2015

tkelman commented Aug 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vtjnash commented Aug 5, 2015

tkelman commented Aug 5, 2015

yuyichao commented Aug 5, 2015

vtjnash commented Aug 5, 2015

tkelman commented Aug 5, 2015

vtjnash commented Aug 5, 2015

tkelman commented Aug 5, 2015

vtjnash commented Aug 5, 2015

tkelman commented Aug 5, 2015

vtjnash commented Aug 5, 2015

tkelman commented Aug 6, 2015

vtjnash commented Aug 6, 2015

Choose a reason for hiding this comment

vtjnash commented Sep 14, 2015

vtjnash commented Sep 14, 2015

tkelman commented Sep 14, 2015

vtjnash commented Sep 14, 2015

tkelman commented Sep 15, 2015

Keno commented Sep 15, 2015

vtjnash commented Sep 15, 2015

Keno commented Sep 15, 2015

vtjnash commented Sep 15, 2015

Keno commented Sep 15, 2015

vtjnash commented Sep 15, 2015

vtjnash commented Sep 15, 2015

vtjnash commented Sep 15, 2015

vtjnash commented Sep 15, 2015

tkelman commented Sep 15, 2015

vtjnash commented Sep 15, 2015

vtjnash commented Sep 15, 2015

mbauman commented Sep 23, 2015

vtjnash commented Sep 23, 2015

mbauman commented Sep 23, 2015

`make O=build` out-of-tree #12463

`make O=build` out-of-tree #12463