Julia support

[ghost]

Adding initial Julia support. Julia and Travis has been a love story for quite some time, with Travis powering the majority of testing for both the official language implementation as well as a good chunk of the available packages. This PR attempts to bring Julia support to Travis. As I this is my first attempt at writing Ruby code, so I suspect that things may be off here and there and any and all feedback is very much appreciated. The tests passed locally using Vagrant, but spec is certainly the part which I have the least understanding of, please, do kick the tires.

[joshk]

Thanks for the great contribution.

Is it possible to install Julia for the travis user without having to use sudo?

[ghost]

Is it possible to install Julia for the travis user without having to use sudo?

I take it that any call to apt should be avoided? Yes, it should be possible using the "generic" Linux binaries. If this is preferable I can take a stab at it shortly. The reason I went with apt-version was that this mirrors the current way Julia users tend to hack into .travis.yml.

[joshk]

The reason it would be good to stay away from sudo is that it would work on our docker setup which has sudo disabled due to security reasons.

[ghost]

Avoiding root access seems sensible to me. New version pushed that uses the "generic" binaries instead. Quick question, are there 32 and 64-bit build machines at Travis? If not, I may be able to cut down on the amount of code.

[meatballhat]

@ninjin Indeed, our VMs are 64-bit 😸 👏

[ghost]

...our VMs are 64-bit

Wonderful! I cut out the 32-bit related code and force-pushed. I still feel shaky if the tests really guarantee that everything is okay, they mirror the tests for Rust though. The best thing to make my worries go away would of course be to see a successful session on a Travis VM.

[joshk]

I was going to say that I would personally prefer that we keep the arch detection, even though it needed to change, but I'm fine with not having it for now.

I'll review this on Monday and see when we can get it merged in.

[ghost]

I'll review this on Monday and see when we can get it merged in.

Thanks @joshk, I will be available on Monday if there are any tweaks necessary to get the PR accepted. Removing the WIP from the title now.

[joshk]

I can look into deploying this later today but think it might be good to change one last thing.

By default I think Travis should install Julia nightly as the latest release will surely change often and mean we will need to update this often.

Thoughts?

[ghost]

By default I think Travis should install Julia nightly as the latest release will surely change often and mean we will need to update this often.

The nightly introduces breaking changes on pretty much a weekly basis and packages are generally not encouraged to keep up with it. Instead, I think we should try to couple this to the latest stable release. Unfortunately, as far as I know, there is no official URL for the generic binary when it comes to the latest stable release. However, I think there is one in the works that is already fully functional. I will patch this in, force-push and drop an e-mail to the man in charge of the build to confirm that the URL will be maintained for the foreseeable future.

[joshk]

Sounds awesome! Thank you!

[tkelman]

What does this do on an osx worker? For reference, our current "default" .travis.yml file that we auto-generate when someone creates a new package is (or would be, subject to an open PR):

language: cpp
os:
  - linux
  - osx
compiler:
  - clang
notifications:
  email: false
env:
  matrix:
    - JULIAVERSION="juliareleases"
    - JULIAVERSION="julianightlies"
before_install:
  - curl http:https://julialang.org/install-julia.sh | sh -s $JULIAVERSION
  - if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
script:
  - julia -e 'versioninfo(); Pkg.init(); Pkg.clone(pwd()); Pkg.test("$pkg")'

where $pkg gets replaced by the package name (repository name minus .jl). We can tweak the install-julia.sh script to prefer avoiding sudo, it's not very widely used yet.

[ghost]

I just force-pushed a version taking into consideration the comments above, tests passing locally, I will grab lunch now and be back in about 30 minutes.

[ghost]

@tkelman: Currently it does nothing for OS X, since I had plenty of examples of how to test on Linux, as well as using Vagrant to test that it worked, I went ahead and implemented it first. Assuming that the Travis OS X boxes are compatible with what we build I can hack in support with a few lines of code. However, I have not used a Mac for more than half a decade now, so I feel a bit shaky if I can not test the code. As a side-note, do you know of a Julia package that has OS X enabled for testing on Travis?

[tkelman]

https://github.com/JuliaOpt/JuMP.jl/blob/master/.travis.yml
It has to be explicitly turned on by sending an email request to Travis support last I checked.

We don't have a tarball binary for OSX, we use a dmg app package, so it's not quite as simple as for Linux. But if you check JuliaLang/www_old.julialang.org#151 you can see the steps under Darwin. Really just a couple more lines of work in the end.

[ghost]

@tkelman: Agreed, those where the lines I was thinking of. After checking with a colleague about the need for root access and hdiutil I hacked in OS X support.

[tkelman]

That looks great! So I think once this is running our default could be:

language: julia
os:
  - linux
  - osx
notifications:
  email: false
julia:
  - release
  - nightly

That'll be pretty damn sweet.

[ghost]

@tkelman: If that becomes the case, it would just be beautiful!

[ghost]

Thanks to the efforts from yesterday by @staticfloat, status.julialang.org is now served using HTTPS. This minor code change to fix a major wart has been force-pushed to the branch.

[joshk]

Before we merge this in, there is one last thing we ask for, and that is two community maintainers.

The Travis CI team does not have any Julia knowledge and would like to promote this addition as a community maintained addition. It would mean adding two maintainers names and emails as comments to the julia.rb file, as well as the log output.

[tkelman]

You (edit: grammar) can add me for that, tony at kelman dot net

[ghost]

You're can add me for that, tony at kelman dot net

Count me in as well, hopefully I should be the one that is most familiar with the workings of these scripts. pontus stenetorp se would be the way to reach me. Adding both myself and @tkelman to the file now, I could not find any community maintainer comments for the other files in the same directory, so I will make up a format as I go.

[tkelman]

Thanks for the help with testing here @BanzaiMan. Unfortunately status.julialang.org is down right now, we're unfortunately single-point-of-failure on our download links right now.

[staticfloat]

@ninjin I have created 0.3-latest copies of each of the latest releases on our AWS servers, so you can always get the latest stable release by downloading e.g.:

https://s3.amazonaws.com/julialang/bin/osx/x64/0.3/julia-0.3-latest-osx10.7+.dmg

Does this satisfy your need? I assume you don't need this for 0.4.0-dev right now? (I'll start doing this for 0.4 once we have stable releases)

[tkelman]

Moving discussion from JuliaLang/julia@68e5cc7#commitcomment-8902845 - @staticfloat should we modify this to use links on AWS instead of status.julialang.org ? Would AWS support something to the effect of a link for latest nightly that the buildbots could overwrite on successful builds?

[staticfloat]

We can do that, yes. We could have a https://s3.amazonaws.com/julianightlies/bin/osx/x64/julia-latest-osx10.7+.dmg or similar. Although S3 doesn't support symlinks, we can just copy the file in twice. ;)

[tkelman]

That would be under julianightlies for julia-latest-nightly, yeah?

[staticfloat]

I just edited it, yes. :)

[staticfloat]

Alright, these should be accessible now. Examples:

Win32: Latest stable 0.3, Latest nightly.
Linux64: Latest stable 0.3, Latest nightly

[ghost]

@staticfloat: Yay! I will take them for a spin as soon as I have verified that the changes from yesterday are working.

[ghost]

Stupid quoting issue fixed, moving on to integrating the new URLs after lunch.

[BanzaiMan]

@c6cf3fc is pushed to staging.

[ghost]

@BanzaiMan: Thanks, yet another trivial quoting fix coming up. Not being able to test this locally really makes you feel silly about the errors you make...

[BanzaiMan]

@31e322f is deployed to staging

[ghost]

@BanzaiMan: Thanks again!

Now, this, is a complete success. Trying out the nightlies now.

[ghost]

Okay, we are green across the board. 31e322f is good.

[ghost]

We should now be good to merge, I will go ahead squash and rebase to the latest master. @tkelman: I am running a bit low on time in regards to some of my other commitments, so I think it is best to merge this now and then hack in the support for release-0.* and 0.*.* etc. using a PR from the fork we have over in the JuliaCI group. This way we can avoid me being a bottleneck.

[BanzaiMan]

@ninjin I've deployed the feature without the squash. See https://github.com/travis-ci/travis-build/commits/master.

I'm closing this.

Thanks for your effort, everyone!

[ViralBShah]

Thanks everyone and thank you Travis. This is really amazing.

[tkelman]

@BanzaiMan I'm having difficulty getting this to work on production travis-ci.org, does the yaml validator need updating to accept the new language keys? https://github.com/tkelman/BLIS.jl/blob/travis-julia/.travis.yml

[BanzaiMan]

@tkelman travis-yaml would need to be told of the new languages. You might also have to push a new commit for the language to be recognized for the build.

[tkelman]

I think it was some crossed wires with builds getting routed to staging instead of production, looks fixed now. Though travis-ci/travis-core#416 doesn't look to have taken effect yet in the UI (or there's some other piece needed for julia: to show up as a new column).

[BanzaiMan]

We'll have to look at the column display issue. Could you open a new ticket? This one is pretty long, and it takes a long time to load. :-)

[ghost]

Could you open a new ticket? This one is pretty long, and it takes a long time to load. :-)

Agreed, thank you everyone for all the hard work put into this. Let's end with a hearty お疲れ様 on my part.