Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use language: julia in default Travis #8744

Merged
merged 3 commits into from
Dec 15, 2014
Merged

Conversation

tkelman
Copy link
Contributor

@tkelman tkelman commented Oct 20, 2014

Edit 12/10/14: redid this to use the new language: julia

former content:


Ref #7364 and JuliaLang/www_old.julialang.org#151 - I put together a shell script to handle installing Julia binaries on different platforms, now I'm proposing to use it in the default .travis.yml from Pkg.generate. I've got a [otherwise unfinished] test package using this script across a matrix of linux and mac on both release and nightlies here, though it looks like the osx builds are going to be queued for a while.

Also add the os: section for osx testing (has to be explicitly enabled by sending an email to [email protected], see http:https://docs.travis-ci.com/user/multi-os, it doesn't do anything if it hasn't been enabled yet).

@staticfloat
Copy link
Sponsor Member

+1, especially since this falls back to Linux-only testing with no user support. Nice work, @tkelman

@tkelman
Copy link
Contributor Author

tkelman commented Oct 20, 2014

There are some corner cases that can make the script do strange things, so I don't recommend it for anything beyond Travis use at this point, but it should serve that particular purpose fairly well. May as well centralize the platform-dependent code in one place rather than in every package's travis config.

@IainNZ
Copy link
Member

IainNZ commented Oct 21, 2014

Nice

@ivarne
Copy link
Sponsor Member

ivarne commented Oct 22, 2014

Shouldn't we just download the whole .travis.yml template from julialang.org/travis.yml instead? Seems like we update this every other month, and it would be good if newly created packages started using the new travis config immediately. That way we could also have a script that updates .travis.yml for an existing package with the latest "best practices".

also: @JuliaBackports?

@tkelman
Copy link
Contributor Author

tkelman commented Oct 22, 2014

I don't know whether there's any advantage to moving the default yml template to a separate repository, there's no benefit for it to live at some short url.

Some type of automatic detection of old idioms in .travis.yml files of existing packages would be nice, but maybe difficult. Often packages may need to manually customize that file to install some distribution packages, etc. Not sure what's best to do there - perhaps just scraping the existing yml's and collecting some statistics on what they look like in terms of age and how modified they are?

@IainNZ
Copy link
Member

IainNZ commented Oct 22, 2014

I'm 👎 on doing anything fancy with the .travis.ymls, keeping the Pkg.generate one up-to-date with best practices and then just fixing older package's ones later is working pretty well.

@kmsquire
Copy link
Member

How about a Pkg.update_travis_yml(pkg) function?

On Wednesday, October 22, 2014, Iain Dunning [email protected]
wrote:

I'm [image: 👎] on doing anything fancy with the .travis.ymls, keeping
the Pkg.generate one up-to-date with best practices and then just fixing
older package's ones later is working pretty well.


Reply to this email directly or view it on GitHub
#8744 (comment).

@IainNZ
Copy link
Member

IainNZ commented Oct 22, 2014

Sure, but I feel like its worth having some of your own understanding of what is going on in the only bit of the .travis.yml that isn't covered by the install script. It's unlikely to change too much from the current clone(pwd) and test.

@tkelman
Copy link
Contributor Author

tkelman commented Oct 22, 2014

On a somewhat related note, one wonders how big of a community we have to be before we could realistically start talking with someone at Travis about adding a first-class language: julia setup. They recently added Rust support, for example.

@StefanKarpinski
Copy link
Sponsor Member

I think we can ask them for this already.

@tkelman
Copy link
Contributor Author

tkelman commented Oct 22, 2014

Looks like it might just take a relatively simple PR to https://github.com/travis-ci/travis-build - the closed one for Rust and an open one for C# don't look that complicated. I'm not quite sure what's going on in the _spec.rb script though.

@ghost
Copy link

ghost commented Oct 27, 2014

I submitted a PR to travis-build over the weekend and Julia support is well on the way. @tkelman: Can you confirm that http:https://status.julialang.org/stable, that you used in the install script, will be maintained as a URL for the latest stable release? I just thought I should confirm this since it is not used at http:https://julialang.org/downloads/.

@tkelman
Copy link
Contributor Author

tkelman commented Oct 27, 2014

Awesome, thanks! You should've cc'ed me on that, would've liked to watch as you worked on it.

@tkelman
Copy link
Contributor Author

tkelman commented Oct 27, 2014

So the question is whether we want to use the PPA, or the generic binaries. The binaries don't need sudo to install, but they're very recent and not yet very widely used.

@ghost
Copy link

ghost commented Oct 27, 2014

You should've cc'ed me on that, would've liked to watch as you worked on it.

My apologies, I was not sure how long it would take and spent most of the time wrestling with RSpec.

So the question is whether we want to use the PPA, or the generic binaries. The binaries don't need sudo to install, but they're very recent and not yet very widely used.

I agree about the PPAs being more along the lines of what we have been doing with Julia. But if the people over at Travis wants us to refrain from requiring root access I do not see any good alternative.

@ghost
Copy link

ghost commented Oct 27, 2014

Also, before I forget, we really need to find a way not to serve Julia over HTTP, but rather over HTTPS.

@tkelman
Copy link
Contributor Author

tkelman commented Oct 27, 2014

Yeah, @staticfloat is in charge of status.julialang.org, I have no idea how hard it would be to use https for it (also the install script, if we want to use it).

@staticfloat
Copy link
Sponsor Member

Those "stable" links will remain stable. ;)

https will take a little bit of time, (have to generate a certificate) but is totally doable.

@staticfloat
Copy link
Sponsor Member

I just thought of a big reason not to host install.sh on julialang.org; we can't use https properly on julialang.org since it's a GitHub-hosted page. If https://julialang.github.io/install.sh worked, that'd be great, but it doesn't for some reason. @StefanKarpinski any way to get that to work? I'm assuming this is due to some setting that says "Forward everything to julialang.org first".

Failing that, I can get HTTPS working on status.julialang.org, so it'd be trivial to add install.sh to that website instead.

@staticfloat
Copy link
Sponsor Member

Alright, everything on status.julialang.org should now be encrypted. HTTP should redirect to HTTPS transparently. I also took the liberty of hosting install-julia.sh on the same site, since it doesn't seem to fit in one place better than the other, and we have HTTPS setup on status.julialang.org now. I also added a checksum file, install-julia.sh.sha256 for the truly paranoid.

@ivarne
Copy link
Sponsor Member

ivarne commented Oct 27, 2014

Seems like a hash hosted on the same encrypted host is just more for you to screw up? I think we should rather/also host it on the github site julialang.org/install-julia.sh.sha256, so that a potential attacker would have two targets to compromize, even tough one of them isn't encrypted.

@staticfloat
Copy link
Sponsor Member

This hash isn't really meant to protect against attackers. If they've compromised status.julialang.org, the game is up already. They could serve malicious binaries and leave install-julia.sh alone, etc... This is meant to allow for a user to double-check that the file downloaded and was written to disk properly, even though the probability of something going wrong is vanishingly small for such a tiny file.

@ghost
Copy link

ghost commented Oct 28, 2014

@staticfloat: Awesome! Thanks for the good work.

About the hashes, agreed, hashes are to mitigate corruption. If you want security, you need signatures.

@ghost
Copy link

ghost commented Nov 3, 2014

@staticfloat: Apologies for slightly hijacking the issue. A while ago, over at the Travis PR, it was requested that we added community maintainers. For now, this is @tkelman and myself. However, given that you are the one performing the actual builds, would you also be interested in being added as a maintainer so that you get notified immediately if something goes wrong?

@staticfloat
Copy link
Sponsor Member

Sure, that's fine.

@tkelman
Copy link
Contributor Author

tkelman commented Nov 3, 2014

No worries on hijacking the issue, this should be redone and simplified quite a bit once Travis rolls out the Julia support anyway.

sorta related but not really, @staticfloat is something up with libgfortran on the travis osx builds today?

@staticfloat
Copy link
Sponsor Member

Yes. gcc 4.9.2 was just released, which broke all my bottles. Waiting for gcc 4.9.2 to finish building on OSX 10.7, (Homebrew doesn't build bottles for 10.7 anymore, so I have to wait for it to build from source now) and then I should be able to rebuild openblas, arpack and suite-sparse.

@staticfloat
Copy link
Sponsor Member

Bottles are fixed now.

and add the os: section for mac testing (has to be explicitly enabled
by sending an email to [email protected], see
http:https://docs.travis-ci.com/user/multi-os )
@tkelman tkelman changed the title Use new generic install script in default Travis Use language: julia in default Travis Dec 10, 2014
@tkelman
Copy link
Contributor Author

tkelman commented Dec 10, 2014

Redid this now that Travis supports language: julia, see http:https://blog.travis-ci.com/2014-12-10-community-driven-language-support-comes-to-travis-ci/

I have the script: section commented out in the generated file since the default setting does exactly this, but the user might want to customize it a bit. versioninfo() gets displayed at the end of the Julia setup, and Pkg.init() doesn't look necessary any more.

@ghost
Copy link

ghost commented Dec 11, 2014

It took about a month and a half, but we got that PR accepted. Thanks to @tkelman for the code review and occasional poking of the Travis devs, it really helped. Also a big thanks to everyone handling the Travis docs. Should I wait before this PR is merged and backported before making an announcement over at julia-users/dev?

@tkelman
Copy link
Contributor Author

tkelman commented Dec 11, 2014

@ninjin nah, this PR only matters for new packages, be my guest and announce on the mailing lists now in case anyone wants to adapt their existing packages.

@vchuravy
Copy link
Sponsor Member

When you make the announcement remind people that run their custom test script to keep:

before_install:
   - if [[ -a .git/shallow ]]; then git fetch --unshallow; fi

otherwise overriding script also seems to override the following section travis-ci/travis-build@88c78ff#diff-b9601304d9fc462f9637f2fdd13bd885R64

- if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
script:
- julia --check-bounds=yes -e 'versioninfo(); Pkg.init(); Pkg.clone(pwd()); Pkg.build("$pkg"); Pkg.test("$pkg")'
#script:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be a good idea to add a comment along the lines of "Uncomment these lines if you wish to override the default Travis script for Julia".

@tkelman
Copy link
Contributor Author

tkelman commented Dec 11, 2014

good call @vchuravy and @ninjin

@tkelman tkelman added domain:packages Package management and loading backport pending labels Dec 13, 2014
@tkelman
Copy link
Contributor Author

tkelman commented Dec 15, 2014

Has language: julia been working well for people? Should we merge this?

@IainNZ
Copy link
Member

IainNZ commented Dec 15, 2014

I'm unaware of any registered projects using it as this point, but I haven't been getting my eyeballs on as many packages lately

@vchuravy
Copy link
Sponsor Member

OpenCL.jl is using it currently and it cut down your travis test time from 5-6min to 2-3

@tkelman
Copy link
Contributor Author

tkelman commented Dec 15, 2014

@vchuravy you got an improvement in test running time out of it? That's awesome, I actually did not expect that.

@jakebolewski
Copy link
Member

Yes, this is working great. +1 for merging.

tkelman added a commit that referenced this pull request Dec 15, 2014
@tkelman tkelman merged commit f91d927 into master Dec 15, 2014
@tkelman tkelman deleted the tk/default-travis-multi-os branch December 15, 2014 19:36
@ivarne
Copy link
Sponsor Member

ivarne commented Dec 19, 2014

Are we ready to backport this? Seems to me like we want it before 0.3.4.

@tkelman
Copy link
Contributor Author

tkelman commented Dec 19, 2014

Yes. AppVeyor's idle, you want to do it?

tkelman added a commit that referenced this pull request Dec 19, 2014
Use language: julia in default .travis.yml

and add the os: section for mac testing (has to be explicitly enabled
by sending an email to [email protected], see
http:https://docs.travis-ci.com/user/multi-os )

(cherry picked from commit 6cb77aa)

add git fetch --unshallow in commented out script section

add explanatory comment

(cherry picked from commit 9b4453a)

turn on coverage in commented-out script section
(cherry picked from commit a92ea42)
@ivarne
Copy link
Sponsor Member

ivarne commented Dec 19, 2014

Done in 5a18cc6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain:packages Package management and loading
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants