ubuntu 13.04 release

[ViralBShah]

Does this mean that julia is included in the ubuntu 13.04 release?

https://packages.ubuntu.com/raring/julia

If so, we should figure out the freeze date for packages, and ensure that we have a 0.1 release in time and sort out graphics on ubuntu (which should not be difficult). In the worst case, we should at least ensure that we get the latest stable build uploaded.

Cc: @sebastien-villemot

[svillemot]

Yes Julia will be part of Ubuntu Raring. The current version is the one that I uploaded in Debian, and is based on commit d9656f4.

The automatic import freeze from Debian is Feb 14. Past that date, I cannot do much. See https://wiki.ubuntu.com/RaringRingtail/ReleaseSchedule

If you want me to upload a version of Julia based on a more recent commit, please let me know soon. I guess having 0.1 in time is too late now.

Concerning graphics, what do you have in mind exactly?

[ViralBShah]

It is not too late to tag 0.1. Would it be fair to say that whatever we have, we should target to upload by Feb 10, giving time for build issues and such?

[svillemot]

I think this is reasonable.

Concerning graphics, I have never tried so far to make anything work. I guess you meant having the winston thing running?

[nolta]

For winston (and gadfly too, i think), we'll need the following packages:

libcairo2
libpango1.0-0
tk-dev

[ViralBShah]

Also we need a dependency on git for the package manager.

[diegozea]

People isn't going to see Ubuntu 13.04 until end of April. Even if Debian import freeze on February, a lot of changes can happen on Julia before April. Maybe the best feature before the freeze is a simple update( ) function inside Julia, for an easy update of Julia to the current version. I don't know how this works, Is it possible?

[timholy]

I think if you want to update frequently, one should just use our existing mechanisms (git + make). The whole point of a distribution-provided package is that you rely on the distribution to make decisions about when it's time to update.

I for one will welcome a bit of medium-term stability in the foreseeable future. On my laptop I update regularly, but I have my lab still running a Julia from October. Once you get a really important large set of functionality working, breakage becomes something you want to avoid. I've been waiting to update our own internal codebase until there's a 0.1 release.

[timholy]

That said, I should have also said that I hugely support all the refactoring that has gone on. Now is much better than later, and it will make Julia much more pleasant in the long run! It's nice that I've been able to have both stability (by not updating) and access to all the amazing things that have happened over the last months.

@StefanKarpinski , for your recently announced "package overhaul," it would be nice to have a mechanism whereby packages can have dependencies on a particular version of Julia; in my experience so far, that is the dependency that actually matters, much more than versions of other packages.

[johnmyleswhite]

+1 for Julia version dependencies

[ViralBShah]

@sebastien-villemot For graphics, if you include those dependencies, then you should be able to get graphics working with:

Build the tk wrapper in deps, which is not part of the default build.

make install-tk-wrapper

Install and use

Pkg.add("Winston")
using Winston
plot(1:10)

For Gadfly:

Pkg.add("Gadfly")
# follow the gadfly examples

[ViralBShah]

@johnmyleswhite @timholy This should certainly push us towards stabilizing a 0.1 release, which will help for a lot of packages. We should push for better documentation of base julia in the meanwhile, and more comprehensive tests.

@dmbates I think we should get the sparse solvers working again, and do the renaming of the linalg functions in the next week or so, as we had discussed in Boston.

[vtjnash]

I think the Tk wrapper in deps is incompatible with the current version of Tk (and that the Tk package only works on Windows). This should probably be fixed before 0.1, however, I haven't had much time to look into it.

On Feb 1, 2013, at 1:03 PM, "Viral B. Shah" [email protected] wrote:

@sebastien-villemot For graphics, if you include those dependencies, then you should be able to get graphics working with:

Build the tk wrapper in deps, which is not part of the default build.

make install-tk-wrapper
Install and use

Pkg.add("Winston")
using Winston
plot(1:10)
For Gadfly:

Pkg.add("Gadfly")

follow the gadfly examples

—
Reply to this email directly or view it on GitHub.

[dmbates]

@ViralBShah re the sparse solvers, I will get back at those. I got diverted by agreeing to give a seminar on "Beyond R: Rcpp and Julia" I'll post the slides.

[nolta]

The deps tk wrapper still works on linux, at least for me.

[svillemot]

I am working on a new upload of Julia to Debian. It will include the Tk wrapper.

However, I cannot do much for Winston. The reason is twofold:

• I find it a bit cumbersome to add a dependency of Julia on Cairo and friends, while nothing in Julia uses it (only Julia packages do, but they are not yet packaged in Debian)
• if I wanted to add dependencies, I would have to depend on the development packages for these libs: the reason is that calls to dlopen() in Julia code do not include the SONAME, so these calls rely on having the *.so symlink available; but this symlink is only in development packages (at least in Debian). For Julia itself I maintain a patch (see https://anonscm.debian.org/gitweb/?p=pkg-julia/julia.git;a=blob;f=debian/patches/use-sonames-with-dlopen.patch;hb=HEAD ) but I obviously can't do the same for code distributed through the Julia package manager; and I don't want to force users of the Debian package to install tons of development packages that they don't necessarily need

So for the time being, users of Debian/Ubuntu will have to manually install (through apt-get) the development packages of cairo and friends if they want to use Winston. I hope we can find a solution to #1692, so that a fully-automated way becomes also available.

[ViralBShah]

Ok, that is fair enough. We can provide clear instructions for users to install Winston / Gadfly.

Also, we are doing a major spree, and will tag 0.1 in a couple of days. When is the latest you would want 0.1 tagged so that you can comfortably upload everything?

[svillemot]

This is good news!

If you can tag it by Sunday 10, as agreed on before, it should be fine. Ubuntu says it will stop importing from Debian on Feb 14, but I don't know how precise this timing is, so I'd rather have the package uploaded by Feb 12 or 13 for security.

Also, some testing would be welcome, if there are users of Debian Sid or Ubuntu Raring around…

[ViralBShah]

I can update my ubuntu 12.10 vbox to raring.

[JeffBezanson]

We use this function jl_lookup_soname that uses ldconfig to find the proper soname of a library if it is not found otherwise. So it may be that your soname patch is not needed any more.

[svillemot]

@JeffBezanson Thanks for pointing me to this function. It may indeed work for most libraries, but not all. For example, under Debian, libtcl.so.XX does not exist: it is either libtcl8.5.so.0 or libtcl8.4.so.0. Same kind of issue for libLLVM in extras/strpack.jl. Also, I like to have some control over SONAMEs, to minimize the risk of breakage when some library experiences a SONAME bump in Debian.

Otherwise, I have just uploaded commit 32ff575 to Debian. It failed to build on i386 (32-bit) because of #2177 (the testsuite is exercised during build, and it crashes in test/math.jl). If you cannot fix this before 0.1, I will temporarily disable that test so that there is a binary for i386.

[svillemot]

The upload failed to build in Ubuntu on both amd64 and i386. For i386, it's again #2177. For amd64, it is another strange error in the testsuite, see https://launchpadlibrarian.net/130573636/buildlog_ubuntu-raring-amd64.julia_0.0.0%2B20130206.git32ff5759-1_FAILEDTOBUILD.txt.gz
Note that both testsuite used to pass in previous uploads.

[ViralBShah]

@sebastien-villemot Could you pick the latest build from master? We will declare it 0.1 once it is accepted into unstable. The message on the mailing list declaring 0.1 release candidate is here:

https://groups.google.com/forum/?fromgroups=#!topic/julia-dev/tCmuD9SfDyw

[svillemot]

I uploaded commit 86fbe98. Unfortunately it failed to build in Debian on i386:

https://buildd.debian.org/status/fetch.php?pkg=julia&arch=i386&ver=0.1~20130211.git86fbe98d-1&stamp=1360573538

(see the "Aborted" messages first after the system image compilation, then in the middle of the testsuite)

This error is similar to the error I got with the previous snapshot on Ubuntu/amd64:

https://launchpadlibrarian.net/130573636/buildlog_ubuntu-raring-amd64.julia_0.0.0%2B20130206.git32ff5759-1_FAILEDTOBUILD.txt.gz

The problem is that I can't reproduce this error on my machine (I can build the package on both amd64 and i386).

At first glance this looks like a memory corruption problem. Note that this problem seems to have appeared recently (i.e. during the last month); it was apparently not there in commit 9656f41.

[svillemot]

I am now able to reproduce the SIGABRT, only when I build the package in an environment identical to Debian build daemons. Still investigating the cause…

[svillemot]

Here is the backtrace:

#0  0x00002b7a5bd07475 in *__GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00002b7a5bd0a6f0 in *__GI_abort () at abort.c:92
#2  0x00002b7a5b226db6 in uv__io_poll (loop=0x2b7a5b532c00, timeout=0) at src/unix/linux/linux-core.c:163
#3  0x00002b7a5b21170a in uv__run (loop=0x2b7a5b532c00) at src/unix/core.c:271
#4  0x00002b7a5b211769 in uv_run (loop=0x2b7a5b532c00) at src/unix/core.c:279
#5  0x00002b7a5b1dedae in uv_atexit_hook () at init.c:277
#6  0x00002b7a5d9cacea in ?? ()
#7  0x0000000001f0d100 in ?? ()
#8  0x00002b7a5c946acc in llvm::JIT::updateFunctionStub (this=0x7ffffa662b80, F=0x2b7a5b18ee70) at /tmp/buildd/llvm-3.2-3.2/lib/ExecutionEngine/JIT/JITEmitter.cpp:1278
#9  0x00002b7a5c938bf5 in llvm::JIT::runJITOnFunctionUnlocked (this=0x0, F=<optimized out>, locked=...) at /tmp/buildd/llvm-3.2-3.2/lib/ExecutionEngine/JIT/JIT.cpp:639
#10 0x00002b7a000000ff in ?? ()
#11 0x00002b7a00000002 in ?? ()
#12 0x0000000002a84378 in ?? ()
#13 0x0000000002b637a0 in ?? ()
#14 0x000000000000000b in ?? ()
#15 0x0000000001f0d1c8 in ?? ()
#16 0x0000000002aa7150 in ?? ()
#17 0x0000000002aea180 in ?? ()
#18 0x00002b7a5cd53f8e in compareMemory (Length=4208304, Rhs=<optimized out>, Lhs=0x2b7a5cd53f8e "\205\300D\213D$\bD\213L$\020L\213T$\030u\266D\211\370\351m\377\377\377\220f\017\037D")
    at /tmp/buildd/llvm-3.2-3.2/include/llvm/ADT/StringRef.h:66
#19 equals (RHS=..., this=<synthetic pointer>) at /tmp/buildd/llvm-3.2-3.2/include/llvm/ADT/StringRef.h:132
#20 operator== (RHS=..., LHS=...) at /tmp/buildd/llvm-3.2-3.2/include/llvm/ADT/StringRef.h:516
#21 llvm::StringMapImpl::FindKey (this=0x82ee179752da224c, Key=...) at /tmp/buildd/llvm-3.2-3.2/lib/Support/StringMap.cpp:141
#22 0x0000000000000000 in ?? ()

In frame 2 (linux-core.c from libuv), the call to uv__epoll_ctl(loop->backend_fd, op, w->fd, &e) fails and returns errno=EPERM, so that libuv calls abort(). Note that w->fd = 0, i.e. STDIN.

I guess this is related to the warning "Using pipes/files as STDIN is not yet supported. Proceed with caution!" that recently appeared. Obviously Debian build daemons do not have a real STDIN.

I am not sure how to move forward, please advise.

[svillemot]

The crash is actually easy to reproduce, using /dev/null as STDIN:

(sid)sebastien@brouzouf:~/debian/julia/base(master)$ LD_LIBRARY_PATH=../usr/lib JL_PRIVATE_LIBDIR=lib/x86_64-linux-gnu/julia /home/sebastien/debian/julia/usr/bin/julia-release-readline -f sysimg.jl </dev/null
Using pipes/files as STDIN is not yet supported. Proceed with caution!
Warning: replacing module Base
Aborted

[mlubin]

@loladiro ^

[vtjnash]

Anyone know how other environments, which expect to be able to open a STDIN, handle receiving /dev/null?

[JeffBezanson]

Thanks for investigating sebastien!
/dev/null should not be a huge problem; it is just always at EOF when reading, I believe..

[svillemot]

Note that having STDIN equal to a file also triggers the crash.

[svillemot]

I did it on purpose. It means that git will be installed automatically when you install julia, unless you say otherwise.
This is meant to reflect the fact that julia is usable without git (so you can have the former without the latter), but that the typical user will want git.

[binarybana]

Ah, good. I was not aware that was the policy for recommended packages.

[ViralBShah]

I am not around for too much, but commit 4bc33bb seems to be a good candidate.

[ViralBShah]

If debian buildd accepts commit 4bc33bb and we do not commit anything significant, perhaps we can mark it 0.1, and @sebastien-villemot can re-upload it as 0.1.

[JeffBezanson]

Yes I think we are in decent shape. One very minor thing might be to make sure helpdb is up to date, but that's not blocking.

[svillemot]

Breaking news: commit 4bc33bb entered Ubuntu (without any dirty hack!).

You can now proceed to tagging, as you see fit.

[StefanKarpinski]

W00T!

[ViralBShah]

I pushed a helpdb update. Let's tag commit 8cc5036 as 0.1.

[StefanKarpinski]

I'm slightly concerned about d3d0f85. Not that I think it's wrong, but I don't quite grok it and I don't want to screw this up. @nolta, can you comment?

[ViralBShah]

Oh boy, I thought commit d3d0f85 only touched non-essential files. This does make me a little jittery.

[nolta]

Sure, w/o that commit examples/shootout/fasta.jl crashes. Plus, it makes the code consistent w/ the rest of base:

$ git grep OUTPUT_STREAM:: base/
base/io.jl:write(x) = write(OUTPUT_STREAM::IO, x)
base/show.jl:show(x) = show(OUTPUT_STREAM::IO, x)
base/show.jl:showcompact(x) = showcompact(OUTPUT_STREAM::IO, x)
base/show.jl:xdump(fn::Function, args...) = xdump(fn, OUTPUT_STREAM::IO, args...)
base/show.jl:xdump(args...) = xdump(xdump, OUTPUT_STREAM::IO, args...)
base/show.jl:dump(args...) = dump(OUTPUT_STREAM::IO, args...)
base/show.jl:showall(x) = showall(OUTPUT_STREAM::IO, x)

[StefanKarpinski]

Oh, right. I read the commit as being the inverse patch, which seemed weird. Pretty sure this is right. Let's go with it. Thanks for testing that.

[StefanKarpinski]

@sebastien-villemot: https://groups.google.com/d/topic/julia-dev/Ld2uzedyx4I/discussion. Please use e9263b7 as the official 0.1 version for Debian/Ubuntu.

[svillemot]

Final 0.1 just entered Ubuntu. Congrats to all who made this possible!

As I already said to some of you, updates to Ubuntu 13.04 should still be possible until March 7, but they will require approval from an Ubuntu developer (which I am not). So if you release 0.1.1 before that date, I will try to push it into Ubuntu 13.04.

[ViralBShah]

Thanks Sebastien. This would not have been possible without your persistence. We should certainly try for 0.1.1 by Mar 7.

[StefanKarpinski]

Yes! Thanks so much for all your work on this. Getting a v0.1.1 in would be good since it should be purely bug fixes.

[JeffBezanson]

Thanks so much Sebastien for all your work and for putting up with all our last-minute bugs!

[ViralBShah]

I just updated my ubuntu vbox to 13.04, and tried installing julia. There are a couple of immediate issues, for which we may need to push 0.1.1. I am reopening this issue.

• There seems to be some issue with installing fftw. Since the filename has juliadeps in it, I am guessing this may be related to installing julia. In any case, we need to either fix this ourselves, or ping the fftw debian devs.

Unpacking libfftw3-3:amd64 (from .../libfftw3-3_3.3.3-juliadeps1_amd64.deb) ...
dpkg: error processing /var/cache/apt/archives/libfftw3-3_3.3.3-juliadeps1_amd64.deb (--unpack):
 trying to overwrite '/usr/lib/x86_64-linux-gnu/libfftw3f_threads.so.3.3.2', which is also in package libfftw3-single3:amd64 3.3.3-2ubuntu1
No apport report written because MaxReports is reached already
                                                              dpkg-deb: error: subprocess paste was killed by signal (Broken pipe)

• This was reported earlier here, but seems to have fallen through the cracks. Leading zeros are printed everywhere. Could it be because debian packages libdouble-conversion and it is built slightly differently? Or it could be something else.

julia> 100
000000000000000100

julia> rand(100,100)
000000000000000100x000000000000000100 Float64 Array:
 0.608919    0.156727   0.66736    …  0.0147032    0.0678617  0.315451 
 0.124284    0.603625   0.205354      0.285233     0.625809   0.280826 
 0.431766    0.211552   0.532288      0.23004      0.245217   0.570659 

[pao]

This was reported earlier here, but seems to have fallen through the cracks.

Sebastien mentioned it went away when he updated Debian's LLVM package.

[svillemot]

@ViralBShah Your FFTW package is apparently not from (official) Ubuntu. I guess your are using some PPA (maybe the one setup by @staticfloat ). I guess this problem will disappear when you install official packages.

I can't replicate the second problem either. It may be that you are also using an unofficial package for LLVM.

[staticfloat]

Anything with juliadeps at the end is from my PPA, although I'm surprised it's affecting you on 13.04, it should only apply to 12.04, as it was meant to backport newer versions of e.g. fftw and llvm to the older Ubuntu releases. If you have 13.04 installed, I would remove the apt repository of my PPA and just use the Ubuntu default packages which should be relatively up to date.

[ViralBShah]

Ok. Good points. I will check and remove the PPA packages.

[ViralBShah]

I did manage to clean out all of the PPA packages provided by @staticfloat and everything works fine.

[ViralBShah]

@sebastien-villemot You had mentioned that there may be the possibility of releasing 0.1.1 into ubuntu. Could you check with the ubuntu folks, and let us know if this is possible? There are a bunch of stuff on the release-0.1 branch, and it would be great if we can get it accepted into 13.04.

[svillemot]

@ViralBShah I just discussed with an Ubuntu developer who said he would do the sync in Ubuntu 13.04. Of course, release 0.1.1 should happen soon for that to be possible (the Ubuntu freeze deadline is March 7 if I understand correctly).

[ViralBShah]

@StefanKarpinski Can we plan to tag 0.1.1 by Mar 4, so that we can go through the debian upload cycle for 0.1.1? I think that your recent checkin for having packages depend on a specific version of julia needs to be included in 0.1.1.

I am not sure if there is any other issue that needs to be addressed for 0.1.1.