RFC: Pkg scaffolding and versioning API #1668

pao · 2012-12-04T06:06:56Z

[original title: Pkg.skeleton() creates package directory in current directory]

Pkg.skeleton creates a directory in the current working directory, which is typically the directory that julia is started from. It should instead create the new directory within the package directory (typically .julia)

aviks · 2012-12-04T01:32:49Z

Actually, looking at pkg.jl, skeleton should probably call create()? Or maybe merge the two?

aviks · 2012-12-04T01:53:51Z

Metadata.gen_hashes("PkgName") also has the same problems, it runs in the current directory rather than in the julia_pkg_dir

StefanKarpinski · 2012-12-04T03:50:33Z

Yes, skeleton was something that @johnmyleswhite added and I haven't really contributed to. All the methods I intended to be "customer facing" cd into the correct directory for you. I think they should be merged too, but I'm not quite sure what the best approach is. My create function is minimal, skeleton is a bit less so.

pao · 2012-12-04T05:06:35Z

I might have a go.

pao · 2012-12-04T06:09:31Z

I didn't push this directly because I'd like to make sure I'm doing the METADATA stuff somewhat correctly. However, it does work insofar as Pkg.new("TestPackage"); require("TestPackage") is not an error.

StefanKarpinski · 2012-12-04T06:20:30Z

Nice. I'm very happy to have you hacking on this :-)

I generally like to make an initial empty commit in all repos because git gets very weird about root commits, so I try to make sure there's an empty root commits that serves as an ancestor to everything interesting.

Making the METADATA entry right away feels off. I think that maybe the patch, minor and major functions should make the appropriate directory for you?

I'm kind of thinking this:

new(pkg): create a new unregistered package repo to work in (if you decide to scrap it, you can just delete it).
version(pkg,ver): tag the current commit of pkg as ver and copy that over to METADATA, creating the directory and other scaffolding there as necessary. patch(pkg), minor(pkg), major(pkg) are shortcuts for calling version(pkg,ver) where ver is the next patch, minor or major version after the current one. If you call patch when the current version is v0 then the version isn't changed.
publish(pkg): push the pkg repo and then push the changes to pkg in METADATA, effectively publishing the latest version of pkg.

pao · 2012-12-04T13:31:04Z

Yes, the best thing is to hand this off to API as much as possible; a versioning API would simplify this. I don't like the immediate METADATA commit either, but gen_hashes() doesn't work without it since it relies on git ls-tree. One other option I was thinking of would be to switch METADATA to a develop branch first.

This should also print a brief, informative message telling you what you should do next and where.

johnmyleswhite · 2012-12-04T13:52:59Z

Please do edit Pkg.skeleton to your heart's content. I just wanted something with R's semantics, which is why it uses the current directory. But I have no particular need for that to be true -- except that you might want to be able to make a skeleton for a package whose name is already in use.

StefanKarpinski · 2012-12-04T16:03:43Z

Another option would be not to rely on git ls-tree but just use ls instead so that whatever's in METADATA now, regardless of whether it's committed or not is used.

pao · 2012-12-04T17:04:43Z

I'll have to read that line more carefully. I thought you were using the hash that ls-tree emits in some way.

StefanKarpinski · 2012-12-04T17:54:04Z

It's currently looking at tagged versions in the installed package repo (not in METADATA). Maybe that's not the right thing to do. Maybe the hashes directory should just be an inversion of the version => sha1 map that's included under each pkg/versions directory. Also possible it should just be kept in memory and recomputed from pkg/versions every time instead of on disk in the hashes directory.

StefanKarpinski · 2012-12-04T17:56:21Z

The hash => version map isn't super important, btw. It's currently only used to print nice readable version names when installing, removing or up/downgrading. Nice to have, but not crucial.

pao · 2012-12-05T03:43:47Z

Changed the git ls-tree invocations to ls, removed the automatic commit to METADATA (but keeps the METADATA scaffolding so the package's files are immediately usable), and added a brief, helpful message:

julia> Pkg.new("TestPackage")
Initialized empty Git repository in /home/patrick/.julia/TestPackage/.git/
[master (root-commit) 26358de] Scaffold for Julia package TestPackage
 0 files changed
 create mode 100644 LICENSE.md
 create mode 100644 README.md
 create mode 100644 REQUIRE
 create mode 100644 src/TestPackage.jl
You have created a new package in

  /home/patrick/.julia/TestPackage

When the package is ready to submit, you can use

  > Pkg.version(TestPackage)
  > Pkg.publish(TestPackage)

to publish the package.

See (link to Pkg docs) for more details.

This also throws in Pkg.obliterate() which we may not actually want, but I thought I'd put it out there. It's helpful for testing.

Versioning API will be next, I guess, and I'll rewrite Pkg.new() again after that patch is ready.

I need to port my existing extras/ code, but the Next Level would be a PkgHub package; PkgHub.new() could additionally establish a remote on GitHub. That would also be useful for sending pull requets to JuliaLang/METADATA.jl. We'd need a Requests package first. PkgBucket and similar could be built for other hosts/services.

pao · 2012-12-05T03:48:20Z

Build failure is #1682, only on clang, and the failure is in the linalg tests.

pao · 2012-12-05T05:47:54Z

Did most of the versioning API and rebased the patchset on top.

Note that the patch bb3c224 contains a horrific workaround due to push deciding to not exist in Pkg.

pao · 2012-12-05T06:01:47Z

This test failure was #1652 this time.

The dependence of each_tagged_version() on `git ls-tree` induces a requirement that scaffolded packages needed to be checked into METADATA. That isn't a friendly thing to do automatically, and there's no obvious reason to use and parse ls-tree.

Pkg.version(pkg, ver) is the general entry point, tagging the package and making the proper METADATA entries for that revision. The .patch(pkg), .minor(pkg), and .major(pkg) provide convenience methods which increment the appropriate level of the version number.

pao · 2012-12-05T13:44:28Z

Fixed horrific workaround after Jeff beat some sense into me.

pao · 2012-12-05T14:33:20Z

Bikeshedding might be called for on this most recent one. Pkg.pkg_origin() should be the last thing needed to keep a package maintainer from having to manually touch the METADATA directory.

pao · 2012-12-05T14:41:03Z

Oops. Initial version of this most recent one had a blindingly obvious injection vulnerability.

pao · 2012-12-05T15:03:29Z

#1682 again.

pao · 2012-12-06T03:51:57Z

I was going to implement .publish() before actually committing this, but I find that I can't figure out how it should work. If a contributor lacks access to JuliaLang/METADATA.jl and isn't using their own, then there's no way to push the change out. Submitting a pull request instead requires both more magic and GitHub API calls of the sort that PkgHub should eventually provide. So I'm going to change the brief message, add Stefan's empty initial commit in .new() having read up on the subject, and commit, since I haven't really gotten any comments on this.

Pkg.new() creates a scaffolded package in .julia and the required directory structure for METADATA.

Pkg versioning API and scaffolding methods

StefanKarpinski · 2012-12-06T22:27:20Z

Thanks for this flurry of work on Pkg, @pao. It's great to have you working on it.

I was going to implement .publish() before actually committing this, but I find that I can't figure out how it should work. If a contributor lacks access to JuliaLang/METADATA.jl and isn't using their own, then there's no way to push the change out. Submitting a pull request instead requires both more magic and GitHub API calls of the sort that PkgHub should eventually provide. So I'm going to change the brief message, add Stefan's empty initial commit in .new() having read up on the subject, and commit, since I haven't really gotten any comments on this.

I don't want to tie package repos or metadata repos in general to GitHub, but since the "official" METADATA.jl will be on GitHub for the foreseeable future, automatically making pull requests when appropriate is probably desirable. A sane strategy would be to try pushing first, no matter what, and only if that fails and the URL is a GitHub one, use their API to create a pull request. For now, we can probably just try to push and provide a helpful message if that fails.

pao · 2012-12-06T23:13:33Z

I just see that getting annoying. I'd like to think about it a bit.

Meanwhile, I'm going to try to package some stuff. I just needed to get the minimum tooling to not make me want to throttle Pkg in a fit of rage. But I need to actually use it a bit.

Incidentally, while a full fledged Requests package would be nice, I'll probably hack together an expedient PkgHub that just calls out to defunkt/hub. Julia calls the GitHub API via a Ruby script using command invocation. It's a match made in hell.

StefanKarpinski · 2012-12-06T23:58:41Z

I just see that getting annoying. I'd like to think about it a bit.

How would it be annoying? I'm talking about doing it automatically – as in the Pkg.publish command tries to push and then if that fails, opens a pull request.

Meanwhile, I'm going to try to package some stuff. I just needed to get the minimum tooling to not make me want to throttle Pkg in a fit of rage. But I need to actually use it a bit.

I appreciate that you're directing your rage at the code and not at me!

Incidentally, while a full fledged Requests package would be nice, I'll probably hack together an expedient PkgHub that just calls out to defunkt/hub. Julia calls the GitHub API via a Ruby script using command invocation. It's a match made in hell.

Hahaha. Yes, we really ought to get a proper Requests package working. Shouldn't be so awful. In its place, we could actually just have a package that calls curl and automatically parses the result as JSON and returns it. That's awfully useful for APIs and using an external process is actually kind of nice since there can't be any resource leakage.

pao · 2012-12-07T00:18:46Z

How would it be annoying? I'm talking about doing it automatically – as in the Pkg.publish command tries to push and then if that fails, opens a pull request.

Opening a pull request requires making the changes public on GitHub, which in general isn't a trivial matter, and it's logic PkgHub will need anyways. I'd rather the plain Pkg.publish() just fail rather than write all that twice. Plus, if I'm running my inside-the-firewall METADATA.jl with references for all our internal packages, I need pushing that on accident to JuliaLang/METADATA.jl to be as impossible as practical.

In its place, we could actually just have a package that calls curl and automatically parses the result as JSON and returns it.

Ah yes, curl. Right, forgot that you could do that. Better choice.

StefanKarpinski · 2012-12-07T16:10:08Z

Yeah, ok. So maybe this is configurable with an environment variable, which, if set causes a pull request to be issued instead. Speaking of which, we need some way of making it easy to switch protocols across the board from git:https:// to https:// or http:https://. Any bright ideas on how to do that? I figure since you're one of the people behind a firewall, you might have some thoughts.

pao · 2012-12-07T17:45:09Z

Is there a reason the url file in METADATA has to be limited to one line? Being able to specify redundant upstreams might be a start.

StefanKarpinski · 2012-12-07T17:49:09Z

That's a very good idea. We could then parse it as a list of URL strings.

pao · 2012-12-07T18:57:54Z

Just leaving this here so I can find it later (germane to .publish()): http:https://docs.pythonpackages.com/en/latest/github-service.html

StefanKarpinski · 2012-12-07T19:10:51Z

That's an interesting idea. We could automatically update METADATA based on pushes to package repos with a specific format like that. At the very least, we could automatically create pull requests or something like that. Of course, we'd have to run a service to do all of that, which I generally try to avoid.

pao · 2012-12-07T19:35:04Z

You wouldn't need to; the service in this case is GitHub, which would be running the required post-commit hook and calling its own API.

StefanKarpinski · 2012-12-07T20:05:13Z

Oh, that's extra cool. Very, very nice. That would be a good way to do it.

pao mentioned this pull request Dec 5, 2012

Base.push() requires explicit import into Pkg #1685

Closed

pao added 2 commits December 5, 2012 07:43

Package versioning API

51b7cf8

Pkg.version(pkg, ver) is the general entry point, tagging the package and making the proper METADATA entries for that revision. The .patch(pkg), .minor(pkg), and .major(pkg) provide convenience methods which increment the appropriate level of the version number.

pao added 3 commits December 5, 2012 22:09

Pkg.new() replaces .create() and .skeleton()

8f67a79

Pkg.new() creates a scaffolded package in .julia and the required directory structure for METADATA.

Pkg.obliterate() is a way to get rid of local packages

975c7b1

Pkg.pkg_origin() method to set remote url in METADATA

ac681a6

pao added a commit that referenced this pull request Dec 6, 2012

Merge pull request #1668 from pao/topic/pkg-scaffold

85f102b

Pkg versioning API and scaffolding methods

pao merged commit 85f102b into JuliaLang:master Dec 6, 2012

pao mentioned this pull request Dec 14, 2012

Safeguards for developing within package repository? #1754

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Pkg scaffolding and versioning API #1668

RFC: Pkg scaffolding and versioning API #1668

pao commented Dec 4, 2012

aviks commented Dec 4, 2012

aviks commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

pao commented Dec 4, 2012

pao commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

pao commented Dec 4, 2012

johnmyleswhite commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

pao commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 6, 2012

StefanKarpinski commented Dec 6, 2012

pao commented Dec 6, 2012

StefanKarpinski commented Dec 6, 2012

pao commented Dec 7, 2012

StefanKarpinski commented Dec 7, 2012

pao commented Dec 7, 2012

StefanKarpinski commented Dec 7, 2012

pao commented Dec 7, 2012

StefanKarpinski commented Dec 7, 2012

pao commented Dec 7, 2012

StefanKarpinski commented Dec 7, 2012

RFC: Pkg scaffolding and versioning API #1668

RFC: Pkg scaffolding and versioning API #1668

Conversation

pao commented Dec 4, 2012

aviks commented Dec 4, 2012

aviks commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

pao commented Dec 4, 2012

pao commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

pao commented Dec 4, 2012

johnmyleswhite commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

pao commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

StefanKarpinski commented Dec 4, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 5, 2012

pao commented Dec 6, 2012

StefanKarpinski commented Dec 6, 2012

pao commented Dec 6, 2012

StefanKarpinski commented Dec 6, 2012

pao commented Dec 7, 2012

StefanKarpinski commented Dec 7, 2012

pao commented Dec 7, 2012

StefanKarpinski commented Dec 7, 2012

pao commented Dec 7, 2012

StefanKarpinski commented Dec 7, 2012

pao commented Dec 7, 2012

StefanKarpinski commented Dec 7, 2012