Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package system on libgit2 #11196

Merged
merged 90 commits into from
Sep 29, 2015
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
865c87c
changing `installed` based to support `libgit2`
wildart May 5, 2015
851c053
merged installed & dependents into `Read`, moved `libgit2` calls into…
wildart May 5, 2015
12a525c
removed `Git` module calls from `Read`
wildart May 5, 2015
b2be2d7
changed string type
wildart May 5, 2015
9c88dbc
renamed to 'libgit2'
wildart May 6, 2015
cdf28d4
fixed to AbstractString
wildart May 7, 2015
c57ec72
added libgit2 error handling
wildart May 8, 2015
7140141
write to repo configuration value with any accepted type
wildart May 8, 2015
f836a71
added `clone` & `fetch`, updated 'Dir' and `Cache`
wildart May 8, 2015
61ba8f0
switching to `Ref` type
wildart May 8, 2015
ba591fe
changed incorrect types & syntax
wildart May 8, 2015
efb5bba
working on `Generate`
wildart May 9, 2015
bb40811
fixed changed api
wildart May 9, 2015
4ca8175
construct type pragmatically
wildart May 9, 2015
25a7d0a
added `with_libgit2` helper function
wildart May 9, 2015
3516046
package generator works!!!
wildart May 10, 2015
3198519
added default value to GitConfig 'get' to prevent exceptions
wildart May 10, 2015
3305ed6
added anyobject type and `checkout` functions
wildart May 10, 2015
071a872
no more `git` in Write
wildart May 10, 2015
87e7fbf
fixed `revparse`, `isdirty` & errors in `Write`
wildart May 10, 2015
409bb27
improved git types description
wildart May 10, 2015
9802b85
transitioned Entry.add
wildart May 11, 2015
e2c1a38
transitioned Entry.clone
wildart May 11, 2015
ff9b5d6
GitError is Exception subclass
wildart May 11, 2015
f55de93
fixed `need_update`, `head_oid`, `revparse`, `CHECKOUT_NONE`, `Checko…
wildart May 11, 2015
6dedbb2
transitioned Entry: `free` & `pin`
wildart May 11, 2015
8fbf905
refactored error handling
wildart May 13, 2015
e4ac8bd
error fix: nonexistent directory case, cannot create empty wrapper
wildart May 13, 2015
8ff8a25
made options structures immutable with convinient constructors
wildart May 13, 2015
273ecfa
rewrote Write module: changed resource management and commit checkout.
wildart May 13, 2015
c1179e7
added proper `checkout` & `branch` functions
wildart May 13, 2015
67435f6
added tests for `libgit2`
wildart May 16, 2015
740194a
removed unnecessary `rethrow` calls
wildart May 16, 2015
30688dc
transitioned `checkout`, `free`, `update`, `register`
wildart May 17, 2015
331f8c2
switched to https
wildart May 17, 2015
dcac546
added `gitdir` (location of repo '.git') and `path` (repo location)
wildart May 17, 2015
52d72cb
import Base for `finalize`s
wildart May 17, 2015
3e957e8
transitioned `tag` & `register`
wildart May 17, 2015
3ff4628
fixed `get` with default for GitConfig
wildart May 18, 2015
c9d04e2
fixed errors, set to run only pkg & libgit2 tests
wildart May 18, 2015
d409e13
added PkgError exception
wildart May 19, 2015
3335bf7
added licence line
wildart May 19, 2015
b5efeef
fixed GitHub saved token retrieval & PkgError visibility
wildart May 19, 2015
efde7ed
added `push`, transitioned `pull_request`
wildart May 19, 2015
f1b5694
fixed import error
wildart May 20, 2015
c1298e8
added `revcount` for counting differences
wildart May 20, 2015
6943209
fixed exceptions in tests
wildart May 20, 2015
8612798
speedup `prefetch` call
wildart May 21, 2015
9016b11
refactored GitConfig methods
wildart May 21, 2015
21fcc2c
removed prefix from `finalize` calls
wildart May 21, 2015
2d456ab
newline at end of file
wildart May 21, 2015
327e989
No more `Git`!
wildart May 23, 2015
307aa8b
partial fix for #11580
wildart Jun 6, 2015
70963a9
adopting CString & refactoring
wildart Jun 27, 2015
f8b0dec
added prototype of git repl
wildart Jun 28, 2015
c0d59a0
added `clone` & `init` command
wildart Jul 2, 2015
09406ad
refactoring commit message call
wildart Jul 8, 2015
50b81f6
rebased, moved `libgit2` to `base`, updated to 0.23 & tests
testurser Jul 9, 2015
c9fcd0b
fixed repl tests
wildart Jul 9, 2015
39aa02b
tuple-based Oid
testurser Jul 9, 2015
8b5dd78
added `Oid` tests, modified `free` with tuple parameters,
testurser Jul 10, 2015
5a4aad0
fixed: string value retrieval from config, resource management in tests
wildart Jul 10, 2015
c254abb
fixed 0.23 compatibility errors, added more commands to git-repl
wildart Jul 13, 2015
f58f762
used new enums, changed location of GITHUB_REGEX, exported PkgError
wildart Jul 13, 2015
fad9263
switch to release build for windows
wildart Jul 14, 2015
93af677
diff & rebase update to v0.23, error fixes
wildart Jul 27, 2015
1347d68
added callbacks, updated push & publish
wildart Jul 28, 2015
25778f7
merge overhaul, added credential & fetchhead callbacks, small fixes &…
wildart Aug 2, 2015
9f3bad9
added extended repo open, ported Base.url
wildart Aug 2, 2015
5fa68d5
fixed Pkg.pull_request, suppress backtrace PkgError & code cleanup
wildart Aug 2, 2015
f3301fb
switch GitRepo to GitRevWalker
wildart Aug 2, 2015
9690fa6
added to GitREPL: branch & tag
wildart Aug 2, 2015
c5350e5
added `commit` to Git REPL & fixed commands help info
wildart Aug 2, 2015
243e1a8
fix Git REPL `commit` output
wildart Aug 2, 2015
7aa333b
added `checkout` to Git REPL
wildart Aug 2, 2015
2f8c7b7
fixed remote branch name
wildart Aug 2, 2015
2853db0
reomoved Git module and its tests
wildart Aug 2, 2015
0ccfdfe
fix error message & add all tests
wildart Aug 3, 2015
4beb9af
fixed cache update, remove redundant `chomp` call from Git REPL `comm…
wildart Aug 3, 2015
7e9caae
fixed: `tag` requirements creation, empty callbacks
wildart Aug 8, 2015
78929ad
fixed tests
wildart Sep 14, 2015
29004a4
removed Git REPL
wildart Sep 14, 2015
116bb4c
fixed bad rebase commit of `inbase`
wildart Sep 14, 2015
b0f07a8
added: tree entry type, tree walker, blob lookup, some refinements
wildart Sep 23, 2015
80f0db7
Union() => Union{}
wildart Sep 23, 2015
0663672
fixed comparisons to `nothing`
wildart Sep 25, 2015
509e4b2
renamed module `GitConst` to `Consts`, added support of ssh-agent
wildart Sep 29, 2015
11f08b5
returned `Pkg.Git` module
wildart Sep 29, 2015
d18efae
fix deprecation warning
jakebolewski Sep 29, 2015
35cd85f
add error branch test for package availablity
jakebolewski Sep 29, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added clone & fetch, updated 'Dir' and Cache
  • Loading branch information
wildart authored and jakebolewski committed Sep 29, 2015
commit f836a71a6854c70a55cd6f342212cbc81e3fb29e
23 changes: 11 additions & 12 deletions base/pkg/cache.jl
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

module Cache

import ..Git, ..Dir
import ..LibGit2, ..Dir
using ..Types

path(pkg::AbstractString) = abspath(".cache", pkg)
Expand All @@ -28,27 +28,26 @@ function mkcachedir()
mkdir(cache)
end


function prefetch(pkg::AbstractString, url::AbstractString, sha1s::Vector)
isdir(".cache") || mkcachedir()
cache = path(pkg)
if !isdir(cache)
info("Cloning cache of $pkg from $url")
try Git.run(`clone -q --mirror $url $cache`)
catch
repo = LibGit2.clone(url, path, bare = true, remote_cb = Pkg.LibGit2.mirror_cb)
if repo == noting
rm(cache, recursive=true)
rethrow()
error("Cannot clone $pkg from $url")
end
else
repo = LibGit2.GitRepo(cache)
end
Git.set_remote_url(url, dir=cache)
in_cache = Git.iscommit(sha1s, dir=cache)
if !all(in_cache)
LibGit2.set_remote_url(repo, url)
if !all(sha1->LibGit2.iscommit(sha1, repo), sha1s)
info("Updating cache of $pkg...")
Git.success(`remote update`, dir=cache) ||
error("couldn't update $cache using `git remote update`")
in_cache = Git.iscommit(sha1s, dir=cache)
isa(LibGit2.fetch(repo), LibGit2.GitError) &&
error("couldn't update $cache using `git remote update`")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potentially worth noting - IIUC - that this is an API change: this no longer errors if there is a problem with fetch (LibGit2.fetch just warns). New behaviour is better IMO.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not intend to warn on a fetch failure. I was a working version. I guess it is fine to leave it this way.

end
sha1s[!in_cache]
filter(sha1->!LibGit2.iscommit(sha1, repo), sha1s)
end
prefetch(pkg::AbstractString, url::AbstractString, sha1::AbstractString...) = prefetch(pkg, url, AbstractString[sha1...])

Expand Down
5 changes: 3 additions & 2 deletions base/pkg/dir.jl
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,9 @@ function init(meta::AbstractString=DEFAULT_META, branch::AbstractString=META_BRA
temp_dir = mktempdir(dir)
Base.cd(temp_dir) do
info("Cloning METADATA from $meta")
run(`git clone -q -b $branch $meta METADATA`) #TODO: LibGit2.clone
LibGit2.set_remote_url("METADATA", meta)
metadata_repo = LibGit2.clone(meta, "METADATA", branch = branch)
LibGit2.set_remote_url(metadata_repo, meta)
LibGit2.free!(metadata_repo)
touch("REQUIRE")
touch("META_BRANCH")
open("META_BRANCH", "w") do io
Expand Down
48 changes: 41 additions & 7 deletions base/pkg/libgit2.jl
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,14 @@ function __init__()
end

include("libgit2/const.jl")
include("libgit2/error.jl")
include("libgit2/repository.jl")
include("libgit2/config.jl")
include("libgit2/clone.jl")

type Ref
type GitRef
ptr::Ptr{Void}
function Ref(ptr::Ptr{Void})
function GitRef(ptr::Ptr{Void})
r = new(ptr)
finalizer(r, r -> ccall((:git_reference_free, :libgit2), Void, (Ptr{Void},), r.ptr))
return r
Expand All @@ -40,10 +42,10 @@ function head(repo::GitRepo)
(Ptr{Ptr{Void}}, Ptr{Void}), head_ptr, repo.ptr)

(err != 0) && return nothing
return Ref(head_ptr[1])
return GitRef(head_ptr[1])
end

function ref_id(ref::Ref)
function ref_id(ref::GitRef)
ref == nothing && return ""

typ = ccall((:git_reference_type, :libgit2), Cint, (Ptr{Void},), ref.ptr)
Expand All @@ -60,7 +62,7 @@ end

head_oid(repo::GitRepo) = head(repo) |> ref_id

function ref_name(ref::Ref)
function ref_name(ref::GitRef)
ref == nothing && return ""

name_ptr = ccall((:git_reference_shorthand, :libgit2), Ptr{UInt8}, (Ptr{Void},), ref.ptr)
Expand Down Expand Up @@ -153,14 +155,14 @@ end
function set_remote_url(repo::GitRepo, url::AbstractString; remote::AbstractString="origin")
cfg = GitConfig(repo)

err = set_config_string(cfg_ptr[1], "remote.$remote.url", url)
err = set!(AbstractString, cfg, "remote.$remote.url", url)
err !=0 && return

m = match(GITHUB_REGEX,url)
m == nothing && return
push = "[email protected]:$(m.captures[1]).git"
if push != url
err = set_config_string(cfg_ptr[1], "remote.$remote.pushurl", push)
err = set!(AbstractString, cfg, "remote.$remote.pushurl", push)
end
end

Expand All @@ -170,6 +172,38 @@ function set_remote_url(path::AbstractString, url::AbstractString; remote::Abstr
LibGit2.free!(prepo)
end

function mirror_callback(remote::Ptr{Ptr{Void}}, repo::Ptr{Void}, name::Ptr{UInt8}, url::Ptr{UInt8}, payload::Ptr{Void})
# Create the remote with a mirroring url
fetch_spec = "+refs/*:refs/*"
err = ccall((:git_remote_create_with_fetchspec, :libgit2), Cint,
(Ptr{Ptr{Void}}, Ptr{Void}, Ptr{UInt8}, Ptr{UInt8}, Ptr{UInt8}),
remote, repo, name, url, fetch_spec)
err != 0 && return Cint(err)

# And set the configuration option to true for the push command
config = GitConfig(GitRepo(repo, false))
name_str = bytestring(name)
err = set!(config, "remote.$name_str.mirror", true)
free!(config)
err != 0 && return Cint(err)

return Cint(0)
end
const mirror_cb = cfunction(mirror_callback, Cint, (Ptr{Ptr{Void}}, Ptr{Void}, Ptr{UInt8}, Ptr{UInt8}, Ptr{Void}))

function fetch(repo::GitRepo, remote::AbstractString="origin")
remote_ptr = [C_NULL]
err = ccall((:git_remote_lookup, :libgit2), Cint,
(Ptr{Ptr{Void}}, Ptr{Void}, Ptr{UInt8}),
remote_ptr, repo.ptr, remote)
err != 0 && return GitError(err)

err = ccall((:git_remote_fetch, :libgit2), Cint,
(Ptr{Void}, Ptr{Void}, Ptr{UInt8}),
remote_ptr[1], C_NULL, C_NULL)
err != 0 && return GitError(err)
end

function normalize_url(url::AbstractString)
m = match(GITHUB_REGEX,url)
m == nothing ? url : "git:https://github.com/$(m.captures[1]).git"
Expand Down
114 changes: 114 additions & 0 deletions base/pkg/libgit2/clone.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
immutable StrArrayStruct
strings::Ptr{Ptr{Uint8}}
count::Csize_t
end
StrArrayStruct() = StrArrayStruct(Ptr{Uint8}(0), zero(Csize_t))

immutable CheckoutOptionsStruct
version::Cuint
checkout_strategy::Cuint
disable_filters::Cint
dir_mode::Cuint
file_mode::Cuint
file_open_flags::Cint
notify_flags::Cuint
notify_cb::Ptr{Void}
notify_payload::Ptr{Void}
progress_cb::Ptr{Void}
progress_payload::Ptr{Void}
paths::StrArrayStruct
baseline::Ptr{Void}
target_directory::Ptr{Uint8}
ancestor_label::Ptr{Uint8}
our_label::Ptr{Uint8}
their_label::Ptr{Uint8}
end
CheckoutOptionsStruct() = CheckoutOptionsStruct(one(Cuint),
GitConst.CHECKOUT_SAFE_CREATE,
zero(Cint),
zero(Cuint), # Cuint(0o755), #
zero(Cuint), # Cuint(0o755), #
zero(Cint),
GitConst.CHECKOUT_NOTIFY_NONE,
Ptr{Void}(0), Ptr{Void}(0),
Ptr{Void}(0), Ptr{Void}(0),
StrArrayStruct(),
Ptr{Void}(0),
Ptr{Uint8}(0),
Ptr{Uint8}(0),
Ptr{Uint8}(0),
Ptr{Uint8}(0))

immutable RemoteCallbacksStruct
version::Cuint
sideband_progress::Ptr{Void}
completion::Ptr{Void}
credentials::Ptr{Void}
certificate_check::Ptr{Void}
transfer_progress::Ptr{Void}
update_tips::Ptr{Void}
pack_progress::Ptr{Void}
push_transfer_progress::Ptr{Void}
push_update_reference::Ptr{Void}
payload::Ptr{Void}
end
RemoteCallbacksStruct() = RemoteCallbacksStruct(one(Cuint),
Ptr{Void}(0),
Ptr{Void}(0),
Ptr{Void}(0),
Ptr{Void}(0),
Ptr{Void}(0),
Ptr{Void}(0),
Ptr{Void}(0),
Ptr{Void}(0),
Ptr{Void}(0),
Ptr{Void}(0))

type CloneOptionsStruct
version::Cuint
checkout_opts::CheckoutOptionsStruct
remote_callbacks::RemoteCallbacksStruct
bare::Cint
localclone::Cint
checkout_branch::Ptr{Uint8}
signature::Ptr{Void}
repository_cb::Ptr{Void}
repository_cb_payload::Ptr{Void}
remote_cb::Ptr{Void}
remote_cb_payload::Ptr{Void}
end

CloneOptionsStruct() = CloneOptionsStruct(one(Cuint),
CheckoutOptionsStruct(),
RemoteCallbacksStruct(),
zero(Cint),
zero(Cint),
Ptr{UInt8}(0),
Ptr{Void}(0),
Ptr{Void}(0), Ptr{Void}(0),
Ptr{Void}(0), Ptr{Void}(0)
)

function clone(url::AbstractString, path::AbstractString;
branch::AbstractString="",
bare::Bool = false,
remote_cb::Ptr{Void} = C_NULL)
# start cloning
clone_opts = CloneOptionsStruct()
clone_opts.bare = Int32(bare)
if !isempty(branch)
clone_opts.checkout_branch = pointer(branch)
end
if remote_cb != C_NULL
clone_opts.remote_cb = remote_cb
end

clone_opts_ref = Ref(clone_opts)
repo_ptr = Ptr{Void}[C_NULL]
err = ccall((:git_clone, :libgit2), Cint,
(Ptr{Ptr{Void}}, Ptr{UInt8}, Ptr{UInt8}, Ref{CloneOptionsStruct}),
repo_ptr, url, path, clone_opts_ref)
err != 0 && return nothing

return GitRepo(repo_ptr[1])
end