Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Package system on libgit2 #11196

Merged
merged 90 commits into from
Sep 29, 2015
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
90 commits
Select commit Hold shift + click to select a range
865c87c
changing `installed` based to support `libgit2`
wildart May 5, 2015
851c053
merged installed & dependents into `Read`, moved `libgit2` calls into…
wildart May 5, 2015
12a525c
removed `Git` module calls from `Read`
wildart May 5, 2015
b2be2d7
changed string type
wildart May 5, 2015
9c88dbc
renamed to 'libgit2'
wildart May 6, 2015
cdf28d4
fixed to AbstractString
wildart May 7, 2015
c57ec72
added libgit2 error handling
wildart May 8, 2015
7140141
write to repo configuration value with any accepted type
wildart May 8, 2015
f836a71
added `clone` & `fetch`, updated 'Dir' and `Cache`
wildart May 8, 2015
61ba8f0
switching to `Ref` type
wildart May 8, 2015
ba591fe
changed incorrect types & syntax
wildart May 8, 2015
efb5bba
working on `Generate`
wildart May 9, 2015
bb40811
fixed changed api
wildart May 9, 2015
4ca8175
construct type pragmatically
wildart May 9, 2015
25a7d0a
added `with_libgit2` helper function
wildart May 9, 2015
3516046
package generator works!!!
wildart May 10, 2015
3198519
added default value to GitConfig 'get' to prevent exceptions
wildart May 10, 2015
3305ed6
added anyobject type and `checkout` functions
wildart May 10, 2015
071a872
no more `git` in Write
wildart May 10, 2015
87e7fbf
fixed `revparse`, `isdirty` & errors in `Write`
wildart May 10, 2015
409bb27
improved git types description
wildart May 10, 2015
9802b85
transitioned Entry.add
wildart May 11, 2015
e2c1a38
transitioned Entry.clone
wildart May 11, 2015
ff9b5d6
GitError is Exception subclass
wildart May 11, 2015
f55de93
fixed `need_update`, `head_oid`, `revparse`, `CHECKOUT_NONE`, `Checko…
wildart May 11, 2015
6dedbb2
transitioned Entry: `free` & `pin`
wildart May 11, 2015
8fbf905
refactored error handling
wildart May 13, 2015
e4ac8bd
error fix: nonexistent directory case, cannot create empty wrapper
wildart May 13, 2015
8ff8a25
made options structures immutable with convinient constructors
wildart May 13, 2015
273ecfa
rewrote Write module: changed resource management and commit checkout.
wildart May 13, 2015
c1179e7
added proper `checkout` & `branch` functions
wildart May 13, 2015
67435f6
added tests for `libgit2`
wildart May 16, 2015
740194a
removed unnecessary `rethrow` calls
wildart May 16, 2015
30688dc
transitioned `checkout`, `free`, `update`, `register`
wildart May 17, 2015
331f8c2
switched to https
wildart May 17, 2015
dcac546
added `gitdir` (location of repo '.git') and `path` (repo location)
wildart May 17, 2015
52d72cb
import Base for `finalize`s
wildart May 17, 2015
3e957e8
transitioned `tag` & `register`
wildart May 17, 2015
3ff4628
fixed `get` with default for GitConfig
wildart May 18, 2015
c9d04e2
fixed errors, set to run only pkg & libgit2 tests
wildart May 18, 2015
d409e13
added PkgError exception
wildart May 19, 2015
3335bf7
added licence line
wildart May 19, 2015
b5efeef
fixed GitHub saved token retrieval & PkgError visibility
wildart May 19, 2015
efde7ed
added `push`, transitioned `pull_request`
wildart May 19, 2015
f1b5694
fixed import error
wildart May 20, 2015
c1298e8
added `revcount` for counting differences
wildart May 20, 2015
6943209
fixed exceptions in tests
wildart May 20, 2015
8612798
speedup `prefetch` call
wildart May 21, 2015
9016b11
refactored GitConfig methods
wildart May 21, 2015
21fcc2c
removed prefix from `finalize` calls
wildart May 21, 2015
2d456ab
newline at end of file
wildart May 21, 2015
327e989
No more `Git`!
wildart May 23, 2015
307aa8b
partial fix for #11580
wildart Jun 6, 2015
70963a9
adopting CString & refactoring
wildart Jun 27, 2015
f8b0dec
added prototype of git repl
wildart Jun 28, 2015
c0d59a0
added `clone` & `init` command
wildart Jul 2, 2015
09406ad
refactoring commit message call
wildart Jul 8, 2015
50b81f6
rebased, moved `libgit2` to `base`, updated to 0.23 & tests
testurser Jul 9, 2015
c9fcd0b
fixed repl tests
wildart Jul 9, 2015
39aa02b
tuple-based Oid
testurser Jul 9, 2015
8b5dd78
added `Oid` tests, modified `free` with tuple parameters,
testurser Jul 10, 2015
5a4aad0
fixed: string value retrieval from config, resource management in tests
wildart Jul 10, 2015
c254abb
fixed 0.23 compatibility errors, added more commands to git-repl
wildart Jul 13, 2015
f58f762
used new enums, changed location of GITHUB_REGEX, exported PkgError
wildart Jul 13, 2015
fad9263
switch to release build for windows
wildart Jul 14, 2015
93af677
diff & rebase update to v0.23, error fixes
wildart Jul 27, 2015
1347d68
added callbacks, updated push & publish
wildart Jul 28, 2015
25778f7
merge overhaul, added credential & fetchhead callbacks, small fixes &…
wildart Aug 2, 2015
9f3bad9
added extended repo open, ported Base.url
wildart Aug 2, 2015
5fa68d5
fixed Pkg.pull_request, suppress backtrace PkgError & code cleanup
wildart Aug 2, 2015
f3301fb
switch GitRepo to GitRevWalker
wildart Aug 2, 2015
9690fa6
added to GitREPL: branch & tag
wildart Aug 2, 2015
c5350e5
added `commit` to Git REPL & fixed commands help info
wildart Aug 2, 2015
243e1a8
fix Git REPL `commit` output
wildart Aug 2, 2015
7aa333b
added `checkout` to Git REPL
wildart Aug 2, 2015
2f8c7b7
fixed remote branch name
wildart Aug 2, 2015
2853db0
reomoved Git module and its tests
wildart Aug 2, 2015
0ccfdfe
fix error message & add all tests
wildart Aug 3, 2015
4beb9af
fixed cache update, remove redundant `chomp` call from Git REPL `comm…
wildart Aug 3, 2015
7e9caae
fixed: `tag` requirements creation, empty callbacks
wildart Aug 8, 2015
78929ad
fixed tests
wildart Sep 14, 2015
29004a4
removed Git REPL
wildart Sep 14, 2015
116bb4c
fixed bad rebase commit of `inbase`
wildart Sep 14, 2015
b0f07a8
added: tree entry type, tree walker, blob lookup, some refinements
wildart Sep 23, 2015
80f0db7
Union() => Union{}
wildart Sep 23, 2015
0663672
fixed comparisons to `nothing`
wildart Sep 25, 2015
509e4b2
renamed module `GitConst` to `Consts`, added support of ssh-agent
wildart Sep 29, 2015
11f08b5
returned `Pkg.Git` module
wildart Sep 29, 2015
d18efae
fix deprecation warning
jakebolewski Sep 29, 2015
35cd85f
add error branch test for package availablity
jakebolewski Sep 29, 2015
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added with_libgit2 helper function
better memory resource handling
`finalize` instead of `free`
  • Loading branch information
wildart authored and jakebolewski committed Sep 29, 2015
commit 25a7d0aaaf22ca6b484cd962b29d569abae63d51
7 changes: 2 additions & 5 deletions base/pkg/dir.jl
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
module Dir

import ..Pkg: DEFAULT_META, META_BRANCH
import ..LibGit2
import ..LibGit2, ..LibGit2.with_libgit2

const DIR_NAME = ".julia"

Expand Down Expand Up @@ -46,11 +46,8 @@ function init(meta::AbstractString=DEFAULT_META, branch::AbstractString=META_BRA
temp_dir = mktempdir(dir)
Base.cd(temp_dir) do
info("Cloning METADATA from $meta")
metadata_repo = LibGit2.clone(meta, "METADATA", branch = branch)
try
with_libgit2(LibGit2.clone(meta, "METADATA", branch = branch)) do metadata_repo
LibGit2.set_remote_url(metadata_repo, meta)
finally
LibGit2.free!(metadata_repo)
end
touch("REQUIRE")
touch("META_BRANCH")
Expand Down
10 changes: 7 additions & 3 deletions base/pkg/entry.jl
Original file line number Diff line number Diff line change
Expand Up @@ -107,9 +107,13 @@ end
function installed(pkg::AbstractString)
avail = Read.available(pkg)
if Read.isinstalled(pkg)
res = typemin(VersionNumber)
prepo = LibGit2.GitRepo(pkg)
res = Read.installed_version(pkg, prepo, avail)
LibGit2.free!(prepo)
try
res = Read.installed_version(pkg, prepo, avail)
finally
LibGit2.finalize(prepo)
end
return res
end
isempty(avail) && error("$pkg is not a package (not registered or installed)")
Expand Down Expand Up @@ -163,7 +167,7 @@ function status(io::IO, pkg::AbstractString, ver::VersionNumber, fix::Bool)
catch
print(io, "broken-repo (unregistered)")
finally
LibGit2.free!(prepo)
LibGit2.finalize(prepo)
end
else
print(io, "non-repo (unregistered)")
Expand Down
27 changes: 10 additions & 17 deletions base/pkg/libgit2.jl
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
module LibGit2

export with_libgit2

const GITHUB_REGEX =
r"^(?:git@|git:https://|https://(?:[\w\.\+\-]+@)?)github.com[:/](([^/].+)/(.+?))(?:\.git)?$"i

Expand Down Expand Up @@ -66,11 +68,11 @@ function isdirty(repo::GitRepo, paths::AbstractString="")
c = ccall((:git_diff_num_deltas, :libgit2), Cint, (Ptr{Void},), diff.ptr)
result = c > 0
end
free!(diff)
finalize(diff)
catch
result = true
finally
free!(tree)
finalize(tree)
end
return result
end
Expand All @@ -97,8 +99,7 @@ function is_ancestor_of(a::AbstractString, b::AbstractString, repo::GitRepo)
end

function set_remote_url(repo::GitRepo, url::AbstractString; remote::AbstractString="origin")
cfg = GitConfig(repo)
try
with_libgit2(GitConfig, repo) do cfg
set!(cfg, "remote.$remote.url", url)

m = match(GITHUB_REGEX,url)
Expand All @@ -108,17 +109,13 @@ function set_remote_url(repo::GitRepo, url::AbstractString; remote::AbstractStri
set!(cfg, "remote.$remote.pushurl", push)
end
end
catch e
warn("set_remote_url: ", e.msg)
finally
free!(cfg)
end
end

function set_remote_url(path::AbstractString, url::AbstractString; remote::AbstractString="origin")
repo = GitRepo(path)
set_remote_url(repo, url, remote=remote)
free!(prepo)
with_libgit2(GitRepo, path) do repo
set_remote_url(repo, url, remote=remote)
end
end

function mirror_callback(remote::Ptr{Ptr{Void}}, repo_ptr::Ptr{Void}, name::Ptr{UInt8}, url::Ptr{UInt8}, payload::Ptr{Void})
Expand All @@ -134,7 +131,7 @@ function mirror_callback(remote::Ptr{Ptr{Void}}, repo_ptr::Ptr{Void}, name::Ptr{
name_str = bytestring(name)
err= try set!(config, "remote.$name_str.mirror", true)
catch -1
finally free!(config)
finally finalize(config)
end
err != 0 && return Cint(err)
return Cint(0)
Expand All @@ -144,14 +141,10 @@ const mirror_cb = cfunction(mirror_callback, Cint, (Ptr{Ptr{Void}}, Ptr{Void}, P
function fetch(repo::GitRepo, remote::AbstractString="origin")
rmt = get(GitRemote, repo, remote)

try
with_libgit2(rmt, warn_on_exception=true) do rmt
@check ccall((:git_remote_fetch, :libgit2), Cint,
(Ptr{Void}, Ptr{Void}, Ptr{UInt8}),
rmt.ptr, C_NULL, C_NULL)
catch err
rethrow(err)
finally
free!(rmt)
end
end

Expand Down
60 changes: 43 additions & 17 deletions base/pkg/libgit2/commit.jl
Original file line number Diff line number Diff line change
Expand Up @@ -20,42 +20,68 @@ function committer(c::GitCommit)
return Signature(ptr)
end

""" Wrapper around `git_commit_create` """
function commit(repo::GitRepo,
refname::AbstractString,
msg::AbstractString,
author::Signature,
committer::Signature,
author::GitSignature,
committer::GitSignature,
tree::GitTree,
parents::GitCommit...)
id_ptr = Ref(Oid())
commit_id_ptr = Ref(Oid())
nparents = length(parents)
parentptrs = Ptr{Void}[c.ptr for c in parents]
err = ccall((:git_commit_create, :libgit2), Cint,
@check ccall((:git_commit_create, :libgit2), Cint,
(Ptr{Oid}, Ptr{Void}, Ptr{Uint8},
Ptr{SignatureStruct}, Ptr{SignatureStruct},
Ptr{Uint8}, Ptr{Uint8}, Ptr{Void},
Csize_t, Ptr{Ptr{Void}}),
id_ptr, repo.ptr, isempty(refname) ? C_NULL : refname,
author, committer,
C_NULL, msg, tree,
commit_id_ptr, repo.ptr, isempty(refname) ? C_NULL : refname,
author.ptr, committer.ptr,
C_NULL, msg, tree.ptr,
nparents, nparents > 0 ? parentptrs : C_NULL)
err !=0 && return GitError(err)
return id_ptr[]
return commit_id_ptr[]
end

"""Commit changes to repository"""
function commit(repo::GitRepo, msg::AbstractString,
refname::AbstractString="",
refname::AbstractString="HEAD",
author::Signature = Signature(repo),
committer::Signature = Signature(repo),
tree_id::Oid = Oid(),
parent_ids::Vector{Oid}=Oid[])
tree = if isempty(tree_id)
get(GitTree, repo)
else
get(GitTree, repo, tree_id)
# Retrieve tree identifier
if iszero(tree_id)
idx = GitIndex(repo)
try
tree_id = write_tree!(idx)
catch err
rethrow(err)
finally
finalize(idx)
end
end
isa(err, GitError)&& return err

parents = [get(GitCommit, repo, parent) for parent in parents]
return commit(repo, refname, msg, author, committer, tree, parents)
# return commit id
commit_id = Oid()

# get necessary objects
tree = get(GitTree, repo, tree_id)
auth_sig = convert(GitSignature, author)
comm_sig = convert(GitSignature, committer)
parents = GitCommit[]
try
for parent in parents
push!(parents, get(GitCommit, repo, parent))
end
commit_id = commit(repo, refname, msg, auth_sig, comm_sig, tree, parents...)
finally
for parent in parents
finalize(parent)
end
finalize(tree)
finalize(auth_sig)
finalize(auth_sig)
end
return commit_id
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newline would be nice

9 changes: 3 additions & 6 deletions base/pkg/libgit2/remote.jl
Original file line number Diff line number Diff line change
@@ -1,24 +1,21 @@
function GitRemote(repo::GitRepo, rmt_name::AbstractString, rmt_url::AbstractString)
rmt_ptr_ptr = Ref{Ptr{Void}}(C_NULL)
err = ccall((:git_remote_create, :libgit2), Cint,
@check ccall((:git_remote_create, :libgit2), Cint,
(Ptr{Ptr{Void}}, Ptr{Void}, Ptr{UInt8}, Ptr{UInt8}),
rmt_ptr_ptr, repo.ptr, rmt_name, rmt_url)
err != 0 && return GitError(err)
return GitRemote(rmt_ptr_ptr[])
end

function get(::Type{GitRemote}, repo::GitRepo, rmt_name::AbstractString)
rmt_ptr_ptr = Ref{Ptr{Void}}(C_NULL)
err = ccall((:git_remote_lookup, :libgit2), Cint,
@check ccall((:git_remote_lookup, :libgit2), Cint,
(Ptr{Ptr{Void}}, Ptr{Void}, Ptr{UInt8}),
rmt_ptr_ptr, repo.ptr, rmt_name)
err != 0 && return GitError(err)
return GitRemote(rmt_ptr_ptr[])
end

function save(rmt::GitRemote)
err = ccall((:git_remote_save, :libgit2), Cint, (Ptr{Void}, ), rmt.ptr)
err != 0 && return GitError(err)
@check ccall((:git_remote_save, :libgit2), Cint, (Ptr{Void}, ), rmt.ptr)
end

function url(rmt::GitRemote)
Expand Down
2 changes: 1 addition & 1 deletion base/pkg/libgit2/repository.jl
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ function GitRepo(path::AbstractString)
(Ptr{Ptr{Void}}, Ptr{UInt8}), repo_ptr_ptr, path)
if err != GitErrorConst.GIT_OK
if repo_ptr_ptr[] != C_NULL
free!(GitRepo(repo_ptr_ptr[]))
finalize(GitRepo(repo_ptr_ptr[]))
end
throw(GitError(err))
end
Expand Down
27 changes: 17 additions & 10 deletions base/pkg/libgit2/signature.jl
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,32 @@ function Signature(ptr::Ptr{SignatureStruct})
offset = sig.when.offset
return Signature(name, email, time, offset)
end
Signature(sig::GitSignature) = Signature(sig.ptr)

function Signature(name::AbstractString, email::AbstractString)
sig_ptr_ptr = Ref{Ptr{SignatureStruct}}(C_NULL)
err = ccall((:git_signature_now, :libgit2), Cint,
@check ccall((:git_signature_now, :libgit2), Cint,
(Ptr{Ptr{SignatureStruct}}, Ptr{UInt8}, Ptr{UInt8}), sig_ptr_ptr, name, email)
err != 0 && return GitError(err)
sig_ptr = sig_ptr_ptr[]
s = Signature(sig_ptr)
ccall((:git_signature_free, :libgit2), Void, (Ptr{SignatureStruct},), sig_ptr)
sig = GitSignature(sig_ptr_ptr[])
s = Signature(sig.ptr)
finalize(sig)
return s
end

function Signature(repo::GitRepo)
sig_ptr_ptr = Ref{Ptr{SignatureStruct}}(C_NULL)
err = ccall((:git_signature_default, libgit2), Cint,
@check ccall((:git_signature_default, :libgit2), Cint,
(Ptr{Ptr{SignatureStruct}}, Ptr{Void}), sig_ptr_ptr, repo.ptr)
err != 0 && return GitError(err)
sig_ptr = sig_ptr_ptr[]
s = Signature(sig_ptr)
ccall((:git_signature_free, :libgit2), Void, (Ptr{SignatureStruct},), sig_ptr)
sig = GitSignature(sig_ptr_ptr[])
s = Signature(sig.ptr)
finalize(sig)
return s
end

function Base.convert(::Type{GitSignature}, sig::Signature)
sig_ptr_ptr = Ref{Ptr{SignatureStruct}}(C_NULL)
@check ccall((:git_signature_new, :libgit2), Cint,
(Ptr{Ptr{SignatureStruct}}, Ptr{Uint8}, Ptr{Uint8}, Cint, Cint),
sig_ptr_ptr, sig.name, sig.email, sig.time, sig.time_offset)
return GitSignature(sig_ptr_ptr[])
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the git signature stuff was terribly memory unsafe. It really should be rewritten.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part should be fine. Memory allocated in git_signature_default is cleared in finalize (Only wrapper types have finalize functions that clear wrapped object to which wrapper holds the pointer). Nothing more is allocated. I use GitSignature as a wrapper around pointer to SignatureStruct and Signature only reads data from SignatureStruct.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see you changed this quite a bit. Yes, making copies everywhere is the right way to go.

55 changes: 38 additions & 17 deletions base/pkg/libgit2/types.jl
Original file line number Diff line number Diff line change
Expand Up @@ -105,25 +105,26 @@ CloneOptionsStruct() = CloneOptionsStruct(one(Cuint),
)

# Common types
for (typ, fnc) in ((:GitRemote, :(:git_remote_free)),
(:GitRevWalker, :(:git_revwalk_free)),
(:GitConfig, :(:git_config_free)),
(:GitReference, :(:git_reference_free)),
(:GitDiff, :(:git_diff_free)),
(:GitRepo, :(:git_repository_free)))
for (typ, ref, fnc) in ((:GitRemote, :Void, :(:git_remote_free)),
(:GitRevWalker, :Void, :(:git_revwalk_free)),
(:GitConfig, :Void, :(:git_config_free)),
(:GitReference, :Void, :(:git_reference_free)),
(:GitDiff, :Void, :(:git_diff_free)),
(:GitIndex, :Void, :(:git_index_free)),
(:GitSignature, :SignatureStruct, :(:git_signature_free)),
(:GitRepo, :Void, :(:git_repository_free)))
@eval type $typ
ptr::Ptr{Void}
function $typ(ptr::Ptr{Void})
ptr::Ptr{$ref}
function $typ(ptr::Ptr{$ref})
@assert ptr != C_NULL
obj = new(ptr)
return obj
end
$typ() = new(C_NULL)
end

@eval function free!(obj::$typ)
@eval function finalize(obj::$typ)
if obj.ptr != C_NULL
ccall(($fnc, :libgit2), Void, (Ptr{Void},), obj.ptr)
ccall(($fnc, :libgit2), Void, (Ptr{$ref},), obj.ptr)
obj.ptr = C_NULL
end
end
Expand All @@ -134,7 +135,7 @@ end
# Object types
abstract GitObject

function free!(o::GitObject)
function finalize(o::GitObject)
if o.ptr != C_NULL
ccall((:git_object_free, :libgit2), Void, (Ptr{Void},), o.ptr)
o.ptr = C_NULL
Expand All @@ -150,14 +151,34 @@ for typ in [:GitCommit, :GitTree]
obj = new(ptr)
return obj
end
$typ() = new(C_NULL)
end
end

# Misc types
# Structure has the same layout as SignatureStruct
type Signature
name::UTF8String
email::UTF8String
time::Int32
name::AbstractString
email::AbstractString
time::Int64
time_offset::Int32
end

""" Resource management helper function
"""
function with_libgit2(f::Function, obj)
try
f(obj)
catch err
rethrow(err)
finally
finalize(obj)
end
end

function with_libgit2{T}(f::Function, ::Type{T}, args...; warn_on_exception::Bool=true)
obj = T(args...)
try
with_libgit2(f, obj)
catch err
warn_on_exception ? warn("$(string(T)) thrown exception: $err") : rethrow(err)
end
end
Loading