Add support for gitea

[zeripath]

This pull-request implements basic support for Gitea.

Still to-do:

1. The Gitea API doesn't provide a way to get the permissions for an arbitrary user on a repository. This means that the cache getter has to rely on the user being the last user to authenticate in this thread. This appears to be the case although this doesn't appear to be specified.
2. I have not checked whether LFS support actually works
3. I have not created any integration tests as yet.

[slonopotamus]

I really doubt this will end up well. We reuse threads for subsequent connections, so this threadlocal will have completely unrelated stuff. Did you investigate where code that uses this is called? Maybe that functionality could just be claimed unsupported for Gitea?

[zeripath]

OK,

An ApiClient is actually just a holder for a username and password so it's really light and doesn't waste too much memory. Collecting the ApiClient is checked on username - so another user of the thread won't be able to get the ApiClient unless it's for the same username. This is because User is final and can't be extended to keep the password around for the remainder of the request.

It's unfortunately needed because at present there is no way to ask the Gitea API what permissions a random user has and lacks a Sudo feature - I wrote a pull-request for a sudo go-gitea/gitea#4809 and to change the collaborator api go-gitea/gitea#4814 but neither particularly seem to be gaining traction.

If there's a better way to keep the session around I'd be happy to use it.

[zeripath]

go-gitea/gitea#4809 has been merged now in to the Gitea mainline so once Gitea v1.6 or a release is made with this patch in I will release java-gitea-api v1.6 (as appropriate) and change this patch to use the sudo. (If go-gitea/gitea#4814 is pulled we could even drop the sudo.)

[slonopotamus]

Okay, I think we can merge this PR when you get rid of threadLocal. Download of all users is weird, but since it is only used for LFS (that most likely doesn't work for different reasons), we can live with it.

[zeripath]

Hmm, it appears that LFS works on the svn:https:// protocol with the default config-gitea.example and doesn't use the lookupByExternal function.

[zeripath]

OK, so I've made some more changes - I noticed that there was a problem with adding repositories from the DirectoryWatcher which required a few changes.

I've also done more testing and discovered that the LFS works out of the box when using the svn:https:// protocol - lookupByExternal isn't used in this.

Is there another way of connecting to the svn repositories?

[slonopotamus]

No, git-as-svn only exposes svn:https:// to users. We cannot do svn+ssh because in svn+ssh user connects to server via ssh and runs svn binary that directly uses svn repository files on server. But 1. native svn obviously cannot use our repository and 2. we do not have a fake svn binary that would talk to git-as-svn server.

Have you tried both uploading and downloading LFS files via git-as-svn? Are they available from git side after that?

[zeripath]

Yah it works both ways. I don't think there's a clear way to install git lfs from within svn, but once it's installed it works.

[slonopotamus]

Okay, I've released 1.6.0 with your changes!

[fcharlie]

No, git-as-svn only exposes svn:https:// to users. We cannot do svn+ssh because in svn+ssh user connects to server via ssh and runs svn binary that directly uses svn repository files on server. But 1. native svn obviously cannot use our repository and 2. we do not have a fake svn binary that would talk to git-as-svn server.

Have you tried both uploading and downloading LFS files via git-as-svn? Are they available from git side after that?

In fact, We can implement a SSH Server and Parse svn handshake, then proxy to git-as-svn. git-as-svn should use ANONYMOUS auth. I implemented SSH server with svn+ssh using C++ (libssh) and Go respectively.

Go use https://github.com/gliderlabs/ssh

Java can use https://github.com/apache/mina-sshd

[zeripath]

Hmm I suspect implementing one's own SSH server could be risky and could conflict with the SSH requirements for gitlab and gitea. I personally wouldn't be too happy running a SSH server from a small project like this - the attack surface is too large.

However, looking at what SVN does for svn+ssh it appears that after the SSH handshake it simply executes svnserve and passes data through stdin and stdout over the SSH tunnel. An option might therefore be to proxy or replace this command instead, and rather like gitea (and presumably gitlab) manage an authorized_keys file. (An example line of which could read:

command="/usr/bin/git-as-svnserve -t -r /data/git",no-port-forwarding,no-pty,
	           no-agent-forwarding,no-X11-forwarding ssh-rsa AAA....

From http:https://coderazzi.net/howto/security/ssh_svn.html)

This would require some way of listening for new public keys and sudoing as the user.

It might be possible to set up the svnserve command in such a way that it could be forced to push its commands across to the SVN port, this would reduce the changes to the git-as-svn project on that front.

[slonopotamus]

Yep, implementing a custom executable for svn+ssh is much more preferred than creatiing a full-blown SSH server.

[slonopotamus]

I really wish there was a "svns:https://" protocol - svn over ssl, i.e. encrypted channel with ordinary svn:https:// traffic inside. Unfortunately, svn devs went http/https route and never implemented this. svn+ssh is problematic wrt identifying the user (we no longer have username but instead have ssh key that needs to be mapped to username somehow) and user access permissions on server. Afaik, when running native svn+ssh, svnserve is run on behalf of connected user so user technically has full write access to repo files and can do whatever he wants with them, including deletion.

Also, svn+ssh has terrible performance due to svn client opening multiple connections for various operations. And unless you're running persistent ssh connections, you spend 0.2-0.3s to establish an ssh connection.

[zeripath]

Yes it seems insane that there is no "svns:https://" protocol. It's somewhat ludicrous that this was never implemented. There was something about the cyrus SASL but I think authentication wasn't protected on this.

OK, well, in terms of getting secure access to the server - and one that doesn't involve passing passwords over the network in plain text I think there are two options:

1. Implement the "svn+ssh:https://" bridge. The drawbacks of the native svnserve regarding access won't really be affect the bridge - however the performance issues will.
2. Implement "http:https://" access and allow proxying behind HTTPS. AFAIK the HTTP protocol was improved in svn 1.7 and the performance problems with this were massively reduced.

[slonopotamus]

Implement "http:https://" access

This should be easier, both in terms of implementation in git-as-svn and server setup.

[zeripath]

So just looking at "svn+ssh:https://" it seems that putting the following in authorized_keys:

command="nc localhost 3690",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa ...

works with the caveat that git-as-svn then asks you to login again...

[fcharlie]

@zeripath @slonopotamus NO. In fact gitea and gogs use golang.org/x/crypto/ssh.

See: https://github.com/go-gitea/gitea/blob/master/modules/ssh/ssh.go

gogs: https://github.com/gogs/gogs/blob/master/pkg/ssh/ssh.go

You only need to add support for svnserve -t in the exec request, and use goroutine io.Copy to exchange the data of ssh.Session and svn socket. Git-as-svn can still use multi-threaded mode to avoid slower response times.
Auth need check write access.

We use github.com/gliderlabs/ssh provider git ssh and svn+ssh feature.

[slonopotamus]

@fcharlie "no" what?

[fcharlie]

@slonopotamus
"No" here:

Hmm I suspect implementing one's own SSH server could be risky and could conflict with the SSH requirements for gitlab and gitea. I personally wouldn't be too happy running a SSH server from a small project like this - the attack surface is too large.

[sapk]

To be more precise gitea and gogs let the admin choose to use internal ssh or the standard sshd deamon by editing the authorized_keys file of user. So the admin can choose what suit best his case.

[slonopotamus]

I don't see how that's related to git-a-svn. If git-as-svn implements its own ssh (I hope not), it will obviously use a separate port.

[zeripath]

@fcharlie I think I understand what you're saying.

Yes, we can use the gitea ssh. The implementation would have to be:

1. We need to watch the git user's .ssh/authorized_keys. (As there's no webhook for gitea server events)

• We could use code similar to DirectoryWatcher to watch the git users .ssh directory. (This should be ok as git-as-svn has to run as the git user.)

2. Munge the command in the authorized_keys for each user

• We could write out an authorized_keys file to another user, possibly the svn user. This would require a set-uid binary for svn or passing information through a pipe to a process running as the svn user
• Another option would be to keep the same user but somehow change the namespace - perhaps by using a symbol that is not allowed in the owner names eg. a prefix like svn+. We would then have to check if that prefix is there and if not pass control back to gitea serv.

3. Quite what the command should be is difficult, but it would need to associate a key with a user

• At present the Gitea API for interrogating keys is not helpful. /api/v1/user/keys/:id will always return a key if present. It will not say what type of key or who the key belongs to. /api/v1/user/keys will return all keys associated with the logged in user but that would require iterating across all users and checking each one individually. Some keys will be deploy keys and to get those would require iterating across every repository in /repos/:owner/:repo/keys
• This could be cached as when the authorized_keys file changes that's when the keys are created.
• I've put a pull request in to help alleviate this at: Adjust keys API go-gitea/go-sdk#121 & Keys API changes go-gitea/gitea#4960

4. The command would then have to communicate with the git-as-svn server - perhaps over a separate port with some extra information.

• I don't know if the svn protocol would allow us to pre-communicate over the standard port before then passing in the remaining stream. i.e. could we just open a connection to the git-as-svn server, do the login, then feed in the rest of the rest of the svn commands from standard-input?

5. If this command is written in java it would mean spinning up a jvm for each connection - at least it was back in the 1.5 days this was quite slow - has this improved? Python appears a common choice for git hooks so perhaps a python script would be reasonable or maybe even a bash script?

[slonopotamus]

spinning up a jvm for each connection
No, this would be awfully slow.

If i understand how things work, when user connects via ssh, they explicitly tell server what command they want to execute (git-receive-pack, svnserve, git-lfs-authenticate, etc). You don't need to separate gitea users from svn users. We just need a small svnserve-like executable that will get ssh key from env, connect to git-as-svn via protected port (pipe, preferrably), pretend it is using a different authentication mechanism (we already have CRAM-MD5 and LOGIN auths) and give ssh key to git-as-svn. Now, on git-as-svn side we ask gitea "what user has this ssh key" and authenticate that way. After that, svnserve just proxies all traffic between remote client and git-as-svn.

So:

1. We don't need to read authorized_keys
2. We don't need scary setuid things

So yes, you can pre-communicate by implementing custom auth mechanism. https://github.com/bozaro/git-as-svn/blob/master/src/main/java/svnserver/server/SvnServer.java#L280 (sorry for comments in RU).

Possibly traffic between our 'svnserve' and git-as-svn doesn't need to be proper svn traffic and instead can be something simpler.

[zeripath]

Ah you'd be right about authorized_keys, except that gitlab, gitea and the like restrict the shells that you can use.

The way gitea works is that you always log in to the git user. On a successful ssh connection the command specified in authorized_keys is run with the environment variable SSH_ORIGINAL_COMMAND containing the command that was sent by the user.

Here's an example:

command="/app/gitea/gitea serv key-1 --config='/data/gitea/conf/app.ini'",no-port-forwarding,no-X11-forwarding,no-agen
t-forwarding,no-pty ssh-rsa AAA...

The gitea serv command will look at the SSH_ORIGINAL_COMMAND, if it's empty i.e. the person wrote ssh [email protected], then it will exit with an error, telling ssh to close the connection. Otherwise, there are a limited number of acceptable commands - git-upload-pack, git-upload-archive, git-receive-pack, and git-lfs-authenticate - anything else leads to it exiting with an error. Gitea will then look at the key argument and determine if that key can read or write to the repository specified in the SSH_ORIGINAL_COMMAND.

Now if you don't want to touch the authorized_keys etc., we need to do option 2b: we shadow the gitea binary and in particular the gitea serv subcommand. We simply check the SSH_ORIGINAL_COMMAND and if it is a svnserve we can take over from there. If not we pass it back to the original gitea and let it do its business there. (When a client does svn svn+ssh:https://... it connects to the server using ssh with svnserve -t as the SSH_ORIGINAL_COMMAND)

Regarding the way to communicate with git-as-svn, I think a pipe isn't going to work - you'd need a pipe per connection. We could just use the standard port and pass a secret token as well as the key number across much like the sudo token for the APIs. We can create a special svnserver.auth.Authenticator as you say for this within git-as-svn. You seem to be saying it's valid to prepend any subversion protocol communication with a forced authorisation like that? (I'll check the protocol.)

The svn client on the client side actually sends the SVN protocol down its pipe to the SSH server, which is just the stdin on the command running on the SSH server. So unless we parse the SVN protocol in our shadow we'll need to parse it in git-as-svn - if therefore we can prepend an authorisation request before the protocol we can make the shadow a lot simpler - hence the way that command="nc localhost 3690" works.

A possible shadow would look like:

#!/bin/bash
REAL_GITEA_PATH="/the_real_unshadowed_gitea
SSH_ORIGINAL_COMMANDS=($SSH_ORIGINAL_COMMAND)

SUBCOMMAND="$1"
if [ "$1" = "serv" ] && [ -n "$SSH_ORIGINAL_COMMAND" ] && [ "${SSH_ORIGINAL_COMMANDS[0]}" = "svnserve" ] ; then
## DO THE MAGIC ##
#  but I think here's an easy working test
exec nc localhost 3690
else
exec "$REAL_GITEA_PATH" "$@"
fi

[zeripath]

OK, so if I move /app/gitea/gitea to /app/gitea/gitea-shadowed and make /app/gitea/gitea:

#!/bin/bash
REAL_GITEA_PATH="/app/gitea/gitea-shadowed"
SSH_ORIGINAL_COMMANDS=($SSH_ORIGINAL_COMMAND)

SUBCOMMAND="$1"
if [ "$1" = "serv" ] && [ -n "$SSH_ORIGINAL_COMMAND" ] && [ "${SSH_ORIGINAL_COMMANDS[0]}" = "svnserve" ] ; then
## DO THE MAGIC ##
#  but I think here's an easy working test
exec nc localhost 3690
else
exec -a "/app/gitea/gitea" "$REAL_GITEA_PATH" "$@"
fi

This will work although it will ask for the password...

[slonopotamus]

Look at this: http:https://svn.apache.org/repos/asf/subversion/trunk/notes/ssh-tricks If we can add custom entries to gitea authorized_keys and gitea won't overwrite them, we could just add separate keys for svn access without messing with shadowing at all.

Also, you do not need to move gitea anywhere, you just need to change default shell that is spawned when user connects via ssh. This is good because we do not make changes to gitea installatiion that would be gone on gitea update.

[slonopotamus]

Also, WRT pipe. Not pipe but named socket! We can create named sockets from git-as-svn (see https://github.com/bozaro/git-as-svn/blob/master/src/main/java/svnserver/ext/socket/config/SocketConfig.java#L33 ). And here is client side: https://github.com/bozaro/git-as-svn/blob/master/tools/git-lfs-authenticate#L67 That particular thing is overcomplicated and setups protobuf on top of that socket, ignore that.

[zeripath]

Look at this: http:https://svn.apache.org/repos/asf/subversion/trunk/notes/ssh-tricks If we can add custom entries to gitea authorized_keys and gitea won't overwrite them, we could just add separate keys for svn access without messing with shadowing at all.

Also, you do not need to move gitea anywhere, you just need to change default shell that is spawned when user connects via ssh. This is good because we do not make changes to gitea installatiion that would be gone on gitea update.

Ah but then you're back to parsing authorized_keys - to keep it updated with the git-as-svn keys as there's no webhook you can use for that at present. (I think gitlab does have a webhook for this kind of thing tho.)

[slonopotamus]

We do not need to parse authorized_keys. ssh will do everything for us and call our svnserve --tunnel-user=bob when user with appropriate key logs in. We do not even have to be named svnserve (which is a good thing too because we do not shadow real svnserve)

[zeripath]

hmm... were you intending to make every user of svn a real user on the machine?

[zeripath]

BTW If moving around gitea isn't right then, munging/parsing the authorized_keys file could actually be very simple. A simple inotify/DirectoryWatcher that does the equivalent of sed -e's+/path/to/gitea serv+/path/to/shadow_like_above serv+' -i authorized_keys would work.

[slonopotamus]

Uff. Let's try again. SSH gives us a key via environment. We (fake svnserve) can take this key, pass key to git-as-svn, which in turn calls gitea api and asks "hey, gitea, tell us what user has this key". Gitea gives us username/email, authentication done, svnserve just proxies traffic between client and git-as-svn. This route will work if we use the same keys as Gitea uses. But this way we either need to shadow gitea shell or teach it that invoking svnserve is allowed.

Option numba two: we use separate keys (from gitea) for svn+ssh, but still same ssh account. This way, we can (via command=... in authorized_keys) tell ssh to invoke us the way we want to be invoked. Users connecting with keys that are made for svn+ssh simply cannot invoke any different command. And the command we force them to invoke is either the same as in option 1 (that still takes key out of env variables, passes it to git-as-svn, which uses gitea api to lookup user data) or git-as-svn-uber-cool-svnserve --user.name=<whatever>. Then we just use this username to lookup other data in gitea.

In both options, no authentication is performed by git-as-svn, it assumes that user was already authenticated and needs to blindly trust user identifier that our svnserve passed to it. That's why connection between svnserve and git-as-svn needs to be secured.

[MichaelJCole]

@zeripath, do you have any thoughts on #349 ?

I'm getting a client error "svn: E170001: Authentication error from server: unknown auth type: KEY-AUTHENTICATOR"