ssh troubleshooting


This document must be read in full the first time. If you start from some nice looking section in the middle it may not help you unless you're already an expert at ssh.

In this document:


This document should help you troubleshoot ssh-related problems in installing and accessing gitolite.

IMPORTANT -- READ THIS FIRST

caveats


naming conventions used

taking stock -- relevant files and directories

normal gitolite key handling

Here's how normal gitolite key handling works:

(Other resources)

common problems

Since I'm pretty sure at least some of you didn't bother to read the "IMPORTANT: PLEASE READ FIRST" section above, let me take a minute to point you there again. Especially the first bullet.

Done? OK, read on...

The following problem(s) indicate that pubkey access is not working at all, so you should start with appendix 1. If that doesn't fix the problem, continue with the other appendices in sequence.

The following problem(s) indicate that your pubkey is bypassing gitolite and going straight to a shell. You should start with appendix 2 and continue with the rest in sequence. Appendix 5 has some background info.

step by step

Since I'm pretty sure at least some of you didn't bother to read the "IMPORTANT: PLEASE READ FIRST" section above, let me take a minute to point you there again. Especially the first bullet.

Done? OK, now the general outline for ssh troubleshooting is this:

random tips, tricks, and notes

giving shell access to gitolite users

We've managed (thanks to an idea from Jesse Keating) to make it possible for a single key to allow both gitolite access and shell access.

This is done by copying the pubkey (to which you want to give shell access) to the server and running either

cd $HOME/.gitolite  # assuming default $GL_ADMINDIR in ~/.gitolite.rc
src/gl-tool shell-add ~/foo.pub

or

gl-tool shell-add ~/foo.pub

The first method is applicable if you installed using the from-client method, while the second method is for any of the other three (see doc/1-INSTALL.mkd, section on "install methods", for more on this)

IMPORTANT UPGRADE NOTE: previous implementations of this feature were crap. There was no easy/elegant way to ensure that someone who had repo admin access would not manage to get himself shell access.

Giving someone shell access requires that you should have shell access in the first place, so the simplest way is to enable it from the server side only.

losing your admin key

If you lost the admin key, and need to re-establish ownership of the gitolite-admin repository with a fresh key, get a shell on the server and use the program called gl-admin-push that comes with gitolite. See instructions here.

simulating ssh-copy-id

don't have ssh-copy-id? This is broadly what that command does, if you want to replicate it manually. The input is your pubkey, typically ~/.ssh/id_rsa.pub from your client/workstation.

[Actually, sshd requires that even directories above ~ (/, /home, typically) also must be go-w, but that needs root. And typically they're already set that way anyway. (Or if they're not, you've got bigger problems than gitolite install not working!)]

problems with using non-openssh public keys

Gitolite accepts public keys only in openssh format. Trying to use an "ssh2" key (used by proprietary SSH software) results in:

WARNING: a pubkey file can only have one line (key); ignoring YourName.pub

To convert ssh2-compatible keys to openssh run:

ssh-keygen -i -f /tmp/ssh2/YourName.pub > /tmp/openssh/YourName.pub

then use the resulting pubkey as you normally would in gitolite.

windows issues

On windows, I have only used msysgit, and the openssh that comes with it. Over time, I have grown to distrust putty/plink due to the number of people who seem to have trouble when those beasts are involved (I myself have never used them for any kind of git access). If you have unusual ssh problems that just don't seem to have any explanation, try removing all traces of putty/plink, including environment variables, etc., and then try again.

Thankfully, someone contributed contrib/putty.mkd.


appendix 1: ssh daemon asks for a password

NOTE: This section should be useful to anyone trying to get password-less access working. It is not necessarily specific to gitolite, so keep that in mind if the wording feels a little more general than you were expecting.

You have generated a keypair on your workstation (ssh-keygen) and copied the public part of it (~/.ssh/id_rsa.pub, by default) to the server.

On the server you have appended this file to ~/.ssh/authorized_keys. Or you ran something, like the gl-setup or gl-easy-install steps during a gitolite install, which should have done that for you.

You now expect to log in without having to type in a password, but when you try, you are being asked for a password.

This is a quick checklist:

appendix 2: which key is which -- running sshkeys-lint

Follow these steps on the client:

This checks every *.pub file in your ~/.ssh, in combination with the server's authkeys file. It then tells you which key gets you what type of access (if any) to the server.

Note that it is not trying to log in or anything -- it's just comparing bits of text (the contents of /tmp/foo and of each of the *.pub files).

Note: It's also a stand-alone program, so even if your gitolite version is old, you can safely bring over just this program from a more recent gitolite and use it, without having to upgrade gitolite itself.

If the pubkey file you're interested in appears to have the correct access to the server, you're done with this step.

Otherwise you have to rename some keypairs and try again to get the effect you need. Be careful:

typical cause(s)

For server-based installs (i.e., not the 'from-client' install mode), the admin often has passwordless shell access to git@server already, and then used that same key to get access to gitolite (i.e., copied that same pubkey as YourName.pub and gave it to gitolite in some way).

As a result, the same key appears twice in the authkeys file now, and since the ssh server will always use the first match, the second occurrence (which invokes gitolite) is ignored.

To fix this, you have to use a different keypair for gitolite access. The best way to do this is to create a new keypair, copy the pubkey to the server as YourName.pub, then run gl-setup YourName.pub on the server. Remember to adjust your agent identities using ssh-add -D and ssh-add if you're using ssh-agent, otherwise these new keys may not work.

NOTE: If you installed using the 'from-client' install method, gl-setup won't work. You will have to use the more generic method described here.

appendix 3: ssh client may not be offering the right key

appendix 4: making git use the right key using ~/.ssh/config

If you have several pubkeys with access to the same server, you need to create/edit a file called ~/.ssh/config to make sure git picks up the right one when it accesses the server. There is no other way to do this, as far as I know.

What you need is to add something like this to ~/.ssh/config:

host gitolite
     user git
     hostname server
     identityfile ~/.ssh/YourName

Now you can use gitolite:reponame as the URL to make ssh use the named key. (Using git@server:reponame will end up using the default key id_rsa and, presumably, bypass gitolite).


appendix 5: why bypassing gitolite causes a problem

When you bypass gitolite, you end up running your normal shell instead of the special gitolite entry point script gl-auth-command.

This means commands (like 'info') are interpreted by the shell instead of gitolite.

It also means git operations look for repos in $HOME.

However, gitolite places all your repos in a subdirectory pointed to by $REPO_BASE in the rc file (default: repositories), and internally prefixes this before calling the actual git command you invoked. Thus, the pathname of the repo that you use on the client is almost never the correct pathname on the server. (This is by design. Don't argue...)

This means that, you get 2 kinds of errors if you bypass gitolite