Multiple SSH identities - beware!

One of my jobs required me to make use of a number of SSH keys for access to various different systems. This was a new experience for me; previous employers as well as university had made extensive use of Kerberos/GSSAPI. I modified my shell's startup fies to load the appropriate keys into my agent using ssh-add on login, and promptly forgot all about it.

The Symptoms

This worked seamlessly for some time, with me adding keys to the list as I was granted access to new systems. However I noticed that after a while, I would get 'Permission denied (publickey)' errors when logging in to some systems. It was pretty random, and I found that ssh-add -D (to remove all my keys from my agent) followed by adding in the one I needed worked around the issue. This happened rarely enough that I was never motivated to find the underlying root cause.

The Cause

It took me a long while to realise why this was the case; it was specifically called out in SSH: The Definitive Guide. As an aside: this is an excellent resource on the SSH protocol, covering the details of both OpenSSH as well as Tectia (the company formerly known as SSH Communications Security, who developed the original SSHv1 protocol).

When you connect to an SSH server specifying publickey authentication, unless there is a specific identity configured via the 'IdentityFile' option, SSH will attempt to use each key loaded into your ssh-agent in order.

OpenSSH's sshd supports a keyword MaxAuthTries which is particularly relevant here To quote the man page:

MaxAuthTries
             Specifies the maximum number of authentication attempts permitted
             per connection.  Once the number of failures reaches half this
             value, additional failures are logged.  The default is 6.

Each key that is presented for authentication is counted as a failed attempt. Thus, with the remote sshd using a default value of MaxAuthTries 6, if the key required to log in is the 7th one loaded into your agent, you will receive a 'Permission denied (publickey)' and your connection attempt will fail.

Note that this also impacts login attempts to servers requiring keyboard-interactive/password authentication, if you're not careful. Looking at the ssh_config man page:

PreferredAuthentications
             Specifies the order in which the client should try protocol 2
             authentication methods.  This allows a client to prefer one
             method (e.g. keyboard-interactive) over another method (e.g.
             password).  The default is:

                   gssapi-with-mic,hostbased,publickey,
                   keyboard-interactive,password

Note the default: publickey authentication is attempted prior to keyboard-interactive or password. If there are 6 or more keys loaded in your SSH agent, they will be presented in sequence until the server's MaxAuthTries value is reached, and the connection will be closed.

Workarounds

If you require more than 6 keys loaded into your SSH agent, or access to publickey and password based systems, there are workarounds. You can specific authentication methods and identity files either on the command line or in your ssh_config files. Some common patterns below:

ssh_config

ssh_config allows you to specify keywords on a per host basis, as well as using limited globbing.

For example, to specify authentication methods and identities for specific hosts, with a fallback of Kerberos for any hosts you haven't called out specifically, try the following:

# For badger, only try a password, don't bother with public keys.
Host badger
  PreferredAuthentications password

# For serenity, use publickey authentication and use the identity in ~/.ssh/keys/browncoat
Host serenity
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/keys/browncoat

# For miranda, use publickey authentication and use the identity in ~/.ssh/keys/reaver
Host miranda
  PreferredAuthentications publickey
  IdentityFile ~/.ssh/keys/reaver

# For any other hosts, only attempt gssapi-with-mic (aka Kerberos) authentication
Host *
  PreferredAuthentications gssapi-with-mic

Command line

The keywords mentioned above can also be specified on the command line with the use of the '-o' command line option to SSH. Multiple keywords can be specified with multiple '-o' arguments. For example:

user@host$ ssh miranda -o "PreferredAuthentications=publickey" -o "IdentityFile=~/.ssh/keys/reaver"

OpenSSH has a vast number of configuration options and features which are of use in different scenarios. I may do another post on this in the future, but in the interim, if you're interested, the OpenSSH man pages or the book referenced above are well worth reading.

GNU Make: a very short introduction

I finally got sick and tired of having a slightly different command line environment in each of the Linux/UNIX environments I use. This led me to put some effort into collating my various dotfiles (for bash, vim and screen) and coming up with a consolidated version of each, tracked using git, that I could install on any machine which I had a unique home directory on and not have to remember which of my preferences I've configured on this particular machine. At some point, I may post annotated version of said files here, along with what lead me to implement the various things in each file.

Today's post, however, is to do with a tool called 'make'. Make is most commonly associated with managing software builds on UNIX/Linux platforms, but more generically it can be described as a dependency tracking engine. A 'makefile' consists of a set of rules of the form:

<target1> [target2 ...]: [dependency(ies)] 
    [command 1]
    [command 2]
    ....

Where 'targets' are files that are being generated, 'dependencies' are other files (or other make targets) on which the current target depends, and 'commands' are commands to execute in order to generate an up to date version of the current 'target'.

In the case of my dotfiles, I wished to be able to do a few things without retyping excessive numbers of commands:

1. For each of the three subdirectories in my 'dotfiles' project, create a gzip compressed TAR archive, named <project>.tgz.


2. Additionally, my 'vim' directory contains several colour scheme definitions. I want two versions of my 'vim' archive, with and without the 'colors' directory.


3. I would like the ability to automatically copy these files to a directory accessible via HTTP, allowing me easy access to them wherever I have internet access.


4. I would like the ability to delete all existing archives in the project directory.

The naive way

So, let's assume that I don't have make available - how would I go about each of these tasks?

Creating archives (ignoring any advanced shell features, for simplicity):

[muz@codd:dotfiles]$ l
bash/ screen/ vim/
[muz@codd:dotfiles]$ tar -czf bash.tgz bash/
[muz@codd:dotfiles]$ tar -czf screen.tgz screen/
[muz@codd:dotfiles]$ tar -czf vim.tgz vim/
[muz@codd:dotfiles]$ l 
bash/ bash.tgz screen/ screen.tgz vim/ vim.tgz
[muz@codd:dotfiles]$

Creating separate archives for vim with and without colors:

[muz@codd:dotfiles]$ ls vim/
colors/ vimrc
[muz@codd:dotfiles]$ tar -cvzf vim-no-colors.tgz vim/ --exclude=colors
vim/
vim/vimrc
[muz@codd:dotfiles]$

Copying to a directory:

[muz@codd:dotfiles]$ cp *.tgz ~/public_html/dotfiles/
[muz@codd:dotfiles]$ l !$
l ~/public_html/dotfiles/
bash.tgz  screen.tgz  vim-no-colors.tgz  vim.tgz
[muz@codd:dotfiles]$

Removing all the tars:

[muz@codd:dotfiles]$ l
bash/  bash.tgz  screen/  screen.tgz  vim/  vim-no-colors.tgz  vim.tgz
[muz@codd:dotfiles]$ rm ./*.tgz
[muz@codd:dotfiles]$ l
bash/  screen/  vim/
[muz@codd:dotfiles]$

The make way

Based on my initial requirements, this was my initial makefile:

all: bash.tgz screen.tgz vim.tgz vim-all.tgz
clean:
        rm -f *.tgz
bash.tgz: ./bash/*
        tar -czf bash.tgz ./bash/
screen.tgz: ./screen/*
        tar -czf screen.tgz ./screen/
vim.tgz: ./vim/*
        tar -czf vim.tgz ./vim/ --exclude=colors
vim-all.tgz: ./vim/*
        tar -czf vim-all.tgz ./vim/
publish: all
        cp *.tgz ~/public_html/dotfiles/
        chmod go+r ~/public_html/dotfiles/*.tgz

This does everything that I initially specified.

make all

Create four .tgz files; bash, screen, vim, and vim-all, if any of the files that an archive contains have changed. (I changed vim to mean vim without the colors directory.)

make clean

Remove all files ending .tgz. This is an example of a 'phony target'; a make target that does not refer to a specific file, but instead causes a set of commands to be executed whenever it is invoked.

make publish

Copy all *.tgz files in the project directory to a web directory, and make them readable.

However, this is fairly verbose, and there is a lot of repetition. Using some variables/macros (some internal to make, some defined by me) my final make file is a little less redundant:

# Make file for tarring and publishing dotfiles.

HTDOCS=~/public_html/dotfiles
ARCHIVES=*.tgz
TARCMD=tar -czf

all: bash.tgz screen.tgz vim.tgz vim-all.tgz
clean:
        rm -f $(ARCHIVES)
bash.tgz: ./bash/*
        $(TARCMD) $@ ./bash/
screen.tgz: ./screen/*
        $(TARCMD) $@ ./screen/
vim.tgz: ./vim/*
        $(TARCMD) $@ ./vim/ --exclude=colors
vim-all.tgz: ./vim/*
        $(TARCMD) $@ ./vim/
publish: all
        cp $(ARCHIVES) $(HTDOCS)
        chmod go+r $(HTDOCS)/$(ARCHIVES)

(The in-built variable `@' refers to the name of the current target.)

Make can be used to simplify any procedure that requires multiple commands to be run as various files change. Software builds, operating system configuration files, document publishing... the list goes on. From a system/infrastructure administration point of view, there is a strong argument to be made that a makefile is the logical endstate of getting your documentation up to date. Rather than having a document that a human operator follows, have a makefile that a system can execute, and remove as much possibility for human error as possible. (Look for another post on the subject of human error soon.)

To quote an anonymous system administrator: "Everything should be a Makefile."

Lab machines on the cheap

While my Linux/UNIX skills as a user are fairly good, my administration skills are virtually non-existent. I know how to 'su -' and 'apt-get', but installing a production system from scratch and understanding how to put a secure, stable OS build together are well beyond my ken. My BSD knowledge is... limited, to say the least.

Fortunately, all of the above lend themselves quite neatly to the use of virtual machines; there's a lot to be said for being able to install a fresh operating system into a clean environment, play with it and break it to your heart's content and then wipe the slate clean when you're finished. All this, and no extra costs for new hardware either. VMWare Server is freely available for download, and allows you to run virtual machines in containers on top of your existing Linux/Windows install.

After the installer finished I was prompted with a web GUI to log in to, through which I could manage the virtual machines running on my desktop. Somewhat obscurely, I was prompted for a username and password to log in to VMWare Infrastructure Web Access, despite not having been prompted to configure one during the installation process. A quick web search revealed that VMWare authenticates against the local OS; once I had input my credentials and installed the requisite Firefox plug-in, I was presented with an inventory:

Half an hour later, I had a functioning FreeBSD installation on a newly created VM:

The plan is to start by using pre-configured installations of BSD, Solaris and Linux, and to move towards my own hand-crafted builds. The end goal is to have minimal, secure images for each OS that perform specific roles within an IT infrastructure. The first two on the list are a DNS server and a Kerberos KDC/TGS.

Why a blog?

In an outbreak of random motivation, I've decided to start keeping a blog. The main goal is to help me keep track of the various things I'm attempting to learn about, most of which are technological. If this ends up being useful to other people, then so much the better.

Things that interest me, and will be covered at some point (I hope):

• IT infrastructure
• Data networks
• UNIX/Linux/BSD
• Scalability
• High availability
• Managing large environments

For another perspective, behold my bookshelf of 'stuff to read', as of a year ago. (Some of these have been read now - once I source a new camera, I shall post an updated photo.)