Thursday, July 19, 2007

Discussion summary on improving SSH

SSH (OpenSSH) is a workhorse for secure web administration and, as such, important for managing Internet servers including email. SSH is, usually, the only way to access remotely located servers. Of course, this makes SSH a valuable attack target.

There are established practices for securing SSH. However, they currently rely substantially on additional systems, that also present their problems and add to the maintenance burden.

After some discussion in the list and in private, I present below a summary of the dialogue and clarifications with a view to improve SSH. I've divided it in sections A-F, to make it easier to discuss.

The free test tool Nessus allows a remote SSH server to be quite easily discovered and explored by automated scans & attacks. The information obtained in SSH server scans can be typically divided (from the view point of a sys admin) in six issues (to be discussed later, this is just a list):

1. no way to block scanning and attacks
2. does not learn from attacks
3. advertises itself and the server OS
4. allows empty authentication requests (eg, requests by invalid users)
5. sends host key fingerprint to invalid users (eg, with no username)
6. sends a reply to invalid users (eg, with no username)

These issues, alone or in combination, can lead to multiple vulnerabilities, some with higher security significance than others (according to operational factors as well).

Let's first state that not all the six issues are created by crypto use in SSH. And this leads us to the main question here:
Could new crypto in SSH help mitigate or solve any of these six issues?
Currently, how are these issues addressed? By non-portable and potentially buggy methods (see below), with more software that has to be patched, updated, configured and tested, all at different times, with added security gaps that can be compromised.

New crypto methods in SSH could do a better job by reducing the problems in the first line of defense at SSH itself and by avoiding security gaps. Non-crypto things such as PAM, tcpwrapper, port-knocking, grey-listing and others can always be added later anyway, if desired.

Some issues could be minimized by turning off password authentication, to use only private/public keys (PKs). SSH allows users to generate PKs without cost (no need for a CA). Users can keep their private keys in their drives and also in a USB dongle, for example. One can also add second-channel authentication (e.g., RSA SecurID) to SSH with a PAM or a patch.

However, turning off password authentication is not desired in many cases, even when PKs and second-channel authentication are used.

Password authentication may be more secure against attacks on the client, to gather information before compromising the server. According to Peter Gutmann, a study of SSH attacks a few years ago showed that nearly two thirds of all SSH private keys were stored on disk with no password protection at all, so that simply being able to read a hard drive will get you access to any number of systems without having to trojan the SSH client or plant a keyboard logger as you'd need for an SSH password. So turning off password authentication could make things less secure, not more.

Another reason for using passwords in SSH, and perhaps more compelling, is that you don't want to risk getting locked out just because your disk died, you forgot your laptop, SecurID or USB dongle, and the private key cannot be found (for security reasons!) in the back-up. Passwords can also easily be given over the phone from sys admin to sys admin, and are easily protected by threshold cryptography. And using a private/public key pair in SSH would anyway require a password for protecting the private key.

Using good, high-entropy passwords turns out to be preferred by many sys admins because it is simpler, more portable, and works fine for security (provided the password is chosen with enough entropy). Sys admins already need to use many passwords for many servers and protocols, and have safe methods for off-line storage, redundant record keeping, and authorized record access (including a handwritten notebook, encrypted USB dongles, and a printed list), that are of course not kept near the computer.

Some of the issues can be solved with SSH PAM, such as pam-abl. However, PAM libraries are usually buggy (buggier than SSH), and might be unstable unless suitable PAM items were added by which the daemon can communicate the client IP address and port number, and maybe other things too.

I believe that each one of the issues 1-6 can be met by a cryptographic protocol deployed in SSH. Some solutions are well-known. BTW, there were some similar issues in the SSL protocol.


#1. no way to block scanning and attacks
Firewall port-knocking can help here. It is hard to guess and defeat; nothing shows up in scanning. However, it is not portable and is hard to debug and use.

While there are also IDS that can turn off the SSH port if a scanning is detected, this can lead to DoS.

Pam-abl can also help, but PAM is usually buggier and adds security gaps.

Possible solution: SSH could easily incorporate some of the pam-abl code, enough to block an IP for some time T after N failed attempts, self-clearing after T.
#2. does not learn from attacks
Possible solution: SSH can incorporate some of the pam-abl code for this purpose.
#3. advertises itself and the server OS
"Need to know" is a valuable security policy.

Advertising the OS may compromise a higher security function (the OS) in the case of a zero-day or unpatched exploit. SSH MUST NOT advertise the OS. It may compromise policies and apps in a lower (more critical) layer than SSH, not just the OS.

Also, advertising SSH's innards in "already dead" authentication has no useful purpose that could not be more securely met.

Possible solution: let the client tell its version to the server, for the server to decide if it is acceptable -- it is a server function anyway (eg, not to accept SSH v1).

Additionally, it would also be useful for the client to send to the server a list of acceptable configurations, for the server to select, so that the server would no longer be groping in the dark as to what algorithm and parameters to use.
#4. allows empty authentication requests (eg, requests by invalid users)
#5. sends host key fingerprint to invalid users (eg, with no username)
#6. sends a reply to invalid users (eg, with no username)

- Some claim that the SSH protocol requires key exchange to occur before the user identity is sent, as it protects the user identity from disclosure to an impersonator.

However, if the user does not have a valid username, why and what should the server have to prove to that user? Invalid data deserves either "no response" (safer) or an error response. The SSH server should not do work or provide information to an user that is known already to be invalid. The SSH protocol needs improvement here.

- Some also claim that if the SSH protocol would permit such differing responses for valid/invalid users, then attackers can use SSH as an oracle to determine username validity.

However, SSH can prevent username enumeration by trial and error. With a simple keyed-hash, if this protection is desired, one would just enter the same key for the server and clients, so they can be all using the same transform. This is a simple mechanism that hackers cannot break by guessing. Note that SSH already uses a hash mechanism to hide domain names in the hosts file.

- Some claim that the issues #4 and #5 are actually "features" that are used by some of the anonymous services, such as anonymous CVS.

However, these "features" could be closed by default, and opened in the sshd config file if desired. This may also need some cryptographic protocol changes.
Because the reported issues could be improved by adding crypto, I am asking for more input first, before taking these issues to the OpenSSH development list. This should help the development effort. The OpenSSH development mailing list is at:

Thanks for all the input! Comments are welcome.

Ed Gerck


Ed Gerck said...

From: Pat Farrell
Subject: Comment on improving ssh

Hi Ed,

> Comments are welcome in here (if crypto) an in the blog in
> general.

I don't see a way to add comments onto the blogspot, perhaps I'm just
not looking in the right place. Or do you want all comments to the
openssh-unix-dev instead?

[Ed: I added to the blog on your behalf].

Here is my half baked idea, its been rattling around in my head for a
while now.

I like the ideas of skey or a SecureId, in general, for logging in.
But skey has no hope of being accepted by non-geeks. And SecureId is
expensive to deploy and a bit of a management hassle.

There have been postings and rumours of SecureId equivalents in Java
enabled cell phones. I've not seen any evidence that this is more than
etherware, but I think the idea is worth exploring.

How to best implement this is clearly an engineering question. Here is
one idea:

Have the SSH login trigger an outgoing SMS message, with a challenge.
This would naturally be delivered out of band. The user could get the
SMS from their cell phone, and enter it to login.

There are many variations, say having one option for challenge/cell
phone, and another requiring a stronger password for use without the


Pat Farrell

Ed Gerck said...


What you suggest is viable to help protect the password but the
password is usually very good in ssh (ssh users are techies).
Also, it would probably not work in the environment where SSH is
used. It would require (for example) an SMTP service to be working.
SSH is a basic sys admin service that should not rely on too many
things already working (as SSH is used to configure these things).

Further, it would not mitigate the issues 1-6.

Ed Gerck

Ed Gerck said...

From: Pat Farrell
Subject: Comment on improving ssh

All true. But your blog talks and good practice talk about high entropy passwords. Which are more often talked about than used.

Clearly adding SMTP opens interdependancies, which are bad. And even good STMP implementations (postfix) are too complex to be analyzed.

As I said, its mostly half baked, but if firmed up, maybe we could move ssh out of pure techy world.

In the non-geek world, hard passwords are kept on yellow stickies on the monitor.


Pat Farrell

Ed Gerck said...


Thanks. Users can also generate private/public keys (PKs) with
SSH, at no cost (no need for a CA), and not use passwords. Some of
the 1-6 issues are less relevant in terms of creating
vulnerabilities if SSH only accepts PKs (which is easy to
configure). Several universities have user instructions for
non-geeks to use SSH with PKs, in Windows, Mac, and Unix/Linux.

Ed Gerck

Ed Gerck said...

From: Chris

Maybe I'm missing something, but it seems many of the problems might be overcome by making at least the first part of the ssh protocol UDP-based. I originally thought that there's no rejection message from UDP, but looking on Google it seems there sometimes is. Some firewalls filter out the returning ICMP error message, too, so probably an attacker wouldn't learn much about the system from the fact that there was no ICMP error. They could try to connect to some rarely open port and see if they get an error - it might be simple enough to add support to generate the fake ICMP in reply, or just disable ICMP replies altogether (I think this is best practice according to some). I don't know what most operating systems (eg windows) actually do - if they failed silently, then you'd be able to 'blend in' with all the servers not running ssh quite easily.

You would have a udp packet to start things off, containing something like a username encrypted to the server's public key. (I think the setup at the moment is to use the server pubkey only for signing? You'd need to work something out.) This would mean that you have to add the server keys manually, but a) you have to verify them anyway and b) you need to have some kind of information not available to the random internet passerby, which would need to be distributed similarly anyway. You could, in principle, just send the username unencrypted, but that gives anyone sniffing information (which ssh is designed to avoid, obviously).

If you can blend in with everyone by just dropping the packet, you can solve #3-6, and #1 is at least made somewhat better. It ends up very like port knocking, but only the server needs a (small) modification.

#3 is fixed because the initial UDP packet can be made very simple, so there should be no bugs needed to be worked round until after there has been at least some kind of user->server authentication. Then they can exchange version numbers, if needed, although there's still privilege escalation to worry about. #4-#6 are no longer true.

imo, #4 is basically (at the moment) a good thing, because otherwise it would allow people to enumerate users. It's not like they're going to succeed authenticating as a non-existent user, right? I think the idea of a keyed-hash being the solution is not bad in principle, but would be unwieldy to administer - another password to remember that everyone who uses the server needs to know. It seems like a bit of bother to solve something that's not really a problem (at least, it's a subset of #6, which is a problem).

#5 is not, as far as I can see, a security problem - I can't think of a way in which that information would be useful to an attacker.

I am not a fan of auto-blacklisting, but I think (for example) detecting rapid changes of username might be a good idea.

Good luck with your improvements,


Ed Gerck said...


Your UDP suggestion is interesting and could help not stand out. To work, one would have to close 22/tcp until the 22/udp authentication process is successful. Let's think more about it and then see what ssh dev hinks.

The keyed hash for the username transform does not use a key that anyone needs to remember, as the key can be stored in the server and clients without much security concern. That key is not used for authentication, just for online encoding.

Auto-blacklisting, even with a relatively short recovery time, should not be used for usernames because it could lead to DoS. We should only use it for IPs. If the username is key-hashed, then (again) only IPs should be used (but not for DoS reasons, as DoS is not likely).


Ed Gerck

Unknown said...

This is only marginally on-topic, but on the crypto mlist you mentioned that OSes have different firewall APIs, making it hard to trigger rule changes.

I have a solution to this: it's called the Dynamic Firewall Daemon.

The goal is to create a firewall-independent API using text-based commands to trigger rule changes. These textual commands can be sent automatically by IDSs such as snort or by a sniffer specifically designed to catch too many SSH login attempts.

Nick Owen said...

A couple of quick thoughts:

I have done a lot with pam_radius ( and find it to be a very reliable way to add two-factor authentication to SSH, though there are many pam libraries and so quality varies greatly. We have one of those Java tokens (including an open source version in j2se). I cannot recommend SMS for a number of reasons, mostly b/c it is often unencrypted and carriers are not incented to protect the accounts.

I also think that embedding two-factor auth into SSH would be limiting and probably add unnecessary complexity. It would be better to add radius support, if you are concerned about pam. Best to support an open standard.

Adding some of the pam_abl capabilities to SSH would be excellent.

While it may be outside of the current scope, I will throw out that SSH has some short-coming vis-a-vis compliance ( I only mention it b/c you might be able to address some of these issues (lack of key expiry, no way to control which user has public key authorization, and no way to enforce password complexity) in this effort. At the same time, add 2FA addresses these, so it's not really needed IMO.

I really like the idea of the hashed username. As a 1st step, could you also pad the username with something that the server recognized and would toss out improperly padded usernames? I'm just wondering if something like that would be easier. So, our protocol is to login with 010nowen, any username without the 010 is ignored or blocked?

HTH, & thanks for all the effort!