February 17, 2008

Say it ain't so? MITM protection on SSH shows its paces...

For a decade now, SSH has successfully employed a simple opportunistic protection model that solved the shared-key problem. The premise is quite simple: use the information that the user probably knows. It does this by caching keys on first sight, and watching for unexpected changes. This was originally intended to address the theoretical weakness of public key cryptography called MITM or man-in-the-middle.

Critics of the SSH model, a.k.a. apologists for the PKI model of the Trusted Third Party (certificate authority) have always pointed out that this simply leaves SSH open to a first-time MITM. That is, when some key changes or you first go to a server, it is "unknown" and therefore has to be established with a "leap of faith."

The SSH defenders claim that we know much more about the other machine, so we know when the key is supposed to change. Therefore, it isn't so much a leap of faith as educated risk-taking. To which the critics respond that we all suffer from click-thru syndrome and we never read those messages, anyway.

Etc etc, you can see that this argument goes round and round, and will never be solved until we get some data. So far, the data is almost universally against the TTP model (recall phishing, which the high priests of the PKI have not addressed to any serious extent that I've ever seen). About a year or two back, attack attention started on SSH, and so far it has withstood difficulties with no major or widespread results. So much so that we hear very little about it, in contrast to phishing, which is now a 4 year flood of grief.

After which preamble, I can now report that I have a data point on an attack on SSH! As this is fairly rare, I'm going to report it in fullness, in case it helps. Here goes:

Yesterday, I ssh'd to a machine, and it said:

zhukov$ ssh some.where.example.net
WARNING: RSA key found for host in .ssh/known_hosts:18
RSA key fingerprint 05:a4:c2:cf:32:cc:e8:4d:86:27:b7:01:9a:9c:02:0f.
The authenticity of host can't be established but keys
of different type are already known for this host.
DSA key fingerprint is 61:43:9e:1f:ae:24:41:99:b5:0c:3f:e2:43:cd:bc:83.
Are you sure you want to continue connecting (yes/no)?

OK, so I am supposed to know what was going on with that machine, and it was being rebuilt, but I really did not expect SSH to be effected. The ganglia twitch! I asked the sysadm, and he said no, it wasn't him. Hmmm... mighty suspicious.

I accepted the key and carried on. Does this prove that click-through syndrome is really an irresistable temptation and the archilles heel of SSH, and even the experienced user will fall for it? Not quite. Firstly, we don't really have a choice as sysadms, we have to get in there, compromise or no compromise, and see. Secondly, it is ok to compromise as long as we know it, we assess the risks and take them. I deliberately chose to go ahead in this case, so it is fair to say that I was warned, and the SSH security model did all that was asked of it.

Key accepted (yes), and onwards! It immediately came back and said:

iang@somewhere's password:

Now the ganglia are doing a ninja turtle act and I'm feeling very strange indeed: The apparent thought of being the victim of an actual real live MITM is doubly delicious, as it is supposed to be as unlikely as dying from shark bite. SSH is not supposed to fall back to passwords, it is supposed to use the keys that were set up earlier. At this point, for some emotional reason I can't further divine, I decided to treat this as a compromise and asked my mate to change my password. He did that, and then I logged in.

Then we checked. Lo and behold, SSH had been reinstalled completely, and a little bit of investigation revealed what the warped daemon was up to: password harvesting. And, I had a compromised fresh password, whereas my sysadm mates had their real passwords compromised:

$ cat /dev/saux foo@...208 (aendermich) [Fri Feb 15 2008 14:56:05 +0100] iang@...152 (changeme!) [Fri Feb 15 2008 15:01:11 +0100] nuss@...208 (43Er5z7) [Fri Feb 15 2008 16:10:34 +0100] iang@...113 (kash@zza75) [Fri Feb 15 2008 16:23:15 +0100] iang@...113 (kash@zza75) [Fri Feb 15 2008 16:35:59 +0100] $

The attacker had replaced the SSH daemon with one that insisted that the users type in their passwords. Luckily, we caught it with only one or two compromises.

In sum, the SSH security model did its job. This time! The fallback to server-key re-acceptance triggered sufficient suspicion, and the fallback to passwords gave confirmation.

As a single data point, it's not easy to extrapolate but we can point at which direction it is heading:

  • the model works better than its absence would, for this environment and this threat.
  • This was a node threat (the machine was apparently hacked via dodgy PHP and last week's linux kernel root exploit).
  • the SSH model was originally intended to counter an MITM threat, not a node threat.
  • because SSH prefers keys to passwords (machines being more reliable than humans) my password was protected by the default usage,
  • then, as a side-effect, or by easy extension, the SSH model also protects against a security-mode switch.
  • it would have worked for a real MITM, but only just, as there would only have been the one warning.
  • But frankly, I don't care. The compromise of the node was far more serious,
  • and we know that MITM is the least cost-effective breach of all. There is a high chance of visibility and it is very expensive to run.
  • If we can seduce even a small proportion of breach attacks across to MITM work then we have done a valuable thing indeed.

In terms of our principles, we can then underscore the following:

  • We are still a long way away from seeing any good data on intercept over-the-wire MITMs. Remember: the threat is on the node. The wire is (relatively) secure.
  • In this current context, SSH's feature to accept passwords, and fallback from key-auth to password-auth, is a weakness. If the password mode had been disabled, then an entire area of attack possibilities would have been evaded. Remember: There is only one mode, and it is secure.
  • The use of the information known to me saved me in this case. This is a good example of how to use the principle of Divide and Conquer. I call this process "bootstrapping relationships into key exchanges" and it is widely used outside the formal security industry.

All in all, SSH did a good job. Which still leaves us with the rather traumatic job of cleaning up a machine with 3-4 years of crappy PHP applications ... but that's another story.

For those wondering what to do about today's breach, it seems so far:

  • turn all PHP to secure settings. throw out all old PHP apps that can't cope.
  • find an update for your Linux kernel quickly
  • watch out for SSH replacements and password harvesting
  • prefer SSH keys over passwords. The compromises can be more easily cleaned up by re-generating and re-setting the keys, they don't leapfrog so easily, and they aren't so susceptible to what is sometimes called "social engineering" attacks.
Posted by iang at February 17, 2008 04:26 PM | TrackBack

ssh client is suppose to honor

PasswordAuthentication no
Protocol 2
StrictHostKeyChecking yes

so that it becomes even harder (explicit effort) to override and fallback to password.

note that PKI was designed to address first time communication between strangers ... where the relying party has no other mechanism for authenticating the stranger (i.e. the offline email scenario from the early 80s ... where relying party dialed local electronic post office, exchanged email and hung up ... this is the analogy to the letters of credit/introduction from the sailing ship days).

PKI doesn't actually make initial key distribution/exchange issues disappear ... they are simply obfuscated under layers of complex, expensive protocols and business processes. First off, the certification authority(s) public key has to be securely distributed to the relying parties (in a trusted manner). Most existing implementations involve certification authority public keys being included in some application distribution (and end-users have to trust that the correct keys were included in the application build and were never subsequently compromised before coming into their possession).

Then the digital certificates that are created by certification authorities ... need a trusted registration authority process ... where individuals and/or systems reliably transmit their keys to the registration authority. the registration authority then has to vet these keys before the digital certificates are created and returned.

All of these (mostly obfuscated) PKI processes don't fundamentally differ with what would happen in a purely SSH scenario (aka a remote server new key acceptance is on par with what would be needed to accept a new certification authority key or a certification authority checking out a key pursuant to creation of new digital certificate).

Any claims that the SSH model couldn't work implied that fundamental, underlying PKI implementations also couldn't work ... once the enormous obfuscating PKI business processes, costs, and complexity were stripped away.

Posted by: Lynn Wheeler at February 17, 2008 05:19 PM

Not to spoil your party, but a compromised SSH installation is hardly an MITM attack. Less the incompetence on the attacker's side, you were thoroughly screwed the moment you entered "yes" on that prompt to accept the changed host key (because, had this been MITM, Mallory would have forwarded the original host's authentic request for your private key, you would have authenticated, and they would have 0wned your session).

Posted by: Thomas Themel at February 18, 2008 07:35 AM

Not sure if you've arrived at the party as yet :) it indeed wasn't an MITM attack, as it says in the conclusions!

It was something far far worse: the machine was compromised by the attacker. The attacker already owned the machine, and used that to acquire something else, again more important than a simple 0wned session: passwords.

If the best attack against SSH is to take over the node, then that's good news for SSH, as it is secure enough. Question then is can we expand SSH to also protect more of the node?


Posted by: Iang at February 18, 2008 08:25 AM

Another observation ... it has been pointed out that the sysadms involved worked as a team, and it was only the combined actions of the team that detected the compromise. That's less good news for the model.

Posted by: Iang at February 18, 2008 08:30 AM

"Remember: the threat is on the node. The wire is (relatively) secure."

I would say they are about equal. We see a lot of phishing, and a lot of script attacks to guess passwords.

We know how to fix the node - the specifications for Bitfrost are a summary of current (unapplied) wisdom.

I don't think most people have grasped what is required to fix the network. On reflection, I need to set up a web page explaining the problem, and what is needed to fix it. My last attempt to explain the problem resulted in considerable indignation.

"The use of the information known to me saved me in this case. This is a good example of how to use the principle of Divide and Conquer. I call this process "bootstrapping relationships into key exchanges" and it is widely used outside the formal security industry."

Trust ascends from below, like a crane, not from above like a skyhook. You don't trust Joe merely because Verisign informs you his true name is Joseph Melgow.

Posted by: James A. Donald at February 18, 2008 07:43 PM

>Etc etc, you can see that this argument goes round and round,
>and will never be solved until we get some data.

I tried to get some data a while back on SSH key checking in response to SSH fuzzy fingerprints (if you're not familiar with fuzzy fingerprints, they create a close-enough fingerprint to the actual target to pass muster in most cases). Because human-subject experimentation requires a lot of paperwork and scrutiny, I thought I'd first try and establish a base rate for SSH fingerprint checking in general. In other words if you set up a new server with a totally different key from the current one, how many people will be deterred?

So I tried to establish the fingerprint-check rate in a population of maybe a few thousand users.

It was zero.

No-one had ever performed an out-of-band check of an SSH fingerprint when it changed.

Given a base rate of zero, I didn't consider it worthwhile doing the fuzzy fingerprint check :-).

So there is a form of data available, but because it's not very interesting it'll never be written up in a conference paper (there's a longer discussion of fuzzy fingerprints and related stuff in my perpetual-work-in-progress http://www.cs.auckland.ac.nz/~pgut001/pubs/usability.pdf). I've seen this type of authentication referred to as leap-of-faith authentication in a few recent usability papers, and that seems to be a reasonable name for it. That's not saying it's a bad thing, just that you have to know what it is you're getting for your money.

Posted by: Peter Gutmann at February 19, 2008 04:14 AM

>So there is a form of data available, but because it's not very interesting
>it'll never be written up in a conference paper

Having said that, if anyone would like to accept the following short paper for a conference, let me know.

-- Snip --

Do Users Check SSH Key Fingerprints?
Peter Gutmann, University of Auckland


-- Snip --

(I may have to include some references or something I guess).

Posted by: Peter Gutmann at February 19, 2008 04:20 AM
Post a comment

Remember personal info?

Hit preview to see your comment as it would be displayed.