March 09, 2008

The Trouble with Threat Modelling

We've all heard it a hundred times: what's your threat model? But how many of us have been able to answer that question? Sadly, less than we would want, and I myself would not have a confident answer to the question. As writings on threat modelling are few and far between, it is difficult to draw a hard line under the concept. Yet more evidence of gaping holes in the security thinker's credibility.

Adam Shostack has written a series of blog posts on threat modelling in action at Microsoft (read in reverse order). It's good: readable, and a better starting point, if you need to do it, than anything else I've seen. Here are a couple of plus points (there are more) and a couple of criticisms:

Surprisingly, the approach is written to follow the practice that it is the job of the developers to do the security work:

We ask feature teams to participate in threat modeling, rather than having a central team of security experts develop threat models. There’s a large trade-off associated with this choice. The benefit is that everyone thinks about security early. The cost is that we have to be very prescriptive in how we advise people to approach the problem. Some people are great at “think like an attacker,” but others have trouble. Even for the people who are good at it, putting a process in place is great for coverage, assurance and reproducibility. But the experts don’t expose the cracks in a process in the same way as asking everyone to participate.

What is written between the lines is that the central security team at Microsoft provides a moderator or leader for the process. This is good thinking, as it brings in the experience, but it still makes the team do the work. I wonder how viable this is for general practice? Outside the megacorps where they have made this institutional mindshift happen, would it be possible to ask a security expert to come in, swallow 2 decades of learning, and be a leader of a process, not a doer of a process?

There are many ramifications of the above discovery, and it is fascinating to watch them bounce around the process. I'll just repeat one here: simplification! Adam hit the obvious problem that if you take the mountain to Mohammad, it should be a small mountain. Developers are employed to write good code, and complex processes just slow that down, and so an aggressive simplification was needed to come up with a very concise model. A more subtle point is that the moderator wants to impart something as well as get through the process, and complexity will kill any retention. Result: one loop on one chart, and one table.

The posts are not a prescription on how to do the whole process, and indeed in some places, they are tantalisingly light (we can guess that it is internal PR done through a public channel). With that understanding, they represent a great starting point.

There are two things that I would criticise. One major error, IMHO: Repudiation. This was an invention by PKI-oriented cryptographers in the 1990s or before, seeking yet another marketing point for the so-called digital signature. It happens to be wrong. Not only is the crypto inadequate to the task, the legal and human processes implied by the Repudiation concept are wrong. Not just misinformed or misaligned, they are reversed from reality, and in direct contradiction, so it is no surprise that after a decade of trying, Non-Repudiation has never ever worked in real life.

It is easy to fix part of the error. Where you see Non-Repudiation, put Auditing (in the sense of logging) or Evidence (if looking for a more juridical flavour). What is a little bit more of a challenge is how to replace "Repudation" as the name of the attack ... which on reflection is part of the error. The attack alleged as repudiation is problematic, because, before it is proven one way or the other, it is not possible to separate a real attack from a mistake. Then, labelling it as an attack creates a climate of guilty until proven innocent, but without the benefit of evidence tuned to proving innocence. This inevitably leads to injustice which leads to mistrust and finally, (if a fair and open market is in operation) rejection of the technology.

Instead, think of it as an attack born of confusion or uncertainty. This is a minor issue when inside one administrative or trust boundary, because one person elects to carry the entire risk. But it becomes a bigger risk when crossing into different trust areas. Then, different agents are likely to call a confusing situation by different viewpoints (incentives differ!).

At this point the confusion develops into a dispute, and that is the real name for the attack. To resolve the dispute, add auditing / logging and evidence. Indeed, signatures such as hashes and digsigs make mighty fine evidence so it might be that a lot of the work can be retained with only a little tweaking.

I then would prefer to see the threat - property matrix this way:

Threat Security Property
Information Disclosure-->Encryption
Denial of Service-->Availability
Elevation of Privilege-->Authorisation

A minor criticism I see is in labelling. I think the whole process is not threat modelling but security modelling. It's a minor thing, which Adam neatly disposes of by saying that arguing about terms is not only pointless but distracts from getting the developers to do the job. I agree. If we end up disposing of the term 'security modelling' then I think that is a small price to pay to get the developers a few steps further forward in secure development.

Posted by iang at March 9, 2008 09:04 AM | TrackBack

Nice post!

We do have a chart, very much like the one you present, with threat, "property we'd like," explanation, and example.

I agree with your critique of Repudiation. The hard part about the move you suggest is it breaks our nice acronym, which seems a silly objection, except pronounceable acronyms (like STRIDE) are great mnemonics. Can you suggest an R word we could slip in, and repudiate repudiation? ;)


Posted by: Adam at March 9, 2008 09:30 PM

the non-repudiation schtick we saw in the 90s with payment transactions ... was that it supposedly would be used to change the burden of proof in consumer disputes ... which was worth a lot to merchants/banks (and implied that they would be willing to fork over a whole lot for digital signature infrastructure).

this is akin to some of the activity currently going on in the UK with respect to payment transaction disputes.

this was later brought home by lawyers that we dealt with when we were brought in to help wordsmith the cal. state (and later federal) electronic signature legislation.

where there seemed to be quite a tendency to promote semantic confusion just because the terms "human signature" and "digital signature", both contained the word "signature".

Posted by: Lynn Wheeler at March 10, 2008 06:33 AM

I got that chart up, with a bit of additional commentary. It's at

Posted by: Adam at March 14, 2008 09:20 PM
Post a comment

Remember personal info?

Hit preview to see your comment as it would be displayed.