How do we measure success in a cryptographic protocol?
Many talk as if it were ordained. This is a resort to religion. There are others that follow the words of their betters. Perhaps runes are cast, tea leaves remained, palms scanned.
All of this remains uninspiring. I mean, in a religious sense, where is the beauty in listening to someone telling you, "Believe, else yea be struck down!"
There has to be some science, some objectivity to this question. Why is it that one crypto protocol rises and another sinks? How can we measure this? How can we decide what is succesfull or not?
This is no mere sidelines question. No fodder for Tired Journalists. If we don't understand what makes a security protocol lead itself and ourselves to success, then how can we write the next one?
I propose these measures:
a. nunber of design "wins" - something that catches the eye. Press releases, deployments, applications that bought in to the secuity vision. They must have done so for a reason, their choice is tangibly measurable by us outsiders.
b. penetration into equivalent unprotected market. This is the easiest. If we have an alternate already in place, find some easy measure of comparison. How many people switch over?
c. number of actual attacks defeated. Now, this may seem like an imponderable, but it is possible to draw upper and lower bounds. It is possible and fruitful to estimate based on analogues.
d. subjective good, as at the application level. That is, a security protocol is naked without its application - what good has it delivered to its masters?
e. placebo effect - the inspiring of the user community to move forward and utilise the system, regardless of the real security so delivered. (This last one is subtle, but very important, as several people have commented).
These are all measures that can be applied externally, without need to worry ourselves over the goodness or otherwise of the cryptography.
From the list, we can exclude such worthless measures as
* deployed copies,
* amount of traffic protected,
* opinions of experts,
We need as professionals, objective, measurable metrics. Measure on!
Posted by iang at September 14, 2003 05:35 AM