Over on Adam's blog he has a developing theme on 'security signalling.' He asks whether a code-checking program like RATS would signal to the world that a product is a good secure product? It's an important question, and if you need a reason, consider this: when/if Microsoft gets done rewriting its current poisoned chalice of an operating system, how is it going to tell the world that it's done the job?
Last night I had occasion to feel the wrath of such a check, so can now respond with at least one sample point. The story starts earlier in the week, when I reinstalled my laptop with FreeBSD 5.3. This was quite a massive change for me, up from 4.9 which had slowly been dying under the imposition of too many forward-compatible ports. It also of course retriggered a reinstall of languages, but this should have been no trouble as I already had jdk1.4.2 installed, and that was still the current. (Well, I say no trouble ... as Java is "unsupported" on FreeBSD, mostly because of Sun's control freak policies creating a "write once, run twice" environment.)
Anyway, my code went crazy (*). A minor change in the compiler checking brought out a squillion errors. Four hours later, and I'd edited about 100 files and changed about 500 lines of code. My eyes were glazed, my brain frizzled and the only cognitive capability I had left was to open beers and dispose of them.
Now, this morning, I can look at the effect, hopefully in the cold hard light of a northern winter's sunny day. At least for another hour.
It's security code (hard crypto payments) so am I more secure? No. Probably actually less secure, because the changes were so many and so trivial that the robot masquerading as me made them without thinking; just trying to get the darn thing to compile so I could get back to my life.
So one answer to whether the RATS proposal could make any difference is that it could make things worse: If thrown at a project, the rush to get the RATS thing to come out clean could cause more harm than good.
Which is a once-off effect or a singularity. But what if you didn't have a singularity, and you instead just had "good coding checks" all the time?
Well. This is just like the old days of C where some shops used Lint and others didn't. (Lint was an old tool for cleaning out "fluff" from your C code.) Unfortunately there wasn't enough of a real discernable difference that I ever saw to be able to conclude that a Lint-using shop was more secure.
What one could tell is that the Lint-using shop had some coding practices in place. Were they good practices? Sometimes, yes. Maybe. On the whole Lint did good stuff, but it also did some stupid things, and the net result was that either you used Lint or you were careful, and not using Lint was either a signal that you were careful and knew more, or you weren't careful, and knew less; whereas using Lint was a signal that you didn't know enough to be careful, but at least you knew that!
We could debate for years on which is better.
As an example of this, at a tender age, I rewrote the infamous strcpy(3) set of routines to eliminate buffer overflows. Doing so cost me a day or two of coding. But from there on in, I never had a buffer overflow, and my code was easy to audit. Massive benefit, and I preferred that to using Lint, simply because *I* knew what I was doing was much safer.
But how to convince the world of that? I don't know ... still an open question. But, I'm glad Adam has brought up the question, and I have the chance to say "that won't work," because the answer is probably worth an extra percentage point on Microsoft's market cap, in a couple of years, maybe.
* The code is WebFunds which is about 100kloc (kilo-lines-of-code) chock full of hard crypto, RTGS payments and various other financial cryptography applications like secure net storage and secure chat.
Posted by iang at December 19, 2004 09:31 AM | TrackBackCould you say a little more about what the change was? Was it one error, 500 times, or 500 errors? Was it a compiler warning, or an error that blocked compilation?
Also, I object somewhat to your description of error prevention as "care." Developers who either don't know about a problem, or are too pressed for time by management are not being careless, they're making an economicly rational tradeoff between perfection and shipping code.
How soon after format string errors were discovered would adding one to your code be "careless?" To say that another way, there's a reasonable level of care and education we should expect today, and then there are new bugs. The output of RATS would allow you to distinguish, somewhat.
> Four hours later, and I'd edited about 100 files and changed
> about 500 lines of code. My eyes eere glazed, my brain frizzled
Using a refactoring tool might have helped...
-- O.L.
Posted by: Olivier at December 19, 2004 03:21 PMTrust me to open my mouth when there are coding experts in the room...
OK, what the Java compiler now grumbles at is variable name shadowing. In the past, it would grumble if a method's variable name was the same as an object's variable name, quite rightly pointing out that one of them was hidden, or 'shadowed'.
This time, however, it is warning about static method variables shadowing object variables. These are however in two totally different spaces, so can not really shadow. Whilst programmer confusion is still a possibility, what is not possible is code confusion ... at least I don't think so.
class Moo {
int notHidden;
static call() { int notHidden; }
}
should generate an error. In my code, I had to change the names of all of those, chosing one or the other (generally the second). (Olivier, you could now postulate on whether a refactoring tool would be able to do that; I'd be impressed if it could as all the code and all the names are different!)
Whether Java is justified in warning is open to debate. Either way, it is instructive in its very debatability: the programmer has no choice in the debate, he just has the grumble to deal with, and that means fixing the code to turn the grumble off.
So, these compiler checks, Lint, RATS and similar are overriding the programmers' decisions. This is either a good thing or a bad thing, depending; it's a bit like the debate about Boeing versus Airbus and whether the pilot gets to fly a plane or a bus.
Now to opinion. In *general*, I think that driving an Airbus is the way to go, and I'm quite happy for "my teams" to run with high warnings set, because it helps to resolve general bugs. But, I think that what I am saying here is that I would stop short of calling the RATS/Lint approach something that effects *security*. I guess it's a bit like saying that security code is like flying a fighter, and you don't want no darn computer telling you how to do it.
(Of course this analogy falls apart very quickly under real world flying circumstances....)
Posted by: Iang at December 19, 2004 03:42 PMI think there's a distinction between overriding and commenting on, which you're blurring: The compiler insists you change these. RATS or Lint have no such power.
And you should read a bio of Boyd: It turns out that you almost do want a computer to fly a fighter plane, because given your energy state and that of your opponent, there are a set of moves that will cause you to win, and a set that will cause you to lose. A computer can "easily" be programmed to win a one-on-one fight between two planes. Human creativity is involved in ensuring that your plane starts with the better energy state and perhaps suprise, and dealing with more than one opponent.
Posted by: Adam Shostack at December 19, 2004 04:32 PMBlurring: well, my point here is that when you have a big coding shop, the rule tends towards "no warnings." So if you are using Lint, or RATS or whatever, it makes little difference whether the changes are voluntary or not. The boss tends not to be satisfied until everything is clean. Now, how this applies to RATS in particular I do not know.
Flying analogy: yes, it was in a tail spin from the moment I wrote it; I need a better analogy.
Posted by: Iang at December 19, 2004 05:16 PMending note in the above with respect to strcpy and buffer overflows and the recent thread going on in sci.crypt re: same
http://www.garlic.com/~lynn/2004q.html#51
(at the end of the following collection)
http://www.garlic.com/~lynn/subpubkey.html#overflow
Posted by: Anne & Lynn Wheeler at December 20, 2004 10:08 AM