July 20, 2007

ROI: security people counting with fingers?

A curious debate erupted over whether there is ROI on security investments. Normally sane Chris Walsh points to normally sensible Richard Bejtlich seems to think that because a security product saves money and cannot make money on its own, therefore it is not an investment, and therefore there cannot be ROI.

The problem the "return on security investment" (ROSI) crowd has is they equate savings with return. The key principle to understand is that wealth preservation (saving) is not the same as wealth creation (return).

If you use your fingers to count, you will have problems. The issue here is a simple one of negative numbers and the distinctions between absolute and relative calculations.

Here's how it works. Invent Widget. Widget generates X in revenue, per unit, which includes some small delta x in shrinkage or loss. Call it 10% of $100 so we are at an X of $90 of revenues.

Now, imagine a security tool that reduces the shrinkage by half. X' improves by $5. As X' of $95 is an improvement in your basic position of X at $90, this can then be calculated as an ROI (however that is done).

What then is the fallacy? One way to put it is like this (edited from original):

The "savings" you get back are what you already own, and you only need to claw them back.

No such, you don't have it, so it isn't yours to calculate, and resting on some moral or legal position is nonsense. The thief laughs at you, even if the blog evidence is that nobody else notices the joke, including economists who should know better! The thing that Richard is talking about is not "savings" in economic terms but "sunk costs."

In business terms, too, all numbers and formulas are just models. As the fundamental requirement is here to compare different investments then as long as we treat "savings" or "shrinkage" or "sunk costs" or whatever the same way in each instance of the model, the result is comparable. Mathematics simply treats minus numbers as backwards of positive numbers, it doesn't refuse to do it. A "savings" is just a negative number taken from another positive number that might be called "ideal maximum".

Having said all that, Richard's other points are spot on:

  • Calculating ROI is wrong, it should be NPV. If you are not using NPV then you're out of court, because so much of security investment is future-oriented.
  • Predicting the "savings" from a security investment is hard. There are few metrics, and they are next to useless. No security seller will give them to you. So you are left predicting from no base of information.
  • Hence excessively hopeful interest in metrics conferences and breach reports. But, I like Richard treat that skeptically. Yes, it will help. No, it won't make the NPV calculations anywhere near useful enough to be accurate.
  • NPV is therefore not going to help that much because they are wildly unfounded in their predictions. NPV therefore suffers from GIGO -- garbage-in-garbage-out! (more)
  • You need something else.

In closing, it still remains the case that security people say their managers don't understand security. And, as above, managers are safe in assuming that security people don't understand business. Another point that is starting to become more widely accepted, thank heavens, again spotted recently from sensibly sane Arthur ( Chris Walsh :).

Posted by iang at July 20, 2007 09:05 AM | TrackBack
Comments

Sorry, but the "sensibly sane" guy you link to last is Arthur.

Amusingly, the original version of my "You can't spell" post said that the right way to go was NPV (Hey, I can read Brealy and Meyers as well as anybody), but I deleted it precisely because of the GIGO problem. As you know, I am trying in my humble way to obtain less garbage-y data (as is Richard -- and I bet his budget is waaaay bigger!), so I think we all agree much more than we may seem to.

Posted by: Chris at July 20, 2007 09:58 AM

You said "Here's how it works. Invent Widget. Widget generates X in revenue, per unit, which includes some small delta x in shrinkage or loss. Call it 10% of $100 so we are at an X of $90 of revenues."

Where does this loss come from? Are you saying that in the course of doing business you suffer loss by not having security? (I am curious here, not trying to be difficult.)

Let's replace your project with another one.

Buy $100 worth of Treasury notes. At end of time period, receive $100 plus x% as indicated by the note.

Where is the loss here? Would you assume a similar "shrinkage" due to poor security?

It sounds like you are just assigning whatever loss a security measure might avoid to actual return-generating projects in the company. In that case it's still the return-generating projects that provide return, not the security measure. The security measure is avoiding loss but the revenue-generating project is generating revenue... AND THERE IS NOTHING WRONG ABOUT THAT.

Why can't security managers say "my security project will help you keep more of the money your revenue-generating project creates" and be respected, without resorting to claiming "ROI"?

Incidentally, I fully appreciate NPV but that is not the central issue I am debating here.

Thank you for commenting.

Posted by: Richard Bejtlich at July 20, 2007 10:26 AM

[quote]
What then is the fallacy? One way to put the fallacy is that the "savings" you get back are what you already own.
[/quote]

I simply do not understand that.

I sell a widget for $100. The total cost of producing that widget, including utilities, wages, taxes, cost of goods, and "shrinkage", is $90. Therefore my profit per widget sold is $10. That is a ten percent margin -- pretty huge actually.

Now I invest in a security tool that reduces shrinkage per unit by $1. My cost per widget falls from $90 to $89. My profit per widget rises from $10 to $11.

I have just increased my profitability by a full TEN PERCENT.

There is no "fallacy" there. I am now a much richer individual. I means *nothing* to me to say that "the savings you get back are what you already own."

That statement is factually incorrect. The savings I get back are NOT what I already own. The $1 I saved in shrinkage formerly when OUT THE DOOR every time I made a widget. Now that $1 stays IN MY POCKET.

(I am ignoring the fact that the greedy rapacious state will want half of the dollar I saved.)

Will someone please explain to me how my simple arithmetic analysis is so wrong-headed, and how I "just don't get" the amazingly advanced economic thinking espoused in the article?

Posted by: Patrick at July 20, 2007 10:33 AM

Taking the easy one first: Patrick, you have spotted that the fallacy statement was badly worded. You are correct in your claims, and that's what I meant to say. I've changed the text of the entry to better reflect that, keep yelling if it isn't clear :)

Posted by: Iang (random accounting paper) at July 20, 2007 11:15 AM

Richard,

the mistake being made here is to assume that the "loss" is something that you own or have, and can account for. In economics, this isn't any good. Instead, we say that "sunk costs" are written off, gone, non-existent, ASAP, before someone mistreats them. In accounting, if you don't have it, and you never had it, it shouldn't be there on the balance sheet. It shouldn't enter into an accounting model like ROI/NPV, etc...

You say: "Where does this loss come from?"

It doesn't matter. Consider it this way. You have a car that can deliver 10 km to the litre. Now, it could deliver 20 km to the litre if you invested lots of money and improved it. Or, it could deliver 20 km to the litre if you invested a little money and put a locking petrol cap on the tank!

Either way, your current position is "10". To get to "20" you have two projects. The fact that one of them might be "saving fuel that was stolen by crooks 50% of the time" and the other is "efficiently utilising diesel/turbo/green/shark skin blah blah" is irrelevant to the model. The model just does maths, it wants to know what today is, what the prediction is, and it will tell you tomorrow for both projects.

The "loss" versus "investment" thing is just a human label.

(Note that "savings does not generate wealth" is a correct statement but it uses the economic definition of savings, being cash you own, already invested in market, *not* "potential savings" extracted from processes, which is definately not "economic savings.")

Posted by: Iang at July 20, 2007 11:31 AM

Also: "It sounds like you are just assigning whatever loss a security measure might avoid to ..."

To be more precise, I am not recognising the loss as existing, for model purposes.

But you are correct to be skeptical about the process: assiging this cost to that process instead of another is an area that is fraught with problems. There is a whole breed of accounting devoted to those games within corporates, but I forget the name of it for now. Basically, a smart operator can generate whatever he likes by doing that across complex systems.

Posted by: Iang at July 20, 2007 11:38 AM

I understand sunk costs as described at

http://en.wikipedia.org/wiki/Sunk_cost

just so we have a common example, i.e., a nonrefundable movie ticket.

Where is the sunk cost for the scenarios I described here

http://taosecurity.blogspot.com/2007/07/glutton-for-roi-punishment.html

specifically, at the point where spam has reduced output from 120 per month to 110 per month?

Again, seeking understanding here, not argument for argument's sake.

Posted by: Richard Bejtlich at July 21, 2007 09:13 AM

Hi Richard,

Yes, a non-refundable ticket is a good example of sunk costs ... a discount airline ticket these days is better because it can't be sold to someone else.

You are right to challenge my use of sunk costs, as I'm not being strict to the words, but more using the concept. But maybe that is what this is about: understanding what the concepts are behind the words.

It is important to realise that sunk costs is a concept, not a "cost" per se. We don't say that such and such a line item on the balance sheet is "sunk costs" but rather, we say that certain things we spent (or did, timewise) are sunk, because we can't get anything like their value back again. Therefore, as the wiki entry states, we should not let the existence of sunk costs effect our future decisions. (This assumes we are rational actors, but that opens a can of worms, so let's assume it today.)

So, back to the security situation: when you have scenario A, and then along comes Hacker H and turns it into scenario B, we have a situation where A is gone, and B is here. We will never see A again. B is now the current situation.

That's "sunk costs" as a concept. Narrowly, whatever we spent on A is irrelevant, but I'm also suggesting that A itself is irrelevant because we now have B. B is the starting point for any decision, and A isn't in any decision.

In designing Cn, we might be informed by A and our experiences, and in this sense the "observed hacking" might help us to design C1 which delivers "savings".

But, in decision making terms it is incorrect to say that C1 delivers savings, because our decision is based on how to move from B to C1, C2, C3, etc, and A is long gone. C1 will never be A, although it might approximate it.

(Now over to your blog to answer you new example.)

Posted by: Iang at July 21, 2007 10:43 AM

Thought the following would be of (some?) interest.

"Someone writes in with the following question:

I've been studying Information Technology risk for some time now and so your work is of great interest. In IT risk we have several problems that a Bayesian approach would seem to help us address. Namely:" Further in the link.

http://www.stat.columbia.edu/~cook/movabletype/archives/2007/09/bayes_and_risk.html

Posted by: Darren points to risk at September 11, 2007 10:34 AM
Post a comment









Remember personal info?






Hit preview to see your comment as it would be displayed.