December 28, 2009
Audits VII: the future of the Audit is in your hands
I established in a series of posts that Audit is in a crisis (I, II, III, IV, V, VI). It didn't perform during the financial crisis, and even if it had, we wouldn't know it. Audit has entered a phase of life where it can not deliver its brand-promise to the buying public, but the cost of the brand is delivered frequently in invoices to us, the buying public. Worse, the cost will go up and the relevance will go down, the machine they built ensures it.
What then do we do in the future? How do we live in a world of Audits without Control? How do we reclaim the control that works to our real needs?
As a user, as a (systems not financial) auditor, as a builder of systems, both financial and Internet, as an investor, as a financial player and as a party reading and relying on audits, I've come across only one person that will provide for your auditing needs. That person is:
You.
In a maxim, it is this: if you the user cannot see it, it is worthless. To you.
It is not entirely true that Audit is worthless, per se, in absolute terms. Many checks and balances can help, and this is the spirit that the audit profession alludes to. These checks and balances are good; we call them governance. But the problem for you is, you can't tell from the outside whether these checks, this audit, are useful or useless. Whether they are coded positively or negatively, whether they are purchased or perverted.
And therefore, your only good strategy is to label an opaque process as useless.
Which leads to a first step: Let's call for an open audit process, not a closed audit process. We know that "open" works from the Internet world, and the claim of many is that "open" can work in many more scenarios than we believed. I emphasise this in a presentation on An Open Audit (which, to close the loop back to the first post of the Audit series, was immediately after Bruce Schneier's apropos talk on the psychology of security).
But, please note, openness is only a first and intermediate step: once we get across the brave step of opening up the entire process, we are inexorably drawn to the fact that if an audit is really open, then the user can do it, herself. An open audit is an audit over open data; if the data is open, she can also audit the data herself.
All of it, or most of it, as much as the user can handle. Which is to say, even my meager attempt at open audit is not going far enough; what you really want is to openly audit the entire system yourself. I as auditor might simply lay the guide posts for you to follow, and in future, you can follow them better than I can.
Say hello to open governance . Yes, this way means more work for the user. But, this is work we already proved we could do. The wider Internet musters thousands of communities of thousands and millions, and a few of those people -- call them the 1% -- are the self-appointed guardians of truth and justice within their communities. Open governance harnesses the vigilantes of Wall Street, the crypto-jihadists of the security world, the peer-to-peer rebels of the intellectual property world, all, as the leaders in a process of checking for everyone else.
What then is the part of the professional auditor? We already recognised over the past couple of years that the proper role of the security expert is to educate programmers and architects to employ more security techniques. Likewise, the proper role of the auditor may be to teach the mechanisms of open governance; rather than opine on their results themselves. To teach, rather than to measure. To lead, rather than to do. To participate, rather than bill.
How would this work? Well, here's one idea. I haven't implemented it, but I want to. Over at the audit I participated in, there is a set of criteria which have to be audited against. Some have green ticks, others have red crosses, signifying OK and not OK. Classical audit process would call for me to investigate all those criteria, find evidence of controls over the criteria, and report on each. That's a lot of work. A lot of billable hours.
Open governance would call for each individual of the body-public to do that instead; in tech terms, each criteria would become a blog post, with comments added by the public, including comments of reliance. In effect, mini-opinions. If you the member-public post that the criteria is good and covered, and you put your monika on to that statement (which is easy to do because it is a CA and client-certs are its business), then that becomes reliable evidence. Once the set of criteria meets some watermark (say 95% green ticks), the audit is done.
By you.
That's just one idea. I know a dozen or so others; but their essence is all the same. Instead of having one person look and attest, have our entire net community look, and share notes. Travelling long distances, checking technical things and making clear reports is now trivial with the net, with cryptography, with protocols, with communities. We no longer need the single trusted third party to do this, we have the trusted members, we have our own stakeholders, we have customers.
It may be that the evolution of open governance, an invention from the world of digital cash, has come just in time to save us. We'll see.
Posted by iang at December 28, 2009 11:30 PM | TrackBackI've mentioned several times being at a european ceo/executive financial & exchange conference several years ago and in session on spreading issues with sarbanes-oxley ... that the audits just catch mistakes ... it has no way of catching determined fraud (at least the audit part, there is the whistle-blower section in the bill).
One of the suggestions was verify financial transaction claims in any corporation audit ... against corresponding information in other corporation audits (independent verification of the information). The claim was that the current public company audit infrastructure has no mechanism to implement such a thing ... since each individual company pays for the auditing of just their books (no verification against independent sources).
part of this is motto "trust, but verify" ... from DTRA (a relative spent a decade at dtra ... in treaty compliance):
http://www.dtra.mil/
The "open audit" concept is interesting. I looked at your WebTrust example on your site. While I'm not convinced at this point, I'd like to see a case study of this actually in action for a technical certification. I am hesitant to believe that a financial audit could be open sourced. How do you avoid the pitfall that if someone knows the audit procedures to be performed that they can then bypass them? One of the key audit concepts that you seem to be overlooking is the concept of unpredictability of procedures.
@Lynn: To what extent are you looking for auditors obtain "independent verification of information"?
Standard financial audit practices dictate that high risk financial statement accounts are verified with external sources. For example, cash is validated with independently provided bank statements, and sales and accounts receivable balances are independently confirmed with customers. In addition, the auditor needs to perform an analysis over journal entries to detect irregularities and fraud. Granted, none of these are foolproof methods, but some frauds were not detected because these procedures were not performed. In these cases, the frauds could prevented or detected earlier (i.e., Satyam provided false bank statements and the auditor did not bother to obtain independent statements fro the banks).
the comments were that current paradigm didn't easily promote the independent verification of every audited transaction because
1) possible conflict of interest ... since the auditing agency was being paid for by the organization that it was auditing
2) lots of the information about every transaction was available in audits of other public companies ... but because of the lack of independent audit process ... there was no obvious way of cross-checking all transactions across all audits
There was something analogous in lack of transparency and visibility in other related activities.
1) supposedly the information about illegal naked short sales transactions is available at DTC (or since merged with NSCC, DTCC) ... which DTCC is refusing to release. There are press items about DTCC being sued to make that information available
2) in year ago congressional hearings into the current financial crisis ... one of the critical components in the transactions resulting in the current financial mess were the rating agencies. The claim was that the seeds for that part of the mess was laid in the early 70s when the rating agencies changed from the buyers paying from the ratings to the sellers paying for the ratings (opening things up for conflict of interest).
Disclaimer: some of the (virtual machine based) online timesharing service bureaus from the early 70s quickly moved up the value chain to financial information. One of them is listed as buying the "Pricing Services" division from one of the rating agencies in the period of changing from buyers paying for the ratings to the sellers paying for the ratings. I had interviewed with them in the late 60s and stayed in touch with some of the people over the years.
In the more recent congressional hearings into the Madoff Ponzi scheme ... it was claimed that tips turn up 13 times more fraud than audits ... and that while the SEC didn't have a "tip" phone line ... they did have a 1-800 number for corporations to complain about investigations (some people pointed out that SOX had almost inverted its focus on what turns up the most fraud and what turns up the least fraud ... there is further mismatch when considering the cost of audit vis-a-vis the amount of fraud it turns up)
It was also stated in the Madoff hearings that transparency and visibility was much more important than new legislation.
Disclaimer: somewhat as result of having participated in the x9.59 transaction standard in x9a10 financial standard working group, in the late 90s, we were asked into NSCC (hadn't yet merged with DTC) to look at defining standard that improved security for all trades. Not very far into the effort, the work was suspended; a side-effort of changes for improving the security on all trades would have also significantly improved visibility and transparency ... something which apparently is not part of the trading culture.
somewhat related recent post in (linkedin) payment systems:
http://www.garlic.com/~lynn/2009s.html#39 Six Months Later, MasterCard Softens a Controversial PCI Rule
As referred to in the above, the countermeasures and the audits ... are enormously more expensive ... as well as the cost of the activities compared to the benefits.
This also gets into past "naked transaction" metaphor discussions that went on here ... some of my posts archived here:
http://www.garlic.com/~lynn/subintegrity.html#payments
also
https://financialcryptography.com/mt/archives/000745.html
https://financialcryptography.com/mt/archives/000744.html
https://financialcryptography.com/mt/archives/000747.html
https://financialcryptography.com/mt/archives/000749.html
oh, the numbers from the Madoff hearings were that audits turn up 4% of the fraud and tips turn up 52% of the fraud (13 times as much). A subtext was that there is a fantastic, enormous cost for something (audits) that show such poor results.
Posted by: Lynn Wheeler at December 29, 2009 10:45 AMNick,
A case study is surely the next step, see link for a sense of that. We see some sense of this with OpenSSL which has gone through FIPS, and has also spent many years out in the open. The question of whether the FIPS process helped overall in comparison to its open work is not something I've looked at, but someone should. We do know FIPS wasn't clearly only positive, there were problems in engineering the RNG (cough) which were not picked up.
Your pitfall of "someone knows the audit procedures" applies equally well to the closed auditor. Your Satyam case in point: probably false bank statements were provided because they knew the auditor wouldn't check them; try that with an aggressive open audit and they would have no such certainty.
Open auditing is conducted in the gold issuance world. Many of the major digital gold currencies publish their reserves and issuance on a daily basis. As temptation builds up to fix internal problems, those publications became material statements which acted as a break. I'm aware of one case where the statements made became false, and therefore fraudulent. If there had been an ability to go to the next step -- check the metal independently -- then that issuer would have been toast.
So clearly, there are some primary evidence things that are hard to open-audit. Bank statements and metal reserves, perhaps. But, having a statement about them is great evidence for later, and actually these aren't as hard as me might fear, we just need to put our heads together and figure out how to do it.
Posted by: Iang (one open audit) at January 2, 2010 06:24 AM