March 01, 2009
Audits III: we don't know enough even to know what we don't know
Are Audits going to help at all? Are they worth the cost? Are they part of the problem or can they be part of the solution? Originally, I claim they can help, especially for an organisation that has never been audited. That's my experience of one data point. But that's surely not sufficient, we need more. We need to know whether we can rely on these things, we need to know how to rely on these things, and when. And in the aftermath of the failure of Sarbanes-Oxley, we need to dismiss the easy answer of "we'll all just work harder."
In short, we need to know what it is we do know. Here is my view: we don't know enough.
Let's see if I can sustain that claim. If we read through the background of the cases of failure before us, whether Madoff, Satyam, Bear-Stearns, Lehman Brothers or all the bailouts, we will (a) find the Auditor, (b) find why he didn't pick up the failure, (c) cry foul, and say it should be like this or that, and (d) be fooled again. Why is this? We need to look beyond the superficial (tweaks like changing the auditor, rewriting the rules, or collapsing all firms down to the Big One) and go deep.
What actually do we the end-user really know about an audit? We can look at this several ways.
- We can read the audit opinion itself. That is, read any audit report of any bank-that-then-failed, and ask yourself what it says? Try these on for size:
- Is there any language in there tells us it is good? Or about to fail?
- Drill further. Do the criteria used for the examination advance your interests or not? Do you understand the criteria? Can you even find the criteria?
- Who was the audit report delivered to? If the opinion wasn't delivered to you how do you know that it is relevant to you?
- Are the opinions summarised, are critical disclaimers included?
- Did the auditor tell the client what was to be provided, or did the client tell the auditor what was wanted? Were the terms of the examination stipulated? Where does it say that? Where does it say it wasn't?
- Is it an audit, an opinion, a review, an attestation, or an attest? Is it a "trust service," an SAS70, or? Compatible with, or compliant with?
- Was it a compliance audit or chosen by discretion? Almost certainly, it was a compliance audit, but what was it in compliance with? How useful is that goal to you?
- Is there "audit language" in there that is only interpretable by another auditor? A "secret code," as it were, for other auditors?
If you didn't quite follow the above, that is precisely the point. To cut a long story short, if you can successfully interpret an audit report, you are probably either very experienced, or in the business yourself. For the most part, the result of the audit is inscrutable to the outsider.
- There wider business issues in the audit. Some are well known signals, frequently commented on in the press: Is the auditor too small or under-resourced to do the job? Is the auditor too big to avoid the channeled result, to avoid being locked in his box? Is the reviewer licensed by a body, tested to some standard, trained to some degree or knowledgeable through street learning? Are any of these relevant? And some are more subtle, but well understood in the industry: whether there are conflicts of interest, whether the auditor was chosen for the result, or more blatantly, whether the auditor is in pocket or for hire, or an out-and-out crook?
Just to ask that stress that last point, I asked a mate this seemingly innocent and easy question: "how do I find a dodgy auditor for hire?" Without a moment's thought, he came back with three recommendations: examine the regulatory filings, look for suspensions, and, ask a crooked lawyer. There followed a much more detailed explanation of how these things will help, which I won't bore people with here. Suffice to say, these dirty tricks reveal the existence of auditors who are easily for hire. Hopefully, they are the exception not the rule, but how do we know?
- It's probably also worth mentioning that the audit itself is only a very specific or narrow thing, yet most people like to think of the audit as a binary signal of saintliness. The public brand of the audit is still very good, indeed, almost unchallenged. The broader public likes to think of an audit as proof of goodliness, investment potential, security etc etc, when anyone who has been close to the situation knows that the gulf between perception and reality is so wide as to be at least wrong, definately troubling and possibly deceptive.
Let me explain what I mean by that point. Auditors if pressed will reveal that their opinion is strictly limited by a number of caveats. Indeed, the opinion is rendered over layers of indirection, such as the management's procedures rather than the assets in question. See point 1 above. However, Auditors will not press home the real conclusions: you yourself do not understand it, nor will you spot when it is no longer useful to you. Meanwhile, those same Auditors are happy to let you believe as the wider public that the audit is a singular, all encompassing stamp of goodliness.
In short, the Audit profession benefits by letting you believe in one very broad and saintly brand, but acts to reduce the scope of the result so far as to make that brand non-representative. To use a polite term, you understand ... the point to fixate on here is not why it is like this, or how far it is from the truth, but that this may explain why you don't really appreciate the limits of audit, let alone understand them.
My claim in today's post then is that the user cannot tell whether an audit is any use or not. Which audit is good for you, and which not, even if good for others? Which audit is good, and which is plain bad? The crux of the matter is that you yourself cannot tell what any of those pronouncements mean, unless you are an insider. You don't know whether you can rely, when to rely or how to rely.
Instead, you are offered a promise of a verified obscurity, within the comfort of a wonderful brand. In this situation, although there is a vague promise of positive results, there are also far too many circumstances in which the results can be positive for others, while negative for you, so obfuscated and confused as to be worthless, or, even as far as downright fraudulent. You will never know, and indeed, you probably can never know.
To put it in terms of the popular security media, the Audit is fully compliant with security-by-obscurity. In the security world, we would say that a tool designed to that standard is generally brittle. Once cracked, it often fails completely, and badly. This is because, although the obscurity gave a measure of protection, that same obscurity hid other weaknesses which could have been easily fixed. For that reason, we in the security field do not advise security-by-obscurity.
What that does to the concept of reliance on Audits is left for another post!
Posted by iang at March 1, 2009 04:35 PM | TrackBackHere is an exercise, useful in understanding auditing. You will need a sheet of paper, and a trash can with a metal or hard plastic bottom. Now, close your eyes and crumple the paper into a ball-- feel the paper crumple and listen to the sound... Now, open your eyes, and toss it into the trash. CRUMPLE.... PLONK. That's the sound of the audit manager discarding the audit findings of junior auditors.
Perhaps you will understand what's happening faster than most junior auditors do. The purpose of financial statements is not disclosure. It is to maintain the greatest possible secrecy and autonomy of movement by principals in the firm.
Posted by: SecretSquirrel at March 1, 2009 01:59 PMI think that people forget the simple idea of "you get what you pay for". You and I do not pay for the audit the principles of the organisation eing audited do.
Now I'm not saying that auditors are "on the take" but there must be considerable presure from the seniors in an audit firm to keep a "good name" on the books. Such presure would be difficult for a junior in an audit organisation to resist.
However I read a news story that even when a group of auditors are apointed by Congress they still find little of worth. This is based on the team tasked with Fredie Mac and Fannie May organisations.
It begs the question as to why a 100 or so "indipendent" auditors missed the shananigins of the principles of the two organisations...
I think it would not be unreasonable to sugest that audit as a process carried out in the way it currently is, is mainly a waste of time and resources .
Which gives rise to the question "how do we get greater oversight without disclosing information that would cause a "competative disadvantage" for those under going the process.
Posted by: Clive Robinson at March 7, 2009 11:58 PMActually I have to agree with that last comment. I have been a professional IT regulatory auditor for the better part of the last decade working both the corporate sector (SOX/HIPAA/PCI DSS/ISO 27002) and government (FISMA/DITSCAP/DIACAP/NIST SP 800-57/NERC CIP). While not auditor frameworks per se, you definitely audit against them given business regulatory and government legal requirements.
The basic problem of auditing is you are a paid investigatory whistleblower. As an internal corporate auditor (which I will also define as external auditors hired by the company being audited) you are always hamstrung by business needs. If you truly audit and present your findings in a unfavorable way your external auditing firm will lose business (costing you your job after it becomes a pattern). If you present them internally you will lose your job for airing dirty laundry; while you will rarely be outright fired you will get railroaded.
In the government sector it's even worse. Agency heads (SES's, Flag grade military officers, directors) have overall unquestioned authority and are highly politically in nature. Audits are often conducted 4 or 5 levels removed and running the results up the food chain is a quick way to get reassigned and/or kill your career permanently as bureaucrats have a long memory. It also doesn't help that everybody between you and the head has a valid professional career growth opportunity by stopping you to protect the head (who will write them favorable reviews). Once you hit the agency head level they have zero motivation to act given most issues, even if illegal, are reflective upon their duties hence they would have to acknowledge fault which isn't going to happen. Given that the GOA and various IG's have insufficient authority to act (or political will as there is little to gain in prosecuting your boss because even if they loss their job you won't get it, their lackey will), the only thing heads have to fear are congressional investigations and we all know how rare that is.
I long ago decided the only way to make audit work is to give them the authority to fire and relieve individuals on the spot without retribution (to include C levels and heads) but equally acknowledge this is unrealistic (prone for unethical abuse) as audit doesn't drive business or the government. Given this will always be an unresolvable issue means audit will always fail except when being used offensively by the folk paying the auditor and/or defensively to satisfy liability needs by shifting it onto the auditor. You still needs audits as it keeps the sheeple in line but it will never detect (or at least report as I am sure Madoff's auditors were well aware of what was going on) Madoff's, Enron's, and Coast Guard Deepwaters until it is too late.
I have hundreds of real worlds examples of illicit behavior but as an auditor, just like with whistleblowers, unless you are willing to lose your job, future employment prospectives, and your family over something that ultimate will have no impact (even if a congressional hearing happened, when was the last time you seen a agency head go to jail for violating federal law .. they simply get early retired) you will always been ineffective so you just deal with it, do the best you can, and if this bothers you, find a new career.
Posted by: Peter at April 6, 2009 03:39 PM...
And this is where we come to the party that up until now hasn’t received its fair share of scrutiny: PricewaterhouseCoopers, which as MF Global’s auditor was supposed to be the first-response regulator.
A week after I wrote in my BankThink column that the relationship between PwC and MF Global was too cozy for my taste, the regulators are catching up. Late Thursday, Bloomberg reported that regulators had subpoenaed PwC for “information on the segregation of assets belonging to clients.”
The CFTC’s action against PwC probably came as a result of a shocking CME Group announcement late Wednesday: "It now appears that the firm [MF Global] made … transfers of customer segregated funds in a manner that may have been designed to avoid detection." These transfers, CME Group said, appeared to have taken place after its audit team showed up last week at MF Global to take a look and found everything to be in order.
CME Group couldn’t have been hoodwinked like that if PwC had been doing its job all along. You can't circumvent controls unless there are none or there are holes. It was PwC’s job to review controls and the adequacy of policies and procedures to support them.
Since MF Global is a broker-dealer and a Futures Commission Merchant, PwC’s job went well beyond a standard audit. The auditor for a firm like this must annually review the procedures for safeguarding customer and firm assets in accordance with the Commodity Exchange Act. The annual audit must include a review of a firm’s practices and procedures for computing the amounts that, by law, have to be set aside in clients’ accounts each day. MF Global also had to send regulators an annual supplemental report from PwC. This report would describe any material inadequacies existing since the date of the previous audit and any corrective action taken or proposed.
...