December 28, 2009
a new way of auditing (A-VII-i)
Yesterday I finally finished/published my essay on Audits. This rant in 7 parts (I, II, III, IV, V, VI, VII) had been brewing for years, and finally came to a head with the tumultuous events of my open audit of a CA; that experience gave me enough of the "other side" knowledge needed to form the critical judgement.
In essence, audit fails audit, as can be seen from the idea that if you cannot see it, it is worthless to you. This is simply audit-talk, as we frequently say things such as "if we cannot see it, it does not exist." Audit fails its own test.
How then do we "open the eyes" of audit? I said it is in your hands, it is also your eyes. Here's that one idea amplified, revealed.
The Classical Audit process would call for an auditor to investigate every one of a set of criteria, and for each, to find evidence of controls over the criteria, document the evidence and derive the report from it. By way of example, here's one such criteria, taken from my audit, with some editing for brevity and style (i.e., you cannot rely on the content here):
| |
|
|
|
| A.2.f | The CP clearly describes how the identity of each certificate subscriber is verified. |
|
|
| WT25 |
This is criteria A.2.f taken from David Ross's criteria drawn up for Mozilla CA policy purposes (known as DRC). The reference WT25 is a cross-reference to the comparable one in WebTrust, an old set from the 1990s.
in my rendition, the text of the criteria itself is in the yellow boxes, and is treated as our input. The Auditor writes in the white boxes; formal statement in the first, expanded commentary in the right.
DRC has around 150 such criteria. One single criteria might take a day of research, if -- and only if -- the work has been done to meet the criteria. That's a lot of work. A lot of billable hours. (WebTrust has around 25 criteria, but there is rumour that they have a deeper, secret set of controls.)
Classical auditing practice would distribute this work to a team of employees within the practice. Employed at a fraction of the rate of the partner, but still billed out at impressive numbers (which is how partners become rich).
How can we distribute this under a more open regime of governance? If the data is open, we can as a community perform this control ourselves. We do not need an audit partner with 20 years of experience, earning a million in fees every year to confirm the presence of a document on the Internet.
We can do this check ourselves, or, we can do most of this check ourselves. Have a look at this very stylised mock-up:
| | |
|
| A.2.f | The CP clearly describes how the identity of each certificate subscriber is verified. |
|
| WT25 | ||
| |
|
|
|
Trinity
|
|
|
|
Ming the Merciless
|
|
|
| New: | Note that all Posts are Reliable Statements. |
The criteria stay the same, but the Auditor's work is now split up. The community confirms the basic elements and sign off on them. In the example, some vigilantes with colourful names such as Trinity and Ming the Merciful investigate and document the facts leading to the criteria, in the Pink boxes. Each box represents a signed statement made by the person. This is easy for the CA to do because it is made of a community of certificate users (but that is a strawman, any community can do this).
Then, once that is done, the Auditor simply confirms the criteria by walking over their statements. He also confirms that the basic form of reliable statement is sound; if they later prove to be false, recovery of some form or other should be available, which is perhaps the meaning of reliance.
This concept splits the audit team, but maintains the same elements. The Auditor might still be the very experienced, senior professional. But his team of reliable workers are now provided by the Community, as volunteers, and without the pyramid of fees.
Posted by iang at December 28, 2009 11:30 PM | TrackBackDo you have any evidence of open audits and similar in the pharmaceutical industry?
As you most probably know, pharmaceutical manufacture is heavily regulated (primarily by markets) by organisations such as the FDA (US) MHRA (UK). This regulation includes regular audits from these organisations - one of the implications of your 'audit rant' is that the conclusion would make these organisations redundant overnight.
Is there any evidence that this is happening?
Posted by: gyges at January 1, 2010 03:23 PMNope, none I've seen. And probably for good reason.
The notion of user-auditing or open governance requires one important thing: open data.
Now, open data can only happen in a few ways. It can happen in a non-competitive environmen, where we don't care if our "competitors" get our data, or it can happen under some convention of equal access, where we all release exactly the same data and therefore any competitive advantage is neutralised.
That latter might happen under regulation, including self-regulation, but both of these get tougher the larger the industry and the larger the spoils.
In the pharmacuetical industry (and others) you can see a sense of this with patents. Companies are rewarded with 20 years monopoly of they document their invention. Over in the CA world you can also see a sense of this, where there is public or open scrutiny of CAs, and there is an association of CAs that create their own regulatory regime. But both these are subject to capture.
Where it is best seen (IMO) is in stock market filings. Companies are required file with the regulator, which puts the filings up on some site somewhere (EDGAR in the USA). But it's pretty poor pickings when you consider that the open auditors are actually the owners of that data (shareholders).
What we really want is the real-time accounting data. Actually companies want that too, and are getting there. It couldn't really happen until around the last decade, because of the lack of infrastructure, but now we have ubiquitous net.
Not every drug is under patent ... there is a massive generic drug industry. Further, the generic industry has gone East and along with it a lack of trust and transparency. Everyone is making these generics the same way, or, at least: they should be.
Perhaps for generics, standard 'open source' processes should be available with the purchaser saying, 'we will only buy your product if you produce it using Standard Process AB23X1 under real time reporting conditions'?
Posted by: gyges at January 4, 2010 03:25 AMgyges: If a drug is listed in the United States Pharmacopeia, it is listed with its synthesis procedures. "We will only buy your product if you produce it using the USP process, under real-time reporting conditions of all precursors and their sources, who is working on which part of the reaction, and who is overseeing the operation."
Ian: This post seems to suggest that everyone has the same capacity to audit. As we've seen, this isn't entirely the case. You and Eddy and Nelson and Frank and Kathleen and I (referring to the dev-sec-policy group) can audit what the CP/CPS say, but we didn't watch the key-generation ceremony (for any given CA). Not everyone would know what to look for.
This goes double and treble for source-code auditing (and I submit that C and C++ are impossible to properly audit -- even Microsoft admits this, and this is why even though they can compile to the Common Language Runtime, they are always marked as 'unsafe'). It's even worse in a banking-type situation, or insurance, or any other fiduciary.
Posted by: Kyle H at January 14, 2010 08:39 PM