Wow - reading that made the hair on the back of my neck stand up! I've been in similar predicaments and it ain't pretty! Best wishes on your speedy escape from 'The Twilight Zone'!

That's how I felt when I finally saw what had really happened! I just had to stop at that point ... hit the beer fridge and leave it until the next day. Luckily I was able to just shut down the effected issues, heaven knows what one would do in a busy system. Which is why I'm thinking on the systemic solution... and not rushing it :)

A lot of food for thought, indeed. Makes one ask "can it happen to me?", but this question is meaningless. Of course it can't, but it couldn't have happened to you either, and yet it did. That's the whole point. But I'll try, nevertheless:
Our payment system guards against the dark forces of evil by making the set of signed receipts and the transaction records actually the same thing; the signed receipts constitute the recorded database. There are no other records. Transactions happen as their signed receipts enter the public records. Thus, signed receipts are distributed not only to all the account's users but to everybody.
This, of course, implies that the signed receipts should be devoid of unencrypted private information, which is quite a challenge by itself, and I have no idea how to formalize this requirement and how to verify it. I hope that I have solved this problem, but I cannot state it with any certainity.
I am perfectly satisfied, however, that in our system there can be no inconsystency between the signed receipts that the users have and the state of the system that the issuer (the minting service, using Ian's terminology) sees, because the two are the same thing. This is certainly part of a systemic solution, but is this enough?
The conservative nature of contractual value can be verified by anyone at all times. This was one of the most important design criteria, though for different reasons (to guard against malicious issuers -- the architects of our system are Hungarians, whose grandparents have witnessed the hyperinflation of 1946 that ended with the exchange rate of 1:4e29).
We also have a fail-stop procedure (for contingencies such as the compromise of the issuer's private signature key), after which the users have to prove title to contractual value using signed receipts from before the triggering of the fail-stop mode.
Thus, if the mint is by accident restarted from an earlier backup with some transactions missing, we have both a proof (two different receipts with the same serial number) and a procedure to follow. Our wallet application reports such inconsistencies (if noticed) to the issuer, triggering the fail-stop. Doing so is in the best interest of the clients, so there are no tragedy of commons issues here; users have no reason to disable the verification code in the wallet application.
Am I right in my assertion that we will always have a regular way out of the twilight zone?

again some background ... financial transactions mapping to database transactions
http://www.garlic.com/~lynn/2005f.html#27
http://www.garlic.com/~lynn/2005f.html#32

If The Twilight Zone exists, it's to myself an issue, I don't know, don't think, if I can fail to dwell on, as a result of its unsolved 'existence', for your own sake, please receive a few more mails of mine, also that my labor situation still deserves to be controlled by myself, so that I can of course tell & e.g. help us both etc. find out & so on, greetings, arentved@in.com, there to be continued.

That's all very 'Heisenberg-y' indeed!