August 28, 2005
Application mirroring - In which I strike another blow against the System Programmer's Guild
Once upon a time I was a systems programmer. I was pretty good too; not brilliant but broad and sensible enough to know my limitations, perhaps the supreme asset of all. I attribute this good fortune to having attended perhaps the best place outside Bell Labs for early Unix work - UNSW - and my own innate skepticism which saved me from the self-glorification of the "guru culture" that pervaded those places and days.
Being a systems programmer is much more technically challenging than their alter ego, the applications programmer. The technology, the concepts, the things you can do are simply at a higher intellectual plane. Yet there is a fundamental limit - the human being. Wherever and however you do systems programming, you always strike up against that immovable yet indefinable barrier of the end user. At the end of the day, your grand complex and beautiful design falls flat on its face, because the user pressed the wrong button.
A good systems programmer then reaches a point where he can master his entire world, yet never advance intellectually. So one day I entered the world of applications programming and looked back only rarely. Still, I carried the ethos and the skepticism with me...
This meandering preamble is by way of reaching here: Recently I hit a tipping point and decided to throw a Core Systems Programming Construct into the trash. RAID, mirrored drives and all that have been a good old standby of well managed systems for a decade or more. In one act, Confined and Plonk. To hell with it, and all this sodding special drivers and cards and SANs and NASes and what-not - out the window. For the last three or four years we'd been battling with mirroring technologies of one form or another, and here's the verdict: they all suck. In one way or another, and I was very glad to spy research (since lost) that claimed that something like one in eight of all mirrors system end in tears when the operator inserts the wrong drive and wipes his last backup... It restores faith to know that we weren't the only ones.
So...
I dived into our secure backend code and discovered that coding up mirroring at the application-level took all of a day. That's all! In part, that's because the data requirements are already strictly refined into one log feed and one checkpoint feed, and they both go through the same hashed interface. Also, another part is that I'd been mulling it over for months; my first attempt had ended in retreat and a renewed attack on the mirroring drivers... In those ensuing months the grey matter had been working the options as nice -n 20 task.
But, still! A day! It is a long time since I'd coded up a feature in only a day - I was somewhat shocked to have got self-testing done by the time it was dark.
OK, I thought, surely more problems will turn up in production? Well, this morning I moved a server from an unprotected machine to a new place. I turned on mirroring and then deliberately and painfully walked through starting it up without the application-defined mirror. Bang, bang, bang, it broke at each proper place, and forced me to repair. Within 30 mins or so, the migration was done, and turning on mirroring was only about 10 mins of that!
So where is all this leading? Not that I'm clever or anything; but the continuing observation that many of the systems, frameworks, methodologies, and what have you are just ... Junk. We would be better off without them, but the combined weight of marketing and journalistic wisdom scare us from bucking the trend.
Surely so many people can't be wrong? we say to ourselves, but the history seems to provide plenty of evidence that they were indeed just that: Wrong. Here's a scratch list:
SSO, PKI, RAID and mirrored drives (!), Corba, SQL, SOAP, SSL, J2EE, firewalls, .NET, ISO, Struts, provable security, AI, 4GLs, IPSec, IDS, CISPs, rich clients,
Well, ok, so I inserted a few anti-duds in there. Either way, we'd probably all agree that the landscape of technology hasn't changed that much in the last N years, but what has changed is the schlockware. Every year portends a new series of must-have technologies, and it seems that the supreme skill of all is knowing which to adopt, and which to smile sweetly at and wait until they fail under their own mass of incongrueties.
Not to hammer the nail in too far...
Posted by iang at August 28, 2005 06:17 AM | TrackBack