Natwest crash - what happened

LincolnshireYokel

zaax wrote: »

Why, precisely, does one mess-up by one employee in front of one computer put your ENTIRE BANKING SYSTEM out of action, nationwide?

In the same way that ONE sailor on a neuclear sub could walk into the reactor room, hit the emergency SCRAM and put the sub out of action for several hours.

Some parts of a system are mission critical, and sometimes its not possible to make them idiot proof. That why you have trained IT staff to run them, but even that can be circumvented by employing monkeys.

In fact i can illustrate that with a true story.

Many years ago I worked for a while in a laboratory, in a foundry. metal samples from the cupolas used to come to the lab for analysis, and identical samples went to another room across the corridor to a device known as the Quantovac.

We analysed the samples by traditional chemical means, the Quantovac did it by making an electric spark between the sample and a newly sharpened solid silver electrode , and then analysing the light with a spectrometer.

One day, it all went wrong. the results in the lab started to vary wildly with the Quantovac. We checked all the reagents fopr contminants, tore the lab apart looking for something wrong.,, went through all the procedures to make sure everyone was doing the same. Nothing.

So they turned to the Quantovac. They tore it apart, tested every part, replaced bits, recalibrate it umpteen times, got the makers to send in two guys who spent a week scratching there heads. All this time production was at a standstill, because we couldn't g'tee the composition of the metal that they were casting with.

This went on for two weeks. Then someone noticed the young lad they had employed a while ago in the quantovac, the trainee, was putting his finger on the tip of the silver electrode after sharpening it in the lathe, before inserting it into the Quantovac chambers, and thus contaminating it with all sorts of dirt, grime, proteins, salts, skin cells, and other organic matter.

Thus one idiot brought the entire foundry to a standstill, and cost it hundreds of thousands of pounds worth of production.

No system is idiot proof, the best you can manage is fool proof, because theres no limit to the ingenuity of an idiot.

googler

paddyrg wrote: »

I refer you to my post here http://forums.moneysavingexpert.com/showpost.php?p=54019271&postcount=44

This is how *every* large and complex system grows. Not at all suspicious, it sounds absolutely spot-on to me.

Also perhaps of relevance to the RBS situation is that, as these systems grow, all sorts of factors come into play which drag focus away from the system.

New functionality is developed as an add-on, a patch, because management don't want to shoulder the cost of redesigning the system from the ground up.

Managers decide on a certain course of action, not because it's of overall benefit to the system or the company, but because it keeps them in a job, those under them in jobs, keeps a certain team or group together, etc etc.

InsideInsurance

It is not simply management wanting to shoulder the cost, it would be an extraordinarily difficult project to do and would take many years to complete and what you do in the interim period becomes very difficult as most companies dont want to stand still.

I did once look at a complete system replacement for an insurer, whos systems are much simpler and under less stress than a banks, and to simply replace the claims function was a 4 year project, to replace the pricing engine was a 3 year project, to replace the document production system was a 2.5 year project (though that was hampered by cost benefit analysis issues). They didnt even bother getting quotes for replacing the core underwriting system or CRM.

Even offering IBM a blank cheque to do the claims replacement project within 12 months got a resounding no, there was no number of zeros that could be added that would make the project possible with an acceptable level of risk.

agrinnall

Having worked for many years (in a previous life) on systems feeding into and receiving feeds from the RBS core banking systems, I echo what II says about the difficulty of replacing those systems. While it's not impossible, it would be astonishingly difficult and risky. Because the systems have evolved over 40+ years they are full of things that nobody now understands, either from an IT or business perspective. Many sections of code will have been redundant for years, but nobody would risk removing them just in case they are doing something that would have unforeseen consequences. The reverse engineering required to completely figure out everything that the systems do would be a mammoth task, and by the time it was finished it would be out of date because of the mandatory changes made in the meantime. There's no obvious way to get out of this situation, and I would expect most, if not all, banks that have been around for 20 or more years to face similar issues with their systems.