How robust ARE banks' core IT systems?

bob792

This one is for techies with some inside knowledge.

If the RBS meltdown was caused was caused by a software upgrade that went wrong, there are obvious questions about how rigorously it was tested before being applied to a live system and also whether outsourcing thousands of IT jobs to contractors in India is part of the problem.

So, is this a disaster waiting to happen at all the clearing banks or have any of them resisted the short term cost pressure and kept an experienced team of their own staff looking after their core IT infrastructure?

Are all the clearing banks working on archaic core IT that is creaking under the strain of too many boltons?

I heard somewhere that Nationwide were nearing the end of a major programme to renew their core IT. Is this true and what about the rest of the big banks?

Gromitt

I don't have any inside knowledge, but I do know from experience that it doesn't matter how much you test upgrades in a testing environment, there is always the problem that it may perform completely different on the production systems. So all you can do is prepare for "What happens if" scenarios. Then you have the choice: You know it didn't work, do you spend time rolling back to what did work, knowing you'll have to do it again at a future date, or do you push forward, hoping to get the system working as quickly as possible?

pinkdalek

It is not about how robust, more like how competent the people involved in the testing and installation of it are.

You would have thought that the system they installed would have been tried and tested for months earlier, in a "dummy" role or even piloted in certain branches or an area.

Outsourcing something to another country is a rather lame excuse, quite offensive in fact, to assume that British would be better and to lie the blame of "johnny foreigner". End of the day there are lame workers and shoddy products anywhere in the world.

agrinnall

Core systems are robust, I've posted elsewhere that in the 15 years I worked in IT at RBSG we never had an issue as bad as this, but of course like all systems they are vulnerable.

I'm out of the loop now, having left 4 years ago, but I would be pretty confident that transferring jobs to India has been a contributory factor in this. Many of the people who had worked for 10, 20, even 30 years on these systems (parts of which date back to the early 1970s) have been made redundant, and however good they are people in India who have only got 2 or 3 years (at best) experience on the systems simply will not have the same level of knowledge.

I also suspect that pressure to reduce costs by cutting testing has also played a part. I know at least one test manager has left recently because of frustration at the constraints placed on what testing could be performed.

I can't say whether other banks are in the same boat but it would seem reasonable to assume that they are all trying to cut their costs, although maybe without quite the pressure that RBSG have been under due to the level of public ownership.

Gentoo365

The probability of an issue like this occurring is probably once every 10 years.

There is no way to stop it happening. There are however ways to plan in advance for what to do when it does happen.

huckster

Many banks systems are held together with sticky tape. I have worked with IT contractors who have been involved with many of the leading banks. They are all vulnerable to having the same problems as RBS. As someone else commented, the banks made a mistake in making redundant staff that have serviced these IT systems for years and understand what they need to do, to stop problems happening.

gadgeteer_2

A bank's IT system should be about as robust as they come. Their will be multiple redundant systems to safeguard against disasters and hardware failures.

I'm sure there will be rigorous change management processes in place which will have had to get approved before anyone is let loose in making changes to a live system

The problem here is that something went wrong during an upgrade. I'm speculating but it could have been as simple as someone typing in the wrong thing. Or some unforseen circumstance which never occurred in the development/testing environment.

Half the time when systems go wrong in my company it is as a result of someone misconfiguring something. And it's the hardest thing in the world to explain to managers whom have no technical skills why all the redundant systems are useless against human error.

bobblebob

I work for the NHS, if you want to know about useless IT systems that dont work, look no further

rb10

bob792 wrote: »

I heard somewhere that Nationwide were nearing the end of a major programme to renew their core IT. Is this true and what about the rest of the big banks?

I believe that Nationwide's programme will only (initially) apply to the system that their current accounts are held on, but is to be extended to savings and mortgages in due course.

I don't know of any other banks that are currently doing the same.

rjohn

pinkdalek wrote: »

Outsourcing something to another country is a rather lame excuse, quite offensive in fact, to assume that British would be better and to lie the blame of "johnny foreigner". End of the day there are lame workers and shoddy products anywhere in the world.

Sorry, but as a software developer who's worked on quite a few projects with outsourcers, it's completely true that "British is better".

It has nothing to do with nationalities of course — Indian developers are undoubtedly capable of the same quality of work as British developers, but only in the same environment.

Outsourcers don't work for the client directly (in this case the bank), they work for an outsourcing company and might work on projects for many clients. This means they don't have any loyalty to the bank, or depth of knowledge on a project.

And when you think about it, of course a developer in this country on £70k in a comfy office is going to produce better work than one in India on a fraction of that working in a call centre-like environment.

GustyGardenGalaxy

The problem in this case seems to stem from NatWest laying off staff, farming out the work to other countries, offsourcing, etc.

The comments here:

http://forums.theregister.co.uk/forum/1/2012/06/22/rbs_natwest_outage_fourth_day/

are a good read as they are largely from technical people.