MSE News: NatWest customers in lurch after meltdown

innovate

I don't subscribe to that "software upgrade gone wrong" theory - - they would be most unlikely to implement a software upgrade during a working day. Must be something else that's gone catastrophically wrong.

JuicyJesus

bobblebob wrote: »

I do agree with what you're saying, but there isnt a business out there that doesnt have issues from time to time, especially when technology is used in the way banks use it. Yea its not your fault if the system goes down, but unfortunately, its you that has to pick up the pieces

There are issues in every business, yes, but this is the fourth day here of some very, very major issues on what (given the RBS Group's size) has turned out to be extremely critical infrastructure, which it's speculated only failed due to RBS Group's negligence. As such, they should be doing a lot more than what they're currently doing, and (in my sole view) looking to do more about consequential losses than is currently being mooted.

MoneySaverLog

innovate wrote: »

I don't subscribe to that "software upgrade gone wrong" theory - - they would be most unlikely to implement a software upgrade during a working day. Must be something else that's gone catastrophically wrong.

According to this post http://forums.moneysavingexpert.com/showpost.php?p=53977475&postcount=44 it would appear that someone in NatWest IT who is in the know has put the cause down to this.

pmduk

JuicyJesus wrote: »

RBS' reputation is going to take a BIG hit on this. The fact that they decided to upgrade a critical piece of payment infrastructure on a major processing day and then seemingly did not have a valid regression plan in place reflects extremely poorly on them (although I am not an expert, and no doubt I am being somewhat short-sighted here.)

With a banking institution the size of RBS every day is a major processing day. We had someone on this very forum a few weeks back complaining that system upgrades at 2am on a Sunday was unreasonable. There is no good time. Let's hope the one outcome of this is to thoroughly test all upgrades first. Somehow I suspect they'll have learned their lesson.

JuicyJesus

Tommaso46 wrote: »

I hope that idiot has now been sacked.

I would definitely not like to be him right now.

innovate

MoneySaverLog wrote: »

According to this post http://forums.moneysavingexpert.com/showpost.php?p=53977475&postcount=44 it would appear that someone in NatWest IT who is in the know has put the cause down to this.

This is neither official nor confirmed. It may or may not be true.

PasturesNew

I used to project manage software and hardware projects in the banking industry, and have done a few bigguns.

I can spot several points within a project that those without experience might c0ck up .... and more points within a project where a pompous boss has over-ruled the "extra time required for testing" request without actually looking at the project plan and the reasons that was asked for.

Maybe the update was applied to the test servers etc... but the dataset used in those test servers didn't take into account some occasional inconsistencies in accounts fields, brought about by a succession of mergers etc in recent years.

A good project needs a PM with the b4lls to say "No ..... " to top management when it's cirtical and to help prevent short cuts. Beyond that, even if you have that then the PM can be immediately dispensed with and the management steam on ahead anyway with a new sap to take the rap. After all, management are in control and make the Go/Stop decisions.... all other monkeys can merely suggest and advise and point out.

Maintaining a pessimistic Risk Log is key... when it comes to risk you can't have a sunny and optimistic outlook or there's no point.

MoneySaverLog

Trying to get something like this officially confirmed by NatWest will be like trying to get blood out of a stone I'd guess.

Still all we have to go on at the moment is this and a few NW staff on here pointing in this direction too in other threads of the same topic.

funkyfreddy9k

Lets remember that RBS got rid of over 1000 of their IT team not so long ago.

This probably means at least some of it was outsourced - which usually means you're depending on people who have no experience and limited training along with not really having their finger on the pulse to what's really going on with the systems.

So before we blame actual RBS IT guys for this, lets remember its entirely possible it was because of dodgy testing done by outsourced app testing teams, or maybe outsourced support people that actually promoted the change.

Without being close to the problem we dont really know.

kittiej

Yes Dazza sort of said this in post 49