MSE News: NatWest customers in lurch after meltdown

tlh858

dazza12 wrote: »

In this case, an update was ran on a live system. It had been tested, but should have been temporarily decommissioned (services stopped etc) prior to this. This wasn't picked up on immediately, and transactions continued to process through an effectively broken system. This then corrupted live data. It took until earlier today to fully roll back the changes and recover the data that had been damaged, and then 'play' the transactions from the point of failure.

Taking several days to roll back changes to mission critical computer systems is quite frankly pathetic.

Whether the problem was caused by human error, hardware failure or whatever else, disaster recovery procedures should be designed to get things working again within hours - not days.

ahxcjb

Hi Dazza,

Apologies if you took my post in wrong faith. It wasn't intended.

I interpreted your post as an admin making an unauthorised change off of their own back, which as we're both aware is very unlikely and actually technically tough to do (the banks / stock exchange I worked for wouldn't allow root rights without a full approval process - and we were the guys running the servers).

Completely agree with you re runbooks sometimes being mis-read, or a script being typo'ed which introduces an unexpected risk.

Essentially it sounds as if an application writing to live data had not been stopped, and a batch process was run which then corrupted that data if i understand you correctly? Shame they didn't write a script that actually tested that the required apps were stopped, but hindsight is a great thing.

Do you know why they couldn't simply switch over to their DR instead of attempting to fix the live system?

emweaver

x_Bunny_x wrote: »

Don't you have any friends to help?. I'm sure you have a helper with the baby if you are housebound, to take baby for a ride out?

i was going to say call your local small shop up, ours is a co-op, explain to them and see if they can do anything

They may deliver some stuff, or just tell tesco, im sure they know whats happening, and im sure they will deliver if you tell them whats happening your side

Please tell me where you live as I really don't believe any shop would provide food on credit and deliver it .

emweaver

Cate1976 wrote: »

I really feel for people like ffghugsf who are struggling due to Nat west tech meltdown.

If I had a neighbour in similar situation to ffghugsf, I'd take some food to them to keep them going over the weekend.

Think the biggest problem is going to be for people who don't bank with Natwest but their employers do.

People on means tested benefits might have been able to get crisis loans from the benefits agency if they couldn't get money from the Emergency cash scheme Natwest has.

I bank with HSBC and when their computers went down last year, it was only down for a day so didn't cause us any bother. If it had gone on longer it would have done.

I have bank accounts with 3 banks, each account is used for different purpose. Easier to manage finances and save for things that way. Also has the advantage that if one ban has tech issues, we won't be without money or shouldn't be.

To all who've been affected by Natwest glitch, i hope you get everything sorted soon.

This is our situation we do not bank with Natwest but my husbands employer does so we have no money and can't even get any emergency cash from our bank like Natwest customers are being able to.

We only returned from holiday today all spent out so no cash at all left and no food here as w e can not afford to fill up a stock cupboard like others suggest we can only afford to buy what we need.

zerog

emweaver wrote: »

This is our situation we do not bank with Natwest but my husbands employer does so we have no money and can't even get any emergency cash from our bank like Natwest customers are being able to.

We only returned from holiday today all spent out so no cash at all left and no food here as w e can not afford to fill up a stock cupboard like others suggest we can only afford to buy what we need.

You can't afford to buy a bit of extra food but you can afford a holiday where you spend until your bank account is dry?

funkyfreddy9k

tlh858 wrote: »

Taking several days to roll back changes to mission critical computer systems is quite frankly pathetic.

Whether the problem was caused by human error, hardware failure or whatever else, disaster recovery procedures should be designed to get things working again within hours - not days.

In my experience, disaster recovery procedures very rarely actually cover every possible scenario. So whilst the DR procedures may allow for recovery in a couple of hours in expected scenarios, it will always be possible that the unexpected will take longer.

An example of unexpected scenarios would be failures of multiple systems.

funkyfreddy9k

ahxcjb wrote: »

Do you know why they couldn't simply switch over to their DR instead of attempting to fix the live system?

I'd guess because the corrupt/bad transactions had already mirrored to the DR systems as well.

OR

Whilst their DR systems were clean, transactions from the live system had already processed through onto other related systems, so switching to DR would mean that DR was then out of line with those other systems so they thought it best to try to fix live.

emweaver

zerog wrote: »

You can't afford to buy a bit of extra food but you can afford a holiday where you spend until your bank account is dry?

I do not have to justify myself to you but who says we even paid for the holiday!

We saved very hard for the spending money and sold loads on ebay and we did not like everyone else expect this to happen and have all our wages gone!

Always get someone who thinks they know it all and wants to criticise and attack how others choose to live!

Agricolae

dazza12 wrote: »

I've got a couple of friends who work for Natwest IT. Feeling sorry for them right now. However, they've told me what happened. Some idiot applied an untested update to their clearing system that corrupted a load of data. It has taken until today to unravel this, and put in place a new system. The delay is now down to approximately 200 million transactions queued in the system. Normally this would run overnight when bank systems are quiet - however it's had to run whilst all hell has broken loose.

This is all entirely plausible. There are rigorous change management procedures in effect at RBSG as there are at all banking groups, however it doesn't mean that human error can be entirely designed out.

I would hate to be the person responsible right now.

Having to run transaction batches during business hours is certainly a massive problem and this is probably responsible for various systems going offline during the day. Work was once again a nightmare because we were not certain when systems would be available. At our nadir today we had absolutely nothing and we could not even serve people over the phone.

However I am not entirely certain what you mean by "clearing system". There are several systems which could be described as such and not all of them going wrong would have caused an incident of this magnitude.

MoneySaverLog

Agricolae wrote: »

There are rigorous change management procedures in effect at RBSG

dazza12 wrote: »

Natwest extensively test any change that needs to be made, on test environments with copies of live data. Various scenarios are tested and only when it's been signed off will any change be made to a live system.

Something went wrong with these rigorous change control procedures then for someone to put untested code into a live production environment. If that is what happened.

tlh858 wrote: »

Taking several days to roll back changes to mission critical computer systems is quite frankly pathetic.

disaster recovery procedures should be designed to get things working again within hours - not days.

They'd have to roll back all the changes, and then put it how it was at some predetermined point in the past where it was OK.

The transactions since that point would have to be reconciled in some way. Depending on the size of the system, and data affected would determine what disaster recovery plan they use, and what tests are then done to ensure the system is again fully operational in accordance with their "rigorous change management procedures" that are in place.