Santander upgrade gone wrong?

tempus_fugit

When I worked in IT, we often did upgrades to software systems and they would fail for some reason on an alarmingly frequent basis. Often the issues could be fixed within a few hours, but if they couldn't then we had to have a tried and tested backout plan that could be put into place at short notice, i.e. set everything back to the way it was and then work on the issues before trying again at a future date. This sort of thing happening so often suggests that they didn't have a proper backout plan or it hadn't been properly tested. Unfortunately these things happen, but they should always be able to put everything back if necessary, it shouldn't all be down for this long.

tempus_fugit

eskbanker said:

molerat said:

Opening some branches tomorrow.  Is this going to be of TSB proportions ?

Use of "12pm" will get the pedants going, are they planning to stay open until midnight?

That's 12am.

Thrugelmir

Not_Scammed said:

My parents suffered at the hands of this incompetent buch of cowboys a few weeks ago. Needless to say the are moving to First Direct. How anyone can continue to have any trust in this bank is beyond me. I feel sorry for the staff, they are being badly let down by the clowns at the top. I fully expect there to be a major news scandal about how this bank is bein run/misrun.

Ever a run a bank or major organisation or project yourself? 

eskbanker

tempus_fugit said:

eskbanker said:

molerat said:

Opening some branches tomorrow.  Is this going to be of TSB proportions ?

Use of "12pm" will get the pedants going, are they planning to stay open until midnight?

That's 12am.

It's never going to be a particularly productive discussion, but with your Latin username you'll doubtless be aware that 'pm' is post meridiem, i.e. after noon - obviously 'am' is correspondingly before noon, so using either am or pm for midday itself (or midnight) is best avoided....

colsten

BBC, The Guardian, Sky News, Bloomberg, Radio 4 Moneybox. 

PR nightmare for Santander. Heads will roll, I am sure.

https://twitter.com/paullewismoney/status/1393542014206099457?s=20

Peter999_2

tempus_fugit said:

When I worked in IT, we often did upgrades to software systems and they would fail for some reason on an alarmingly frequent basis. Often the issues could be fixed within a few hours, but if they couldn't then we had to have a tried and tested backout plan that could be put into place at short notice, i.e. set everything back to the way it was and then work on the issues before trying again at a future date. This sort of thing happening so often suggests that they didn't have a proper backout plan or it hadn't been properly tested. Unfortunately these things happen, but they should always be able to put everything back if necessary, it shouldn't all be down for this long.

It's always a good idea to have four eyes checking on every upgrade and have a proper backout plan.   I've worked in IT finance server teams for a few years and the SOE (plan) is usually pages long and we make it very simple to use for people doing the upgrade and full instructions on how to back it out.      However, we did once have a case where we were upgrading the payment systems and an off-shored chap was to turn off the relevant 16 servers.   Unfortunately he turned off the wrong 16 servers (even though the SOE actually detailed the server names).     The whole system went down for about 30 minutes - hell of a !!!!!! storm that day.

Thrugelmir

colsten said:

BBC, The Guardian, Sky News, Bloomberg, Radio 4 Moneybox. 

PR nightmare for Santander. Heads will roll, I am sure.

https://twitter.com/paullewismoney/status/1393542014206099457?s=20

Trial by social media yet again.  :'(

molerat

App & web banking back up

colsten

molerat said:

App & web banking back up

Faster payments in and out working fine, too

EssexExile

Just think, in the good old days before instant everything no one would have noticed.