Santander upgrade gone wrong?

colsten

Mickey666 said:

colsten said:

Mickey666 said:

colsten said:

mr_accountant said:

Right shall I be the first, I don't need to use santander today nor have I been inconvenienced but which social media platform can I put my boring story so that the world knows and so help me claim my compensation because its my birth right? 

You should not assume that your requirements are the same for the other 14 million Santander customers. There have been plenty of completely credible reports from people stranded at petrol stations, unable to buy food or pay for their train/underground ticket or their taxi etc etc. Those people have a right to compensation, and they will no doubt keep the Santander complaints team busy for some time.

Instead of bragging with an I‘m alright Jack attitude, it would be constructive to remind people of the importance of always having alternative payment means. This can be anything from carrying cash to having one or more other current accounts with a positive balance  to one or more credit cards. Goes (almost) without saying that one should have at least one each of a VISA and a Mastercard, in case one of them has a problem (as has happened in the past).

As aside, having my main current account with Santander: I won‘t be making any compensation claim myself as I was not materially inconvenienced by this outage. I will also not close my Santander account. 

Looks like you've answered your own rant there

Yes, it SHOULD go without saying that people should have more than one means of paying for stuff, but I guess some people just like risking it.  Their choice, their responsibility.

I haven't ranted. I merely took exception with a post that tried to ridicule those who were unfortunate enough to be left without a means to pay for essentials.

Mickey666 said:

IvanOpinion said:

tempus_fugit said:

This sort of thing happening so often suggests that they didn't have a proper backout plan or it hadn't been properly tested. Unfortunately these things happen, but they should always be able to put everything back if necessary, it shouldn't all be down for this long.

That depends on the nature of what has gone wrong. Several years back we hit a problem which meant it took us almost 80 hours to get a stable system back online.  The press and social media was rife with BS about software updates, incompetence etc. etc. and 101 "experts" providing simple solutions and pointing the finger of blame - but not one was even close to guessing what had actually happened.

The problem was a set of unpredictable conditions.  For commercial reasons I can't go into great detail but  ... despite all the sensors, our primary site had suffered undetected damage that brought the servers down over a period of about 1-2 weeks (as best as we can guess).  This meant they were still syncing with our DR site (several hundred miles away) using an industry standard best practice process that had been tested many times in the past, but (unknown to us) the data contained corruptions.  In fact we had only audited and tested our fail over process 6 weeks previously and it worked perfectly.  When the primary site went down it went down big time in a way that did not trigger DR to kick in.  We spent the first few hours not realising the problem and trying to get the primary site up and running, eventually we sent someone to the primary site itself (120mile round trip) and realised the scale of the problem (there was no way it would be recoverable or brought back up for many weeks). 

We then manually fired up the DR site and it was online within the hour - but reports started flooding in of data problems so we had to take it back down.  Realising the problem we restored from a backup (a process that takes several hours - and took us 3 goes before we found a stable data set) but the regulator would not allow us to bring the site back up until we had done a load of additional actions and checks.  The board did what boards do best, ranted, raved and demanded hourly updates (which meant we were basically a man down just to provide them with their updates).  Our entire team did not leave the office for those 80 hours with us often seen sleeping in our chairs or under desks as we waited for various processes to complete.  On the plus side it made us all valuable resources in the market and targets for head hunters.

Well said.  The 'expert's who pop out from under their stones when things like this happen have no clue about the complexity of such systems, the planning that goes into upgrades, or the fall-back procedures required.  Things go wrong and, as far as I can see, the problem was dealt with one way or another.   A few hours outage is certainly inconvenient but it's hardly the end of the world.

If it wasn't for 24/7 rolling news or the Twitterati, 99% of the population wouldn't even have known about this 'problem'.
Storm in a teacup.

It's not about 99% or 100% of the population (most of whom probably still don't know, nor have a need to know) but about those Santander customers who were adversely affected by the outage. Whilst nobody would argue that it takes some time and effort to fix the systems one way or another, your dismissive remarks simply show that you are another I'm-alright-Jack.

Sorry to disappoint your uninformed assumption but I was actually affected as I had a rather large invoice to pay by bank transfer and wasn't able to.  I suppose I could have paid it from a different bank account but since I already had the payment details set up in my Santander account I took the easier option of simply waiting until today instead.

I guess that makes me an 'alright-Jack' in your eyes when the reality is that I could see the situation for what it was - a temporary glitch - and didn't feel the need to hit the panic button or leap aboard the compensation bandwagon.  Takes all sorts I suppose

As for being 'adversely affected' by a few hours of bank outage, I suppose it depends on your definition of 'adverse' but my suspicion is that we're talking about first-world problems here.  Like I said, storm in a teacup. 

Thanks for confirming that you are indeed an I‘m-Alright-Jack. As you say, you did not have an urgent need on Saturday, such as paying for food, or for petrol to get to work or to pick up your kids, It‘s a mystery to me (also a Santander customer who didn‘t need my money on Saturday) why you appear to be unable to see the severe issues the outage has caused for some. 

username

kuratowski said:

jimjames said:
Is it just people don't think they can be trusted with a credit card? 

If you only have one card/account, is there a reason why you don't have another?

Anecdotally, I think there is some truth in that.  But wary of drawing conclusions on the wider population.

I tried to find some research findings on this but what I found was fairly old: https://www.fca.org.uk/firms/consumer-credit-research-cards/consumer-use
As of 2013, 90% of households with income more than £50k have a credit card, but only 33% of households with income less than £10k.  It's presumably the case that these lower income households are simply unable to access credit cards.  And as a result the impact of a banking failure on these, already vulnerable, households could - as Colsten suggests upthread - be really severe.

Most banks can open basic bank account with debit card - so low income should not really be a barrier to having a backup.

Perhaps I consider myself to be somewhat lucky to have a diversified setup (and I certainly don't earn £50k) -

Bank A, Visa Debit (everyday spending), Bank B, Visa Credit, Bank C Mastercard Credit, Bank D Mastercard Debit.

All banks not part of the same group and completely separate, the only thing tying them together is the card schemes, but then again you don't have much choice (apart from Amex).

In order of where an issue might occur to prevent a transaction, from likely to least likely, it will be the retailer (faulty comms/hardware), followed by the issuing bank and then the card scheme.

Whilst I agree that banks do have responsibility to ensure their computers are working, it is also prudent on your part to ensure that if you carry little cash that you ensure you are covered as much as can be for payment methods.

Mickey666

colsten said:

Mickey666 said:

colsten said:

Mickey666 said:

colsten said:

mr_accountant said:

Right shall I be the first, I don't need to use santander today nor have I been inconvenienced but which social media platform can I put my boring story so that the world knows and so help me claim my compensation because its my birth right? 

You should not assume that your requirements are the same for the other 14 million Santander customers. There have been plenty of completely credible reports from people stranded at petrol stations, unable to buy food or pay for their train/underground ticket or their taxi etc etc. Those people have a right to compensation, and they will no doubt keep the Santander complaints team busy for some time.

Instead of bragging with an I‘m alright Jack attitude, it would be constructive to remind people of the importance of always having alternative payment means. This can be anything from carrying cash to having one or more other current accounts with a positive balance  to one or more credit cards. Goes (almost) without saying that one should have at least one each of a VISA and a Mastercard, in case one of them has a problem (as has happened in the past).

As aside, having my main current account with Santander: I won‘t be making any compensation claim myself as I was not materially inconvenienced by this outage. I will also not close my Santander account. 

Looks like you've answered your own rant there

Yes, it SHOULD go without saying that people should have more than one means of paying for stuff, but I guess some people just like risking it.  Their choice, their responsibility.

I haven't ranted. I merely took exception with a post that tried to ridicule those who were unfortunate enough to be left without a means to pay for essentials.

Mickey666 said:

IvanOpinion said:

tempus_fugit said:

This sort of thing happening so often suggests that they didn't have a proper backout plan or it hadn't been properly tested. Unfortunately these things happen, but they should always be able to put everything back if necessary, it shouldn't all be down for this long.

That depends on the nature of what has gone wrong. Several years back we hit a problem which meant it took us almost 80 hours to get a stable system back online.  The press and social media was rife with BS about software updates, incompetence etc. etc. and 101 "experts" providing simple solutions and pointing the finger of blame - but not one was even close to guessing what had actually happened.

The problem was a set of unpredictable conditions.  For commercial reasons I can't go into great detail but  ... despite all the sensors, our primary site had suffered undetected damage that brought the servers down over a period of about 1-2 weeks (as best as we can guess).  This meant they were still syncing with our DR site (several hundred miles away) using an industry standard best practice process that had been tested many times in the past, but (unknown to us) the data contained corruptions.  In fact we had only audited and tested our fail over process 6 weeks previously and it worked perfectly.  When the primary site went down it went down big time in a way that did not trigger DR to kick in.  We spent the first few hours not realising the problem and trying to get the primary site up and running, eventually we sent someone to the primary site itself (120mile round trip) and realised the scale of the problem (there was no way it would be recoverable or brought back up for many weeks). 

We then manually fired up the DR site and it was online within the hour - but reports started flooding in of data problems so we had to take it back down.  Realising the problem we restored from a backup (a process that takes several hours - and took us 3 goes before we found a stable data set) but the regulator would not allow us to bring the site back up until we had done a load of additional actions and checks.  The board did what boards do best, ranted, raved and demanded hourly updates (which meant we were basically a man down just to provide them with their updates).  Our entire team did not leave the office for those 80 hours with us often seen sleeping in our chairs or under desks as we waited for various processes to complete.  On the plus side it made us all valuable resources in the market and targets for head hunters.

Well said.  The 'expert's who pop out from under their stones when things like this happen have no clue about the complexity of such systems, the planning that goes into upgrades, or the fall-back procedures required.  Things go wrong and, as far as I can see, the problem was dealt with one way or another.   A few hours outage is certainly inconvenient but it's hardly the end of the world.

If it wasn't for 24/7 rolling news or the Twitterati, 99% of the population wouldn't even have known about this 'problem'.
Storm in a teacup.

It's not about 99% or 100% of the population (most of whom probably still don't know, nor have a need to know) but about those Santander customers who were adversely affected by the outage. Whilst nobody would argue that it takes some time and effort to fix the systems one way or another, your dismissive remarks simply show that you are another I'm-alright-Jack.

Sorry to disappoint your uninformed assumption but I was actually affected as I had a rather large invoice to pay by bank transfer and wasn't able to.  I suppose I could have paid it from a different bank account but since I already had the payment details set up in my Santander account I took the easier option of simply waiting until today instead.

I guess that makes me an 'alright-Jack' in your eyes when the reality is that I could see the situation for what it was - a temporary glitch - and didn't feel the need to hit the panic button or leap aboard the compensation bandwagon.  Takes all sorts I suppose

As for being 'adversely affected' by a few hours of bank outage, I suppose it depends on your definition of 'adverse' but my suspicion is that we're talking about first-world problems here.  Like I said, storm in a teacup. 

Thanks for confirming that you are indeed an I‘m-Alright-Jack. As you say, you did not have an urgent need on Saturday, such as paying for food, or for petrol to get to work or to pick up your kids, It‘s a mystery to me (also a Santander customer who didn‘t need my money on Saturday) why you appear to be unable to see the severe issues the outage has caused for some. 

But that's exactly my point - "for some".

I've not denied the outage didn't cause issues for SOME people, but it didn't affect the vast majority of people in the country at all and of the tiny minority it did affect (like you and me) the inconvenience was trivial, so why all the media hype?

Storm in a teacup.

colsten

Decent reaction from Santander

masonic

colsten said:

Decent reaction from Santander

The apology is decent. The commitment to make sure no customer is out of pocket is a regulatory obligation.