Santander upgrade gone wrong?

Mickey666

IvanOpinion said:

tempus_fugit said:

This sort of thing happening so often suggests that they didn't have a proper backout plan or it hadn't been properly tested. Unfortunately these things happen, but they should always be able to put everything back if necessary, it shouldn't all be down for this long.

That depends on the nature of what has gone wrong. Several years back we hit a problem which meant it took us almost 80 hours to get a stable system back online.  The press and social media was rife with BS about software updates, incompetence etc. etc. and 101 "experts" providing simple solutions and pointing the finger of blame - but not one was even close to guessing what had actually happened.

The problem was a set of unpredictable conditions.  For commercial reasons I can't go into great detail but  ... despite all the sensors, our primary site had suffered undetected damage that brought the servers down over a period of about 1-2 weeks (as best as we can guess).  This meant they were still syncing with our DR site (several hundred miles away) using an industry standard best practice process that had been tested many times in the past, but (unknown to us) the data contained corruptions.  In fact we had only audited and tested our fail over process 6 weeks previously and it worked perfectly.  When the primary site went down it went down big time in a way that did not trigger DR to kick in.  We spent the first few hours not realising the problem and trying to get the primary site up and running, eventually we sent someone to the primary site itself (120mile round trip) and realised the scale of the problem (there was no way it would be recoverable or brought back up for many weeks). 

We then manually fired up the DR site and it was online within the hour - but reports started flooding in of data problems so we had to take it back down.  Realising the problem we restored from a backup (a process that takes several hours - and took us 3 goes before we found a stable data set) but the regulator would not allow us to bring the site back up until we had done a load of additional actions and checks.  The board did what boards do best, ranted, raved and demanded hourly updates (which meant we were basically a man down just to provide them with their updates).  Our entire team did not leave the office for those 80 hours with us often seen sleeping in our chairs or under desks as we waited for various processes to complete.  On the plus side it made us all valuable resources in the market and targets for head hunters.

Well said.  The 'expert's who pop out from under their stones when things like this happen have no clue about the complexity of such systems, the planning that goes into upgrades, or the fall-back procedures required.  Things go wrong and, as far as I can see, the problem was dealt with one way or another.   A few hours outage is certainly inconvenient but it's hardly the end of the world.

If it wasn't for 24/7 rolling news or the Twitterati, 99% of the population wouldn't even have known about this 'problem'.
Storm in a teacup.

Mickey666

colsten said:

mr_accountant said:

Right shall I be the first, I don't need to use santander today nor have I been inconvenienced but which social media platform can I put my boring story so that the world knows and so help me claim my compensation because its my birth right? 

You should not assume that your requirements are the same for the other 14 million Santander customers. There have been plenty of completely credible reports from people stranded at petrol stations, unable to buy food or pay for their train/underground ticket or their taxi etc etc. Those people have a right to compensation, and they will no doubt keep the Santander complaints team busy for some time.

Instead of bragging with an I‘m alright Jack attitude, it would be constructive to remind people of the importance of always having alternative payment means. This can be anything from carrying cash to having one or more other current accounts with a positive balance  to one or more credit cards. Goes (almost) without saying that one should have at least one each of a VISA and a Mastercard, in case one of them has a problem (as has happened in the past).

As aside, having my main current account with Santander: I won‘t be making any compensation claim myself as I was not materially inconvenienced by this outage. I will also not close my Santander account. 

Looks like you've answered your own rant there

Yes, it SHOULD go without saying that people should have more than one means of paying for stuff, but I guess some people just like risking it.  Their choice, their responsibility.

colsten

Mickey666 said:

colsten said:

mr_accountant said:

Right shall I be the first, I don't need to use santander today nor have I been inconvenienced but which social media platform can I put my boring story so that the world knows and so help me claim my compensation because its my birth right? 

You should not assume that your requirements are the same for the other 14 million Santander customers. There have been plenty of completely credible reports from people stranded at petrol stations, unable to buy food or pay for their train/underground ticket or their taxi etc etc. Those people have a right to compensation, and they will no doubt keep the Santander complaints team busy for some time.

Instead of bragging with an I‘m alright Jack attitude, it would be constructive to remind people of the importance of always having alternative payment means. This can be anything from carrying cash to having one or more other current accounts with a positive balance  to one or more credit cards. Goes (almost) without saying that one should have at least one each of a VISA and a Mastercard, in case one of them has a problem (as has happened in the past).

As aside, having my main current account with Santander: I won‘t be making any compensation claim myself as I was not materially inconvenienced by this outage. I will also not close my Santander account. 

Looks like you've answered your own rant there

Yes, it SHOULD go without saying that people should have more than one means of paying for stuff, but I guess some people just like risking it.  Their choice, their responsibility.

I haven't ranted. I merely took exception with a post that tried to ridicule those who were unfortunate enough to be left without a means to pay for essentials.

Mickey666 said:

IvanOpinion said:

tempus_fugit said:

This sort of thing happening so often suggests that they didn't have a proper backout plan or it hadn't been properly tested. Unfortunately these things happen, but they should always be able to put everything back if necessary, it shouldn't all be down for this long.

That depends on the nature of what has gone wrong. Several years back we hit a problem which meant it took us almost 80 hours to get a stable system back online.  The press and social media was rife with BS about software updates, incompetence etc. etc. and 101 "experts" providing simple solutions and pointing the finger of blame - but not one was even close to guessing what had actually happened.

The problem was a set of unpredictable conditions.  For commercial reasons I can't go into great detail but  ... despite all the sensors, our primary site had suffered undetected damage that brought the servers down over a period of about 1-2 weeks (as best as we can guess).  This meant they were still syncing with our DR site (several hundred miles away) using an industry standard best practice process that had been tested many times in the past, but (unknown to us) the data contained corruptions.  In fact we had only audited and tested our fail over process 6 weeks previously and it worked perfectly.  When the primary site went down it went down big time in a way that did not trigger DR to kick in.  We spent the first few hours not realising the problem and trying to get the primary site up and running, eventually we sent someone to the primary site itself (120mile round trip) and realised the scale of the problem (there was no way it would be recoverable or brought back up for many weeks). 

We then manually fired up the DR site and it was online within the hour - but reports started flooding in of data problems so we had to take it back down.  Realising the problem we restored from a backup (a process that takes several hours - and took us 3 goes before we found a stable data set) but the regulator would not allow us to bring the site back up until we had done a load of additional actions and checks.  The board did what boards do best, ranted, raved and demanded hourly updates (which meant we were basically a man down just to provide them with their updates).  Our entire team did not leave the office for those 80 hours with us often seen sleeping in our chairs or under desks as we waited for various processes to complete.  On the plus side it made us all valuable resources in the market and targets for head hunters.

Well said.  The 'expert's who pop out from under their stones when things like this happen have no clue about the complexity of such systems, the planning that goes into upgrades, or the fall-back procedures required.  Things go wrong and, as far as I can see, the problem was dealt with one way or another.   A few hours outage is certainly inconvenient but it's hardly the end of the world.

If it wasn't for 24/7 rolling news or the Twitterati, 99% of the population wouldn't even have known about this 'problem'.
Storm in a teacup.

It's not about 99% or 100% of the population (most of whom probably still don't know, nor have a need to know) but about those Santander customers who were adversely affected by the outage. Whilst nobody would argue that it takes some time and effort to fix the systems one way or another, your dismissive remarks simply show that you are another I'm-alright-Jack.

briskbeats

RG2015 said:

surreysaver said:

briskbeats said:

colsten said:

briskbeats said:

colsten said:

briskbeats said:

greyteam1959 said:

Same here............
Always like to have an alternative bank account & credit / debit card.

^^ This is why it’s important to have a backup.

Hate when a bank is having problems as customers blame US for it! 

Who is us?

Me and my colleagues 

So are you Santander IT?

No - shop worker. Had customers shouted at us when Lloyds’ systems went down and another bank went down earlier this year. They blamed us even though the problem was with the bank NOT us

As a shop, you should have a system in place to work around the situation. How did shops function with credit cards before everything was online?

Before credit card data was online, there was a machine that ran a roller or press over the embossed card to take an imprint of the details. These are of course long gone. The shop would have a backup system, either cash or cheques. It is not the fault of the shop. The bank is clearly to blame for providing systems that are not robust.

The bank should have a backup and most prudent customers would also have a backup but I cannot see what other backup a shop could have.

We still have these flatbed card machines. Only dug out when needed. One episode happened three years ago - when I was  on holiday, so missed this. As internet went down around the area. So could not pay with card, buy phone vouchers, buy 3rd party gift vouchers. The ATMs in the area affected were down. My teammates said it was a nightmare. Customers were yelling at them asking which is the nearest ATM which was working. How on earth were my teammates supposed to know this when the internet went down 40 mins previously? As had no idea how far the problem was.

It's best not to bank with one bank just in case things like this happened. I am with Santander myself for current accounts. Have a Halifax credit card and a Nationwide bank account with £70 in. That reminds me, must get a new PIN for Nationwide.

briskbeats

IanManc said:

RG2015 said:

surreysaver said:

briskbeats said:

colsten said:

briskbeats said:

colsten said:

briskbeats said:

greyteam1959 said:

Same here............
Always like to have an alternative bank account & credit / debit card.

^^ This is why it’s important to have a backup.

Hate when a bank is having problems as customers blame US for it! 

Who is us?

Me and my colleagues 

So are you Santander IT?

No - shop worker. Had customers shouted at us when Lloyds’ systems went down and another bank went down earlier this year. They blamed us even though the problem was with the bank NOT us

As a shop, you should have a system in place to work around the situation. How did shops function with credit cards before everything was online?

Before credit card data was online, there was a machine that ran a roller or press over the embossed card to take an imprint of the details. These are of course long gone. The shop would have a backup system, either cash or cheques. It is not the fault of the shop. The bank is clearly to blame for providing systems that are not robust.

The bank should have a backup and most prudent customers would also have a backup but I cannot see what other backup a shop could have.

I agree with you. And of course more and more cards these days aren't embossed and are flat, so the old manual machines wouldn't work with them even if a shop still had them.

With this, you write the numbers manually on the slip! My first job in the late 90s was working in a now longer exist high street chain. Had a customer from abroad with a non embossed card. Had to fill in the slip manually. 

colsten

briskbeats said:

RG2015 said:

surreysaver said:

briskbeats said:

colsten said:

briskbeats said:

colsten said:

briskbeats said:

greyteam1959 said:

Same here............
Always like to have an alternative bank account & credit / debit card.

^^ This is why it’s important to have a backup.

Hate when a bank is having problems as customers blame US for it! 

Who is us?

Me and my colleagues 

So are you Santander IT?

No - shop worker. Had customers shouted at us when Lloyds’ systems went down and another bank went down earlier this year. They blamed us even though the problem was with the bank NOT us

As a shop, you should have a system in place to work around the situation. How did shops function with credit cards before everything was online?

Before credit card data was online, there was a machine that ran a roller or press over the embossed card to take an imprint of the details. These are of course long gone. The shop would have a backup system, either cash or cheques. It is not the fault of the shop. The bank is clearly to blame for providing systems that are not robust.

The bank should have a backup and most prudent customers would also have a backup but I cannot see what other backup a shop could have.

We still have these flatbed card machines. Only dug out when needed. One episode happened three years ago - when I was  on holiday, so missed this. As internet went down around the area. So could not pay with card, buy phone vouchers, buy 3rd party gift vouchers. The ATMs in the area affected were down. My teammates said it was a nightmare. Customers were yelling at them asking which is the nearest ATM which was working. How on earth were my teammates supposed to know this when the internet went down 40 mins previously? As had no idea how far the problem was.

It's best not to bank with one bank just in case things like this happened. I am with Santander myself for current accounts. Have a Halifax credit card and a Nationwide bank account with £70 in. That reminds me, must get a new PIN for Nationwide.

A few years ago, I was caught up in a total power failure one morning. I had just filled up in a busy petrol station when everything went off - lights, the petrol pumps and their displays, the tills (no cash!), the CCTV cameras, the lot. Staff scrambled and found the card swipe machines but nobody knew how much people owed as the pumps were out. Those customers who had their tanks partially or fully filled were asked for an honest guess about the amount of petrol/diesel they have had, and most, incl. myself, signed a swipe machine slip. One guy drove off in a huff, after a tirade of mindless abuse for the staff, without paying anything. I was lucky that I had already filled up - goodness knows what those waiting for petrol did, as this was the only petrol station in town. The power in the entire town was off for several more hours, as the problem was caused by a substation in which something had blown up. I can only imagine the havoc in the town's shops and homes.

Mickey666

colsten said:

Mickey666 said:

colsten said:

mr_accountant said:

Right shall I be the first, I don't need to use santander today nor have I been inconvenienced but which social media platform can I put my boring story so that the world knows and so help me claim my compensation because its my birth right? 

You should not assume that your requirements are the same for the other 14 million Santander customers. There have been plenty of completely credible reports from people stranded at petrol stations, unable to buy food or pay for their train/underground ticket or their taxi etc etc. Those people have a right to compensation, and they will no doubt keep the Santander complaints team busy for some time.

Instead of bragging with an I‘m alright Jack attitude, it would be constructive to remind people of the importance of always having alternative payment means. This can be anything from carrying cash to having one or more other current accounts with a positive balance  to one or more credit cards. Goes (almost) without saying that one should have at least one each of a VISA and a Mastercard, in case one of them has a problem (as has happened in the past).

As aside, having my main current account with Santander: I won‘t be making any compensation claim myself as I was not materially inconvenienced by this outage. I will also not close my Santander account. 

Looks like you've answered your own rant there

Yes, it SHOULD go without saying that people should have more than one means of paying for stuff, but I guess some people just like risking it.  Their choice, their responsibility.

I haven't ranted. I merely took exception with a post that tried to ridicule those who were unfortunate enough to be left without a means to pay for essentials.

Mickey666 said:

IvanOpinion said:

tempus_fugit said:

This sort of thing happening so often suggests that they didn't have a proper backout plan or it hadn't been properly tested. Unfortunately these things happen, but they should always be able to put everything back if necessary, it shouldn't all be down for this long.

That depends on the nature of what has gone wrong. Several years back we hit a problem which meant it took us almost 80 hours to get a stable system back online.  The press and social media was rife with BS about software updates, incompetence etc. etc. and 101 "experts" providing simple solutions and pointing the finger of blame - but not one was even close to guessing what had actually happened.

The problem was a set of unpredictable conditions.  For commercial reasons I can't go into great detail but  ... despite all the sensors, our primary site had suffered undetected damage that brought the servers down over a period of about 1-2 weeks (as best as we can guess).  This meant they were still syncing with our DR site (several hundred miles away) using an industry standard best practice process that had been tested many times in the past, but (unknown to us) the data contained corruptions.  In fact we had only audited and tested our fail over process 6 weeks previously and it worked perfectly.  When the primary site went down it went down big time in a way that did not trigger DR to kick in.  We spent the first few hours not realising the problem and trying to get the primary site up and running, eventually we sent someone to the primary site itself (120mile round trip) and realised the scale of the problem (there was no way it would be recoverable or brought back up for many weeks). 

We then manually fired up the DR site and it was online within the hour - but reports started flooding in of data problems so we had to take it back down.  Realising the problem we restored from a backup (a process that takes several hours - and took us 3 goes before we found a stable data set) but the regulator would not allow us to bring the site back up until we had done a load of additional actions and checks.  The board did what boards do best, ranted, raved and demanded hourly updates (which meant we were basically a man down just to provide them with their updates).  Our entire team did not leave the office for those 80 hours with us often seen sleeping in our chairs or under desks as we waited for various processes to complete.  On the plus side it made us all valuable resources in the market and targets for head hunters.

Well said.  The 'expert's who pop out from under their stones when things like this happen have no clue about the complexity of such systems, the planning that goes into upgrades, or the fall-back procedures required.  Things go wrong and, as far as I can see, the problem was dealt with one way or another.   A few hours outage is certainly inconvenient but it's hardly the end of the world.

If it wasn't for 24/7 rolling news or the Twitterati, 99% of the population wouldn't even have known about this 'problem'.
Storm in a teacup.

It's not about 99% or 100% of the population (most of whom probably still don't know, nor have a need to know) but about those Santander customers who were adversely affected by the outage. Whilst nobody would argue that it takes some time and effort to fix the systems one way or another, your dismissive remarks simply show that you are another I'm-alright-Jack.

Sorry to disappoint your uninformed assumption but I was actually affected as I had a rather large invoice to pay by bank transfer and wasn't able to.  I suppose I could have paid it from a different bank account but since I already had the payment details set up in my Santander account I took the easier option of simply waiting until today instead.

I guess that makes me an 'alright-Jack' in your eyes when the reality is that I could see the situation for what it was - a temporary glitch - and didn't feel the need to hit the panic button or leap aboard the compensation bandwagon.  Takes all sorts I suppose

As for being 'adversely affected' by a few hours of bank outage, I suppose it depends on your definition of 'adverse' but my suspicion is that we're talking about first-world problems here.  Like I said, storm in a teacup. 

jimjames

greyteam1959 said:

Same here............
Always like to have an alternative bank account & credit / debit card.

Only just heard about this from BBC
https://www.bbc.co.uk/news/uk-57131476
I'm amazed how many people rely on just one bank card for their life. I know it's unusual to have the number of accounts many people here have but to rely on a single card for everything seems crazy. At least with a credit card from a different bank you aren't reliant on that platform. One guy interviewed for that article doesn't seem to realise the irony of what he was saying - "I work in IT so I know about backups & upgrades" - yet relies on a single card for payments! Is it just people don't think they can be trusted with a credit card? 

If you only have one card/account, is there a reason why you don't have another?

kuratowski

jimjames said:
Is it just people don't think they can be trusted with a credit card? 

If you only have one card/account, is there a reason why you don't have another?

Anecdotally, I think there is some truth in that.  But wary of drawing conclusions on the wider population.

I tried to find some research findings on this but what I found was fairly old: https://www.fca.org.uk/firms/consumer-credit-research-cards/consumer-use
As of 2013, 90% of households with income more than £50k have a credit card, but only 33% of households with income less than £10k.  It's presumably the case that these lower income households are simply unable to access credit cards.  And as a result the impact of a banking failure on these, already vulnerable, households could - as Colsten suggests upthread - be really severe.

colsten

Mickey666 said:

colsten said:

Mickey666 said:

colsten said:

mr_accountant said:

Right shall I be the first, I don't need to use santander today nor have I been inconvenienced but which social media platform can I put my boring story so that the world knows and so help me claim my compensation because its my birth right? 

You should not assume that your requirements are the same for the other 14 million Santander customers. There have been plenty of completely credible reports from people stranded at petrol stations, unable to buy food or pay for their train/underground ticket or their taxi etc etc. Those people have a right to compensation, and they will no doubt keep the Santander complaints team busy for some time.

Instead of bragging with an I‘m alright Jack attitude, it would be constructive to remind people of the importance of always having alternative payment means. This can be anything from carrying cash to having one or more other current accounts with a positive balance  to one or more credit cards. Goes (almost) without saying that one should have at least one each of a VISA and a Mastercard, in case one of them has a problem (as has happened in the past).

As aside, having my main current account with Santander: I won‘t be making any compensation claim myself as I was not materially inconvenienced by this outage. I will also not close my Santander account. 

Looks like you've answered your own rant there

Yes, it SHOULD go without saying that people should have more than one means of paying for stuff, but I guess some people just like risking it.  Their choice, their responsibility.

I haven't ranted. I merely took exception with a post that tried to ridicule those who were unfortunate enough to be left without a means to pay for essentials.

Mickey666 said:

IvanOpinion said:

tempus_fugit said:

This sort of thing happening so often suggests that they didn't have a proper backout plan or it hadn't been properly tested. Unfortunately these things happen, but they should always be able to put everything back if necessary, it shouldn't all be down for this long.

That depends on the nature of what has gone wrong. Several years back we hit a problem which meant it took us almost 80 hours to get a stable system back online.  The press and social media was rife with BS about software updates, incompetence etc. etc. and 101 "experts" providing simple solutions and pointing the finger of blame - but not one was even close to guessing what had actually happened.

The problem was a set of unpredictable conditions.  For commercial reasons I can't go into great detail but  ... despite all the sensors, our primary site had suffered undetected damage that brought the servers down over a period of about 1-2 weeks (as best as we can guess).  This meant they were still syncing with our DR site (several hundred miles away) using an industry standard best practice process that had been tested many times in the past, but (unknown to us) the data contained corruptions.  In fact we had only audited and tested our fail over process 6 weeks previously and it worked perfectly.  When the primary site went down it went down big time in a way that did not trigger DR to kick in.  We spent the first few hours not realising the problem and trying to get the primary site up and running, eventually we sent someone to the primary site itself (120mile round trip) and realised the scale of the problem (there was no way it would be recoverable or brought back up for many weeks). 

We then manually fired up the DR site and it was online within the hour - but reports started flooding in of data problems so we had to take it back down.  Realising the problem we restored from a backup (a process that takes several hours - and took us 3 goes before we found a stable data set) but the regulator would not allow us to bring the site back up until we had done a load of additional actions and checks.  The board did what boards do best, ranted, raved and demanded hourly updates (which meant we were basically a man down just to provide them with their updates).  Our entire team did not leave the office for those 80 hours with us often seen sleeping in our chairs or under desks as we waited for various processes to complete.  On the plus side it made us all valuable resources in the market and targets for head hunters.

Well said.  The 'expert's who pop out from under their stones when things like this happen have no clue about the complexity of such systems, the planning that goes into upgrades, or the fall-back procedures required.  Things go wrong and, as far as I can see, the problem was dealt with one way or another.   A few hours outage is certainly inconvenient but it's hardly the end of the world.

If it wasn't for 24/7 rolling news or the Twitterati, 99% of the population wouldn't even have known about this 'problem'.
Storm in a teacup.

It's not about 99% or 100% of the population (most of whom probably still don't know, nor have a need to know) but about those Santander customers who were adversely affected by the outage. Whilst nobody would argue that it takes some time and effort to fix the systems one way or another, your dismissive remarks simply show that you are another I'm-alright-Jack.

Sorry to disappoint your uninformed assumption but I was actually affected as I had a rather large invoice to pay by bank transfer and wasn't able to.  I suppose I could have paid it from a different bank account but since I already had the payment details set up in my Santander account I took the easier option of simply waiting until today instead.

I guess that makes me an 'alright-Jack' in your eyes when the reality is that I could see the situation for what it was - a temporary glitch - and didn't feel the need to hit the panic button or leap aboard the compensation bandwagon.  Takes all sorts I suppose

As for being 'adversely affected' by a few hours of bank outage, I suppose it depends on your definition of 'adverse' but my suspicion is that we're talking about first-world problems here.  Like I said, storm in a teacup. 

Thanks for confirming that you are indeed an I‘m-Alright-Jack. As you say, you did not have an urgent need on Saturday, such as paying for food, or for petrol to get to work or to pick up your kids, It‘s a mystery to me (also a Santander customer who didn‘t need my money on Saturday) why you appear to be unable to see the severe issues the outage has caused for some.