massive IT outage hits the world

Olinda99

looks like it might be a problem with Microsoft cloud 

https://www.bbc.co.uk/news/live/cnk4jdwp49et

MattMattMattUK

Olinda99 said:

looks like it might be a problem with Microsoft cloud 

https://www.bbc.co.uk/news/live/cnk4jdwp49et

From the reports it appears to be an issue with a third-party anti-virus supplier called Crowdstrike who have issued an update which is bricking devices, from tills, to laptops and desktops, to servers. 

Neil_Jones

This outage is very unlikely to affect computer domestic users.  Directly anyway.  Notes about card payments not working, well I didn't have a problem this morning so...

Anyway not the first time this has happened and it won't be the last.

Sea_Shell

We can only hope that this is solved quickly, as an inconvenience today, soon becomes a major problem after a few days. ☹️

Undervalued

Neil_Jones said:

This outage is very unlikely to affect computer domestic users.  Directly anyway.  Notes about card payments not working, well I didn't have a problem this morning so...

Anyway not the first time this has happened and it won't be the last.

I did! All of the tills in the supermarket I was in soon after 8am this morning were hanging for ages then declining contactless payment. Inserting card, then entering PIN was accepted, certainly for smallish amounts. 

Neil_Jones

Undervalued said:

Neil_Jones said:

This outage is very unlikely to affect computer domestic users.  Directly anyway.  Notes about card payments not working, well I didn't have a problem this morning so...

Anyway not the first time this has happened and it won't be the last.

I did! All of the tills in the supermarket I was in soon after 8am this morning were hanging for ages then declining contactless payment. Inserting card, then entering PIN was accepted, certainly for smallish amounts. 

Perhaps I should have added I wasn't in a major supermarket (I was in Farmfoods) , or maybe I just happened to wave my contactless card around when the system happened to be working.

Other people's mileage may vary of course in this regard.

MouldyOldDough

It's affecting many different apps - including Ancestry - Sync works although very slowly but HINTS don't !

Ancestry are not admitting that there are any problems but a "banner" appears on the app - telling users that there is an error with the system..........

RumRat

Neil_Jones said:

Undervalued said:

Neil_Jones said:

This outage is very unlikely to affect computer domestic users.  Directly anyway.  Notes about card payments not working, well I didn't have a problem this morning so...

Anyway not the first time this has happened and it won't be the last.

I did! All of the tills in the supermarket I was in soon after 8am this morning were hanging for ages then declining contactless payment. Inserting card, then entering PIN was accepted, certainly for smallish amounts. 

Perhaps I should have added I wasn't in a major supermarket (I was in Farmfoods) , or maybe I just happened to wave my contactless card around when the system happened to be working.

Other people's mileage may vary of course in this regard.

B&Q were cash only today as was Asda home....But, the Range was working fine.
The Doctors surgery was completely offline...
Guess it'll be hit and miss until it's fixed.

TELLIT01

The wonders of modern technology.  I wonder how those who brag about never carrying cash are getting on.

prowla

The mistake is in considering a PC software supplier to be a global enterprise service provider.

Their cloud offering, "Azure", had an outage a few years back, where an Active Directory update propagated globally; it was almost like one of those disaster movies where you see a map of the worlds and the lights gradually go out.

This time it looks like they embedded a 3rd party piece into their services such that it comprised a Single Point Of Failure (SPOF) across their entire systems, so when it had a bug it went everywhere.

The days of having controlled roll-outs, QA testing, contingency plans, redundant services, resilient systems, and so-on are long gone.

Some might say that replication is a resiliency feature, but the risk is that a service which relies on replication sufers from the pitfal that it will also replicate errors.

As for customers, Microsoft showed that 80% of the product sold cheaply is what sells; if you view the tech as a cost rather than an enabler and have a "that'll do" mentality, then you're putting yourself in a risky position.

PHK

prowla said:

The mistake is in considering a PC software supplier to be a global enterprise service provider.

Their cloud offering, "Azure", had an outage a few years back, where an Active Directory update propagated globally; it was almost like one of those disaster movies where you see a map of the worlds and the lights gradually go out.

This time it looks like they embedded a 3rd party piece into their services such that it comprised a Single Point Of Failure (SPOF) across their entire systems, so when it had a bug it went everywhere.

The days of having controlled roll-outs, QA testing, contingency plans, redundant services, resilient systems, and so-on are long gone.

Some might say that replication is a resiliency feature, but the risk is that a service which relies on replication sufers from the pitfal that it will also replicate errors.

As for customers, Microsoft showed that 80% of the product sold cheaply is what sells; if you view the tech as a cost rather than an enabler and have a "that'll do" mentality, then you're putting yourself in a risky position.

That's not what happened. There was a brief Microsoft outage around midnight our time. Then at just after 4am an update to customers of Crowdstrike caused PCs to crash. By 530am Crowdstrike had identified and solved the issue. But:

Some news outlets (especially BBC) reported this as a Microsoft Outage until about 9am. Even though the exact cause was known much earlier. 

The fix is either to restart the PC up to fifteen times or go in and delete a file. 

The problem is that all 24,000 of Crowdstrike customers are big organisations with many thousands of PCs and Servers each. All of which will need fixing essentially manually (I exclude virtual machines here because they can be rolled back remotely). Some of these PCs are embedded and it will take days to get around to each one. 


The problem here isn't a single point of failure but putting Compliance ahead of risk assessment. Compliance insists that systems like Falcon are in place - that box is ticked but the organisation doesn't do a risk assessment and so there's no plan to swiftly resolve issues. 

This isn't the first time it's happened, in 2010 a similar Mcaffee update knocked out thousands of PCs. But organisations didn't learn.