• Due to ongoing issues caused by Search, it has been temporarily disabled
  • Please excuse the mess, we're moving the furniture and restructuring the forum categories
  • You may need to edit your signatures.

    When we moved to Xenfora some of the signature options didn't come over. In the old software signatures were limited by a character limit, on Xenfora there are more options and there is a character number and number of lines limit. I've set maximum number of lines to 4 and unlimited characters.

Crowdstrike update brings internet down

zooterkin

Nitpicking dilettante, Administrator
Staff member
zooterkin
Joined
Mar 21, 2007
Messages
60,298
Location
Berkshire, mostly
Airlines, supermarkets, health providers, media outlets and many more businesses worldwide have been brought down by an update installed on Microsoft Windows servers.

It seems to be due to a bug in Crowdstrike software, the CEO tweeted:
@George_Kurtz said:
CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.
Click to expand...

No apology was included, and it doesn't explicitly say the bug was in CrowdStrike code.
 
They should have done more and better testing. Then release it in stages. That way if anything goes wrong it is detected early and does not cause such chaos.
 
I’d planned to have a nice quiet Friday WFH but instead I’ve been spending time dealing with this. My organisation isn’t directly affected but many of our business partners are.
 
One of my sons is a senior IT Manager at a big university. He tells me that the damage is not so bad as it was Friday afternoon, and they have a work around for desktop users. But rather ominously he said this was not unexpected and there will be a lot more to follow.
 
It is extremely serious yet it has not effected me -- so far -- and it does not appear to have effected ISF (so far).
A worldwide Windows glitch has taken much of the world’s infrastructure offline. Flights are grounded and TV channels including Sky News have been taken off air. Everything from banks and payment companies to airlines and train companies said that they would see delays and technical issues. Microsoft 365 said that it was investigating the problem and “continue to take mitigation actions”. UK Independent news link (with live updates)
Click to expand...


Seems to be mostly effecting commercial users.
 
Three quarters of my office got a BSD when starting their state laptops and the internal NYS website for our Department is down
 
Oh dear. How unfortunate. Not my problem any more.....

Perhaps "The Cloud" fad will pass? A sudden return to 'on prem'?
 
catsmate said:
Oh dear. How unfortunate. Not my problem any more.....

Perhaps "The Cloud" fad will pass? A sudden return to 'on prem'?
Click to expand...

Doubt it, it's simply too expensive for most companies to have their own "data centre" to power their IT requirements. I do however think governments should take a different approach, they can quite easily fund their own data centres which 1) would be cheaper 2) offer more control. But even that doesn't really provide security if you are using 3rd party software - a glitch in that software would still effectively bring down your hardware.
 
Cloudstrike sounds like the type of name hackers would give themselves.
 
Tech details here https://en.wikipedia.org/wiki/2024_CrowdStrike_incident

An update to CrowdStrike's Falcon Sensor security software at 04:09 (UTC) included the faulty kernel driver csagent.sys causing Windows machines a blue screen of death with the message PAGE_FAULT_IN_NONEPAGED_AREA.[2][3] This left machines in bootloops or recovery mode.[4][5]

Affected machines were restored by booting into safe mode and deleting the %windir%\System32\drivers\CrowdStrike\C-00000291*.sys files.[6] This process, while trivial to complete, requires manual application on each computer affected, which could take days to be completed for larger organizations.[7][8]

CrowdStrike reverted the content update at 05:27 (UTC).[citation needed] Devices booted after the update was reverted are expected to not be affected.[2] As of 09:45 (UTC), the CEO confirmed the fix was deployed.
Click to expand...
 
Darat said:
Doubt it, it's simply too expensive for most companies to have their own "data centre" to power their IT requirements. I do however think governments should take a different approach, they can quite easily fund their own data centres which 1) would be cheaper 2) offer more control. But even that doesn't really provide security if you are using 3rd party software - a glitch in that software would still effectively bring down your hardware.
Click to expand...
You don't need a datacentre. You may need managed hosting. Not dumping everything into "The Cloud" and abdicating responsibility. Or at least use a more reliable cloud provider.

If you have access to systems then your, or your expensive consultants, can fix them.

Wudang said:
Do we know it's cloud related? Last I heard EMIS have their own data centre as I used to work with the person who planned it.
Click to expand...
The glitch is two, potentially related, issues. A major Azure <incident> followed by a massive CloudStrike <incident> which effected physical PCs but mostly virtualised 'Cloud PCs".
The former are fixable by a smart IT person. The latter are not.

In Ye Olde Days this <incident> is the kind of thing consultants like me would have been called in to fix and we would have fixed it; a kludged together workround that would have gotten systems up and running.

BTW anyone else short-selling CloudStrike?
 
catsmate said:
In Ye Olde Days this <incident> is the kind of thing consultants like me would have been called in to fix and we would have fixed it; a kludged together workround that would have gotten systems up and running.
Click to expand...

I have experience - I’m sure I’ve mentioned it on here before - of returning to a site a decade later in a different capacity and finding one of my own such kludges still running silently in a corner, unmodified and undocumented.
 
I wonder if that's why my Alexa wasn't working for several hours yesterday. I could connect my laptop to the wireless internet, but Alexa said it couldn't connect.
 
It's Crowdstrike, not Cloudstrike. It's not cloud related. It's 3rd party firewall, running on Windows. Used exclusively in corporate sphere. But quite popular in corporate sphere. Also many Azure data centers run it.
So normal users are usually fine. Data centers running on Linux are fine. Everybody not using this specific firewall or something running on Azure is fine.
Corporations using Azure are somewhat fine, as MS will fix it for free, and relatively fast.
Corporations using the firewall on all their machines are screwed the most, as every single device has to be manually fixed.
 
malbui said:
I have experience - I’m sure I’ve mentioned it on here before - of returning to a site a decade later in a different capacity and finding one of my own such kludges still running silently in a corner, unmodified and undocumented.
Click to expand...
My longest was 1998 to 2016. Still had my note, added a week after the kludge, about a better fix.
 
I would have thought that it would be standard procedure in larger corporate or government organizations to test updates and push them out in waves, to smaller groups of computers at first. I guess not, because this was a pretty obvious issue that could have been spotted by testing on just one machine before unleashing it on the rest of the organization.
 
Wudang said:
The idea that a 3rd party software vendor could push updates to customer systems is one that would have had salesmen escorted out by security at the data centres I worked at.
Click to expand...

Did you ever work at datacenters where third party updates were pushed out by your central IT department on a regular cadence, sometimes with unexpected results?

Because that's how it's been at every datacenter I've ever worked at.

Also I've never worked at a datacenter where the third party software salesmen came to the site to make their sales pitch. Usually they went to the corporate headquarters. If they showed up in person at all. Most software purchases were done over the phone or via email.
 
Crowdstrike better have its EULAs well written and giving them pretty good immunity from any damage claims, otherwise the company is dead.
 
jadebox said:
I would have thought that it would be standard procedure in larger corporate or government organizations to test updates and push them out in waves, to smaller groups of computers at first. I guess not, because this was a pretty obvious issue that could have been spotted by testing on just one machine before unleashing it on the rest of the organization.
Click to expand...

It's a firewall, and it's security updates. If you want react on threats fast, you need the updates fast.
But I guess at least some basic delay and testing if ANY update kills the machine will now be quickly implemented everywhere.
 
theprestige said:
Did you ever work at datacenters where third party updates were pushed out by your central IT department on a regular cadence, sometimes with unexpected results?
Click to expand...
Yes but we'd test the updates on our OAT & UAT systems first
Because that's how it's been at every datacenter I've ever worked at.

Also I've never worked at a datacenter where the third party software salesmen came to the site to make their sales pitch. Usually they went to the corporate headquarters. If they showed up in person at all. Most software purchases were done over the phone or via email.
Click to expand...

2 departments I worked at in HSBC had their own budgets. I was directly involved as a techie with discussing our contract with IBM when they bought Omegamon and peripherally with others like Interlink Software. The deal was basically signed by my manager subject to approval by his manager and finance.
A salesman pushing an alternative to the latter was escorted from the site when he promised us a live demo and came with a PowerPoint presentation.
 
Dr.Sid said:
It's a firewall, and it's security updates. If you want react on threats fast, you need the updates fast.
But I guess at least some basic delay and testing if ANY update kills the machine will now be quickly implemented everywhere.
Click to expand...

You have to balance a threats potential impact against the risk of an update causing problems. Outsourcing that decision making to a 3rd party who have much less at stake than you is a bold choice.
 
Wudang said:
You have to balance a threats potential impact against the risk of an update causing problems. Outsourcing that decision making to a 3rd party who have much less at stake than you is a bold choice.
Click to expand...

Clearly a bold choice which many took. My experience is windows updates are typically postponed on servers .. but firewalls are updated ASAP.
Also dead server is better than hacked server. That said, there is no guarantee the update wont bring new vulnerability. But with firewalls, you simply have to trust the 3rd party.
 
theprestige said:
Which has nothing to do with third-party software salesmen coming into a datacenter to pitch unsupervised direct updates to your servers. Because that's not actually a thing in software sales.
Click to expand...



Why did you cut out the part where I described what happened? HSBC paid me to be a subject expert and then used my expertise to evaluate software offerings in my specialist areas. My office and those of 3 layers of management above me were in the data centre.
 
Last edited:
Dr.Sid said:
Clearly a bold choice which many took. My experience is windows updates are typically postponed on servers .. but firewalls are updated ASAP.
Also dead server is better than hacked server. That said, there is no guarantee the update wont bring new vulnerability. But with firewalls, you simply have to trust the 3rd party.
Click to expand...

I can see that. Where I first worked they hammered home the difference between involved and committed and that's hard coded in my risk assessment still.
 
Dr.Sid said:
It's Crowdstrike, not Cloudstrike. It's not cloud related. It's 3rd party firewall, running on Windows. Used exclusively in corporate sphere. But quite popular in corporate sphere. Also many Azure data centers run it.
So normal users are usually fine. Data centers running on Linux are fine. Everybody not using this specific firewall or something running on Azure is fine.
Corporations using Azure are somewhat fine, as MS will fix it for free, and relatively fast.
Corporations using the firewall on all their machines are screwed the most, as every single device has to be manually fixed.
Click to expand...

Thank sweet Jesus for you saying this because it's being missed in this thread.

The corporations didn't update anything themselves. If I'm reading this right an update went out from crowdstrike to the individual end user devices via an installed agent. This is all on crowdstrike and I can't think of any real way a company could have prevented this other than not using crowdstrike.

It sucks, Crowdstrike is going to take a massive hit, suffer a bunch of lawsuits, and I'd bet get some new C-suite's.
 
Last edited:
arthwollipot said:
"Silently" until they kill half the internet.
Click to expand...
I was reminiscing with the Old Guard (aka Usual Suspects), think of prior disasters, who'd caused/fixed them and one pointed out a stat, from CrowdStrike, that 46% of sysadmins admit to never testing software patches.
:rolleyes:
 
You must log in or register to reply here.
Back
Top Bottom