Crowdstrike update brings internet down

This is worth a look if you are interested in what Clowdstrike client's were signing up for:

CROWDSTRIKE TERMS AND CONDITIONS

8.5 No Guarantee. CUSTOMER ACKNOWLEDGES, UNDERSTANDS, AND AGREES THAT CROWDSTRIKE DOES NOT GUARANTEE OR WARRANT THAT IT WILL FIND, LOCATE, OR DISCOVER ALL OF CUSTOMER’S OR ITS AFFILIATES’ SYSTEM THREATS, VULNERABILITIES, MALWARE, AND MALICIOUS SOFTWARE, AND CUSTOMER AND ITS AFFILIATES WILL NOT HOLD CROWDSTRIKE RESPONSIBLE THEREFOR.

8.6 Disclaimer. EXCEPT FOR THE EXPRESS WARRANTIES IN THIS SECTION 8, CROWDSTRIKE AND ITS AFFILIATES DISCLAIM ALL OTHER WARRANTIES, WHETHER EXPRESS, IMPLIED, STATUTORY OR OTHERWISE. TO THE MAXIMUM EXTENT PERMITTED UNDER APPLICABLE LAW, CROWDSTRIKE AND ITS AFFILIATES AND SUPPLIERS SPECIFICALLY DISCLAIM ALL IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NON-INFRINGEMENT WITH RESPECT TO THE OFFERINGS AND CROWDSTRIKE TOOLS. THERE IS NO WARRANTY THAT THE OFFERINGS OR CROWDSTRIKE TOOLS WILL BE ERROR FREE, OR THAT THEY WILL OPERATE WITHOUT INTERRUPTION OR WILL FULFILL ANY OF CUSTOMER’S PARTICULAR PURPOSES OR NEEDS. THE OFFERINGS AND CROWDSTRIKE TOOLS ARE NOT FAULT-TOLERANT AND ARE NOT DESIGNED OR INTENDED FOR USE IN ANY HAZARDOUS ENVIRONMENT REQUIRING FAIL-SAFE PERFORMANCE OR OPERATION. NEITHER THE OFFERINGS NOR CROWDSTRIKE TOOLS ARE FOR USE IN THE OPERATION OF AIRCRAFT NAVIGATION, NUCLEAR FACILITIES, COMMUNICATION SYSTEMS, WEAPONS SYSTEMS, DIRECT OR INDIRECT LIFE-SUPPORT SYSTEMS, AIR TRAFFIC CONTROL, OR ANY APPLICATION OR INSTALLATION WHERE FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL INJURY, OR PROPERTY DAMAGE. Customer agrees that it is Customer’s responsibility to ensure safe use of an Offering and the CrowdStrike Tools in such applications and installations. CROWDSTRIKE DOES NOT WARRANT ANY THIRD PARTY PRODUCTS OR SERVICES.
Click to expand...

They can't say they were not warned.
 
arthwollipot said:
Nobody reads the T&Cs.
Click to expand...


True. Back in 2005, PC PitStop included the following clause in their End User License Agreement:
SPECIAL CONSIDERATION
A special consideration which may include financial compensation will be awarded to a limited number of authorize licensee to read this section of the license agreement and contact PC Pitstop at consideration@pcpitstop.com. This offer can be withdrawn at any time.
Click to expand...
It took four months and some 3,000 downloads before someone sent an email to the listed address. The lucky person received $1,000.
Source [makeuseof.com]
 
Last edited:
HansMustermann said:
Yes, but the only actual kernel device driver is CSagent.sys, so that's the only one which needs to be there and thus named. The others don't have to be either named .sys nor in the drivers directory, much less both. The latter is what I consider mis-behaviour on their part.
Click to expand...

I don't agree. Yes the other files don't have to be named .sys, but I think they probably do have to be in the drivers directory because the driver needs to be able to find and read them easily. If you look in the drivers directory on a Windows box, you'll see other drivers do it. It's probably standard practice.
 
jeremyp said:
I don't agree. Yes the other files don't have to be named .sys, but I think they probably do have to be in the drivers directory because the driver needs to be able to find and read them easily. If you look in the drivers directory on a Windows box, you'll see other drivers do it. It's probably standard practice.
Click to expand...

I'd settle for them just not being named sys, really.

But I'll point out that:

- Windows is not CPM. It does have such a thing as sub-directories, to keep files organized. Which they actually did. So there's nothing to keep them from having a subdirectory called, dunno, CSChannels or CSData or something in there.

- other programs manage NOT to do it. For example I have Eset (another anti-virus/security package) currently installed, and while it does have an ehdrv.sys in that drivers directory, the data is NOT under "drivers." So it's not impossible. Because...

- Windows is not CPM. It has a registry. And it CAN be accessed by the kernel or any drivers, including kernel level drivers. You can store anything you want in there, including the location of your data files :p

Also, I have looked on my Windows machine, and there's a distinct lack of many other drivers doing that.

So all in all, I think they've done a rather sloppy job for a supposed top security company. Is all I'm saying.
 
Last edited:
Having worked in a large software development lab it can be amazing the gaps between people who decide how it should work and people implementing how it will work. And from there and a major financial institution how many programmers have no idea of the run time environment their programs run in.
 
Actually, I kinda stopped finding it amazing about 20 years ago. Now I'd find it more amazing if that wasn't the case :p
 
Last edited:
HansMustermann said:
I'd settle for them just not being named sys, really.

But I'll point out that:

- Windows is not CPM. It does have such a thing as sub-directories, to keep files organized. Which they actually did. So there's nothing to keep them from having a subdirectory called, dunno, CSChannels or CSData or something in there.
Click to expand...
They were in a subdirectory called "Crowdstrike". That seems reasonable to me.
- other programs manage NOT to do it. For example I have Eset (another anti-virus/security package) currently installed, and while it does have an ehdrv.sys in that drivers directory, the data is NOT under "drivers." So it's not impossible. Because...

- Windows is not CPM. It has a registry. And it CAN be accessed by the kernel or any drivers, including kernel level drivers. You can store anything you want in there, including the location of your data files :p
Click to expand...
The registry is really a hot mess. They probably chose not to use it.

So all in all, I think they've done a rather sloppy job for a supposed top security company. Is all I'm saying.
Click to expand...

We can agree on that. Two egregious faults that I can think of are

1) not validating the input files for its driver

2) allowing a corrupt input file to be distributed to its customers.

Compared to these two issues, your quibbles about where they put their files and how they name them is like complaining that your steak is overdone on the Hindenburg.
 
jeremyp said:
They were in a subdirectory called "Crowdstrike". That seems reasonable to me.

The registry is really a hot mess. They probably chose not to use it.



We can agree on that. Two egregious faults that I can think of are

1) not validating the input files for its driver
2) allowing a corrupt input file to be distributed to its customers.

Compared to these two issues, your quibbles about where they put their files and how they name them is like complaining that your steak is overdone on the Hindenburg.
Click to expand...

In a way, it was validating the input. If it was bad, the driver would crash. :p


If I'm writing software for someone other than me, I validate the input and have it toss out bad stuff, at the same time logging an error. More to the point, it would work only on valid data: a file with a correct header value and known good values everywhere else. Input that doesn't meet the expected criteria is ignored and error or warning messages generated.

I'm surprised these sort of safeguards weren't in place for a driver that could bring down the entire system if it got bad data. Did this never happen during testing? Or did they even do harsh testing/fuzzing?
 
Last edited:
How should the loader proceed if it can't load security software? If it completes booting the system is vulnerable. If it halts how does it signal to ops that there's a problem? Wouldn't it be loaded before monitoring software like BMC's Patrol etc?
 
From what I understand, the driver was loaded all right. Then it started running its own code to parse those channel files. And accessed unallocated memory. (Basically a null pointer error of sorts. Wasn't strictly speaking null, but close enough for gummint work;))

What I'd add there though is that Windows doesn't technically know at that point if it's loading security software, or the chipset drivers, or the ACPI driver, or even the floppy disk or filesystem drivers. (Yeah, "flpydisk.sys" is in that directory too.) Or the encryption driver for the filesystem. It's A kernel module that severely malfunctioned. Is it safe to go on without it? Is the system even in a stable state?

As you seem to say too, there is no easy one-size-fits-all answer there.
 
Last edited:
Wudang said:
How should the loader proceed if it can't load security software? If it completes booting the system is vulnerable. If it halts how does it signal to ops that there's a problem? Wouldn't it be loaded before monitoring software like BMC's Patrol etc?
Click to expand...

It should fail gracefully with a Big Scary Warning that parts of the anti-virus (or other user protections) aren't available and the machine's security may be compromised. I'm unsure if mouse and keyboard can be used--by this stage of the boot we're no longer in 16 bit mode, so the BIOS services are probably no longer accessible--but if they are there should be a acknowledgement from the user. Corporate setups might want to be able to tweak this, so that for high security systems the warning can't be bypassed. But for everyday systems, let them continue.
 
Blue Mountain said:
It should fail gracefully with a Big Scary Warning that parts of the anti-virus (or other user protections) aren't available and the machine's security may be compromised. I'm unsure if mouse and keyboard can be used--by this stage of the boot we're no longer in 16 bit mode, so the BIOS services are probably no longer accessible--but if they are there should be a acknowledgement from the user. Corporate setups might want to be able to tweak this, so that for high security systems the warning can't be bypassed. But for everyday systems, let them continue.
Click to expand...

Will firewall servers even have keyboards and mice?

Eta: the little I know of virtual servers was writing some code to integrate the banks automation software with blade servers to automate shutdown and restart and as I recall the thinking behind the blade management software was very no intuitive to me.
 
Last edited:
Wudang said:
Will firewall servers even have keyboards and mice?
Click to expand...

Well, thankfully enough, Linux survived that crap. Not sure if anyone is actually using Windows for firewall servers.

That said, IIRC most firewalls do have USB ports, so there's nothing to stop you from plugging a keyboard and mouse in... well, except your own IT fellows which often disable those.

That said, I'm not an IT support guy, I'm a programmer and an ass, so I could well be wrong.
 
HansMustermann said:
Well, thankfully enough, Linux survived that crap. Not sure if anyone is actually using Windows for firewall servers.

That said, IIRC most firewalls do have USB ports, so there's nothing to stop you from plugging a keyboard and mouse in... well, except your own IT fellows which often disable those.

That said, I'm not an IT support guy, I'm a programmer and an ass, so I could well be wrong.
Click to expand...
As an ex-IT support guy, what usually happens is that a tech will RDP into the server from their own workstation. You don't need a mouse and keyboard on the box itself.
 
Wudang said:
But that needs the RDP server to be running. I have almost zero memory of what things that blade controllers can do. All I looked at was start/stop.
Click to expand...
I was only answering the question of whether you need a mouse and keyboard to operate a server. And the answer is no, in most cases, there will not be a mouse and keyboard, and often not even a monitor, so the box would have to be manually hard restarted.
 
HansMustermann said:
Well, thankfully enough, Linux survived that crap. Not sure if anyone is actually using Windows for firewall servers.
Click to expand...

Maybe not entirely: CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes [The Register]
The Register said:
CrowdStrike's now-infamous Falcon Sensor software, which last week led to widespread outages of Windows-powered computers, has also been linked to crashes of Linux machines.

Red Hat in June warned its customers of a problem it described as a "kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process" that impacted some users of Red Hat Enterprise Linux 9.4 after (as the warning suggests) booting on kernel version 5.14.0-427.13.1.el9_4.x86_64.
Click to expand...
 
HansMustermann said:
Yeah, that was my concern too. You can't remote login into a server that bluescreened.
Click to expand...


Servers these days usually have a built in monitor system that seems to be something like a Raspberry PI running a management app. They are pretty easy to use and can restart, change the boot, update firmware etc. They have their own network connection. Once power is on they start up automatically.
 
a_unique_person said:
Servers these days usually have a built in monitor system that seems to be something like a Raspberry PI running a management app. They are pretty easy to use and can restart, change the boot, update firmware etc. They have their own network connection. Once power is on they start up automatically.
Click to expand...

Assuming your datacenter hasn't locked those interfaces down for security reasons.
 
Wudang said:
They should still be available to ops, the question is whether they're available at the right scale to ops. We had a lot of clumps of servers inherited from various functions and not always brought into the fold. Islands of automation was the phrase.
Click to expand...

It's been a while since I worked in a datacenter, but the ones I worked in, those interfaces weren't available to anyone. Most of the time, the feature was disabled or turned off as part of the system baselining process. To use it, you'd have to go into the datacenter, boot a serve to BIOS, enable the feature... then plug a network cable into the corresponding port, plug the other end into an available port on the switch, activate the port on the switch, assign it to a suitable VLAN...

I suspect that nowadays, since datacenters tend to work at scales of entire racks or rows, the usual approach when an instance bluescreens is to simply bring up a new instance of the VM on the nearest available hypervisor. And if the entire hypervisor goes bad, you just move all its VMs to others, and flag the offending hardware for troubleshooting or replacement during the next maintenance cycle. Which is fine, unless all your hypervisors and all your management tools are bluescreening at the same time.
 
Yeah, I was wondering why no one's suing them for damages. Not shareholders --- users, I mean to say, who've suffered losses as a result. Both end-users, as well as businesses who've had to accommodate the end-users, or maybe who did not accommodate end-users and therefore suffered loss of goodwill.

(Or have they done that already, sued them I mean, and I haven't noticed those news items?)
 
Chanakya said:
Yeah, I was wondering why no one's suing them for damages. Not shareholders --- users, I mean to say, who've suffered losses as a result. Both end-users, as well as businesses who've had to accommodate the end-users, or maybe who did not accommodate end-users and therefore suffered loss of goodwill.

(Or have they done that already, sued them I mean, and I haven't noticed those news items?)
Click to expand...

Do you know how to start up a class action lawsuit, when a systemic failure costs you thousands? I doubt many of end users affected know how either.

Businesses are probably evaluating their legal options, letting their insurance underwriter deal with it, pursuing some remedial arrangement directly with Crowdstrike, or some combination.

Also, it's only been two weeks. Folks have plenty of time, still, to consider their options and consult with a lawyer.
 
theprestige said:
Do you know how to start up a class action lawsuit, when a systemic failure costs you thousands? I doubt many of end users affected know how either.

Businesses are probably evaluating their legal options, letting their insurance underwriter deal with it, pursuing some remedial arrangement directly with Crowdstrike, or some combination.

Also, it's only been two weeks. Folks have plenty of time, still, to consider their options and consult with a lawyer.
Click to expand...


Makes sense. Maybe people are actually evaluating their options, as you say, and the lawsuits will begin one of these days.

Incidentally, not just class actions. Large businesses that suffered losses might decide to go it alone, I suppose, large airlines for instance, or any large business with lawyers on their payroll. They might be able to move quicker than class actions might.

Also, would they necessarily sue Crowdstrike? Or might they sue Microsoft, since they'd be their proximate supplier/vendor? And also because they're the one that'll make the juicier target for lawsuits. (Leaving Microsoft to sue Crowdstrike, if they choose.)
 
Chanakya said:
Makes sense. Maybe people are actually evaluating their options, as you say, and the lawsuits will begin one of these days.

Incidentally, not just class actions. Large businesses that suffered losses might decide to go it alone, I suppose, large airlines for instance, or any large business with lawyers on their payroll. They might be able to move quicker than class actions might.

Also, would they necessarily sue Crowdstrike? Or might they sue Microsoft, since they'd be their proximate supplier/vendor? And also because they're the one that'll make the juicier target for lawsuits. (Leaving Microsoft to sue Crowdstrike, if they choose.)
Click to expand...

The proximate supplier/vendor of Crowdstrike products is Crowdstrike.
 
theprestige said:
Assuming your datacenter hasn't locked those interfaces down for security reasons.
Click to expand...

That's a backwards way of looking at it, at least for the systems I was familiar with. You had to actively connect and configure the iLO hardware, and its purpose was to remove the need for anyone to visit the datacentre just to reboot a system or run checks on it.
 
zooterkin said:
That's a backwards way of looking at it, at least for the systems I was familiar with. You had to actively connect and configure the iLO hardware, and its purpose was to remove the need for anyone to visit the datacentre just to reboot a system or run checks on it.
Click to expand...

You've reminded me of the bad old days, when we installed solenoids/servos onto our remote servers so that we could power cycle them by telephone call.

And by remote, I mean equipment in the desert or on the rocket range.
 
theprestige said:
The proximate supplier/vendor of Crowdstrike products is Crowdstrike.
Click to expand...


Happened to be speaking with my tech guy about something, and we got to gabbing about the outage, and I guess I said something like what I'd said there about Microsoft's liability. At which point he got it, the point about which I was mistaken; and, instead of clever and cryptic, he opted for clear, and in a sentence or two explained it to me.

So yeah, right you are, I get that, now.
 
You must log in or register to reply here.

Back
Top Bottom