• Quick note - the problem with Youtube videos not embedding on the forum appears to have been fixed, thanks to ZiprHead. If you do still see problems let me know.

Catastrophic Internet Failure

rjh01 said:
Problems with this.
1. How does a person boot up in safe mode?
2. How does a person delete the bad file?
3. What is the path name of the bad file?
4. If I have only one computer how do I Google the above questions when my computer is not working? This question assumes that the information is on the Internet and I have the ability to find it.
5. If I am the IT person at work I may have many computers, some in remote locations. That will keep me employed for a long time.
Click to expand...

Look up safe mode. I've booted it in the past long ago when i had a bad driver. Easy peasy.

As for virtual machines:
Load the vs boot filesystem. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory. Locate the file matching “C-00000291*. sys” and delete it. Detach the volume from the new virtual server.
 
HansMustermann said:
Like, seriously, this is what Y2K wished it were.
Click to expand...

Not even close. No machines were bricked, no permanent damage was done to the software or hardware. The fix itself is trivially easy to do and instant in efficaciousness: just delete the affected file and let the OS boot up. It's just taking a long time to deploy because it can't be done remotely or in mass, it has to be done in person at the affected machines one at a time.

It's a software problem, but a logistics disaster.

Mine wasn't the only company to discover the real issue was fumbling with communication and coordination in order to deploy the right people to the right places in the right numbers at the right times. (Also they should never have let Betsy have access to delete things from a command line; bless her heart, she tries her best, but she handles computers like a chimp wielding a flamethrower.)
 
marting said:
Look up safe mode. I've booted it in the past long ago when i had a bad driver. Easy peasy.

As for virtual machines:
Load the vs boot filesystem. Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory. Locate the file matching “C-00000291*. sys” and delete it. Detach the volume from the new virtual server.
Click to expand...

How do I look up safe mode when my computer is not working? How do I even know that this is part of the solution?

The real solution is for the computer to realize that the update did not work and then undo the update automatically. Then ensure that it does not do that update again. Does not matter what the update is.
 
rjh01 said:
How do I look up safe mode when my computer is not working? How do I even know that this is part of the solution?

The real solution is for the computer to realize that the update did not work and then undo the update automatically. Then ensure that it does not do that update again. Does not matter what the update is.
Click to expand...

Use your smartphone :duck:
 
Nice video about the crash: https://www.youtube.com/watch?v=wAzEJxOo1ts

New things (for me at least): the driver itself was not updated. The driver loaded other modules, which were not tested by MS nor signed, and the updated module was one of these.
That does not explain why it was full of zeros. And it also means the "last know configuration" should not work, as the driver itself was unchanged compared to last known configuration.
 
marting said:
Yep. Croaked at the git go. So no damage. Just need to boot up in safe mode and remove the bad file. A major screw up but not malware injection.
Click to expand...


Unfortunately, the laptops issued to NYSDOT employees (and I assume most other NY employees) have security that prevents users from booting in safe mode, so each user has to personally be helped by IT.
 
It seems it's not driver rollback after all, though it's more like the situation is less clear than more.

First, as the driver itself was not changed, rollback wouldn't fix it.

Here people claim it's a race condition. If networking takes too long to start, the crowdstrike soft will fail to check for updates, and will use the faulty module. If the networking starts quick enough though, crowdstrike will check for updates first, download a fix, and will not run the fault module.

https://www.reddit.com/r/technology...dstrike_fixes_start_at_reboot_up_to_15_times/

Many places also state reboot "p to" 15 times. And it's also quite clear it doesn't help everybody. If the issue was solvable by driver rollback, it would help everybody.
So it's just "try rebooting again and again, it might actually fix itself .. or not".
 
rjh01 said:
How do I look up safe mode when my computer is not working? How do I even know that this is part of the solution?
Click to expand...
You take it to your friendly neighborhood computer repair shop and let them sort it out. There's no shame in not being an innate expert in fixing a modern marvel of advanced technology.

The real solution is for the computer to realize that the update did not work and then undo the update automatically. Then ensure that it does not do that update again. Does not matter what the update is.
Click to expand...
The computer is not as smart as we wish it were, or plan it to be at a later date. It's still on us to make up the difference, with basic human ingenuity and adaptability.
 
Well, maybe it should, and maybe it shouldn't, but it turns out this wasn't an update that Windows didn't do itself, and possibly even know about. ClownStroke downloaded, installed and executed it on its own. As far as Windows was concerned, the last time it had a driver update from them was when last the CSagent.sys driver itself was updated. The C-*.sys files are, as far as Windows knows, just data files that CSagent.sys works with.
 
jeremyp said:
I think that shows a distinct lack of imagination. Just off the top of my head: a cyber attack targeting DNS on Linux would very likely make large swathes of the Internet unusable.
Click to expand...

You can imagine the havoc at the software lab I worked at when some numpty misread the instructions to configure TCP/IP and set his PC with the main DNS server IP address. Changes were made. This was back in the '90's.
 
jeremyp said:
I think that shows a distinct lack of imagination. Just off the top of my head: a cyber attack targeting DNS on Linux would very likely make large swathes of the Internet unusable.
Click to expand...

Ironically, the title of the thread is dead wrong. There was no internet failure here. The networks were fine. Ecommerce and social media websites stayed up and accessible. Maps and streaming services were readily available.

What actually got affected by this one were the vast amount of Not-Internet computer systems that power corporations large and small. This was pretty much the opposite of an Internet failure.
 
theprestige said:
Ironically, the title of the thread is dead wrong. There was no internet failure here. The networks were fine. Ecommerce and social media websites stayed up and accessible. Maps and streaming services were readily available.

What actually got affected by this one were the vast amount of Not-Internet computer systems that power corporations large and small. This was pretty much the opposite of an Internet failure.
Click to expand...

What actually failed was a piece of software designed to counter malware attacks from bad actors on the Internet and it failed because of an automatic update that is only possible because of the Internet.

Whilst I agree that this is not a failure of the Internet nor any part of its infrastructure (unlike the theoretical DNS attack I thought up), it is not strictly the opposite of an Internet failure.
 
jeremyp said:
I think that shows a distinct lack of imagination. Just off the top of my head: a cyber attack targeting DNS on Linux would very likely make large swathes of the Internet unusable.
Click to expand...

Well, I think that while you are correct, he kind of gets the saving grace that IMHO no attack SO FAR has caused as much damage. Some, like the infamous I Love You back then came close in term of blast radius, but at least they tended to leave the servers alone and the machines were otherwise usable.
 
Seems like a fantastic example of "never put all your eggs in one basket." How to mitigate this one? Use more than one OS, use more than one endpoint protection package, set some of your systems to delayed updates. Any one of which would have limited this to an annoyance at most.
 
You must log in or register to reply here.

Back
Top Bottom