I heard from a friend who knows someone in Crowdstrike that this bug had been sitting there in the kernel driver for years before being hit. Turns out that the flawed data was added in a post-processing step of the configuration update, which is after it's been tested internally but before it's copied to their update servers.
Their test setup was fine for the update data itself, it's just that they didn't catch it before it was sent out to production because they were testing the wrong thing. Oops.
A test order randomizer could have caused a meta-QA-test to write the test-artifact-file over the actual build artifact file. If so, these meta tests were not supposed to run at this stage, but laziness prevails, and the same test runner looks to have been used for both the main and the meta QA tests.
> Update as of 10:30 UTC on 19 July 2024: We have received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines.
> We've received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.
I think I'm well past 15 on my laptop now. My desktop was unaffected, but I reach over and click restart every once in a while to see if something different happens.
I'm not the one to bash Microsoft "just because", but this "up to 15 reboots" thingy reminds me that MS most likely doesn't know what's going on in their own damn OS anymore.
Related picture - not long ago such a message would've been completely unacceptable, yet here we are.
Well, don't use windows but if you can bypass Bitlocker to modify a kernel extension it seems like you could just bypass Bitlocker to install a keylogger / whatever?
With the utility of in a shared computer environment (say school library) you can get the administrative password with enough time.
I heard from a friend who knows someone in Crowdstrike that this bug had been sitting there in the kernel driver for years before being hit. Turns out that the flawed data was added in a post-processing step of the configuration update, which is after it's been tested internally but before it's copied to their update servers.
Their test setup was fine for the update data itself, it's just that they didn't catch it before it was sent out to production because they were testing the wrong thing. Oops.
"post-processing step" aka regex? :)
HN comments from a green username citing a friend who knows someone aren't the greatest source, but this sounds so believeable
A test order randomizer could have caused a meta-QA-test to write the test-artifact-file over the actual build artifact file. If so, these meta tests were not supposed to run at this stage, but laziness prevails, and the same test runner looks to have been used for both the main and the meta QA tests.
Reboot harder.
> Update as of 10:30 UTC on 19 July 2024: We have received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines.
> We've received feedback from customers that several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage.
https://status.cloud.microsoft/
Any indication that this might work for physical computers?
My physical workstation (a Precision laptop) was affected and a single reboot fixed it. Most of my colleagues weren’t as lucky.
I think I'm well past 15 on my laptop now. My desktop was unaffected, but I reach over and click restart every once in a while to see if something different happens.
Sounds like a race condition that some systems hot and others don't
Worked for me. Working for a few or my colleagues. Reboot again and again.
It's amazing how "Did ye try tairnin' it off and on again?" is still the best first-line advice.
Update to: "Did you try turning it off and on again...15 times?"
Important advice since normally I'd resort to percussive maintenance way before getting to 15.
I'm not the one to bash Microsoft "just because", but this "up to 15 reboots" thingy reminds me that MS most likely doesn't know what's going on in their own damn OS anymore.
Related picture - not long ago such a message would've been completely unacceptable, yet here we are.
https://i.imgur.com/4rQLcb7.png
sounds like a race condition that you have a fair chance of recreating after 15 reboots?
Bitlocker instructions in a reply, direct link if you don't have a Twitter account: https://twitter.com/Syndikalist/status/1814281141265846772
Well, don't use windows but if you can bypass Bitlocker to modify a kernel extension it seems like you could just bypass Bitlocker to install a keylogger / whatever?
With the utility of in a shared computer environment (say school library) you can get the administrative password with enough time.
To save everyone going to that abomination of a site:
vx-underground
@vxunderground
How to fix the Crowdstrike thing:
1. Boot Windows into safe mode
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Delete C-00000291*.sys
4. Repeat for every host in your enterprise network including remote workers
5. If you're using BitLocker jump off a bridge
Or if you actually kept track of these things: pull out the bitlocker recovery key, even if it's on an old USB at the back of a drawer. Ask How I Know