UKC

Microsoft has gone down. No 360. Total chaos in the USA and elsewhere. I've shut down the computer and gone running instead!

To be fair to Microsoft, the issue seems to have been caused by Crowdstrike. Microsoft just seemed like the ones at fault initially because the computers Crowdstrike was running on were Windows ones.

Biggest global IT outage probably since WannaCry? Hopefully they can stop the update before it rolls out further across the world. 

Huge queues at a number of airports across the UK….or just another normal day if you are Birmingham Airport 

Luckily UKC hasn't gone down! That's the real crisis averted.

I notice the statement from Crowdstrike didn't include an apology!

The Office 365 outage that @apache is referring to was caused by a problem on Microsoft's Azure cloud platform †. Nothing to do with the Cloudstrike global clusterf*k (which was itself nothing to do with Microsoft). Just a coincidence they happened on the same day.

† https://www.bleepingcomputer.com/news/microsoft/major-microsoft-365-outage-...

A case study for the ages in the making. How on earth did this get through the various testing stages, and on Windows of all platforms not some esoteric random syatem, and only 8% hit on Crowdstrike shares at the time of writing.

Just bonkers

Yeah, you're right. It was pretty confusing this morning with the news reports struggling to differentiate between those two different issues. The Crowdstrike one does seem to have accounted for the majority of the disruption worldwide. As far as I can make out, the Azure issue largely happened in the night UK time and was resolved before many people here would have been working. Plus lots of services for UK folk wouldn't have been hosted on US servers anyway.

It is however Microsoft’s fault that a third party application is able to brick their operating system in a way that can’t be tribally recovered…

I'm not so sure. Third party security apps, by their nature, require privileged systems access. As such, if corporations aren't testing updates to those apps before allowing them into the production environment (as they would with a windows security patch) then they have to take a fair chunk of the blame.

There’s privilege and there’s privilege.

No third party security has ring zero privilege, and you shouldn’t be able to BSOD from outside ring zero.

I agree, but the idea Microsoft have no culpability doesn’t wash with me.  

I don't know whether I'd hold them all that responsible, I'm dubious whether any normal operating system would be impervious to bricking by sufficiently broken software. And at least most devices still seemed to boot in safe mode for a fix.

Whoever is to blame for this it should be a wake-up call. Just wait until the electricity goes off. 

A key point of modern operating systems is to restrict the mischief one process can cause to others or to the kernel.   

Something in the privilege and security model is broken if this can happen.

Not exactly a revelation when it comes to Windows, so much as an illustration.

Yes/no

Mistakes happen, "it" happens. That's why in the old days patching was done midweek for Dev, the weekend for UAT, following weekend for Prod and then only after Prod was OK for a week or more would DR be patched (that way there was always a known-good DR to fail back to)

Why did everything get patched one Thursday evening, that's the far bigger concern

Nonsense man, look at bashdoor, XZ Utils and all the rest, no OS is invulnerable. It's not a Linux versus Windows versus whatever contest (as much as I enjoy kicking Windows)

Windows may not help , but idiocy combined with laziness is the underlying problem. People no longer care about getting the basics right. DevOps is brilliant but the ops part is increasingly getting neglected and this is the payback 

You're right, many questions need some quick answers.

I work in cybersecurity and also previously worked in automated SW testing.

I was also a previous investor in Crowdstrike and nearly ended up working for them too.

My employer has just emailed me saying we have issues too - we use Crowdstrike on our endpoints - but I've been thorning myself to buggery on an overgrown pyracantha today so won't know if my machine is affected until Monday morning. 

Bricking is by design, the software (Crowd strike Falcon Sensor) detects issues (a bit like intrusion detection, antivirus, etc) but goes a step further to "brick" well.. isolate.. the affected host if there's a serious problem it isolates it from your network although still allows communication between the host and Crowd strike cloud.

This means that an e.g. compromised/infected/vulnerable box can be taken off network. Necessarily it needs to be tightly integrated so that malware can't just disable the Falcon Sensor service. It's a huge advance in security... When it works that is.

But now you have a cloud service that can "brick" your servers (unless you restart in safe mode and fix that). You're trusting a someone else somewhere else not to brick your servers accidentally... Ooops!!!

There are reports of similar incidents impacting some Linux systems pretty recently.

Unless you go down the ultra locked down route, which has its own problems, if something has root access then it can cause mayhem.

I didn’t say any of the others were perfect.

Indeed

Agree. Far to much security and protection is bolted on afterthought, where as a project should start with the secure platform and then add features.

… and yet there has never been a time when it’s been more important.

I still don't understand why ClownStrike didn't do at least a staged or a rolling update of their faulty definitions file, apart from some proper internal testing. Could have at least prevented some of the mess.

Oh well, they did in fact do a rolling update – it rolled all the way from Australia, Asia, Europe and USA by their respective time zones

Any climbers in IT having their weekend plans ruined by this Friday clusterf*ck, I'm sending you a virtual beer...

Crowdstrike installs a device driver because, as an endpoint security product, it needs to do "stuff" with kernel-level privileges. Nothing particularly unusual about that. But what seems to have happened is that (a) businesses deployed Cloudstrike extensively on business critical systems, (b) Cloudstrike updates automatically by downloading malware definitions over the net, (c) Crowdstrike updates every installation at the same time (bad, no staging), and last night they pushed-out a malware definition "channel file" that crashes the driver and takes down the os. Reboot and it crashes again. And again...

I'd argue that this is very little to do with Windows and a lot to do with end-user orgs exposing themselves to an uncontrolled risk (single vendor auto-updating software), and to Crowdstrike's shoddy testing and all-or-nothing update processes.

Linux isn't immune. Over on hackernews a guy is describing† how Crowdstrike took down their (presumably large) fleet of Debian-based servers recently. But because Crowdstrike  is not widely deployed on Linux, the effect wss contained. So this isn't a Windows problem per-se.

Anyway, they crashed the world. Quite a thing to do. Reminds me of SqlSlammer back in the day.

† https://news.ycombinator.com/item?id=41005936

Why does that need a device driver than appropriate ACLs to give the required privileges?  

> I'd argue that this is very little to do with Windows and a lot to do with 

I agree with all the other things you list, but...  A well designed OS wouldn’t require a device driver parsing content from the internet.  There would be a user process with appropriate access to disable network interface etc.  I don’t know if windows lacks that or if Crowdstrike are just not very capable.

Didn’t say it was.

I wrote a windows device driver once for a piece of hardware I made and sold. The Mac OS X (as it was then known) didn’t need a device driver because after setting the fine grained permissions I could access everything from userland. Forcing things in to device drivers due to poor ACLs and userland APIs opens the door to this stuff.  If I ever write another windows driver it’ll be too soon.

Do you actually know how/what this defect did, or are you just basing this on conjecture/bias?  

I’m basing my last reply to Andy Johnson based on what they said.

Otherwise I’m basing it on the idea that automatically updating third party stuff shouldn’t be able to BSOD.  The replies I’ve got amount to accepting this status quo because reasons.  I find that really odd.  

I do have a bias about windows, and both past security incidents and my experience of writing a device driver for it factor strongly. I’m not arguing other OSs are immune or perfect but that the response shouldn’t be just blame the provider but the OS for not being suitably robust.  

I think it’s quite reasonable to conjecture they too much code was privileged given what’s its done.

Because the driver (presumably) need to do things with kernel-level privileges that arent possible in userland, and there arent ACLs for. Idk what but maybe packer filtering or process injection or the like. We're talking about an endpoint security product here, so...

Having written (only) one windows device driver myself, I find it hard to believe the Crowdstrike devs would choose to build one unless they needed to. They're a pain to debug.

Didnt say you did

(Something like FUSE? Ive done veey little development on Macs.)

I take your point. But that model gives the entire process  (albeit constrained by the ACLs) elevated privileges. At least the Linux/Windows approach partitions the privileged code in the driver.

(Just read this)

Again I think you make good points. If this had been an accounting application downloading new tax rules then there wouldnt be a problem. But the combination of it being a security product requiring privileged access, and frequent updates due to emerging threats - together with poor data sanitation and parsing by Crowdstrike, is imo why we have this mess.

As a developer I'm extremely glad I'm sat under a tree in the garden with a beer right now, rather than having to help sort this out.

Exactly.

Exactly.

Quite possibly.  I have some stuff of my own doing packet filtering.  It runs in userland on OpenBSD where I have /dev/pf

This is part of my point, bare with me…

The possible horrors of driver code with access to do injection downloading content off the Internet would surely put any sane firm off…. Right…

Exactly… and if Windows doesn’t provide safe, segmented, ACLd access to things security plugins need, it’s forcing the situation where stuff like this happens.

I absolutely hated doing it.  What I’d give to having modern static analysis tools.  Part of the problem was mistakes crashing the machine….  But many/most device drivers don’t need the kind of performance that requires an environment where that can crash the machine.  But that was the way it was.

This was USB 2 endpoints for controlling an external device and receiving a lot of data from it.  The endpoints are available with a userland API on OS X vs requiring driver code in Windows 2k (no idea on current situation for windows).  FUSE is a similar approach I suppose as there’s a generic kernel driver that presents an API to userland, although the kernel part wasn’t robust enough to bad userland requests when last I used it.

Problem is it forces a lot of unrelated code in to the driver as well.  If the code is trusted, being in the driver is surely worse than userland with adequate APIs and ACLed? 

The whole bloody field is terrifyingly complex and far too trust based and despite the critical importance of computers to so much stuff, far to much security is reactive.  This has been massive and it was an accident.  Some states are preparing for deliberate action and many doors are wide open.

Edit to reply to your second post:

Oh I’m clear they’ve effed up big time, but there’s 29 years of history going back to NT 3.51 as to how we got here, why such products are needed and what’s apparently forced them in to a catastrophically unreliable design.

I don’t get much time to write lower level code these days so it was quite fun to get in to /dev/pf recently for some operational stuff, but I’d agree I’m happy sat in my garden with a drink.

this is all rather mild compared to what would have happened if we hadn't worked so hard to pre-fix the Y2K bugs in advance.

Allegedly it was "all a fuss about nothing" according to some moronic lordly Brexiteer numbskulls!

It would have taken years to get everything working again with inability to process dates emebedded in millions of lines of legacy code, not all of it in high level languages (we had some low level routines called from PL/1 that couldn't do Y2K as it turned out.) Flights would have been grounded indefinitely from our "favourite airline".

You've still got the wrong end of the stick. This is software, agent or whatever you want to call it that is designed to be really nailed down so malicious stuff can't interfere with it.

It's nothing to do with NT3.51 heritage or ordinary user mode stuff or what you can do in /dev/pf. It's some horrible sh!!ty software that lurks deeply in crevices that are hard to get to.. by design, because of what it does. If it was "nice" and non privileged, malicious code could side step it or block it or hobble it somehow.

I'm not defending the code or the company or Windows (shudder...) but it's not written nicely how you might do it, by design because of what it does. If you think otherwise you're still missing the point

No, I haven’t.

It sounds an awful lot like code was running some of its tasks with a higher privilege than it needed to do them.

That’s the point.  The windows heritage doesn’t expose a load of stuff through /dev/xx (which isn’t a windows thing BTW).  That forces more stuff deeper into privilege than it should be.

Disagree.  With proper cryptographic verification of the payloads being downloaded, much of the code can live in less trusted layers with crypto validation but not processing of large datasets done in the kernel space.  You’re reaching for excuses when there are known ways of handling untrusted layers.

Only because the OS has a flawed system that doesn’t do sufficient to protect processes from each other.  This product is trying to polish a turd, and when you do that you end up covered in shit.

You are conflating asking for protection with asking for privilege.  If as speculated by AJ and me the designers were forced in to that conflation, that tells you the problem right there.  Asking for privilege should be independent by design of being protected.  Bad designed forced by the history of Windows NT.

No, I don’t think I am missing the point, at all.  A bad design forcing a conflation of protection and privilege.

“Just shove it all in the kernel, downloads and all” isn’t smart.  I’m surprised your taking such a defeatist view that it has to be this way, or that an OS should be so insecure people are willing to install third party divers to manage it.  We should have higher aspirations here.

I'm in part playing devil's advocate here and I'm not saying you're "wrong" per se, as much as I don't agree with your perspective. It just seems a bit glass half empty. I attribute most of the mess to widespread operational/management shortcuts as much as the underlying  design.

As you mention, there's the saying: "you can't polish a turd" but the retort is "but you can roll it in glitter". As an engineer I do find a lot of my time is spent glitter rolling people's stuff. Maybe I'm glass half full from too much glitter rolling

I look at it that way because the level is dropping.   

For sure, it’s taken a lot of bad management decisions to get to the point a third party kernel driver downloading updates from the net is seen as a way to boost the security of its host OS.

Plenty of blame to go around.  But my point that kicked this all off was that Microsoft have some of that share.  An OS so insecure it needs this kind of kernel driver?  An architecture that doesn’t orthogonalise protection and privilege?    The windows NT line has had 29 years of those management decisions you pin the blame on baked in to it now.  Mac Os is going the same way lately IMO.

The number of compromised open source libraries recently - and the OpenSSH vulnerability show it’s not great away from Microsoft either.

The recent ransomeware attack on the NHS was treated as almost inevitable.  I saw a US study in to how much mortality rates rise after such an attack.  It happens so often they can do statistics on the consequences.

I’m agog at how normalised this has become and how low people’s expectations are for OS security. It’s not healthy.

Not really. Or are you blaming Linux and Apple as well? Both of which Crowdstrike provide software for? Oddly enough in the tech forums been several comments about how similar things have happened to Linux in the past its just its hit one or two particular distro versions vs across the board.

The problem is a properly secure OS is also one which is a complete arse to use day to day.  You could go for sandboxing every app but then that would have a few awkward repercussions.

MS have done a reasonable job in regaining ground. Its why standard AV and firewall companies are struggling now with Defender doing the job. However enterprise in particular tend to want bells and whistles.

Not for the crash of windows computers, no.  That would seem pretty dumb, would it not?

It’s hard to blame “Linux” given there isn’t a single clear entity behind its many forms, but I also haven’t once held Linux up as a positive counter example…

The gulf between where we are with Windows, macOS and Linux and a properly security hardened OS is almost immeasurable,  there’s a big juicy middle ground where security is baked in from the start, not bolted on in a half arsed way with third part programs and now kernels drivers/kexts.  

Microsoft are doing a reasonable job at beating AV and firewall companies at fixing the mess Microsoft enabled.

If MS were doing a reasonable job all round on the security of their OS, they would be no mess to fix.

To be fair, any enterprise that settles on a kernel driver that auto updates as part of its “solution” is as much to blame…

It’s 2024 and a whole lot of kernel code is still written in a pair of languages that don’t bounds check anything.  I just can’t comprehend why so many people’s expectations are so low. 

Where can I find a detailed description of what exactly Crowdstrike's 'Falcon' product does? I'm a time-served Unix sysadmin, but I haven't yet got any handle on what's going on.

If it's a kernel module, it will have.

Yes. See the later posts between myself and Andy Johnson.  I was previously wrongly assuming nobody would be insane enough to make a kernel module that downloaded and parses lists from the Internet or auto updates itself….  Because that would be really dumb…

Glad I've got some cash under the bed

Weird that you should get a downvote for this!

1st thing yesterday morning Azure service health page was reporting that a network device misconfiguration in US Central (aka DNS change) had "spread" to other regions in the US and was taking Azure hosts offline, impacting M365 and other services. MS managed to stop the spread so UK hosted services weren't affected. As a lot of multinationals use US hosting by default - some UK and worldwide users were impacted.

Not sure what the status of this incident is.

Just signed in again and there is no mention of the Azure outage - all it mentions is the Crowdstrike Falcon fiasco - no mention in history of the US Central outage.

The US is more affected than anywhere else - how much of this is down to the Azure outage and how much due to Crowdstrike?

This is very surreal. I'm on Lundy, with limited Internet and very little sense that there is a real world across the water.

Yeah, agree, this is what seems really problematic. Critical systems exist and get updated, updates sometimes fritz servers, you mitigate against this by having redundancy and staged updates. You don't put yourself in a position where a risky update - which is to say any update, really - gets applied to more than one node until you've verified that it works on the first one. I don't see why AV should be different in this respect.

To BSOD one server might be considered misfortune, to BSOD all of them is definitely carelessness. 

Microsoft blame the EU competition policy for the outage !!??

https://www.euronews.com/next/2024/07/22/microsoft-says-eu-to-blame-for-the...

Yes. They were legislatively forced to allow ring-0 access to AV makers.

Edit: There is an interesting chicken and egg situation here. The AV makers want access so that they can provide a solution that operates at a low enough level to detect rootkits etc. Allowing access makes it easier for root kits to even exist.

Well if we want fair markets the likes of Microsoft should be able to cope. It's hardly like they could be trusted to never release partially operating software that was easy to hack 

If MS, or any OS maker for that matter, were to release a fully secure OS, nothing but boot drivers would run in kernel mode. 

Crowdstrike managed to brick some Linux servers a few months ago with a similar mishap.

Edit: IMO, that should teach companies how good an idea it is to trust ex-McAffee people… 😉

Personally I think that my first few days working at Crowdstrike didn't go too badly, after all, everyone makes a few mistakes.....................

Vulnerability is what Microsoft made its name with.  Boot secta infectas were possible by Microsoft design from the very beginning.  No reason why the tradition shouldn't continue.

I've always liked the idea of an encrypted CPU instruction set with a unique key for every CPU

A bit like no 2 brains being wired the same way

Were you also in charge of the $10 ubereats giftcard for their partners who have been mildly inconvienced?

https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-apology-gift-card...

Got to wonder who thought that was a good idea.