A combination of factors caused the Falcon EDR sensor to crash, resulting in the massive outage that affected more than 8.5 million Windows systems in July, CrowdStrike said in a root cause analysis of the incident released last week. At the same time, CrowdStrike CTO George Kurtz and President Michael Sentonas were in Las Vegas with a public my fault.

CrowdStrike documented in his root cause analysis that there was a mismatch between input validated by a Content Validator and that provided to a Content Interpreter, as well as an out-of-bounds reach issue in the Content Interpreter. And there was an issue with the way the update was tested.

“Sensors receiving the new version of Channel File 291 containing the problematic content were exposed to a latent out-of-bounds read issue in the Content Interpreter. The next IPC notification from the operating system evaluated the new IPC Template Instances, specifying a comparison to the 21st input value. The Content Interpreter was only expecting 20 values,” CrowdStrike said. “Therefore, the attempt to access the 21st value produced an out-of-bounds memory read past the end of the input data array and resulted in a system crash.”

While CrowdStrike says this exact scenario will not recur, the company is making changes to its process and mitigation measures to “ensure further enhanced resilience,” the company said. CrowdStrike has also engaged two software security vendors to conduct a comprehensive review of the Falcon sensor code for security and quality assurance, and an independent review of the end-to-end quality process from development to deployment is underway.

Owning his mistakes

During the Innovators & Investors Summit at the Black Hat USA conference in Las Vegas, moderator Chenxi Wang opened her panel by asking CrowdStrike CTO George Kurtz: “What happened?” Kurtz apologized to the room, a move that was well-received by the audience, and noted that the company had released the results of its root cause analysis.

The company acknowledged its failures again a few days later, when CrowdStrike president Michael Sentonas attended the DEF CON hacker convention on Saturday to accept the 2024 Pwnie Award for Most Epic Fail. The Pwnie Awards recognize the most notable achievements and failures in cybersecurity of the past year. The Most Epic Fail category is for a “spectacularly epic fail — the kind of fail that brings the entire infosec industry to its knees,” according to the Description of the Pwnie Awards.

The Pwnie Awards said in July that the massive global outage CrowdStrike is automatically the winnerThe global impact of the outage was highlighted by the fact that CrowdStrike was presented with a two-tiered trophy instead of the traditional small pony-shaped trophies awarded to winners in other categories. Sentonas said the trophy will be displayed at the company’s headquarters in Austin, Texas, as a reminder to staff that “this kind of thing can’t happen.”

“Definitely not a prize to be proud of,” Sentonas said in his acceptance speech. “I think the team was surprised when I said straight away that I was going to get it. We did this horribly wrong, we’ve said that a number of times. It’s super important to acknowledge when you do things right, it’s super important to acknowledge when you do things horribly wrong, which we did in this case.”