{"id":225420,"date":"2024-07-23T06:22:34","date_gmt":"2024-07-23T13:22:34","guid":{"rendered":"https:\/\/zpesystems.com\/?p=225420"},"modified":"2025-02-06T08:55:54","modified_gmt":"2025-02-06T16:55:54","slug":"the-crowdstrike-outage-how-to-recover-fast-and-avoid-the-next-outage","status":"publish","type":"post","link":"https:\/\/zpesystems.com\/the-crowdstrike-outage-how-to-recover-fast-and-avoid-the-next-outage\/","title":{"rendered":"The CrowdStrike Outage: How to Recover Fast and Avoid the Next Outage"},"content":{"rendered":"

[et_pb_section fb_built=”1″ _builder_version=”4.17.6″ _module_preset=”default” custom_margin=”0px||||false|false” custom_padding=”0px||||false|false” da_disable_devices=”off|off|off” global_colors_info=”{}” da_is_popup=”off” da_exit_intent=”off” da_has_close=”on” da_alt_close=”off” da_dark_close=”off” da_not_modal=”on” da_is_singular=”off” da_with_loader=”off” da_has_shadow=”on”][et_pb_row _builder_version=”4.17.6″ _module_preset=”default” width=”100%” global_colors_info=”{}”][et_pb_column type=”4_4″ _builder_version=”4.17.6″ _module_preset=”default” global_colors_info=”{}”][et_pb_image src=”https:\/\/zpesystems.com\/wp-content\/uploads\/2024\/07\/CrowdStrike-Outage-BSOD.jpg” alt=”CrowdStrike Outage BSOD” title_text=”CrowdStrike Outage BSOD” _builder_version=”4.27.0″ _module_preset=”default” global_colors_info=”{}”][\/et_pb_image][et_pb_text admin_label=”Text” _builder_version=”4.27.4″ _module_preset=”default” hover_enabled=”0″ global_colors_info=”{}” sticky_enabled=”0″]<\/p>\n

 <\/p>\n

On July 19, 2024,<\/strong> CrowdStrike, a leading cybersecurity firm renowned for its advanced endpoint protection and threat intelligence solutions, <\/span>experienced a significant outage<\/span><\/a> that disrupted operations for many of its clients. This outage, triggered by a software upgrade, resulted in crashes for Windows PCs, creating a wave of operational challenges for banks, airports, enterprises, and organizations worldwide. This blog post explores what transpired during this incident, what caused the outage, and the broader implications for the cybersecurity industry.<\/span><\/p>\n

What happened?<\/h2>\n

The incident began on the morning of July 19, 2024, when numerous CrowdStrike customers started reporting issues with their Windows PCs. Users experienced the BSOD (blue screen of death), which is when Windows crashes and renders devices unusable. As the day went on, it became evident that the problem was widespread and directly linked to a recent software upgrade deployed by CrowdStrike.<\/span><\/p>\n

Timeline of Events<\/b><\/p>\n

    \n
  1. Initial Reports: Early in the day, airports, hospitals, and critical infrastructure operators began experiencing unexplained crashes on their Windows PCs. The issue was quickly reported to CrowdStrike’s support team.<\/span><\/li>\n
  2. Incident Acknowledgement: CrowdStrike acknowledged the issue via their social media channels and direct communications with affected clients, confirming that they were investigating the cause of the crashes.<\/span><\/li>\n
  3. Root Cause Analysis: CrowdStrike’s engineering team worked diligently to identify the root cause of the problem. They soon determined that a software upgrade released the previous night was responsible for the crashes.<\/li>\n
  4. Mitigation Efforts: Upon isolating the faulty software update, CrowdStrike issued guidance<\/a> on how to roll back the update and provided patches to fix the issue.<\/li>\n<\/ol>\n

    What caused the CrowdStrike outage?<\/h2>\n

    The root cause of the outage was a software upgrade intended to enhance the functionality and security of CrowdStrike’s Falcon sensor endpoint protection platform. However, this upgrade contained a bug that conflicted with certain configurations of Windows PCs, leading to system crashes. Several factors contributed to the incident:<\/span><\/p>\n

      \n
    1. Insufficient Testing: The software update did not undergo adequate testing across all possible configurations of Windows PCs. This oversight meant that the bug was not detected before the update was deployed to customers.<\/span><\/li>\n
    2. Complex Interdependencies: The incident highlights the complex interdependencies between software components and operating systems. Even minor changes can have unforeseen impacts on system stability.<\/li>\n
    3. Rapid Deployment: In the cybersecurity industry, quick responses to emerging threats are crucial. However, the pressure to deploy updates rapidly can sometimes lead to insufficient testing and quality assurance processes.<\/li>\n<\/ol>\n

      We need to remember one important fact: whether software is written by humans or AI, there will be mistakes in coding and testing. When an issue slips through the cracks, the customer lab is the last resort to catch it. Usually, this can be done with a controlled rollout, where the IT team first upgrades their lab equipment, performs further testing, puts in place a rollback plan, and pushes the update to a less critical site. But in a cloud-connected SaaS world, the customer is no longer in control. That\u2019s why they sign waivers stating that if such an incident occurs, the company that caused the problem is not liable. Experts are saying the only way to address this challenge is to have an infrastructure that\u2019s designed, deployed, and operated for resilience. We discuss this architecture further down in this article.<\/span><\/p>\n

      How to recover from the CrowdStrike outage<\/h2>\n

      CrowdStrike gives two options for recovering:<\/strong><\/p>\n