But the company depends on deep access to millions of computers to defend them against new attacks, and instructions sent by CrowdStrike to those machines running Microsoft’s Windows operating system overnight rendered them unusable Friday morning.
As banking, airline and 911 emergency call systems struggled to recover, CrowdStrike apologized and blamed the failure on an error rather than a technical glitch. a hacking attack on its internal systems.
“This was not a cyberattack,” CrowdStrike said on its blog. The Austin-based company said it identified the issue and provided a solution to customers to help their employees get back to work.
But the breach was so large and its consequences so far-reaching that not all security experts were convinced it was simply human error. CrowdStrike has grown rapidly over the past year and just last month joined the S&P 500 index of the world’s largest publicly traded companies. But the company has made enemies around the world by exposing hacking operations like the one by Russian intelligence that stole emails from the Democratic National Committee and Hillary Clinton’s campaign manager in 2016.
“I doubt it was an accident. There are too many gaps,” said Matthew Hickey, founder of training company Hacker House. He said the offending file contained random data, was not digitally signed and was not properly tested.
A U.S. federal official speaking on condition of anonymity to discuss national security matters said there was no evidence of sabotage or foreign involvement.
TO CATCH UP
Stories to keep you informed
Some analysts said they were waiting to learn more about CrowdStrike and that the complexity of cutting-edge anti-hacking defenses made them dangerously fragile.
Jake Williams, a former National Security Agency hacker, said that “endpoint detection” products like CrowdStrike’s Falcon tool often send not only updated credentials to block malware, but also active lines of code to thwart more complex attack scenarios. He added that it’s possible that CrowdStrike’s systems for testing the code before rolling it out everywhere weren’t “diverse enough” to catch the error.
While computer network outages are not unusual, experts were stunned Friday that one company’s error spread across so many systems.
“We’ve never seen a cascading failure like this,” said Chuck Herrin, an executive at digital security firm F5 Inc.
THE the extent of technological failures around the world friday exposed the risks inherent in the type of security software that many consider essential for businesses ransomware and other devastating hacks.
To be effective, these programs need to be able to see everything that happens on a machine. But that access can make their failure catastrophic, as it did on Friday, and the solution the company then provided was complex: Many organizations had to manually reboot each machine one by one and delete the faulty update file.
This privileged access also makes security programs a prime target for spies and ordinary hackers. Last month, U.S. authorities banned Russian antivirus software company Kaspersky Lab from launching new operations in the country, after accusing it of playing a role in data theft. secrets NSA employees and others.
Friday’s problems canceled or delayed thousands of flights and forced hospitals to postpone surgeries. The worst cyberattacks, such as Russia’s NotPetya attack on Ukrainian businesses and North Korea’s WannaCry virus, have caused more lasting damage, permanently damaging computers. But even those haven’t spread as quickly or as far.
The extent of the financial damage caused by the outages, as well as the distribution of costs, will not be known for some time. Most software vendors are exempt from legal liability for damage caused by their programs, which are licensed rather than sold. But they usually have service agreements with their largest customers that may require repair assistance, discounts or other compensation.
CrowdStrike’s failure is striking in part because the company’s executives have been among the most prominent voices in the industry to criticize Microsoft for repeated security failures. The software giant has been accused of major recent breaches of U.S. agencies, including the theft of government employees’ emails last year. including Commerce Secretary Gina Raimondo. A biting April report The Cyber Safety Review Board, led by an official from the Cybersecurity and Infrastructure Security Agency, cited “a corporate culture that deprioritized both corporate security investments and rigorous risk management.”
Beyond these failings at Microsoft, CrowdStrike said the company’s dominant position in the operating systems and productivity software market makes any weakness potentially catastrophic.
As one of the few top security companies, some experts are now saying the same thing about CrowdStrike, one of the few network security companies with such broad reach and power.
“It’s obvious this is very serious, it’s going to take weeks. You have to get your hands on keyboards,” said Bryan Palma, CEO of Trellix, a rival security firm. “It shows the need for redundancy and defense in depth.”
The Cybersecurity and Infrastructure Security Agency said it was assisting with recovery efforts and warned that criminals posing as CrowdStrike were trying to convince customers to download malware or give up access to their computers.
Marie Vasek, an assistant professor in the department of computer science at University College London, said the widespread IT outages showed how much the world’s technology systems depend on software from a small number of companies, including Microsoft and CrowdStrike.
“The problem here is that Microsoft is a standard software that everyone uses, and the bug in CrowdStrike is deployed on every system,” she said.
Vasek said technology networks have become so sprawling, complex and interdependent that they increase the risk that a single misfired line of software code could bring down entire computer networks.
The flaw only affected computers running Windows, which powers hundreds of millions of personal computers and many back-end systems for airlines, digital payments, emergency services, call centers and more.
In a statement, CrowdStrike said it is “working with all affected customers to ensure systems are back up and running and can provide the services their customers rely on.”
Some businesses affected by the CrowdStrike issue, including banks and emergency service centers, said Friday that they had implemented CrowdStrike’s fixed software and were beginning to recover.
Vasek said both Microsoft and CrowdStrike need to review their procedures to prevent such widespread technology failures from happening again.
CrowdStrike, she said, should think about how to securely update its software for millions of computer networks. And Microsoft, she added, needs to do more to ensure that other companies’ software updates don’t cripple Windows machines.
“Microsoft needs to think about how to verify that software is what it should be,” she said.
Microsoft did not respond directly to the criticism, but said in a statement that the company is “actively supporting our customers to help them recover.”
The company also reported outages of some of its popular web-connected software for corporate and government technology networks.
It was not immediately clear how many of the computer network outages that occurred Friday were due to the faulty CrowdStrike software update and how many were the result of problems that began Thursday with Microsoft’s online services and its enterprise cloud computing service, Azure.
A Microsoft spokesperson said the company does not believe the CrowdStrike software bug was related to the outage that affected a “subset of Azure customers.” The issue has been resolved, he said.
correction
A previous version of this article misspelled Bryan Palma’s first name as Ryan. The article has been corrected.
