A massive outage at content delivery network Cloudflare, which took down services including ChatGPT, Canva and parts of AWS, was traced to a routine change in database permissions linked to Cloudflare’s Bot Management system, which inadvertently knocked offline a significant portion of the web.
The November 18 outage, which lasted approximately six hours, has once again highlighted how ordinary software updates have become one of the leading triggers of large-scale internet outages in recent years.
The issue was triggered by a change to one of Cloudflare’s database systems’ permissions that caused the database to output multiple entries into a feature file used by the Bot Management system, Matthew Prince, co-founder & CEO of Cloudflare, explained in a blog post.
As this feature file propagated across Cloudflare’s global network, its doubled size exceeded the limits it was designed to handle. The oversized file caused those systems to fail. Once identified, Cloudflare replaced the larger feature file with an earlier version of the file.
What went wrong
Initially, Cloudflare had wrongly suspected it to be a hyper-scale DDoS attack as the volume of the 5xx error HTTPS status code served by the Cloudflare network started to fluctuate. However, the real cause was that every ClickHouse node generated a bad feature-configuration file, pushing the system into a failing state.
Prince explained that every request to Cloudflare takes a well-defined path within the company’s network. The requests are first terminated at the HTTP and TLS layer, then flow into the core proxy system (called FL for Frontline), and finally through Pingora, which performs cache lookups or fetches data from the origin if needed.
As traffic moves through the core proxy, it applies each customer’s unique configuration and settings, from enforcing WAF rules and DDoS protection to routing traffic to the Developer Platform and R2. These configuration and policy rules are enforced by domain-specific modules. One of those modules, Bot Management, resulted in the outage.
The Bot Management module uses a machine-learning model to assign bot scores to every request. The model relies on a feature configuration file, which is a collection of traits that help determine whether a request is automated. This feature file is refreshed every few minutes, allowing Cloudflare to react to new types of bots and new bot attacks.
A change in ClickHouse query behaviour that generates this file created large numbers of duplicate feature rows, increasing the size of the previously fixed-size feature configuration file, causing the bots module to trigger an error. As a result, Cloudflare’s core proxy began returning HTTP 5xx errors for any traffic relying on the bots module. Services like Workers KV and Access were also impacted, as they depend on the same core layer.
Once the underlying issue was identified, Cloudflare halted the generation and propagation of the bad feature file and manually inserted a known good file into the feature file distribution queue, and then forced a restart of its core proxy.
“This is like the butterfly effect in modern interconnected systems. A butterfly can flap its wings to fly, and its wing-flapping is the basis for the butterfly effect, a concept in chaos theory where a tiny change can have massive, unpredictable consequences. Same thing we are seeing here,” said Pareekh Jain, CEO at EIIRTrend & Pareekh Consulting.
Jain noted that such deep dependencies between security layers and AI modules for traffic routing are now common in large-scale cloud environments. Integrating powerful, dynamic modules directly into request processing pipelines means a flaw anywhere in the path can disrupt everything downstream.
More outages are now self-inflicted
This Cloudflare incident highlights a broader trend as some of the largest platforms now experience more frequent failures due to routine system updates than external attacks. In the last month alone, Microsoft Azure experienced a global outage due to an inadvertent tenant configuration change in Azure Front Door (AFD), and AWS suffered due to a latent defect within the service’s automated DNS management system.
“The real tension sits between the speed at which modern cloud platforms deploy changes and the maturity of the mechanisms that are meant to validate those changes,” said Sanchit Vir Gogia, chief analyst and CEO at Greyhound Research. “Cloud and network providers push updates constantly, often through automated pipelines that touch hundreds of services. Yet the dependency maps, failure simulations, and test harnesses behind these pipelines are still catching up to the complexity of the systems they support.”
Gartner’s Senior Principal Analyst Bhuvie Chhabra said that the Cloudflare outage underscores that operational errors arising from any kind of configuration changes in cloud services, not cyberattacks, represent the most significant systemic risk today. “The cascading failure, where a bot-management feature broke core traffic processing, exposes the inherent critical dependency within interconnected layers.”
Cloudflare patches the faultline, but bigger work remains
Apologizing for the disruption caused to its customers and to the Internet at large, Cloudflare said that beyond fixing the immediate problem, it has begun work on preventing similar incidents in the future.
The company is hardening the ingestion of Cloudflare-generated configuration files in the same way it validates user-generated input. It is also enabling more global kill switches for features, eliminating the ability for core dumps or other error reports to overwhelm system resources and reviewing failure modes for error conditions across all core proxy modules.
“These actions address the immediate root cause. These are good first steps. Cloudflare’s fixes are necessary but not sufficient. The outage exposed a far deeper issue,” said Gogia. “The broader challenge sits with enterprises. They must design systems that operate cleanly even when a major intermediary stumbles. That means more isolation between critical application flows and optional security modules, more graceful degradation modes, and more explicit regional containment of feature rollouts. It also means treating multi-provider architectures as strategic investments rather than discretionary costs.”
Chhabra added that while Cloudflare’s remediation is crucial, the wider industry must resist overreacting by partitioning applications or providers and instead prioritize pragmatic resilience. They should apply diversification sparingly and only for critical systems where downtime has a material business impact.
SOCIAL SHARE CARD GENERATOR