Cloudflare explains Tuesday’s outage that temporarily took down ChatGPT

Cloudflare’s bot controls are supposed to help deal with problems like crawlers scraping information to train generative AI. It also recently announced a system that uses Generative AI to build the “AI Labyrinth, a new mitigation approach that uses AI-generated content to slow down, confuse, and waste the resources of AI Crawlers and other bots that don’t respect ‘no crawl’ directives.”

However, it says the problems today were due to changes to the permissions system of a database, not the generative AI tech, not DNS, and not what Cloudflare initially suspected, a cyber attack or malicious activity like a “hyper-scale DDoS attack.”

According to Prince, the machine learning model behind Bot Management that generates bot scores for the requests that travel over its network has a frequently updated configuration file that helps ID automated requests; however, “A change in our underlying ClickHouse query behaviour that generates this file caused it to have a large number of duplicate ‘feature’ rows.”

There’s more detail in the post about what happened next, but the query change caused its ClickHouse database to generate duplicates of information. As the configuration file rapidly grew to exceed preset memory limits, it took down “the core proxy system that handles traffic processing for our customers, for any traffic that depended on the bots module.”

As a result, companies that used Cloudflare’s rules to block certain bots returned false positives and cut off real traffic, while Cloudflare customers who didn’t use the generated bot score in their rules remained online.