Internet company Fastly watches the growth of traffic from AI bots with dismay. The increase puts pressure on the infrastructure and security of the internet.
AI bots are currently responsible for a significant portion of internet traffic. According to a report from Fastly, the majority comes from crawlers of major AI companies such as Meta, Google, and OpenAI. Meta stands out in particular, accounting for more than half of all AI crawler traffic.
Crawlers and Fetchers
In the second quarter, Fastly analyzed over 6.5 trillion web requests through its security products. This shows that 80 percent of AI bot traffic comes from crawlers, which systematically search websites to “collect” information for models. More than half comes from Meta, while Google and OpenAI each account for a good twenty percent.
The geographical distribution of this traffic is not evenly distributed. In Europe, the share of crawlers is much smaller than in North America. This difference can have implications for the representation of regions in AI models, which are oriented toward North America.
The remaining 20 percent of AI bot traffic comes from so-called fetchers, which retrieve content in real-time based on user queries. OpenAI tops the list here: the company’s bots account for 98 percent of fetcher traffic.
read also
GPT-5 Tested: Structured and “No-Nonsense” Thinker
Fetchers are no less aggressive. In some cases, Fastly recorded up to 39,000 requests per minute on a single website, comparable to a DDoS attack. Such peaks can put significant pressure on websites with limited protection, even without malicious intent.
Internet under Pressure
The increase in AI traffic not only creates additional strain on infrastructure but also makes it more difficult to distinguish between legitimate bots and malicious imitators. Sectors such as e-commerce, media, entertainment, and technology in particular experience a lot of scraping activity.
Fastly recommends that organizations proactively manage AI traffic. Robots.txt files can provide guidelines to bots about which parts of the site are accessible. However, some bots ignore these instructions. Therefore, Fastly also advises technical measures such as IP blocking, rate limiting, and CAPTCHA systems. For more control, companies can use bot management solutions that recognize and manage AI traffic in real-time.
Transparency from bot operators is equally crucial. Bots should clearly identify themselves and publish verifiable IP ranges. OpenAI reportedly sets a good example. Finally, organizations can offer their content through licensing platforms, which provides the opportunity to turn AI traffic into a source of revenue.
AI bots are not inherently harmful, but without visibility and rules, they can unintentionally cause overload and costs, Fastly warns. Companies that want to protect their content and infrastructure would do well to make clear choices about who gets access, and under what conditions.