Cloudflare is now giving all customers the ability to categorize bot traffic into three categories: search, agents, and training. Starting September 15, 2026, training and agent bots will be blocked by default on pages with advertisements, while search bots will retain access.
Cloudflare gives customers more granular options to allow or block bot traffic. The company now distinguishes between three types of automated traffic. Search refers to bots that index content to answer questions later, such as search engines. Agent refers to bots that perform tasks in real-time on behalf of a user, such as ChatGPT-User or browser-driven AI assistants. Training includes crawlers that permanently use content to train AI models.
Cloudflare customers can now select behavior for these three categories. This also applies to those on a free subscription. There are three options for each: allow, block on pages with advertisements, or block bots entirely. Companies that use bots for multiple purposes are encouraged to split their crawlers so that websites can see more clearly why a bot is visiting.
New default settings
Starting September 15, 2026, new default settings will apply to all new domains on Cloudflare. On pages with advertisements, Training and Agent bots will then be blocked by default, while Search bots will retain access. According to Cloudflare, advertisements are a signal that human attention is the goal, which is why bots that can disrupt this are being kept out.
Bots that combine multiple purposes, such as Googlebot, Applebot, and Bingbot, will henceforth fall under all applicable rules. The most strict setting will prevail. Those who block Training will therefore also block these bots, even if they use the same traffic for search results. Websites can disable this behavior via their security settings before the September 15 deadline.
Cloudflare launched a feature last year to block AI bots with a single click, supplemented by a marketplace where websites can be paid for crawls. Since then, the demand for more nuance has grown, which is why this new system is the answer.
BotBase provides an overview
Cloudflare is also introducing BotBase, a new database containing all known and verified bots. Enterprise customers with Bot Management will thus get a searchable overview of every crawler and its classification. Later this year, Cloudflare will expand the feature into a full-fledged control center for automated website traffic.