According to Cloudflare, AI agents are the new norm, which is why it is focusing on Markdown for better crawling of web content.
Cloudflare recognizes the shift from traditional search engines to AI agents by automatically offering websites in the Markdown markup language. This allows machines to process web content more efficiently and accurately.
From SEO to AI Discovery
In the past, you had to be discoverable online by attracting human visitors to your website via SEO and search engines. Today, that is increasingly shifting toward AI crawlers and agents. These systems struggle with HTML pages full of scripts, navigation, and formatting. According to Cloudflare, this requires a fundamentally different approach: prioritizing agents.
Why Markdown Makes the Difference
Markdown is more compact than HTML. A simple heading costs only a few tokens in Markdown, whereas the same content in HTML uses much more token space due to tags and formatting. Cloudflare states that a blog post in Markdown can consume up to 80 percent fewer tokens than the HTML version. This results in cheaper, faster, and more accurate processing for AI systems.
With Markdown for Agents, Cloudflare can now directly convert HTML pages into Markdown. AI agents only need to send an Accept: text/markdown header. The Cloudflare network retrieves the HTML from the source, automatically converts it, and serves back a Markdown version, including a header that provides the estimated token count.
Control via Content Signals
Markdown responses include standard Content Signals that indicate whether the content may be used for AI training, search, or agent applications. This allows content creators to maintain more control over how their information is used, while AI systems gain clarity on permitted use cases.
Markdown for Agents is available as a free beta for Cloudflare’s Pro, Business, and Enterprise customers. For organizations that want to remain visible in an AI-driven web, this is a first concrete step toward content that is not only readable for humans but also optimally usable for machines.