Cloudflare can now keep robots.txt aligned with AI bot policy

Cloudflare announced Bot Preference Sync on August 21, 2026. The feature uses a site’s existing zone-level AI bot settings for Search, Agent and Training traffic to generate or update corresponding robots.txt directives, while keeping the edge enforcement policy and the public preference signal aligned. It will be available on every Cloudflare plan. For existing robots.txt files, Cloudflare prepends the generated block and preserves the site’s existing Disallow directives. New customers will have sync enabled by default, while existing users of Cloudflare’s legacy managed robots.txt feature will be prompted to review and confirm their preferences when the new feature launches.

Publishers and other site owners increasingly have two separate control planes for AI crawlers: a voluntary robots.txt signal and an enforceable CDN or bot-management rule. Those can drift apart, creating ambiguity for compliant crawlers and operational mistakes for site owners. Bot Preference Sync makes one category-level policy feed both layers. The practical consequence is easier separation of search discovery, agent access and model training, especially for sites that want to remain discoverable while declining training. The limitation is equally important: Cloudflare is synchronizing category-level policy, not arbitrary custom per-bot exceptions.

Cloudflare’s AI bot settings can Allow, block on ad-supported pages or block Search and Agent traffic, while Training can be disallowed. Bot Preference Sync translates those category choices into robots.txt while edge controls continue to provide actual enforcement where configured. The feature therefore reduces policy drift without pretending that robots.txt itself is an access-control mechanism.

Cloudflare says a Training disallow writes a no-training preference into robots.txt. Mixed-use crawlers that meet its transparency criteria can still access content for search indexing while honoring the training preference. Cloudflare’s criteria include respecting a no-training signal, offering an AI-summary opt-out, giving site owners visibility into training/search usage and demonstrating that declining training does not hurt traditional search results.

Generated directives are prepended to an existing robots.txt, so existing Disallow entries remain. But Bot Preference Sync does not read or reproduce individual custom rules with more complex exceptions. Sites with negotiated crawler access, granular bot rules or other hand-maintained logic should review the generated result and may need to disable sync.

Cloudflare says new publisher or ad-supported customers can choose an onboarding option that defaults Training to Disallow while leaving search available. Other new customers start without category blocks or disallows. Those defaults are editable, but they make business model part of the initial crawler-policy configuration.