Cloudflare can now keep robots.txt aligned with AI bot policy
Bot Preference Sync turns Cloudflare’s AI crawler controls into both an enforcement policy and a published robots.txt signal. It is rolling out to every plan, with different defaults for publishers and other sites and important limits around custom per-bot rules.