Find published research by company, product, platform or technology.
Showing 1–4 of 4 dossiers
Cloudflare now sits on both sides of the policy boundary. Site owners can publish and enforce crawler preferences, while developers using Cloudflare’s own crawl API must declare intended uses and can receive hard failures when a target site disallows them.
beehiiv has moved AI crawler policy from a voluntary robots.txt signal to an enforceable publisher control for Max and Enterprise custom-domain sites. Its new dashboard tracks 22 AI and search crawlers, while separate structured-data and llms.txt features target AI discovery rather than access control.
Blocking OAI-SearchBot is not a complete removal signal for ChatGPT Atlas. OpenAI says a disallowed page can still appear as a link and title when the URL is learned elsewhere and judged relevant; publishers that want to suppress that result need a crawlable noindex directive, while GPTBot remains the separate training control.
Render is reshaping Workflows economics as it reaches GA: most small and I/O-heavy tasks should get cheaper under Flex, while task-state retention becomes a new line item and fixed-size Pro tiers remain for heavier compute.