beehiiv gives publishers edge-enforced control over individual AI crawlers

On August 20, beehiiv launched an AI Discovery package for publisher websites. The most consequential part is AI Crawl Control: publishers on Max or Enterprise using the Website Builder and a custom domain can inspect AI-bot traffic and block individual crawlers, with enforcement applied at the network edge through Cloudflare rather than relying only on robots.txt. beehiiv says its catalog currently tracks 22 AI-training, AI-search, AI-assistant and traditional search crawlers. The release also adds automatic structured data, customizable llms.txt support and richer AEO controls for content that publishers do want crawled.

Publishers increasingly face two separate decisions: whether an AI service may fetch their work at all, and whether exposed content is structured to be discoverable or citable. beehiiv now puts both decisions inside the publishing platform. Edge enforcement gives smaller publishers a practical access-control layer without separately configuring a CDN bot product, while crawler analytics makes otherwise invisible machine traffic measurable. The trade-off is distribution: blocking search or assistant crawlers can reduce visibility, and the strongest controls require a paid beehiiv plan plus a custom domain.

beehiiv distinguishes AI Crawl Control from its existing robots.txt-based discoverability switch. A blocked crawler is stopped at Cloudflare’s network edge before it reaches the site, so the setting does not depend on the bot voluntarily honoring robots.txt. Publishers can allow or block crawlers individually rather than choosing one blanket policy.

The crawler dashboard reports total requests, blocked requests, AI-crawler share, unique crawler types, top-crawled pages and response codes over selectable periods. beehiiv groups known bots into model-training, AI-search, AI-assistant and traditional search categories, giving publishers a clearer basis for deciding what to permit.

beehiiv also generates llms.txt files and adds structured-data tooling intended to help search and answer engines understand content that remains accessible. Those features do not enforce access. Conversely, blocking a crawler does not remove material it collected previously or guarantee removal from an existing model.

Crawler analytics is broadly visible, but per-bot blocking requires a custom domain and Max or Enterprise. Blocking Googlebot, Bingbot or AI-search/assistant crawlers can also reduce or eliminate discovery in the products they power. Publishers therefore need a policy by crawler purpose rather than treating every AI bot as interchangeable.