ChatGPT Atlas can still surface blocked URLs unless publishers use noindex

OpenAI’s current Publishers and Developers FAQ documents a more precise control model for ChatGPT Atlas and search visibility. Publishers should allow OAI-SearchBot if they want page content eligible for summaries, snippets and citations. However, disallowing a page in robots.txt does not guarantee that its URL disappears from Atlas: OpenAI says that if it learns a disallowed URL from a third-party search provider or another crawled page and has relevance signals, Atlas may still surface the page’s title and link. OpenAI tells publishers to use a `noindex` meta tag when they do not want that link-level result to appear, while noting that the crawler must be allowed to fetch the page in order to read the meta tag. GPTBot remains the separate opt-out mechanism for potential model training.

Publishers often treat robots.txt as a single switch for AI visibility, but OpenAI’s documented behavior shows that crawl permission, search-result inclusion and training are separate decisions. A site can block OAI-SearchBot from reading a page and still have a bare title/link surfaced from external discovery signals. Conversely, adding `noindex` behind a robots.txt block can fail because the crawler cannot read the directive. For publishers managing private-ish archives, licensed content, staged migrations or selective AI-search visibility, the ordering and combination of these controls now has a concrete distribution consequence.

OpenAI says public websites can appear in ChatGPT search and that OAI-SearchBot access is needed for content to be included in summaries and snippets. But a disallowed URL may still be known through another search provider or links found elsewhere. In that case Atlas can surface the URL and page title without using the blocked page content.

For publishers that do not want even the title and link surfaced, OpenAI recommends the standard `noindex` meta tag. The implementation detail matters: the page must be crawlable for OpenAI’s crawler to observe the directive. Blocking the crawler at robots.txt while relying on an unseen noindex tag can therefore produce the opposite of the intended control state.

The FAQ separately tells publishers to block GPTBot for pages they want excluded from potential training, including content encountered through user interactions in Atlas. That makes OAI-SearchBot, `noindex` and GPTBot three distinct controls: retrieval for search, inclusion in result surfaces and potential training.

OpenAI says referrals from ChatGPT search include `utm_source=chatgpt.com`, giving publishers a direct way to measure inbound search traffic. That makes selective access decisions testable: sites can compare referral value against the distribution or licensing reasons for restricting particular sections.