Roshanak Kavian
Co-Founder and CEO of Indigo Mars, a Los Angeles marketing agency.
Getting robots.txt for AI crawlers right comes down to one distinction most businesses miss entirely: training crawlers and citation crawlers are not the same thing, and blocking the wrong one can make a business invisible to an entire AI platform's real-time answers, regardless of how good its content is. A site that blocks OpenAI's search-specific crawler, for example, simply won't be pulled into ChatGPT's cited web results — no exceptions, no matter how strong the underlying content.
The Crawlers That Actually Matter
A handful of named crawlers cover the platforms most businesses actually care about:
OpenAI: GPTBot (training), OAI-SearchBot (live ChatGPT search citation), ChatGPT-User (real-time browsing triggered by a user)
Anthropic: ClaudeBot and anthropic-ai (training and crawling), Claude-SearchBot and Claude-User (search and live browsing)
Perplexity: PerplexityBot (search indexing), Perplexity-User (real-time browsing)
Google: Google-Extended (covers Gemini and AI features outside standard Search)
Training Bots vs. Citation Bots: The Distinction That Matters Most
Training crawlers collect content to help shape a model's underlying knowledge over time. Citation and search crawlers are what actually retrieve content in the moment to answer a real, live question — and those are the ones that determine whether a business shows up in an actual AI-generated answer today. A business can reasonably choose to block pure training crawlers, if it's not comfortable with content feeding future model training, while still allowing the search and citation crawlers that drive real, present-day visibility. These are two separate decisions, not one.
A Working Configuration
A reasonable starting point for a business prioritizing AI visibility, allowing both training and citation crawlers:
If you'd rather allow citation visibility while opting out of training specifically, the search and live-browsing crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User) get "Allow," while the pure training crawlers (GPTBot, ClaudeBot, anthropic-ai, Google-Extended) get "Disallow."
The Mistake That Silently Blocks Everything
If a site already has a broad wildcard rule — something like a blanket "Disallow" applied to all crawlers, often left over from a staging environment or a cautious default — every AI crawler gets blocked by that rule unless it's explicitly exempted with its own named "Allow" entry above it. This is an easy, common way a business ends up invisible to AI search without ever intending to block anything specific.
How to Verify It's Actually Working
A direct way to confirm access: send a test request using a specific crawler's user-agent string and check the response. This can be done from a command line with a tool like curl, specifying the user-agent for GPTBot, ClaudeBot, or PerplexityBot directly, to see exactly what that crawler would receive when it visits.
One honest caveat worth knowing: some reporting has found Perplexity using undeclared, less transparent crawling methods that don't always respect standard no-crawl directives the way the officially documented bots do. robots.txt is a real, meaningful signal, but it isn't a perfect guarantee of behavior from every platform.
Where to Go From Here
This is a technical companion to the broader structured-data foundation covered in Why Schema Markup Matters More Than Ever for AI Search.
Not sure whether your own site's robots.txt is quietly blocking AI visibility? Get in touch and we'll check it for you.
