Article / Control access by purpose
Which AI Crawlers Should You Allow?
Search, training, and user-requested retrieval are different jobs. This guide maps the major AI crawlers to the controls and checks that actually apply.
The phrase “block AI bots” sounds decisive until you look at what the bots do. Search indexing, answer grounding, model training, and a live fetch requested by a user are different jobs. A serious policy has to decide which business outcome the site wants, then map the exact crawler, page control, edge rule, and verification method to that decision.
The decisions to make before editing robots.txt
- Decide by crawler purpose instead of using one blanket AI rule.
- A robots.txt directive is only policy. Confirm the real response and edge behavior.
- Blocking training does not automatically remove a page from AI search or user-requested retrieval.
Crawler purpose map
Separate search, training, and user-requested access
The platform controls are not symmetrical. The safest policy starts with purpose and verifies the current first-party documentation for each token.
| Purpose | Typical control surface | What the control does not do |
|---|---|---|
| Search indexing and AI search features | Standard search crawler access plus index and snippet controls on the page. | A guarantee of indexing, retrieval, citation, or ranking. |
| Separate model-training use | A platform-specific training token when the platform documents one, such as GPTBot, ClaudeBot, or Google-Extended. | Removal from previous collections or identical control over search products. |
| User-requested live retrieval | The platform's user-fetch documentation, WAF policy, authentication, rate limits, and the permissions of the requested action. | That robots.txt will be honored in the same way as an automatic crawler. |
| Content-level preview control | noindex, nosnippet, max-snippet, or supported data-nosnippet markup on the actual response. | Control over every external copy, previous crawl, training use, or unsupported platform. |
| Bot verification at the edge | Official IP data or reverse and forward DNS checks where documented, combined with the expected user agent and behavior. | A permanent identity guarantee when ranges, tokens, products, and providers can change. |
Start with the business outcome
Before writing a directive, I ask whether the content should be eligible for search and AI answer discovery, whether separate model-training use is permitted, and whether a user-requested live fetch is permitted. A publisher, store, private support portal, and public documentation site can reasonably answer those questions differently.
The policy can also differ by content class. Public product facts may need discovery. Licensed research may require stricter reuse controls. Account pages should stay behind authentication regardless of the bot name. The rule should follow the content and risk, not the current mood around AI.
- List the content classes and the owner of each decision.
- Name the desired search, training, and user-fetch outcome separately.
- Record the legal, contractual, privacy, security, and commercial reason for the rule.
The platforms do not use one shared control model
Google says Googlebot governs Search, including its AI search features. Google-Extended is a control token for specified Gemini training and grounding uses, not a separate crawler identity, and Google says it does not affect Search inclusion or act as a Search ranking signal.
OpenAI separates OAI-SearchBot for ChatGPT Search from GPTBot for possible foundation-model training. Anthropic documents Claude-SearchBot, ClaudeBot, and Claude-User as separate purposes. Perplexity documents PerplexityBot for search visibility and Perplexity-User for user-requested fetches. Those labels are similar enough to confuse and different enough that copying a random robots block is risky.
- Use the exact, case-sensitive token from the current official source.
- Do not assume a training rule controls search or a search rule controls user fetches.
- Recheck every subdomain because robots.txt policy is host-specific.
robots.txt is a policy file, not proof of the outcome
A correct directive can still fail operationally. The CDN may serve a different robots.txt, the WAF may block an allowed crawler, a redirect may change hosts, or the crawler may receive a challenge page instead of the content. I test the public response from the real hostname and inspect the edge decision.
I also remember what robots.txt cannot do. It controls crawling behavior for compliant automatic crawlers. It does not guarantee removal from an index, erase previous collections, or force a user-requested fetcher to behave like a scheduled bot. Blocking a crawler can also prevent it from seeing a page-level noindex directive.
- Fetch robots.txt through the public hostname and record its status, headers, and body.
- Test one allowed and one blocked canary route for each crawler class.
- Confirm important CSS, JavaScript, images, and API responses are not accidentally blocked.
Use page-level controls when the problem is the preview
Sometimes the goal is not to block discovery. It is to keep a sensitive passage out of a preview or generated answer. Google documents normal Search controls such as noindex, nosnippet, data-nosnippet, and max-snippet for its AI search features. Bing supports data-nosnippet for excluding selected sections from snippets and AI-generated answers while leaving the rest of the page discoverable.
These controls have tradeoffs. Removing a useful passage from previews can reduce how much context the system has. A crawler also needs access to read a page-level directive. I use the narrowest supported control and test the rendered result instead of assuming a meta tag has universal meaning.
- Choose route-level blocking only when the entire route should be unavailable to that crawler.
- Choose noindex when the page itself should not appear in the supported search index.
- Choose snippet controls when discovery is acceptable but a specific preview is not.
Do not trust a user-agent string by itself
Any requester can send a famous bot name. Bing and Google explicitly warn about spoofing, and several platforms publish IP data or a verification method. At the WAF, I combine the expected user agent with official network verification when the platform provides it.
That check also needs maintenance. IP ranges, crawler versions, and infrastructure can change. A copied allowlist that never refreshes becomes either a security hole or an accidental block. The policy register should name the official source and review date beside every exception.
- Use official IP JSON, CIDR ranges, or documented DNS verification.
- Apply rate limits and behavioral monitoring even to verified bots.
- Never bypass authentication or application authorization because a crawler identity looks familiar.
Treat crawler governance like an operating policy
I keep a register with one row per crawler and columns for owner, purpose, allowed paths, blocked paths, robots rule, WAF rule, identity source, review date, and rollback plan. That turns an invisible text file into something the business can actually govern.
After a change, I watch verified crawler requests for at least seven days and record route, status, bytes, response time, robots decision, and WAF decision. I refresh official identity data automatically where possible, review documentation monthly, and revisit the business policy quarterly.
- Version every robots and WAF change with an owner and reason.
- Monitor 2xx, 3xx, 4xx, 5xx, challenge, and timeout behavior separately.
- Keep a tested rollback and confirm it from the public edge after use.
Method
How I audit an AI crawler policy
The audit checks the written intent, public files, page controls, edge enforcement, and logs. A policy is not complete until those records agree.
Build the purpose register
List every relevant bot, owner, business purpose, content class, desired access, official source URL, and review date.
Read the live controls
Fetch robots.txt, response headers, meta robots, snippet markup, authentication behavior, and canonical state from the real hostname.
Inspect the edge layer
Review CDN, WAF, rate limit, bot product, redirect, and challenge rules that can override the written crawler policy.
Test canary routes
Use one intentionally allowed and one intentionally blocked route per class, including required assets and rendered content.
Verify identity and logs
Match official network verification where available and inspect real requests, statuses, bytes, latency, and enforcement decisions.
Monitor and review
Watch at least seven days after a change, refresh identity data, keep rollback proof, and revisit platform documentation monthly.
Limits
What crawler controls cannot guarantee
robots.txt is voluntary for compliant crawlers and does not erase data already collected. Platform-specific controls can change, and user-requested fetchers may not follow the same rules as automatic crawlers.
An allow rule does not guarantee discovery, indexing, answer inclusion, or traffic. A block can also have wider consequences when a search product and an AI feature share the same crawler.
- Blocking crawl does not necessarily remove a known URL from an existing index.
- A crawler that cannot fetch the page may never see its noindex directive.
- Google-Extended does not control Google Search AI features.
- Microsoft does not present the same simple search-versus-training crawler split as OpenAI or Anthropic.
- CDN and WAF products can change behavior outside the robots.txt file.
Sources
Primary sources I checked
- Google common crawlersGoogle Search Central, reviewed Opens in a new tab
- AI features and your websiteGoogle Search Central, reviewed Opens in a new tab
- OpenAI botsOpenAI Developers, reviewed Opens in a new tab
- How Anthropic crawls the web and how site owners can block its crawlersClaude Help Center, reviewed Opens in a new tab
- Perplexity crawlersPerplexity Documentation, reviewed Opens in a new tab
- Bing crawler verificationBing Webmaster Tools, reviewed Opens in a new tab
- Bing introduces support for the data-nosnippet HTML attributeBing Webmaster Blog, reviewed Opens in a new tab
Updates
What changed
Published the May research article after verifying the current Google, OpenAI, Anthropic, Perplexity, and Bing control surfaces.