AI Crawlers — Who Scans Your Site and How to Manage It (robots.txt, Pay-Per-Crawl)

In two years, AI bot traffic has gone from a curiosity to one of the heaviest line items in server logs. The trouble is that "AI bot" is not a single category. The same label covers training bots (collecting a corpus to teach models), search bots (building the index AI answers cite from) and on-demand bots (fetching a page because a specific user just asked about it). Each of these calls for a different decision — and conflating them produces the single most expensive mistake in this area: one misaddressed line in robots.txt can cut your brand out of ChatGPT's answers while protecting nothing. The scale of the imbalance shows clearly in Cloudflare's data: for every real user referral, Google crawls on the order of a dozen pages, OpenAI — thousands, and Anthropic — tens of thousands. Crawling has stopped being a symmetric "content for traffic" trade, and a new economy is forming around that shift: default blocks, pay-per-crawl and machine-readable licences.

GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended — each does something different, and blocking the wrong one means vanishing from AI answers. The complete AI bot map, a robots.txt decision matrix, verifying impersonating scrapers, and the new crawl economy: Cloudflare's pay-per-crawl (HTTP 402) and the RSL licensing standard.

What follows is the whole thing in order: the bot map, a decision matrix, a reference robots.txt, impersonation checks, and what's gathering pace next — HTTP 402 and RSL.

Three types of AI bots — the most important distinction everyone forgets

• Training bots (GPTBot, Google-Extended, ClaudeBot, Meta-ExternalAgent, CCBot) collect content to train future models. Blocking means: your content won't make it into upcoming model versions. It does not affect whether an AI search engine cites you today.
• Search bots (OAI-SearchBot, PerplexityBot, Googlebot — also for AI Overviews, Bingbot — also for Copilot) build the index models pull sources from in real time. Blocking means: you vanish from citations and answers.
• On-demand bots (ChatGPT-User, Perplexity-User, Claude-User) fetch a specific URL because a user pasted a link or asked for an analysis. That's effectively "almost-traffic" — there's a human on the other side. Blocking means: the agent tells the user it can't open your site.

The bot map — who, what for, and what controls it

Two things in this table get confused most often. First, GPTBot is not what decides whether ChatGPT cites you — that's OAI-SearchBot's job; GPTBot is training only. Second, Google-Extended is not an "AI kill-switch for Google" — it only controls Gemini training.

Google is a special case — you can't "leave AI but stay in search"

AI Overviews and AI Mode use the same index and the same Googlebot as classic results. There is no directive saying "don't show me in AI Overviews but keep my links" — at most you can limit the quoted snippet length (nosnippet, max-snippet), paying for it with worse snippets everywhere. So the Googlebot decision is binary: either you're in Google (AI features included) or you're not. How AI Mode works and how to optimize for it is covered in the post on AI Overviews and AI Mode.

What to block and what not to — the decision matrix

The practical rule looks like this:

• You care about AI visibility (services, B2B, e-commerce — i.e. most businesses): allow all search and on-demand bots. This is an acquisition channel now, and traffic from models converts remarkably well.
• Training bots are a business-philosophy decision. Blocking costs nothing in today's visibility, but your content won't reach future models — and having your brand's knowledge "in the model weights" is a form of value too. There's no single right answer; it just has to be a conscious one.
• You publish premium content (research, data, paywalled work): block training bots, consider licensing (RSL, pay-per-crawl) and enforce at the CDN level, not just robots.txt.

How to write it down — a reference robots.txt

# --- AI search & citations: allow ---User-agent: OAI-SearchBotAllow: /User-agent: PerplexityBotAllow: /User-agent: ChatGPT-UserAllow: /# --- Model training: a conscious choice (here: blocked) ---User-agent: GPTBotDisallow: /User-agent: Google-ExtendedDisallow: /User-agent: ClaudeBotDisallow: /# --- Everything else: general rules ---User-agent: *Allow: /Sitemap: https://yoursite.com/sitemap.xml

Three rules when editing. First, a bot obeys the most specific matching User-agent group, not the sum of rules — GPTBot with its own section will ignore the wildcard section. Second, don't block directories with JS and CSS files, or you'll cripple Googlebot's rendering — what bots see without JavaScript is covered in the JavaScript SEO post. Third, remember robots.txt controls crawling, not indexing — removing pages from the index takes noindex, which I explain in the crawl budget guide.

Impersonation — how to verify that GPTBot is really GPTBot

Any scraper can write whatever it wants into the User-agent header — and they do, en masse, because "pretending to be GPTBot" can be a free pass through firewalls. Verification is thankfully simple: the major operators publish official IP ranges for their bots (OpenAI, Google, Perplexity), and Google's bots can additionally be verified via reverse DNS. If a request claims to be GPTBot but comes from outside the published ranges — it's not OpenAI. For the same reason, policy enforcement belongs in the CDN/WAF (Cloudflare, Fastly and others do this for you), and robots.txt should be treated as a declaration for the honest: it's a request, not a wall.

The new crawl economy: pay-per-crawl and RSL

For years an informal deal held: you let a crawler fetch your content, and in return you got traffic from search. AI bots have broken that exchange — they fetch a lot and refer back little. Hence three developments worth knowing:

• Default blocks. Since July 1, 2025, Cloudflare blocks AI training bots by default for newly onboarded domains — the owner has to consciously let them in. That flips the old opt-out logic to opt-in.
• Pay-per-crawl. Cloudflare launched a mechanism where a bot without an agreement gets an HTTP 402 Payment Required response with a per-request price — and can pay or walk away. The publisher sets one crawl rate and decides per bot: allow / charge / block.
• RSL (Really Simple Licensing). An open standard from September 2025, backed by major publishers and tech companies: a machine-readable licence file where you define usage terms for your content (free, with attribution, paid per crawl or — interestingly — paid per inference).

For a typical services business or store the takeaway is simple: exposure in AI answers is worth more today than hypothetical crawl fees — these mechanisms are weapons for premium content publishers. But it pays to know they exist, because the infrastructure (HTTP 402, machine licences) will keep spreading.

How to monitor AI bots on your site

1. 1.Server logs. Filter by User-agent (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Bingbot) and measure: share of requests, frequency, which URLs they visit. If a search bot skips your most important pages — you have an internal linking or sitemap problem.
2. 2.The CDN panel. Cloudflare and peers show AI bot traffic without digging through logs, including a training/search breakdown.
3. 3.Load. On large sites AI bots can generate noticeable cost — cache and a CDN solve 90% of it; aggressive bots can be trimmed at the WAF without touching robots.txt.
4. 4.The end result. A bot's presence is a means, not the goal. Whether models actually see and recommend your brand is what you check with the method from the AI visibility audit.

---

I audit bot access — from robots.txt through server logs to an AI bot policy — as part of technical SEO, and model visibility strategy as part of GEO services. I also teach this in the SEO & GEO course. Get in touch — I'll start with a review of your robots.txt and logs.

Worth reading next:

• JavaScript SEO — what bots see without rendering
• AI Browsers and Agent Experience (AX) — can an agent use your site?
• Crawl budget — how Google indexes a large site
• Google AI Overviews & AI Mode — how to optimize your site
• llms.txt — does it actually help AI visibility?