AI SEO Glossary
What is AI Crawler?
Ask AI for a business like yours. Are you in the answer?
ChatGPT, Claude, and Perplexity only recommend sites they can read - and most can't read yours. Scan it free and see exactly where you stand.
Free · no signup · scores the 15 signals AI uses to find you · ~20 seconds
TL;DR - An AI crawler is any automated agent an AI company runs against the web, and the category splits into three functionally different types: training crawlers that collect data for future models (GPTBot, ClaudeBot), search-index crawlers that build retrieval indexes for grounded answers (OAI-SearchBot, PerplexityBot), and user-triggered fetchers that retrieve a page live because a user's prompt asked for it (ChatGPT-User, Perplexity-User). Each type honors robots.txt separately, so access policy is a three-way decision - and blocking the wrong type silently removes a brand from AI answers.
AI crawlers in practice
Decide which AI crawlers can read your site
AI companies send crawlers to read the web, some to power search answers and some to train models. We help you decide which to allow, write the rules, and check they work as intended.
- ›Search and training crawlers treated separately
- ›Public pages open, private areas closed
- ›Server logs checked for new crawlers
Definition & scope
The commercial calculus differs by type: blocking training bots is a defensible IP stance with little visibility cost today; blocking search and user-fetch agents directly removes the site from citation and recommendation surfaces buyers actually use.
Operationally, AI-crawler management means auditing robots.txt against the current published user-agent list of every major engine, watching server logs for real crawl behavior, and pairing access with llms.txt guidance so the crawlers that do visit spend their fetches on the pages that matter.
Related terms
- GPTBot - OpenAI's training crawler.
- ClaudeBot - Anthropic's training crawler.
- PerplexityBot - Perplexity's index crawler.
- OAI-SearchBot - OpenAI's search-index crawler.
Related services
- llms.txt Implementation - curated crawl guidance for AI agents.
- AI SEO Services - crawler access policy inside the AI-visibility program.