AI SEO Glossary
What is Crawl Budget?
Ask AI for a business like yours. Are you in the answer?
ChatGPT, Claude, and Perplexity only recommend sites they can read — and most can't read yours. Scan it free and see exactly where you stand.
Free · no signup · scores the 15 signals AI uses to find you · ~20 seconds
TL;DR — Crawl budget is the number of URLs a search engine's crawler is willing and able to fetch from a site within a given period, determined by the site's server capacity (crawl rate limit) and how much the engine wants its content (crawl demand). Small sites rarely exhaust it; large eCommerce catalogs with faceted navigation, session parameters, and duplicate paths absolutely do. When crawl budget is wasted on low-value URLs, new products and updated pages wait longer to be discovered — in every index, including the ones AI engines retrieve from.
Definition & scope
Crawl-budget optimization is subtractive: canonicalize or block parameter permutations, prune soft-404s and thin duplicates, fix redirect chains, and keep the XML sitemap an honest list of canonical, indexable URLs. The goal is for every fetch to land on a page worth indexing.
The AI-era wrinkle is that training and retrieval crawlers (GPTBot, ClaudeBot, PerplexityBot) spend fetches on the same site infrastructure — a site that wastes crawler attention on junk URLs is harder for every machine consumer to cover thoroughly.
Related terms
- Canonical URL — the primary duplicate-control mechanism.
- Internal Linking — distributes crawl attention to priority pages.
- IndexNow — change pings that reduce wasted recrawls.
Related services
- eCommerce SEO Audit — crawl-waste diagnosis on large catalogs.