Crawl budget and indexation for large catalogs

Google doesn't crawl your store infinitely - it allocates attention based on your site's size, health, and demonstrated importance. Small stores never notice this budget. Past a few thousand URLs, it becomes real: if Googlebot spends its visits on parameter noise and near-duplicates, your actual catalog gets crawled slowly, new products index late, and updates take weeks to register.

Where crawl waste comes from on Shopify

Shopify stores generate far more URLs than they have pages. The usual suspects: filtered and sorted collection URLs (?sort_by=, ?filter.v.option.color= - one collection with rich filtering can spawn thousands of combinations), collection-path product URLs (/collections/dresses/products/x duplicating /products/x), pagination tails (?page=14), market/language variants multiplying everything, and vendor/type auto-collections nobody curates. Each URL individually harmless; collectively, they can outnumber real pages twenty to one.

Seeing the problem

Two instruments. GSC's Crawl stats (Settings → Crawl stats) shows what Googlebot actually fetches - scan the sampled URLs and count how many are parameter noise versus real pages. GSC's Page indexing report shows the outcome: large counts under "Duplicate without canonical," "Alternate page with proper canonical," and "Crawled - currently not indexed" are the signature of a store generating more URLs than substance. The ratio worth writing down: indexed pages versus pages that should exist (products + collections + content). That's your real coverage number, tracked monthly.

Spending budget deliberately

The toolkit, in order of preference. Internal linking concentration (the positive lever): link generously to the pages you want crawled often; crawl frequency follows link prominence. Canonicals consolidate legitimate duplicates - Shopify handles most cases natively, verify rather than rebuild. Robots.txt blocks crawling of pure-noise patterns; Shopify's default already covers a lot, and you can extend via robots.txt.liquid - with care, because blocking is a blunt instrument (a blocked URL can't pass its equity anywhere, and can even linger indexed without content). Noindex for pages that should be reachable but not indexed. And genuinely empty pages - zero-product collections left published - simply shouldn't exist; they're the cheapest waste to eliminate.

One reframe to keep: "crawled - currently not indexed" at scale is usually not a crawl problem but a worth problem - Google visited and judged thin (Phase 4's opening lesson). Budget optimization gets pages seen. Only data quality gets them kept. The two levers work together, and only one of them lives in robots.txt.