Crawl traps are infinite URL spaces: calendars, sort combinations, session IDs and pagination that never ends. They starve real pages of discovery. Find them in Crawl stats, fix the link graph first, return 404 for out-of-range parameters, and block only the truly infinite spaces.
The extended brief explains when this works, when it does not, and the trade-offs. Join the waitlist and it unlocks for every tip when sign-in opens; MCP access is the paid part.
Group crawl requests by URL pattern and response. A trap shows thousands of requests with few unique pages and mostly 200 or soft 404 responses.
Check: Crawl stats shows at least one pattern with more than 90% of requests returning the same thin template.
Remove the links that create the space General best practice
Stop linking to empty filter results, future calendar dates and out-of-range pagination. Render those interactions without crawlable href values.
Check: The rendered HTML for an empty filter result contains no anchor pointing to further empty combinations.
Return 404 and block only truly infinite spaces
Return 404 for out-of-range parameters, and add robots.txt rules for parameter patterns that can never produce unique content. Keep indexable pages crawlable.
Check: curl -sI your production domain/search?page=999 returns 404, and robots.txt disallows the parameter pattern.
Verify indexing with URL Inspection — URL Inspection shows the indexed version Google holds for one URL: crawl time, selected canonical, status code and blocked resources.
Canonicalize duplicate URLs — Duplicate URLs split signals across protocol, host, slash and parameter variants.
Make JavaScript content crawlable — Google renders JavaScript, but rendering costs time and AI retrieval systems often read raw HTML.