--- title: "Crawling & Indexing — Technical SEO — seo-mcp" description: "Crawling, indexing, rendering, performance and everything the bots must do to find and understand your site." url: "https://seo-mcp.io/c/technical-seo/crawling-indexing" --- # Crawling & Indexing — Technical SEO Crawling, indexing, rendering, performance and everything the bots must do to find and understand your site. Underlying category: [Technical SEO](https://seo-mcp.io/c/technical-seo) (50 tips). - [Fix crawl traps in filters and calendars](https://seo-mcp.io/t/fix-crawl-traps-in-filters-and-calendars) — Crawl traps are infinite URL spaces: calendars, sort combinations, session IDs and pagination that never ends. They starve real pages of discovery. Find them in Crawl stats, fix the link graph first, return 404 for out-of-range parameters, and block only the truly infinite spaces. - [Make JavaScript content crawlable](https://seo-mcp.io/t/make-javascript-content-crawlable) — Google renders JavaScript, but rendering costs time and AI retrieval systems often read raw HTML. Server-render primary content, keep titles and links in the initial response, and avoid gating content behind interactions. Verify by curling the URL and by loading it with JavaScript disabled. - [Fix soft 404 pages properly](https://seo-mcp.io/t/fix-soft-404-pages-properly) — A soft 404 returns HTTP 200 while showing missing-content copy. Google keeps it indexed and wastes crawl budget on empty templates. Find these URLs in the Page indexing report, then make missing content return a real 404 or 410 and keep 200 only for pages with genuine value. - [Canonicalize duplicate URLs](https://seo-mcp.io/t/canonicalize-duplicate-urls) — Duplicate URLs split signals across protocol, host, slash and parameter variants. Pick one canonical form, emit one absolute self-referencing canonical per page, and align internal links, sitemap entries, hreflang and og:url with it. Use redirects when the duplicate should not exist at all. - [Ship app/sitemap.ts and app/robots.ts](https://seo-mcp.io/t/ship-app-sitemap-ts-and-app-robots-ts) — Next.js App Router turns app/sitemap.ts and app/robots.ts into /sitemap.xml and /robots.txt at build time. Return only canonical, indexable URLs from sitemap.ts with honest lastModified values, and set the rules plus the sitemap location from robots.ts. Both files must be reachable without authentication and must not be shadowed by static files in public/. - [Add prerendering and a sitemap endpoint](https://seo-mcp.io/t/add-prerendering-and-a-sitemap-endpoint) — SvelteKit renders pages on the server by default, so crawlers already receive HTML, but every request still needs a render. Enable prerendering for content routes with export const prerender = true and generate a sitemap from src/routes/sitemap.xml/+server.ts so the file exists at build time. - [Fix redirect chains and loops](https://seo-mcp.io/t/fix-redirect-chains-and-loops) — A redirect chain adds latency and leaks crawl signals. Point each legacy URL directly at the final destination, use 301 or 308 for permanent moves, and keep the map in source control. Audit with curl -IL and update internal links so chains stop growing. - [Submit your sitemap in Search Console](https://seo-mcp.io/t/submit-your-sitemap-in-search-console) — A sitemap is a hint, not a guarantee, but submitting it in Search Console shortens discovery for new URLs and reports parse errors. Submit under the property that owns the URLs, keep entries canonical and indexable, and resubmit only when the file or its lastmod values actually change. - [Monitor index coverage in Search Console](https://seo-mcp.io/t/monitor-index-coverage-in-search-console) — The Page indexing report groups URLs by status and reason. Review it weekly, sample each reason, and separate intentional exclusions from regressions. Track counts over time instead of reacting to a single spike, and use URL Inspection to confirm individual cases before changing code. - [Return true 404s with notFound()](https://seo-mcp.io/t/return-true-404s-with-notfound) — A route that renders an empty template with status 200 is a soft 404: crawlers index it and users see nothing. Call notFound() when a record is missing or the slug is invalid, add app/not-found.tsx for the layout, and confirm the response status over HTTP. Keep redirects for moved content and reserve notFound() for genuinely absent pages. - [Return correct status codes for removed pages](https://seo-mcp.io/t/return-correct-status-codes-for-removed-pages) — Status codes are the first thing a crawler reads. Use 404 for URLs that never existed or may return, 410 for content removed for good, and 301 only when a real replacement exists. Check the status at the edge, because CDNs and rewrites routinely turn a correct framework 404 into a 200. - [Move redirects into next.config redirects()](https://seo-mcp.io/t/move-redirects-into-next-config-redirects) — next.config redirects run before rendering and return 308 for permanent rules, so old URLs keep their signals without shipping JavaScript. Move server-side redirect maps into redirects(), mark permanent changes permanent, scope rules with has and missing, and keep temporary campaigns separate. Verify with curl that each old path returns one hop and the expected status. - [Manage crawl budget on large sites](https://seo-mcp.io/t/manage-crawl-budget-on-large-sites) — Crawl budget matters most for large or fast-changing sites. Segment server logs by Googlebot, measure requests per template, and remove the causes of waste: parameter URLs, soft 404s and duplicate paths. Improving response speed and internal linking also raises how much Google can crawl. - [Set up Bing Webmaster Tools and IndexNow](https://seo-mcp.io/t/set-up-bing-webmaster-tools-and-indexnow) — Bing Webmaster Tools covers Bing and Copilot surfaces that Search Console reports do not. Verify the site, submit the sitemap and enable IndexNow so new URLs are pushed instead of waiting for a crawl. Keep the IndexNow key file at the site root, ping each changed URL after deploy and review crawl errors monthly. - [Verify indexing with URL Inspection](https://seo-mcp.io/t/verify-indexing-with-url-inspection) — URL Inspection shows the indexed version Google holds for one URL: crawl time, selected canonical, status code and blocked resources. Run it before rewriting content or changing redirects. It separates indexing failures from ranking problems and confirms whether Google sees the same page your browser renders. - [Write a crawlable robots.txt](https://seo-mcp.io/t/write-a-crawlable-robots-txt) — robots.txt controls crawling, not indexing. Serve one file per host at the root with a 200 status and a sitemap reference, allow CSS and JavaScript so rendering matches users, and block only infinite or private URL spaces. A single Disallow: / line can remove the entire site from search. - [Add sitemap.xml with accurate lastmod](https://seo-mcp.io/t/add-sitemap-xml-with-accurate-lastmod) — An XML sitemap lists canonical, indexable URLs so crawlers discover them without following every link. Build it from the content store, use absolute URLs that match the page canonical, and set lastmod only when content meaningfully changed. Split large sites into child sitemaps under an index. - [Audit Googlebot crawl behavior with server logs to find crawl traps](https://seo-mcp.io/t/audit-googlebot-crawl-behavior-with-server-logs-to-find) — Server log analysis reveals how Googlebot actually spends crawl budget, exposing infinite URL spaces, thin faceted combinations and orphan pages that internal tools miss. The tip walks through filtering verified Googlebot requests, grouping them by URL pattern, then using robots rules, canonicals and internal links to redirect crawl attention toward pages that earn traffic.