Internal links to 404 pages waste crawl budget and strand visitors. Crawl the site, list every target that returns 404 or 410, and trace the linking pages. Then either update the link to a live URL or add a redirect when a real replacement exists. Re-crawl to confirm zero broken internal links.
The extended brief explains when this works, when it does not, and the trade-offs. Join the waitlist and it unlocks for every tip when sign-in opens; MCP access is the paid part.
Crawl and group broken targets General best practice
Run a crawl that reports internal links with 4xx targets. Sort by the number of linking pages and by the traffic the source pages receive.
Check: The crawl export lists each 4xx target URL with its count of internal referrers.
Update links at the source General best practice
Change hrefs in templates, navigation and CMS fields to the correct live URL. Use a redirect only when the original content moved and the new URL is clearly equivalent.
Check: The source template no longer contains the broken path, confirmed by grep in the repository.
Re-crawl and automate the check General best practice
Re-run the crawl after deploying and add a broken-link step to CI. Keep a baseline so new 4xx links fail the check.
Check: The post-deploy crawl reports zero internal links to 4xx URLs.
Verify indexing with URL Inspection — URL Inspection shows the indexed version Google holds for one URL: crawl time, selected canonical, status code and blocked resources.
Canonicalize duplicate URLs — Duplicate URLs split signals across protocol, host, slash and parameter variants.
Make JavaScript content crawlable — Google renders JavaScript, but rendering costs time and AI retrieval systems often read raw HTML.