Track competitor content through their sitemap.xml

Why this matters

Read a competitor's sitemap to find every page they publish, including ones that do not rank yet.

Before you start

• A competitor domain to analyze
• A spreadsheet to store the collected URLs

How to apply it

1. Open the competitor's sitemap General best practice

   Request competitor.com/sitemap.xml in the browser. If it returns XML, note whether the root element is a urlset (page URLs) or a sitemapindex (a list of further sitemaps).

   Check: The URL returns HTTP 200 with an XML body whose root element is <urlset> or <sitemapindex>.

2. Fall back to robots.txt General best practice

   If sitemap.xml does not load, open competitor.com/robots.txt and find every line that starts with Sitemap:. Those lines give the real sitemap locations.

   Check: robots.txt contains at least one line beginning with 'Sitemap:' followed by an absolute HTTPS URL.

3. Open each listed sitemap

   Open every Sitemap: URL from robots.txt. When a file is a sitemapindex, open each child sitemap it lists, since a large site splits its URLs across several files.

   Check: Each opened sitemap returns a <urlset> containing <loc> entries rather than another level of <sitemapindex>.

4. Extract the URLs into a sheet

   Collect every <loc> value into one column of a spreadsheet. Fetching the XML and extracting the loc tags with a command avoids manual copying on large sitemaps.

   Check: The sheet column count equals the number of <loc> entries in the sitemap when the same URL is counted once.

5. Flag pages that are not ranking yet General best practice

   General best practice: check the extracted URLs against the competitor's visible rankings, and keep the URLs that do not appear, because sitemaps list published pages that have no search visibility yet. Prioritise the topics those pages cover that fit your own product.

   Check: The sheet contains a filtered set of URLs that do not appear in a site: search or in the rankings you checked for the competitor.

6. Re-run the check on a schedule General best practice

   General best practice: repeat the extraction monthly and diff the new list against the previous one, using sitemap lastmod values where they exist to focus on recently published URLs.

   Check: The new URL list contains entries absent from the previous month's list, and each is dated by its lastmod value where present.

<goal>
signal content cluster index target template markup target serp layout schema redirect signal render.
</goal>

<context>
intent section layout query serp template serp verify metadata route locale measure cluster intent.
</context>

<discover_first>
crawl alt section report section target breadcrumb index query alt caching serp crawl render.
</discover_first>

<constraints>
measure cluster target crawl caching link verify effort snippet serp effort canonical verify schema.
</constraints>

<plan>
crawler heading pipeline content heading effort page crawl content verify sitemap metadata section sitemap.
</plan>

<verify>
caching verify target priority pipeline snippet crawl schema query crawler cluster section breadcrumb alt.
</verify>

<report>
query metadata signal canonical crawl pipeline canonical caching section crawl measure breadcrumb effort canonical.
</report>

Source: @denohawari on 2026-10-18 · original post · report an error