Website indexation audit and crawlability guide

The intended set of indexed pages

Start with pages the business needs customers to find. Combine the sitemap with internal crawl results and Search Console page records, then group URLs by template. Include an affected page and a working comparison from each relevant group. Keep filters, internal search pages and duplicate variants in separate groups so the audit does not treat every excluded URL as a lost opportunity.

Record the intended destination beside each URL. A redirected page or a duplicate pointing to a useful canonical may be behaving as planned. The audit needs to distinguish that decision from an important page excluded by an accidental directive.

A reproducible diagnosis for each affected URL

For URL diagnosis, URL Inspection separates indexed-page data from the live test. The live test checks the current page's potential indexability and cannot predict Google's canonical selection. Save both observations so a later reviewer can distinguish the stored result from the current page test.

1. Save the URL, observation date and response status. Follow any redirects to the final page, then inspect its content and indexing directives, including the HTTP headers.
2. Check the applicable robots.txt rules. A crawl block can stop Google from reading the page even when the page opens in a normal browser.
3. Inspect the URL in Search Console and save its indexed-page result with the last crawl date. Record the declared canonical and Google's selected canonical where available.
4. Run the live test after the current-page checks. Compare its result with the saved indexed-page data before deciding that a release has fixed the problem.

Repairs tied to the observed cause

Use noindex to control indexing when exclusion is the intended result. Robots.txt manages crawl access and does not reliably keep a URL out of search. Those controls need different acceptance checks.

• For an accidental noindex, correct the generating template or response header, then verify the affected pages again.
• For a page deliberately excluded with noindex, let Google crawl it to read that directive. A robots.txt block can prevent that reading.
• For the wrong canonical destination, review the page relationship and the signals produced by internal links, sitemaps and the template before changing the tag.
• For a fetch or rendering problem, repair the failing response or required resources and retest the page Google receives.

Validation after the release

Recheck the original sample and retain the before-and-after responses with the release date. Confirm the technical repair immediately, then check later indexed-page observations separately. A successful live test or an indexing request does not establish that Google has included the page.

Expand testing across the template after a shared defect is repaired. If the issue began during a platform or URL change, use the migration checklist to review the mapping and launch controls. Keep a wider traffic-drop investigation separate when important pages remain indexed but lose visibility.

Brief checklist

• Save the URL, observation date and response status. Follow any redirects to the final page, then inspect its content and indexing directives, including the HTTP headers.
• Check the applicable robots.txt rules. A crawl block can stop Google from reading the page even when the page opens in a normal browser.
• Inspect the URL in Search Console and save its indexed-page result with the last crawl date. Record the declared canonical and Google's selected canonical where available.
• Run the live test after the current-page checks. Compare its result with the saved indexed-page data before deciding that a release has fixed the problem.