Modern crawl insights for search engines and AI bots

What Is an SEO Crawl Audit?

An SEO crawl audit simulates how a search engine — or an AI crawler like GPTBot or ClaudeBot — moves through your site, following links, reading robots.txt rules, and checking whether each page can actually be indexed. Unlike a manual page-by-page review, a crawl audit surfaces site-wide patterns: pages accidentally blocked, broken internal links, orphaned pages with no incoming links, and mismatches between your sitemap and what is actually crawlable.

Why Run a Website Audit

A site can look fine to human visitors while quietly losing search visibility. If you have ever asked, “why isn’t my website showing up on Google,” the answer is almost always found in a crawl audit, not in your content — a page cannot rank if it was never indexed, and it cannot be indexed if a crawler could not reach or read it in the first place.

Core Website Audit Checklist

A thorough crawl audit checks these areas:

How to Check Your Google Index Status

Beyond crawling, it is worth confirming which pages are actually indexed versus just crawlable:

1. Google Search Console — the Coverage/Indexing report shows exactly which URLs are indexed, excluded, or have errors, and is the most reliable free indexing source
2. The site: search operator — running site:yourdomain.com gives a rough, non-exact estimate of indexed pages
3. URL Inspection Tool — checks a single URL’s live indexing status directly in Search Console
4. View page as Googlebot — rendering a page the way Googlebot sees it (via Search Console’s URL Inspection “View Crawled Page”) reveals content that JavaScript may be hiding from crawlers

Bulk or rapid index checkers can estimate indexing status across many URLs at once, but Search Console remains the ground truth since it reflects Google’s own data rather than an inferred site: search result.

Common Reasons Pages Aren’t Indexed

• Blocked by robots.txt (check for accidental Disallow: / on a live section)
• noindex meta tag left on from staging/development
• Canonical tag pointing to a different URL
• Thin or duplicate content Google chooses not to index
• Page not discoverable — no internal links, missing from sitemap
• Crawl budget exhausted on very large sites before reaching the page

Fixing Broken Links Found in a Crawl Audit

Broken internal links waste crawl budget and pass no value to the pages they used to point to. When your audit flags broken links:

1. Fix the link if the destination moved — update the href to the new URL
2. 301 redirect if the destination is permanently gone but a close replacement exists
3. Remove the link if there is no reasonable replacement — do not redirect broken links to your homepage as a catch-all, since it dilutes relevance signals
4. Re-crawl after fixing to confirm the audit dashboard clears the issue

Robots.txt Issues That Block Crawlers Unexpectedly

• A rule meant for staging (Disallow: /) accidentally left in production
• CSS/JS files blocked, preventing proper rendering and mobile-friendliness evaluation
• AI crawlers (GPTBot, ClaudeBot, PerplexityBot) unintentionally blocked by a wildcard Disallow: / rule meant for scrapers
• No Sitemap: line, leaving crawlers to discover pages purely through links

Use our Robots.txt Generator to rebuild a clean, correct file once your audit identifies problems.

Sitemap Coverage Gaps

“Sitemap coverage” means comparing what is in your sitemap.xml against what is actually indexable and crawled. Common gaps a crawl audit reveals:

• Pages in the sitemap that are blocked by robots.txt (crawlers ignore these entries)
• Important pages missing from the sitemap entirely, relying only on internal links for discovery
• Stale lastmod dates that do not reflect real content updates
• Old URLs still listed in the sitemap long after being removed or redirected

Use our Sitemap.xml Generator to rebuild an accurate sitemap once these gaps are identified.

Auditing for AI Crawler Visibility

Traditional crawl audits focus on Googlebot and Bingbot, but AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot) follow the same robots.txt rules and can be blocked the same way — often unintentionally, if a blanket bot-blocking rule was added without listing exceptions. A modern crawl audit should check AI bot access separately from traditional search bot access, since site owners frequently want different rules for each. Pair this with an LLMS.txt Generator file to help AI tools understand your site once they are able to reach it.

SEO Crawl Audit FAQ