Free Robots.txt Generator

What Is a Robots.txt File?

A robots.txt file is a plain text file placed at the root of your domain (for example, https://yoursite.com/robots.txt) that tells search engine and AI crawlers which parts of your site they are allowed to access. It follows the Robots Exclusion Protocol — a voluntary standard that reputable crawlers such as Googlebot, Bingbot, ClaudeBot, GPTBot, and PerplexityBot generally respect, but it is not a security mechanism.

Important: robots.txt controls crawling, not indexing. A page can still appear in search results even when it is disallowed — use noindex meta tags or headers for that instead.

Basic Robots.txt Syntax

User-agent: * Disallow: /admin/ Disallow: /cart/ Allow: /admin/public/ Sitemap: https://yoursite.com/sitemap.xml

Core Directives

Robots.txt Wildcards Explained

• * matches any sequence of characters. For example, Disallow: /*.pdf$ blocks all PDF files.
• $ anchors the match to the end of the URL. For example, Disallow: /search$ blocks only /search.

User-agent: * Disallow: /*?sort= Disallow: /*.json$

Robots.txt for WordPress

WordPress generates a virtual robots.txt by default. To customize it, either edit the physical file in your root directory if it exists, or use an SEO plugin such as Yoast or Rank Math.

User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Disallow: /?s= Sitemap: https://yoursite.com/sitemap_index.xml

Avoid blocking /wp-content/uploads/ because that hides your images from Google Image Search.

Robots.txt for Shopify

Since 2021, Shopify allows direct editing of a robots.txt.liquidtheme file for full control over crawl rules, letting you add custom disallow rules while keeping Shopify's default protections for cart, checkout, and account pages.

Robots.txt and Sitemap.xml: How They Work Together

Robots.txt tells crawlers what not to crawl, while sitemap.xml tells them what to crawl and how it is structured. Always reference your sitemap inside robots.txt:

Sitemap: https://yoursite.com/sitemap.xml

If your sitemap contains URLs also blocked in robots.txt, crawlers will ignore those entries — a common and easily overlooked mistake.

Blocking or Allowing AI Crawlers (2026 Update)

Allow AI crawlers for visibility in AI answers:

User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: /

Block AI training crawlers but keep search crawlers:

User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Googlebot Allow: /

Pair this with our LLMS.txt Generator for a structured, machine-readable alternative aimed specifically at LLMs.

Robots.txt Disallow: Directories, Subdomains & Folders

• Disallow a folder: Disallow: /folder-name/
• Disallow a subdomain: rules only apply to the (sub)domain they are hosted on — staging.yoursite.com needs its own robots.txt at staging.yoursite.com/robots.txt.
• "Disallow" ≠ noindex: robots.txt cannot noindex a page; use <meta name="robots" content="noindex"> instead — and don't block that page in robots.txt, or Google cannot even see the noindex tag.

"Indexed, Though Blocked by Robots.txt" — How to Fix It

This Google Search Console warning means Google found links to a blocked page and indexed the URL anyway without content. Fix it by either:

1. Removing the Disallow rule and adding a noindex tag instead, or
2. Keeping it blocked, removing internal links to it, then requesting removal in Search Console.

Common Robots.txt Mistakes

• Blocking CSS and JS files needed for Google to render your page properly
• Using Disallow: / accidentally on a live production site
• Forgetting the Sitemap: directive
• Case-sensitivity errors such as /Folder/ versus /folder/
• Assuming Disallow prevents indexing (it does not — use noindex)
• Placing the file anywhere other than the domain root

Robots.txt FAQ