robots.txt Configuration Guide: Control Search Crawlers

robots.txt Configuration Guide: Control Search Crawlers

robots.txt is a plain-text file placed in your site's root directory that tells search engine crawlers which parts of your site they may or may not crawl. It won't directly improve rankings, but it is the most fundamental step in SEO optimization: misconfigured rules waste crawl budget at best and block important pages from being indexed at worst. This guide follows the official Google Search Central specification (RFC 9309, the Robots Exclusion Protocol), starting from where the file must live, through setting separate rules for AI crawlers, and ending with config you can copy directly.

File Location and Basic Format

robots.txt must be placed in the top-level directory of your site and the URL is case-sensitive. A valid file URL looks like https://example.com/robots.txt and only applies to that host, protocol, and port — the robots.txt on www.example.com cannot control crawling on example.com, and http and https do not share rules. The file must be UTF-8 encoded plain text with lines separated by newlines. Google ignores unparseable lines, and the file size limit is 500 KiB.

Once placed, verify it is publicly reachable:

curl -s https://example.com/robots.txt

If you get a 404, the file is not where crawlers expect it (commonly because it was uploaded to a subdirectory, or the static file server is not configured). The simplest format looks like this:

User-agent: *
Disallow: /admin/
Allow: /public/
Sitemap: https://example.com/sitemap.xml

Core Directives and a Complete Example

Get the four core directives straight first:

• user-agent: identifies which crawler the rules apply to; field names and values are case-insensitive. * matches all crawlers.
• disallow: paths that may not be crawled. Remember that robots.txt only blocks crawling, not indexing.
• allow: paths that may be crawled, useful for permitting specific paths within a broader block.
• sitemap: declares the absolute URL of your sitemap; it is independent of user-agent and can be followed by all crawlers.

A typical blog config that blocks the admin and search pages while letting rendering resources through:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /?s=          # on-site search result pages
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Rules accumulate: within the same user-agent group, later rules add to earlier ones rather than replacing them, and conflicts are resolved by the most specific rule winning.

Wildcards, Precedence and Matching Details

Google, Bing, and other major search engines support a limited form of wildcards: * matches any sequence of characters, and $ marks the end of the URL. A few concrete examples:

Matching uses the most specific rule first: Allow: /p is more specific than Disallow: /, so /page is allowed. When two rules are equally specific and conflict, Google applies the least restrictive one. Matching is also case-sensitive: /fish matches /fish.html and /fish/salmon.html, but not /Fish.asp.

Common Combinations

Here are the go-to patterns for common situations:

• Allow Google only: User-agent: Googlebot + Allow: /, and Disallow: / for everyone else. Good for staging sites or when you only want Google indexing — but be aware it also keeps Bing traffic out.
• Block the whole site temporarily: User-agent: * + Disallow: /. Useful while a site is still under construction; remove it after launch or nothing will be crawled.
• Block parameterized URLs: Disallow: /*?*. Helpful for e-commerce filters or on-site search that generate many URLs — but check your URL patterns first, because legitimately parameterized pages will be caught too.

One detail people overlook: the Crawl-delay directive no longer works for Google or Bing. To control crawl rate, rely on crawl statistics, sitemap update frequency, and other indirect levers instead.

Manage AI Crawlers Separately from Search Engines

Sites now face AI crawlers such as GPTBot and ClaudeBot in addition to search engines, and their default user-agents differ:

If you don't want your content used for training, declare it explicitly in the file:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

To understand how robots.txt and llms.txt work together, and whether to offer an AI-friendly structured entry point, refer to the robots.txt and llms.txt configuration guide.

robots.txt vs. noindex

This is the most commonly confused point: blocking with robots.txt is not the same as blocking indexing. Google may still index a URL based on links from other pages, even without crawling the body content. If you want a page to stay completely out of search results, use a noindex meta tag, or declare it via a response header:

X-Robots-Tag: noindex

For the full crawl-to-index pipeline, see the website indexation guide.

Common Mistakes

• Using robots.txt to take a page down: it cannot block indexing, so the page may still show a title and snippet, which is harder to clean up; use noindex or a 404 instead.
• Blocking CSS and JS too: common in the past; it now hurts rendering-based content. Re-check with the URL Inspection tool after changing it.
• Getting the case wrong: /Admin and /admin are different paths, so the block may simply not apply.
• Forgetting the Sitemap: line: putting your sitemap in robots.txt is one of the cheapest ways to submit URLs.

How to Verify After Changes

After editing robots.txt, first curl it to confirm it is reachable, then validate each rule in the robots.txt tester in Google Search Console, and finally watch the crawl statistics for a while. To systematically hunt down other crawling problems, work through the technical SEO checklist.

Reference: Google robots.txt specification https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt; RFC 9309 (REP) https://www.rfc-editor.org/rfc/rfc9309