robots.txt and llms.txt Configuration Guide
In the search-engine era, site owners used robots.txt to tell crawlers where they could and could not go. Now there are AI crawlers too, and the rules have suddenly gotten complicated. OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot each fetch pages under their own User-Agent for training and question-answering. llms.txt takes a different angle: instead of passively blocking, you proactively hand the AI a list of "which pages to look at". Used together, the two files protect sensitive content without wasting the traffic your public pages generate.
First, clear up a common misconception: robots.txt is not a security mechanism. It is a "gentleman's agreement" that relies on crawler cooperation — genuinely malicious scrapers never look at it. Its job is to manage the behavior of "normal crawlers", not to protect sensitive data. Sensitive data should be protected by access control: logins, permissions, IP allowlists. Once you understand this, you know what robots.txt should and should not contain.
Basic robots.txt Setup
User-agent: *
Disallow: /admin/
Disallow: /debug/
Allow: /public/
Sitemap: https://example.com/sitemap.xml
The basic rules are: block admin and debug paths, allow public content pages, and specify the sitemap URL. Note that Disallow means "do not crawl", not "do not index"; if content itself should not appear in search, you also need a noindex tag. A real example: many sites put the login page at /login, but the signup page at /signup should be indexed — treat the two differently. The Sitemap: directive is not honored by every crawler, but it costs nothing to add and helps search engines discover new pages faster.
Separate Rules for AI Crawlers
User-agent: GPTBot
Disallow: /private/
User-agent: ClaudeBot
Disallow: /private/
User-agent: PerplexityBot
Disallow: /private/
Common AI crawlers include GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. If you want only some pages available for training, restrict to those paths; if you do not want a specific crawler at all, use Disallow: /. Whether to allow AI crawling depends on your content strategy: content sites usually welcome AI citations for referral traffic, while sites built on tools or private data tend to close the door.
The decision comes down to three questions: will your content get cited by AI and drive clicks? Does your content have irreplaceable, exclusive value? Could AI crawling leak the core asset your business is built on? A tutorial site that lets GPTBot crawl may find ChatGPT citing its articles with links, producing steady referral traffic; a SaaS selling data-analysis reports, on the other hand, loses a reason to subscribe if its core reports are scraped and summarized — that site should use Disallow: /.
Common AI Crawlers
| User-Agent | Owner | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training ChatGPT / GPT models |
| ClaudeBot | Anthropic | Training Claude models |
| PerplexityBot | Perplexity | Search Q&A citations |
| Google-Extended | Gemini training and AI summaries |
What Is llms.txt
llms.txt is a "site manual" for large models, hosted at the domain root, listing the site intro, main pages, and key resource links in Markdown. It complements robots.txt: robots.txt decides whether a page can be fetched; llms.txt decides how the page is understood once fetched.
llms.txt Example
# Example.com
> One-stop website building service introduction.
## Main Pages
- [Home](https://example.com/)
- [Pricing](https://example.com/pricing)
- [About Us](https://example.com/about)
## Help Center
- [FAQ](https://example.com/faq)
The format has a few spec points to follow. The first line must be an H1 title (# site name), immediately followed by a blockquote (>) with the site description — both are required, and AI reads them first to understand "what this site is". Then use H2 sections (##) with Markdown links listed under each. Links can be relative or absolute URLs, but absolute URLs are recommended so the AI can fetch them directly. llms.txt is meant to be curated, not exhaustive: listing 5-20 of the most valuable pages is enough. Stuffing every page in dilutes the AI's attention on the pages that matter.
Configuration Reference
| Scenario | Recommendation |
|---|---|
| Admin, debug, or temporary pages | Block in robots.txt, add noindex if needed |
| Public content pages | Allow in robots.txt, list as main pages in llms.txt |
| Sensitive or paid content | Exclude from both, keep consistent with privacy terms |
| Let AI see only curated pages | Allow in robots.txt, list only curated links in llms.txt |
Post-Launch Checklist
After publishing the config files, do three things. First, use curl to confirm both files return normally (curl https://example.com/robots.txt) — some CDNs or frameworks block direct access to txt files. Second, validate rule syntax in Search Console's "robots.txt tester"; one wrong wildcard can block your entire site from crawling. Third, review quarterly — the roster and purpose of AI crawlers change fast; a crawler that was active six months ago may have switched User-Agents, and you need to update your rules accordingly.
Frequently Asked Questions
- How fast do changes take effect? Crawlers re-fetch robots.txt anywhere from hours to days later; it is not immediate.
- Does editing robots.txt affect existing index entries? It only affects future crawls; already-indexed pages are re-processed only after the next crawl, and pairing with noindex makes the intent clearer.
- Does llms.txt need to list every page? No, curate it. Too many pages raises the AI's crawling cost and dilutes the weight of your key pages.
References
Reference: robots.txt documentation https://developers.google.com/search/docs/crawling-indexing/robots/intro
Reference: llms.txt proposal https://llmstxt.org/
Reference: Managing AI crawlers at Google https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
For a systematic review of crawl and indexing strategy, combine the robots.txt configuration guide and the XML sitemap guide; for AI search specifically, see the generative AI search optimization guide.