robots.txt Configuration Guide: Control Search Crawlers
robots.txt is the core file that controls how search engine crawlers access your site. This guide explains file location, user-agent/allow/disallow/sitemap syntax, wildcards, precedence, and common configuration examples.