Online Robots.txt Generator
Generate search-engine-compliant robots.txt directives to control crawler access, crawl delays, and XML sitemap locations.
The Mechanics of the Robots Exclusion Protocol (RFC 9309)
The Robots Exclusion Protocol—formally standardized by the IETF under RFC 9309 in 2022—governs how automated search engine crawlers, indexers, and archiving bots access public web content. Placed at the root of a domain (e.g., https://example.com/robots.txt), this plain text configuration file defines crawling permissions across your site's directory structure.
Every reputable search engine crawler (including Googlebot, Bingbot, and Applebot) requests and evaluates robots.txt before fetching pages. While robots.txt does not provide security or password protection, it helps manage crawler crawl budgets and prevents search engines from indexing administrative directories and staging environments.
Core Directives in robots.txt
User-agent: Specifies which web crawler the following rules apply to. UsingUser-agent: *applies directives universally to all automated crawlers. Specific bot identifiers (e.g.,User-agent: Googlebot) target individual search engines.Disallow: Tells crawlers not to fetch URLs matching the specified path prefix. For example,Disallow: /admin/blocks all files within the admin directory.Allow: Carves out explicit exceptions within disallowed directories (e.g., allowingAllow: /admin/public-guide.htmlinside an otherwise disallowed/admin/directory).Sitemap: Specifies the absolute URL of your XML sitemap, helping search engines discover newly published content.Crawl-delay: Supported by engines like Bing and Yandex to pace crawler requests and reduce server load (ignored by Googlebot, which manages crawl rate dynamically via Search Console).
Crucial Difference: Disallow vs. Noindex
A common misconception among web developers is assuming that adding a URL to robots.txt with Disallow will remove it from search results. This is incorrect.
A Disallow directive tells search engines not to crawl the page. However, if external websites link to that URL, search engines can still index the page and show it in search results without crawling its content. To guarantee a page is excluded from search results, remove the Disallow rule from robots.txt and add a <meta name="robots" content="noindex"> tag directly to the page's HTML <head>.
Frequently Asked Questions
Where should the robots.txt file be uploaded?
Upload robots.txt to the root directory of your web server so it is accessible at https://example.com/robots.txt. Subdirectory locations (like /tools/robots.txt) are ignored by search crawlers.
Will Disallowing a page remove it from Google search results?
No. Disallowing prevents crawlers from fetching the page content, but Google can still index the URL if other sites link to it. Use a meta 'noindex' tag to ensure complete removal from search results.
Does Googlebot respect the Crawl-delay directive?
No. Googlebot ignores Crawl-delay directives, managing its crawl rate dynamically. Bingbot, Yandex, and Baidu still support Crawl-delay settings.
Can malicious scrapers bypass robots.txt rules?
Yes. The Robots Exclusion Protocol relies on voluntary cooperation. Legitimate search engines adhere to its rules, but malicious scrapers and bad actors ignore them. Use server-side authentication for sensitive content.
How can I verify that my robots.txt file works properly?
Use the Robots.txt Report tool in Google Search Console to test your file and confirm that crawlers interpret your directives correctly.
Is my site data shared when generating directives?
No. All directive generation runs locally inside your browser's runtime. No directory structures or URLs are transmitted to external servers.