Robots.txt SEO Guide: Rules, Examples & Testing

A small robots.txt mistake can block an important section of a site from crawling, while a correct rule can help manage crawler access to URL areas that do not need to be fetched. Because the file controls crawling rather than access or indexing, it is important to use it for the right purpose.
This guide explains how Google reads robots.txt, how user-agent groups and Allow/Disallow rules work, how to add a sitemap directive, and how to diagnose common errors with Search Console and URL Inspection. For affected URLs, investigate blocked by robots.txt in Search Console alongside the current crawl rules and URL Inspection result.
For a broader technical review, see the Technical SEO Audit Checklist and check robots.txt alongside your status codes, canonical signals, and indexability settings.
Ready to Grow Your Business with SEO?
Let's work together to improve your rankings, drive more traffic, and grow your business online.
What Is robots.txt?
robots.txt is a plain-text file at the root of a host that gives instructions to compliant crawlers about which URL paths they may request. For example, the file for https://www.example.com/ is normally available at https://www.example.com/robots.txt.
Rules apply to the host and protocol where the file is served. A file on example.com does not automatically govern www.example.com, another subdomain, or a different protocol. Check the actual production host variants your site serves and make sure redirects and robots files behave as intended.
Robots.txt is not a security control. It does not authenticate users or keep confidential content private. Do not list secrets or sensitive paths in a publicly readable file; protect private resources with authentication and server-side access controls.
What robots.txt Does—and Does Not Do
Robots.txt controls whether a compliant crawler may fetch a matching URL. It does not reliably remove a URL from search results. If a blocked URL is discovered through links or other signals, Google may still show the URL without a description because it cannot crawl the page to read its content.
If a page should be removed from search, allow crawling so Google can see a noindex directive, or require authentication. Do not disallow the URL at the same time that you expect Google to read its page-level noindex or canonical tag.
Robots rules also do not replace canonical tags, redirects, or internal links. A robots.txt disallow is not a canonical signal, and it does not guarantee that link signals consolidate to another URL. Decide separately whether crawlers should fetch the page, whether it should be indexed, and which URL is the preferred version. For a comparison of the two URL signals, see Canonical Tags in SEO. Filter-generated paths also need a deliberate policy; see Faceted Navigation SEO.
Robots.txt Location and File Format
Serve one plain-text robots.txt file at the root of each applicable host, for example https://www.example.com/robots.txt. Use UTF-8 text, valid line breaks, and the exact host and protocol you intend to control. A robots.txt file is not placed inside a subdirectory to control the rest of the site.
Google supports robots.txt files up to 500 KiB and ignores content beyond that limit. Keep the file concise and remove obsolete rules instead of adding layers of increasingly broad exceptions. Other crawlers can have different limits and syntax support.
Common directives include:
User-agentselects a crawler group.Disallowidentifies paths the selected crawler should not request.Allowprovides a more specific allowed path when it overlaps with a disallow rule.Sitemapidentifies an absolute sitemap URL for crawlers that support the directive.
Robots.txt Examples
The following examples are illustrative only. Replace the example hostname and paths with rules that have been tested against your own site. A broad disallow can have significant effects.
Allow Crawling and Declare a Sitemap
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xml
An empty Disallow value does not block any paths for this group. The sitemap directive uses an absolute URL. A sitemap helps discovery, but listing a URL does not override a robots rule that prevents crawling it.
Disallow a Non-Search Path
User-agent: *
Disallow: /internal-tools/
Use a path rule only after confirming that the path contains no pages or resources Google needs to crawl. The robots.txt rule is not access protection; if the path contains private information, secure it at the server or application level instead.
Allow a More-Specific Path
User-agent: *
Disallow: /catalog/
Allow: /catalog/help/
Google compares the matching rules and applies the most specific path. Here, the more specific Allow rule permits crawling under /catalog/help/ even though the broader catalog path is disallowed. If two matching rules have the same specificity, Google uses the less restrictive rule.
Googlebot User-Agent Group Precedence
Google selects the most specific user-agent group that matches the crawler. A group specifically for Googlebot does not automatically inherit rules from a separate wildcard (*) group. Do not assume that directives in the general group are combined with a more specific Googlebot group.
User-agent: *
Disallow: /private/
User-agent: Googlebot
Disallow: /tmp/
In this example, the Googlebot group is more specific, so its rules do not automatically include the wildcard group’s Disallow: /private/. If both restrictions are intended for Googlebot, repeat the relevant rule in the Googlebot group or design and test the groups accordingly. Rules for other user-agent products are interpreted according to that crawler’s documented behavior.
Group matching is separate from path-rule specificity. First determine the applicable user-agent group; then compare matching Allow and Disallow paths within that group. Google’s parser also supports certain extensions such as wildcards and end-of-URL matching; do not assume every crawler supports the same syntax.
Allow and Disallow Rule Precedence
When multiple rules in the applicable group match a URL, Google applies the rule with the most specific matching path, generally the longest matching path. If an Allow and Disallow rule match equally specific paths, Google chooses the less restrictive rule, which is Allow.
Test representative URLs, not only the rule text. Check the parent path, a nested path, URLs with and without trailing slashes, and any relevant query-string variations. Rules are path matching instructions, not a way to express every application-level permission or URL normalization policy. For large URL inventories, pair this review with Crawl Budget Optimization and a crawl of filtered URL patterns.
Add a Sitemap Directive
You can include a Sitemap directive containing an absolute URL, such as Sitemap: https://www.example.com/sitemap.xml. The directive is not tied to a specific user-agent group. You can also submit a sitemap through Search Console; these methods complement each other.
Keep the sitemap and robots policy consistent. A sitemap should normally list canonical URLs you want considered for indexing, not URLs you have intentionally blocked from crawling. A sitemap does not grant permission to crawl a disallowed URL or guarantee indexing.
For sitemap setup and coverage checks, see the XML Sitemap SEO guide.
How HTTP Status Codes Affect robots.txt
Google’s behavior depends on whether it can fetch a usable robots.txt file:
- Successful 2xx response: Google processes the file returned by the server.
- Most 4xx responses: Google generally treats robots.txt as unavailable and assumes there are no crawl restrictions. HTTP 429 is an exception and is treated as a server error for this purpose.
- 5xx or network errors: Google may temporarily stop crawling the site while it cannot fetch robots.txt. It can use a previously cached copy for a limited time; prolonged failures can change the fallback behavior.
- Redirects: Google follows a limited number of redirects to find the file. Make sure the final response is accessible, stable, and serves the intended robots.txt for the host.
Do not intentionally rely on an error status to allow or block crawling. Return the correct robots.txt file with a successful response and monitor it when changing CDN, hosting, or redirect configuration. For URL redirect issues, see Redirect Chains and Broken Links.
Common robots.txt SEO Mistakes
- Blocking the whole site: A leftover
Disallow: /can stop crawling of all paths for the matching group. - Trying to deindex with robots.txt: A crawler cannot read page-level noindex or canonical directives on a blocked page.
- Blocking JavaScript or CSS needed for rendering: Google may not be able to render or understand pages that depend on those resources. See the JavaScript SEO Guide.
- Assuming wildcard rules are inherited: A specific user-agent group does not automatically inherit the wildcard group’s rules.
- Using unsupported directives: Google does not support
Crawl-delay. Other crawlers may implement their own extensions, so check their documentation. - Blocking a URL already indexed: The URL can remain known to Google without a snippet if external links or other signals still expose it.
- Publishing conflicting or stale rules: Old staging, filter, or temporary rules can block valuable pages after deployment.
How to Test robots.txt
1. Check the Live File
Open /robots.txt on each production host and verify the response status, content type, final URL after redirects, and exact file contents. Check HTTP and HTTPS and both www and non-www hosts if they resolve separately. Confirm there is no CDN error page, authentication challenge, or stale cached version being served.
2. Review the Search Console robots.txt Report
Google Search Console’s robots.txt report is available for domain properties. It shows robots.txt files Google found for top hosts, when they were last crawled, and reported warnings or errors. Use the report to verify the version Google has fetched and to identify file availability problems; it is not a substitute for checking every URL’s indexing status.
3. Use URL Inspection for a Specific URL
Use URL Inspection to examine Google’s information about a particular URL and, where available, run a live test. The live test can help determine whether Google can fetch that URL now and whether a robots restriction is reported for it. It does not replace checking the robots.txt file, auditing all rule groups, or testing a representative set of paths.
A crawler’s robots.txt tester can help simulate how a parser interprets candidate rules, but implementations can differ. For Google-specific confirmation, combine rule review with the robots.txt report and URL Inspection. Re-test after publishing the file and allow time for Google to fetch an updated copy.
Troubleshoot a robots.txt Blocking Issue
- Identify the affected canonical URL and the exact host that serves it.
- Fetch the live robots.txt file from that host and note its status, redirects, and contents.
- Find the most specific user-agent group that applies to Googlebot.
- Compare all matching Allow and Disallow paths, including their specificity.
- Check whether a CDN, framework, or deployment process is serving a different file than expected.
- Remove or adjust only the rule that causes the unwanted block; avoid broad allow-all or disallow-all changes without testing.
- Re-test the affected URL and important neighboring paths, then confirm the updated file is being fetched.
If Google reports that it cannot crawl an important page, also inspect the page response, redirects, robots meta tag, X-Robots-Tag header, and canonical. Robots.txt is only one part of crawl and indexing diagnosis. The guide to why Google isn’t indexing your website covers the wider troubleshooting sequence.
Robots.txt Checks During a Site Migration
Before launch, compare the production robots policy with staging and confirm that a staging-wide block will not be copied to the live host. After launch, check every relevant host, review the live file, and confirm that production pages and required rendering resources are not blocked.
Keep redirects, canonicals, sitemap entries, and crawl rules aligned as routes change. The Website Migration SEO Checklist includes the broader launch and post-launch checks.
Robots.txt SEO Checklist
- Serve a valid plain-text file from the root of each relevant host.
- Use user-agent groups intentionally; do not assume wildcard rules are inherited by specific groups.
- Check the most-specific matching path and resolve equal-specificity conflicts with the correct Allow or Disallow rule.
- Do not use robots.txt for security or as a substitute for noindex.
- Keep required JavaScript and CSS resources crawlable.
- Use an absolute sitemap URL and keep sitemap entries consistent with crawl and indexing intent.
- Check response codes, redirects, CDN behavior, and Google’s cached or fetched version.
- Use the Search Console robots.txt report and URL Inspection for their distinct purposes.
- Do not rely on Google-supported Crawl-delay; Google does not support that directive.
FAQs
Does robots.txt stop a page from being indexed?
No. It controls crawling, not indexing. A blocked URL can still appear in search if Google discovers it elsewhere. Use noindex on a crawlable page or require authentication when the goal is exclusion or privacy.
Do Googlebot-specific rules inherit rules from User-agent: *?
No. Google applies the most specific matching user-agent group, and a specific group does not automatically inherit directives from the wildcard group.
Which rule wins if Allow and Disallow both match?
Google uses the most specific matching path. If the matching Allow and Disallow rules have equal specificity, the less restrictive Allow rule wins.
How large can a robots.txt file be for Google?
Google processes up to 500 KiB of a robots.txt file and ignores content beyond that limit.
Does Google support Crawl-delay?
No. Google’s robots.txt parser does not support Crawl-delay. Check the documentation of other crawlers before relying on crawler-specific extensions.
What is the difference between the robots.txt report and URL Inspection?
The robots.txt report shows files Google found for domain-property hosts and reports fetch warnings or errors. URL Inspection provides information and live testing for a specific URL; it is not a sitewide robots rule audit.
Need help checking crawl restrictions before they block important pages? Request a Technical SEO Audit to review your robots.txt, rendering resources, and indexability signals.
Official Google Documentation
Frequently Asked Questions
Does robots.txt stop a page from being indexed?
No. It controls crawling, not indexing. A blocked URL can still appear in search if Google discovers it elsewhere. Use noindex on a crawlable page or require authentication when the goal is exclusion or privacy.
Do Googlebot-specific rules inherit rules from User-agent: *?
No. Google applies the most specific matching user-agent group, and a specific group does not automatically inherit directives from the wildcard group.
Which rule wins if Allow and Disallow both match?
Google uses the most specific matching path. If the matching Allow and Disallow rules have equal specificity, the less restrictive Allow rule wins.
How large can a robots.txt file be for Google?
Google processes up to 500 KiB of a robots.txt file and ignores content beyond that limit.
Does Google support Crawl-delay?
No. Google’s robots.txt parser does not support Crawl-delay. Check the documentation of other crawlers before relying on crawler-specific extensions.
What is the difference between the robots.txt report and URL Inspection?
The robots.txt report shows files Google found for domain-property hosts and reports fetch warnings or errors. URL Inspection provides information and live testing for a specific URL; it is not a sitewide robots rule audit.