robots.txt is the smallest file on your site and one of the easiest to get catastrophically wrong. This guide covers what it actually controls, what it cannot do, the exact rules Google applies, and how to configure it for AI crawlers like GPTBot and ClaudeBot without losing search visibility.
Quick answer
robots.txt is a plain text file at the root of your domain that tells automated crawlers which URLs they may fetch. It controls crawling, not indexing, and not ranking.
Key points before you touch the file:
- It manages crawl traffic. It does not keep a page out of search. A blocked URL can still be indexed if other pages link to it.
- To keep a page out of results, use
noindexon a page that Google is allowed to crawl. To keep content private, use a password or server-side access control. - Google supports only four fields:
user-agent,disallow,allow, andsitemap. Everything else, includingcrawl-delay, is ignored by Google. - In 2026 the file also decides whether AI answer engines can read and cite you. Blocking the wrong bot removes you from AI search results.
- The file is voluntary. Reputable crawlers obey it. Scrapers do not. For enforcement you need a firewall or bot management.
Table of contents
- What is robots.txt?
- Crawl, index, rank: the distinction that explains robots.txt
- robots.txt vs noindex vs authentication
- Where to put robots.txt: location and scope
- File format and size limits
- robots.txt syntax and supported directives
- User-agent groups and precedence
- Wildcards and rule precedence
- How Google handles robots.txt errors
- robots.txt and crawl budget
- robots.txt for AI crawlers (GPTBot, ClaudeBot, Google-Extended)
- Common robots.txt mistakes
- How to test and deploy robots.txt
- robots.txt examples and templates
- Frequently asked questions
- Sources and further reading
What is robots.txt?
A robots.txt file is a set of rules that tells crawlers which parts of your site they may fetch. It is part of the Robots Exclusion Protocol, standardized as RFC 9309 and supported by Google, Bing, and other major engines. Google's own summary is in its robots.txt introduction.
The most useful way to think about the file is as a budget tool. A search engine will not spend more to process a page than the page is worth to it. Google crawls, then renders, then indexes, and only then considers ranking, and each step costs machine time. During the Google antitrust trial, Pandu Nayak, then Google's head of Search, testified that the most expensive ranking systems do not run on every URL. A page first has to clear cheaper checks to earn the costly evaluation.
robots.txt is the earliest lever you have over that spending. It works before rendering, before indexing, and long before any ranking model sees the page. Get it wrong and nothing downstream matters, because the crawler either never reaches the content or wastes its budget on pages that should never have been fetched.
Crawl, index, rank: the distinction that explains robots.txt
Three separate stages decide whether a page shows up in search, and robots.txt only touches the first one.
Crawling is fetching the page. Indexing is storing and understanding it. Ranking is deciding where it appears for a query. robots.txt is a crawling instruction. It says "do not fetch this," not "do not index this," and not "do not rank this."
This sounds like a technicality until you see the failure it causes. If you disallow a URL, Google will not fetch it, but it can still index that URL if other pages link to it. The result is a listing in search results with no snippet and a note that the description is unavailable. You blocked the crawl and accidentally kept the page in the index. Google states this directly: robots.txt is not a mechanism for keeping a page out of Google.
There is a second trap. To keep a page out of the index you use a noindex rule, but Google can only see that rule if it crawls the page. If the same URL is also disallowed in robots.txt, Google never fetches it, never sees the noindex, and the page can stay indexed. Blocking crawling and blocking indexing on the same URL cancel each other out. Pick one. Google explains the correct approach in its guide to blocking indexing with noindex.
robots.txt vs noindex vs authentication
Use the tool that matches the goal.
| Your goal | Use this | Not this |
|---|---|---|
| Manage crawl traffic and crawl budget | robots.txt disallow |
noindex |
| Keep a web page out of search results | noindex on a crawlable page |
robots.txt |
| Keep a non-HTML file (PDF, image) out of search | X-Robots-Tag: noindex HTTP header |
robots.txt |
| Keep content private or secure | Authentication or access control | robots.txt |
The noindex and X-Robots-Tag rules are documented in Google's robots meta tag specification. One important point from that page: a robots meta tag or X-Robots-Tag is only discovered when the URL is crawled, so a URL that carries an indexing rule must not be disallowed in robots.txt.
A note on security. robots.txt is public. Anyone can open yourdomain.com/robots.txt and read it. Listing a sensitive path there tells the world exactly where to look, so it is worse than useless as a privacy measure. Protect private content with authentication.
Where to put robots.txt: location and scope
The file must sit at the top level of a host, reachable at https://example.com/robots.txt. Crawlers do not look in subdirectories, so a file in a subfolder is ignored. Google's steps for publishing one are in create and submit a robots.txt file.
Scope is stricter than most people expect. A robots.txt file applies only to the exact host, protocol, and port it lives on.
| robots.txt URL | Valid for | Not valid for |
|---|---|---|
https://example.com/robots.txt |
example.com and its subdirectories |
www.example.com, shop.example.com, http://example.com |
https://www.example.com/robots.txt |
www.example.com only |
example.com |
https://example.com/folder/robots.txt |
nothing (invalid location) | everything |
Standard ports are treated as their default host, so https://example.com:443/robots.txt also covers https://example.com/. Google supports the file over HTTP, HTTPS, and FTP. If you run separate subdomains for a blog, a shop, or staging, each one needs its own file, and you should confirm each serves what you expect.
File format and size limits
The file must be UTF-8 encoded plain text. Lines can end in CR, LF, or CR/LF. Google ignores invalid lines, including a byte order mark at the start of the file, and parses only the lines it recognizes. If you accidentally serve HTML instead of plain text, Google tries to extract rules and ignores the rest, which usually means your rules do not work the way you intended.
Two limits cause real problems:
- Encoding. If the file is not UTF-8, Google may drop characters outside the UTF-8 range and break rules that rely on them.
- Size. Google enforces a limit of 500 kibibytes. Anything past that point is ignored, with no error. On large sites with sprawling rules, the lines at the bottom of an oversized file simply stop working. If you are near the limit, consolidate. Moving excluded material into a single directory lets you block it with one rule instead of hundreds.
robots.txt syntax and supported directives
Every valid line is a field, a colon, and a value, with an optional comment after #. Field names are case-insensitive. Spaces are optional but help readability.
# This is a comment
User-agent: Googlebot
Disallow: /private/
Allow: /private/public-file.html
Sitemap: https://example.com/sitemap.xml
Google supports exactly four fields, described in full in how Google interprets the robots.txt specification. Everything else is ignored.
| Directive | What it does | Example |
|---|---|---|
user-agent |
Names the crawler the rules apply to | user-agent: Googlebot |
disallow |
A path the crawler must not fetch | disallow: /admin/ |
allow |
A path the crawler may fetch, used to carve an exception out of a broader disallow | allow: /admin/public.html |
sitemap |
The absolute URL of a sitemap or sitemap index | sitemap: https://example.com/sitemap.xml |
The field most people expect and Google does not honor is crawl-delay. Google ignores it. Bing and some other crawlers still read it, and Anthropic honors it too, so it is not useless everywhere, but it does nothing for Googlebot. To slow Google's crawling, use the guidance in reduce Google's crawl rate instead. The same applies to host, noindex, nofollow, request-rate, and visit-time. Through 2026 Google began publicly naming the unsupported directives it sees most often, because so many files carry leftover lines that owners assume are policy. For Google, unsupported means unsupported, even when the word looks official.
The sitemap field behaves differently from the rest. It is not a crawl rule and is not tied to any user agent. It must be an absolute URL, it does not have to live on the same host as the robots.txt file, and you can list as many as you want.
User-agent groups and precedence
You group rules by repeating user-agent lines. You can stack several user agents before one set of rules to apply the same rules to all of them.
The precedence logic is where subtle bugs live. Only one group applies to a given crawler, and Google picks the group with the most specific matching user agent. All other groups are ignored, including the global * group. Google does not merge a specific group with the wildcard group.
User-agent: *
Disallow: /admin/
Disallow: /cart/
User-agent: Googlebot
Disallow: /no-google/
In this example Googlebot does not block /admin/ or /cart/, because it follows its own group and never reads the wildcard group. If you want Googlebot to avoid those directories too, list them again under Googlebot.
One more rule worth memorizing: the * wildcard matches all crawlers except Google's AdsBot crawlers. AdsBot must be named explicitly, or your ad landing page checks keep crawling paths you thought you had blocked. Every Google user agent is listed in Google's common crawlers reference.
Wildcards and rule precedence
Rule values are case-sensitive, even though field names are not. /Fish and /fish are different paths. This alone causes a surprising number of blocks that do not fire, or fire on the wrong URL.
Google supports two wildcards:
*matches zero or more of any character.$matches the end of the URL.
| Pattern | Matches | Does not match |
|---|---|---|
/fish |
/fish, /fish.html, /fishheads, /fish/salmon.html |
/catfish, /Fish.asp |
/fish/ |
/fish/salmon.html, /fish/?id=1 |
/fish, /fish.html |
/*.php$ |
/index.php, /folder/name.php |
/name.php?x=1, /name.php5 |
/$ |
/ (the root only) |
/page, /folder/ |
When two rules could apply to the same URL, Google uses the most specific rule, measured by the length of the rule path. When rules conflict at the same specificity, Google uses the least restrictive rule, which almost always means allow beats disallow. Google's own examples make this concrete:
- For
/page, withallow: /panddisallow: /, the allow wins because it is more specific. - For
/folder/page, withallow: /folderanddisallow: /folder, the allow wins because on a tie Google takes the least restrictive rule. - For
/page.htm, withallow: /pageanddisallow: /*.htm, the disallow wins because its path matches more characters and is more specific.
If you rely on a broad disallow to block a section, make sure a stray allow or a short wildcard elsewhere does not quietly reopen it.
How Google handles robots.txt errors
The HTTP status your server returns for the robots.txt request changes crawler behavior, and the differences are large enough to cause outages.
| Status | Google's behavior |
|---|---|
2xx success |
Use the file as served |
4xx (except 429) |
Treat as if no robots.txt exists and assume no crawl restrictions |
429 or 5xx |
Pause crawling for about 12 hours, then fall back to the last good cached copy for up to 30 days |
3xx redirect |
Follow at least five redirect hops, then treat as a 404 |
| DNS or network error | Treat the same as a server error |
The practical lesson: an unstable robots.txt endpoint is dangerous in both directions. Serve a 403 by accident and Google may crawl everything you meant to block. Do not use 401 or 403 to throttle crawling. Serve a 503 for too long and Google may stop crawling your site entirely. This file needs to be as reliable as your homepage.
Google caches robots.txt for up to 24 hours, and longer if it cannot refresh the file. It respects max-age in the Cache-Control header, so you have some control over how quickly changes take effect. When you need an urgent refresh, the robots.txt report in Search Console can request a recrawl.
robots.txt and crawl budget
For most small sites, crawl budget is not something to worry about. Google usually crawls a few thousand URLs efficiently on its own, and the useful work is keeping your sitemap current and your internal links clean.
Crawl budget becomes a real constraint on large sites, on the order of hundreds of thousands or millions of URLs, and on medium sites that generate many new or frequently changing pages. Google's guidance on this is in optimize your crawl budget. The classic budget sink is not your article pages. It is the machine-generated long tail: faceted navigation, filter and sort parameters, session IDs, internal search pages, calendars that stretch to infinity, and endless query-string combinations that all resolve to near-identical content. Left open, these can absorb most of a crawler's attention and starve the pages you actually want fetched and refreshed.
This is where robots.txt pays off. Disallowing parameter patterns and infinite spaces keeps the crawler focused on canonical, valuable URLs. It is a direct application of the cost logic: you lower the price of crawling your site so more of the budget lands on pages worth ranking. Google has grown less tolerant of sites that scale thin, auto-generated pages, and its guidance points toward more selective indexing. A clean robots.txt that stops the crawler from drowning in low-value URLs is part of keeping a large site economical to process.
Two cautions:
- Disallowing a URL does not remove it from the index if it is already indexed and linked. For cleanup, allow the crawl and serve
noindexuntil the pages drop out, then block the pattern. - Never disallow the CSS, JavaScript, or API endpoints Google needs to render your pages. Blocking rendering resources can make Google see a broken or empty page, which hurts far more than the crawl saving helps.
robots.txt for AI crawlers (GPTBot, ClaudeBot, Google-Extended)
Until recently, robots.txt was almost entirely about search engine crawlers. That is no longer true. AI companies now run their own crawlers, and the biggest current mistake in this file is treating all AI bots as one thing and blocking them together. They are not one thing.
Most major providers run separate bots for separate jobs, and each reads its own user-agent token:
- Training crawlers collect content in bulk to build model weights. They run continuously, with no user behind them, and provide no attribution or referral traffic.
- Search and retrieval crawlers fetch pages when a user asks a question, then cite the source with a link back.
- User-triggered fetchers act on behalf of a specific person who asked the assistant to visit a URL.
Here is how the main vendors split their bots.
| Vendor | Training crawler (block to opt out of training) | Search or retrieval crawler (allow to stay citable) | User-triggered fetcher |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | (indexing handled by PerplexityBot) | PerplexityBot | Perplexity-User |
| Common Crawl | CCBot (feeds many open training datasets) | not applicable | not applicable |
| Google-Extended (token for Gemini and Vertex training) | Googlebot (also powers AI Overviews) | Google user-triggered fetchers |
Google's setup deserves its own note. Googlebot handles Search. Google-Extended is not a crawler at all. It is a token that controls whether your content can train Gemini and Vertex AI, and Google has confirmed that blocking it has no effect on your Search rankings or on AI Overviews. So you can opt out of Google's model training without touching search visibility. The only bot you must never block if you want to appear in Google Search is Googlebot itself.
The strategic default for most brands in 2026 is straightforward. Block the training crawlers if you do not want your content used to train models, and allow the retrieval crawlers so you stay eligible for citations in AI answers. Blocking the retrieval bots is the expensive error, because it removes you from the AI surfaces that increasingly send qualified traffic. Audits through 2025 and 2026 found large shares of sites blocking retrieval bots by accident, usually as leftovers from the 2023 "block everything" reaction or from restrictive defaults injected by a CDN or CMS.
A balanced configuration looks like this:
# Traditional search engines: allow
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# AI training crawlers: block
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# AI retrieval and search crawlers: allow, so you stay citable
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Default for everyone else
User-agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml
Four things to keep in mind with AI crawlers:
User-agent strings drift. Providers add, rename, and retire bots. Anthropic's current split is ClaudeBot, Claude-SearchBot, and Claude-User, and the older anthropic-ai and Claude-Web strings are deprecated, so a file that only blocks those old names is not blocking anything current. Check the official documentation, which for the big two is OpenAI's bots overview and Anthropic's crawler help article, and cross-reference a community list like Dark Visitors. Review the file quarterly.
User-triggered fetchers vary by vendor. Anthropic states that Claude-User honors robots.txt. OpenAI notes that ChatGPT-User is user-initiated, so robots.txt may not apply in the same way, and Perplexity says the same about Perplexity-User. Google's user-triggered fetchers generally ignore robots.txt. Do not assume the rules apply uniformly.
robots.txt is a request, not a wall. It is voluntary. Reputable crawlers obey it, but scrapers, stealth crawlers, and pipelines that source content from third-party datasets ignore it. If you need enforcement rather than a stated preference, you need firewall rules or a bot management service. Many CDNs also block AI bots at the network layer regardless of what your robots.txt says, so a page can be allowed in the file and still return a 403 to the bot. Test the actual fetch, not just the file.
Content Signals are emerging but not standard. Cloudflare and some providers are pushing machine-readable signals in robots.txt that describe permitted use rather than access, with categories like search, ai-input, and ai-train. Cloudflare documents this in its managed robots.txt guide. Google Search Console may flag these as syntax it does not understand, with no observed effect on crawling. Treat them as an optional, evolving layer, not a replacement for the access rules above.
Common robots.txt mistakes
- Using robots.txt to deindex a page. It does the opposite of what you want. The URL can stay in the index without a snippet. Use
noindexon a crawlable page instead. - Disallowing a page that carries a
noindex. Google never fetches the page, never sees thenoindex, and the page can stay indexed. Let it be crawled until it drops, then block if you must. - Blocking CSS, JavaScript, or rendering endpoints. Google sees a broken page. This quietly damages rankings for a tiny crawl saving.
- Case mistakes in paths.
/Downloads/and/downloads/are different. Match your real URL casing exactly. - Shipping a staging rule to production.
User-agent: *withDisallow: /belongs on staging and nowhere near your live site. This single line has deindexed entire sites. Make its removal part of every launch checklist. - Relying on the wildcard group for a named bot. If Googlebot has its own group, it ignores the
*group entirely. Repeat the rules you need inside each group. - Trusting robots.txt for privacy or security. The file is public and voluntary. Use authentication.
- Letting the file exceed 500 kibibytes. Rules past the limit are ignored with no warning. Consolidate.
- Accidentally blocking AI retrieval bots. A broad wildcard disallow, an old restrictive default, or a CDN rule can remove you from AI answers. Check each bot's real fetch.
How to test and deploy robots.txt
Before and after any change, verify the file rather than trusting it.
- Open
https://yourdomain.com/robots.txtin a private browser window to confirm it is public and serves plain text. - Use the robots.txt report in Search Console to see which files Google found for your hosts, when they were last fetched, and any warnings, and to request an emergency recrawl after a fix. If you need Google to refresh the file quickly, follow update your robots.txt file.
- For AI and non-Google bots, test the fetch directly per user agent. A quick check with curl shows whether a specific bot would be served the page:
curl -A "PerplexityBot" -I https://yourdomain.com/your-page/
Expect a 200 and no blocking headers. Repeat for GPTBot, ClaudeBot, and OAI-SearchBot, and check your CDN or firewall bot settings separately, since they can override the file. If you write code, Google's open-source robots.txt parser lets you test files locally against the same logic Google uses.
robots.txt examples and templates
Copy-paste starting points. Adjust paths to your site, and see Google's list of useful robots.txt rules for more.
Allow everything (an empty or missing file has the same effect):
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xml
Block a single directory for all crawlers:
User-agent: *
Disallow: /private/
Block URL parameters and internal search while keeping clean pages open:
User-agent: *
Disallow: /*?
Disallow: /search
Note that Disallow: /*? blocks any URL containing a question mark, so confirm no important pages depend on query strings before using it.
Block an entire staging subdomain (place this in the staging host's own robots.txt, never on production):
User-agent: *
Disallow: /
AI strategy, block training and allow retrieval:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
Block all AI crawlers, training and retrieval:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: CCBot
Disallow: /
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. It stops crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it, usually without a snippet. To keep a page out of the index, allow crawling and use a noindex rule.
Is robots.txt case sensitive?
Field names are not, but path values are. /Page and /page are treated as different URLs, so match your real casing.
Does Google support crawl-delay?
No. Google ignores crawl-delay. Bing and some other crawlers honor it, and Anthropic does too, but for Google you control crawl rate through Search Console instead.
Where does the robots.txt file go?
At the root of each host, for example https://example.com/robots.txt. A file in a subdirectory is not valid. Each subdomain and protocol needs its own file.
Will blocking GPTBot or ClaudeBot hurt my Google rankings? No. Those are AI crawlers, separate from Googlebot. Blocking them has no effect on Google Search. Blocking Google-Extended is also safe for Search, since it only controls Gemini and Vertex training.
Should I block AI crawlers? It depends on your goal. If you want citations and referral traffic from AI answers, allow the retrieval bots such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot. If you want to keep your content out of model training, block the training bots such as GPTBot, ClaudeBot, and CCBot. Blocking retrieval bots by accident is the most common costly mistake.
What happens if I have no robots.txt file? Google treats a missing file as permission to crawl everything. A missing file is fine for many sites. You only need one when you want to restrict crawling or point crawlers to a sitemap.
How large can robots.txt be? Google reads up to 500 kibibytes. Anything after that is ignored, so keep the file lean.
How long until Google sees my changes? Google caches robots.txt for up to 24 hours. You can request a faster recrawl through the robots.txt report in Search Console.
Tito Azizul
Digital marketing expert at Logink, helping Indonesian brands grow through SEO, content, and performance marketing.
