What is robots.txt
Also known as: robots file, Robots Exclusion Protocol
Robots.txt is a text file at the root of a site that tells search engine crawlers which URLs they may or may not crawl. It is mainly used to manage crawler traffic and keep crawlers from wasting time on unimportant pages. It is not a way to hide a page from Google: a blocked URL can still appear in results if other sites link to it.
A store blocks crawling of its cart and internal search with Disallow for all crawlers. Later, it creates a group just for Googlebot, with a single line allowing the images folder.
Result: Googlebot now follows only its own group and ignores the general one. The cart and internal search, which were blocked, are crawled by it again. The fix is repeating in the Googlebot group the rules that should also apply to it.
How the file works
The file always lives at example.com/robots.txt, is plain UTF-8 text and is organized in groups. Each group starts by naming the crawler it applies to, followed by the rules. The protocol was standardized by the IETF as RFC 9309.
| What it does | Example | |
|---|---|---|
| user-agent | Names the crawler the group applies to | User-agent: Googlebot |
| disallow | A path that may not be crawled | Disallow: /cart/ |
| allow | An exception inside a block | Allow: /cart/help |
| sitemap | The full sitemap URL | Sitemap: https://example.com/sitemap.xml |
What robots.txt does not do
- It does not remove a page from Google. For that, use the noindex rule or password protect the page.
- It does not protect information. The file is public and only reputable crawlers obey it. Hiding something requires a password on the server.
- It does not work with noindex on the same page. If the page is blocked, Google never crawls it and never sees the noindex.
- It has no crawl-delay on Google. Google only understands user-agent, allow, disallow and sitemap.
The mistake that catches most sites: groups do not add up
Under Google's specification, only one group applies to each crawler: the most specific one matching its name. The others are ignored. A group made for Googlebot inherits nothing from the general group, the asterisk one, and the rules of the two are not combined.
In practice, anyone who creates a group for a specific crawler has to copy into it every rule that should still apply. It is a silent mistake: the file looks right and nothing warns that the block stopped working.
Robots.txt and AI crawlers
Google offers the Google-Extended token to decide whether crawled content may be used to train Gemini models and to ground their answers. Blocking Google-Extended does not affect a site's inclusion in Search or its ranking.
Other companies publish their own names. OpenAI, for example, separates OAI-SearchBot, which powers ChatGPT's search results, from GPTBot, used for training, and each can be allowed or blocked independently.
Limits and how to test
- Google reads up to 500 KiB of the file and ignores anything beyond that.
- The content is usually cached for up to 24 hours; a change can take that long to apply.
- Search Console's robots.txt report shows which files Google found, when it read them and any errors.
Frequently asked questions
What is robots.txt?
A text file at the root of a site that tells search engine crawlers which URLs they may or may not crawl.
Does robots.txt remove a page from Google?
No. A blocked URL can still appear in results if other sites link to it. To remove the page, use noindex or a password.
Where is robots.txt located?
Always at the root of the domain, at /robots.txt. Each subdomain has its own.
How do I block AI crawlers?
With groups for the names each company publishes, such as Google's Google-Extended and OpenAI's GPTBot. Each name controls a different use, and blocking Google-Extended does not remove the site from Search.
Why did my block stop working?
A common reason is creating a group for a specific crawler. That crawler then follows only its own group and ignores the general group's rules.
Related terms
- SitemapA sitemap is a file, almost always in XML, that lists a site's pages, videos and other files and provides...
- IndexingIndexing is the stage where Google analyzes the content of a page it has already crawled, works out what it...
- Canonical tagA canonical URL is the version Google picks as the main one when the same content exists at several...
- llms.txtllms.txt is a Markdown file, published at the site root, that summarizes the content and points AI agents to...
- SERPSERP stands for search engine results page, the page a search engine shows after a query. On Google, it...
- Structured dataStructured data is a block of code, invisible to visitors, that describes a page's content in a standard...
- Google Search Central: robots.txt introduction and guide
- Google Search Central: how Google interprets the robots.txt specification
- Google Search Central: robots meta tag specifications
- Google: Google common crawlers (includes Google-Extended)
- Search Console Help: robots.txt report
- OpenAI: overview of OpenAI crawlers
- IETF RFC 9309: Robots Exclusion Protocol
Rodrigo Fávaro
Founder of ROO3, a marketing and technology agency in São José do Rio Preto, Brazil. Builds AI products running in production (Tobia, gerar.app, Pense Mercado) and maintains the AI Benchmark, a public ranking of AI models.
X @rodmf LinkedIn rodrigofavaroMarketing you can measure, from the ad to the sale.
ROO3 builds and runs paid media, SEO and websites with real targets, dashboards and numbers. The first conversation is free.