SEO
What Is Robots.txt? How to Set It Up, Including AI Crawlers
The syntax Google understands, how to test it in Search Console, the AI crawler list straight from each company and what 12 major news sites in the US, Spain and Brazil block.
What is robots.txt? It's a plain text file at the root of your site that tells crawlers, from Google or from an AI company, which parts they may visit. It won't hide pages from Google or protect anything: it's a request that reputable bots honor and bad ones ignore.
We checked it in late October 2026, by reading the robots.txt files of 12 major news sites in the US, Spain and Brazil: 10 block ClaudeBot, 7 block GPTBot and none block Googlebot. Notablia went the other way and lets every crawler in.
Here's how the file works, how to test it, which AI crawlers exist according to each company and the mistakes that knock a whole site out of Google. It's the front door of technical SEO.
On this page
What is robots.txt, and what can't it do?
Robots.txt is a plain text file that tells crawlers which URLs on your site they can request and which they can't.
The robots.txt file is a text file that lives at the root of each domain or subdomain and follows the Robots Exclusion Protocol, standardized as RFC 9309 in 2022. It groups rules by crawler with user-agent, allow and disallow lines, and its main job is managing crawl traffic so bots don't overload your server.
What it can't do, per Google's robots.txt introduction: keep a page out of search results. If other sites link to a blocked page, it can still show up, just without a description. It isn't security either: the file is public and following it is voluntary.
Is robots.txt still relevant?
More than ever. On top of managing Googlebot, it's now where you set your policy for AI crawlers, a job that didn't exist a few years ago. Google was still updating its robots.txt specification in October 2026.
Robots.txt example: user-agent, disallow, allow and sitemap
A robots.txt file is read in groups: each group starts with one or more user-agent lines, followed by its allow and disallow rules.
Here's a robots.txt example for an online store that doesn't want crawlers wasting time on the cart or internal search.
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /
Sitemap: https://yourstore.com/sitemap.xml| Directive | What it does | Watch out for |
|---|---|---|
User-agent | Which crawler the group applies to | * covers every bot without its own group |
Disallow | A path that shouldn't be crawled | Case-sensitive: /Blog/ isn't /blog/ |
Allow | An exception inside a blocked path | The longest, most specific rule wins |
Sitemap | Where your sitemap lives | Always the full URL |
Crawl-delay | A pause between requests | Google ignores it; Anthropic honors it |
The robots.txt sitemap line tells any crawler where your XML sitemap is, even if you never use its webmaster tools.
One detail almost nobody mentions: once you create a group for a specific bot, that bot stops reading the * group. Give GPTBot its own group and forget to copy your general rules, and GPTBot can crawl everything else.
What does disallow mean in robots.txt?
A robots.txt disallow rule means "don't crawl this path." Disallow: /admin/ blocks that folder, Disallow: / blocks the whole site and an empty Disallow: blocks nothing. When allow and disallow rules clash, Google applies the most specific one and, in a tie, the least restrictive.
How to set up robots.txt and test it, step by step
To set up robots.txt you need a text editor, access to your site's root folder and ten minutes to test it in Search Console.
-
Look at the one you already have
Open yourdomain.com/robots.txt. Many platforms generate a virtual one even if you never uploaded anything.
-
Decide what shouldn't be crawled
Cart, internal search, filters with endless parameters, staging areas. Never your CSS or JavaScript.
-
Write the rules in groups
One group for everyone with
*, and separate groups only for bots you want to treat differently. Remember: a bot's own group replaces the general one. -
Add the Sitemap line
With the full URL, at the end of the file. You can list several.
-
Upload it to the root of each host
It must be named robots.txt, lowercase, in UTF-8. Every subdomain and protocol needs its own, and Google ignores anything past 500 KiB.
-
Test it in Search Console
The robots.txt report shows which file Google found, when and with what errors. If you unblocked something important, request a recrawl from there: Google caches the file for up to 24 hours.
Looking for a robots.txt checker? Google's toolkit is now three pieces: that report, URL Inspection to see whether a specific URL is blocked, and its open-source library for testing the file on your own machine.
Robots.txt tester: how to test your file and a single URL
Google spreads testing across a few tools, so use them in this order.
- Is it public? Open yourdomain.com/robots.txt in a private window. If you see the text, Google can request it too.
- Does Google read it right? The robots.txt report in Search Console only works for a file that's already live on your site.
- Is this URL blocked? Paste it into URL Inspection.
- Do my rules say what I think? Check them by hand against the logic in Google's specification.
Rules are case-sensitive, * matches any run of characters and $ marks the end of the URL. Here's how three rules behave, following that specification.
| Rule | Blocks | Doesn't block |
|---|---|---|
Disallow: /blog | /blog, /blog.html, /blogs, /blog/sale, /blog?id=3 | /Blog, /my-blog, /?section=blog |
Disallow: /blog/ | /blog/, /blog/sale | /blog, /blog.html |
Disallow: /*.pdf$ | /catalog.pdf, /docs/prices.pdf | /catalog.pdf?v=2, /catalog.pdf/ |
Developers can also build Google's open-source robots.txt library and run the file locally. And if a third-party tester says "allowed" while Search Console says "blocked," believe Search Console.
Robots.txt generator: 3 templates for the file you need
You can skip the generator. Google reads only four kinds of lines (user-agent, allow, disallow and sitemap) and quietly drops anything else, according to Gary Illyes and Cherry Prommawin in an October 21, 2026 Search Central video. Comments start with # and are ignored too. A generator mostly saves typing, so copy the template that fits your site and swap in your own paths.
What should be in my robots.txt file?
Only what you need. A blog or brochure site can let everything in:
User-agent: *
Disallow:
Sitemap: https://yourdomain.com/sitemap.xmlAn empty Disallow: blocks nothing. An online store skips the cart, the checkout and sort parameters:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sort=
Sitemap: https://yourstore.com/sitemap.xmlWhatever a generator hands you, check three things. No stray Disallow: /. AI bots named by their exact tokens. A clean run through the Search Console report. And to keep price-list PDFs out of the crawl, add Disallow: /*.pdf$.
WordPress robots.txt: where it lives and what to change
Your WordPress robots.txt is virtual by default: WordPress generates it on the fly and blocks /wp-admin/, except the admin-ajax.php file.
To edit it, the easiest route is your SEO plugin: Yoast and Rank Math both include a robots.txt editor in their tools. You can also upload a real robots.txt file to the root folder via FTP, and it replaces the virtual one. Whatever you do, don't block /wp-content/ or /wp-includes/: that's where the CSS, JavaScript and images Google needs to render your pages live.
AI crawlers in robots.txt: GPTBot, ClaudeBot, Google-Extended and the rest
AI crawlers are managed like Googlebot, by user-agent, but each company separates the bots that train models from the ones that search and cite.
Here's the list from each company's own documentation, checked in late October 2026.
| Company | Bot | What it's for | If you block it |
|---|---|---|---|
| OpenAI | GPTBot | Training its models | Your content isn't used for training |
| OpenAI | OAI-SearchBot | ChatGPT search | You drop out of its search answers |
| OpenAI | ChatGPT-User | Visits a user asks for | robots.txt may not apply |
| Anthropic | ClaudeBot | Training its models | Excludes your future content |
| Anthropic | Claude-SearchBot | Claude's search | Less visibility in its results |
| Anthropic | Claude-User | Visits a user asks for | Claude can't read you when asked |
| Google-Extended | Gemini training and grounding | No effect on Google Search | |
| Perplexity | PerplexityBot | Its search engine | You vanish from its links |
| Perplexity | Perplexity-User | Visits a user asks for | Generally ignores robots.txt |
If you want to opt out of training without disappearing from AI search, this is the pattern.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Notablia chose the opposite: three lines, everyone welcome. We're a blog that wants AI tools to read and cite it, and blocking training gave us nothing in return.
Heads up if your site runs behind Cloudflare. According to its documentation, updated in July 2026, starting September 15, 2026, new domains block training and agent bots by default on pages that show ads. Your robots.txt can say "come in" while your CDN says "no."
Does blocking GPTBot remove you from ChatGPT?
No. GPTBot only collects content for training, while ChatGPT search uses OAI-SearchBot. OpenAI's overview of its crawlers says each setting is independent and that changes take about 24 hours to kick in. The "-User" bots are a different story: they act when someone asks, the way an AI browser does, and they don't always read robots.txt.
Robots.txt vs noindex: which one keeps a page out of Google
If you want a page out of Google, use noindex and let Google crawl it: if robots.txt blocks the page, Google never sees the noindex.
| You want... | Use |
|---|---|
| A page kept out of Google | A noindex tag or X-Robots-Tag header |
| Google to skip endless filter URLs | Disallow in robots.txt |
| Private content nobody can see | A password |
| Your content kept out of AI training | Disallow for GPTBot or ClaudeBot, plus Google-Extended |
| To stay out of Google's AI answers | nosnippet or noindex, same as the rest of Search |
Google stopped supporting noindex inside robots.txt in 2019. And be careful blocking pages that earn backlinks: Google won't be able to read them or follow their links.
"Disallow all" and other robots.txt mistakes that sink a site
The most expensive robots.txt mistake is pushing your staging file to production, complete with a Disallow: / that blocks the entire site.
Act as a technical SEO specialist. Here's my robots.txt: [paste it]. And here are the sections of my site I want in Google: [list]. Tell me which rule blocks something it shouldn't, which AI crawlers end up allowed or blocked according to the official documentation from OpenAI, Anthropic, Google and Perplexity, and suggest a fixed version with a comment on each rule.
Frequently asked questions about robots.txt
How do I check if a website has a robots.txt file?
Type the domain followed by /robots.txt in your browser, for example nytimes.com/robots.txt. If a text file with User-agent lines shows up, the site has one; if you get a 404 page, it doesn't. Remember that each subdomain needs its own, so shop.example.com and example.com can have different rules. For your own site, the robots.txt report in Search Console shows which file Google found, when it last fetched it and any errors.
Should I have a robots.txt file?
It isn't required. If there's no file and the server returns a 404, Google simply assumes it can crawl everything. Still, it's worth having a short one: it's where you declare your sitemap, keep crawlers out of carts or internal search results, and set your policy for AI crawlers. A three-line file that allows everything and lists your sitemap is a perfectly good robots.txt for most small business sites.
How to fix blocked by robots.txt error?
First decide whether the page should be in Google at all. If it should, find the rule that blocks it, usually a Disallow line that's broader than you meant, and remove or narrow it. Then open the robots.txt report in Search Console and request a recrawl, since Google can cache the file for up to 24 hours, and use URL Inspection to request indexing. If the page should stay out, leave it and stop worrying about the warning.
Is ignoring robots.txt illegal?
In most places, robots.txt itself isn't a law: it's a voluntary standard, published as RFC 9309, that well-behaved crawlers follow. That said, ignoring it can still get a scraper in trouble when the scraping breaks a site's terms of service, copyright rules or computer misuse laws, and courts look at each case differently. This isn't legal advice. For site owners, the practical point is that robots.txt is a request, so real protection needs logins or firewall rules.
What is crawl budget?
Crawl budget is roughly how many URLs Google is willing to crawl on your site in a given period, based on what your server can handle and how much demand there is for your pages. It mainly matters for very large sites, with tens of thousands of URLs, or for stores whose filters create endless URL combinations. For a typical small business or blog, it isn't a problem. There, robots.txt helps by blocking internal search and low-value filter pages.
What is Google-Extended?
Google-Extended is a product token, not a separate crawler. You add it to robots.txt to decide whether content Google crawls can be used to train future Gemini models and for grounding answers in Gemini apps and Vertex AI. Google says it has no effect on your inclusion or ranking in Google Search, and Googlebot keeps crawling as usual. To limit how your pages appear in AI Overviews, Google points to nosnippet or noindex instead.