Robots.txt is one of those things that sounds more technical than it is.
It’s a simple text file that controls which pages search engines and AI crawlers can access on your site. A small mistake in it can have a big impact on how your site performs in search and in AI chatbots like ChatGPT, Gemini, and Claude.
In this guide, you will learn what robots.txt is, how it works, how to set it up correctly, and how to use it to improve your SEO and manage AI crawler access.
What is a robots.txt file?
A robots.txt file is a plain-text file placed at the root of your domain. It tells web crawlers which pages or sections of your site they are allowed to access.
Every crawler, including Googlebot, Bingbot, and AI training bots, reads and respects this file before crawling your site. They respect the rules inside it as well.
What does a robots.txt file look like?
A robots.txt file is made up of simple directives, which are instructions that tell crawlers what to do.
Here’s a simple example of a robots.txt file:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.webyes.com/sitemap.xmlThere are four core directives in a robots.txt file:
- User-agent specifies which crawler the rule applies to. Using * means the rule applies to all crawlers. You can also target a specific one, like User-agent: Googlebot.
- Disallow tells the crawler which paths it cannot access. In this example, the entire /wp-admin/ directory is off-limits.
- Allow overrides a Disallow rule for a specific path. Here, /wp-admin/admin-ajax.php is permitted even though the rest of /wp-admin/ is blocked.
- Sitemap points the crawler to your XML sitemap, so it knows what pages exist on your site.
Note: The rules in robots.txt are only respected by compliant crawlers. Malicious bots may ignore them entirely.
How does robots.txt work?
When a crawler visits your site, it follows a predictable sequence before accessing any page.
- The crawler fetches yourdomain.com/robots.txt.
- It parses the file and finds the rules that apply to its user-agent.
- For each URL it wants to visit, it checks whether the rules allow or block access.
- If allowed, it crawls the page. If blocked, it skips it.
Crawlers don’t read your robots.txt on every visit. They cache its contents for a set duration, so they don’t need to read it each time. For example, Googlebot caches it for up to 24 hours. Which also means any update to your robots.txt won’t reflect immediately.
Why does robots.txt matter for SEO?
Robots.txt matters for SEO because it controls where crawlers spend their time on your site.
Search engines allocate a limited crawl budget to each site. If crawlers waste it on low-value pages like cart, login, or filter pages, your important pages get crawled less frequently.
By blocking the right pages, you direct that budget toward your most valuable pages. For large websites (10,000+ pages), this can have a significant impact.
How to create a robots.txt file
A robots.txt file is easy to create. Decide what to block and which crawlers to allow, write your rules in a text file, and upload it to your site’s root directory.
Just follow these steps:
Step 1: Decide what to block
Before writing anything, decide which pages or directories you want to block or allow. Common candidates for blocking include login pages, thank-you pages, etc.
Step 2: Choose which crawlers to target
By default, your rules apply to all crawlers using User-agent: *. If you want to target a specific crawler, like Googlebot, you can define separate rules for each one.
Step 3: Create your file
Use a simple text editor like Notepad (Windows) or TextEdit (Mac) to write your directives and save them as “robots.txt.” Alternatively, use a free tool like the SEOptimer Robots.txt Generator. It lets you set your rules through a simple interface and generates the file for you.
Step 4: Upload it to your site
Upload the file to your domain root so it lives at yourdomain.com/robots.txt.
Note: Most CMS platforms have a built-in robots.txt editor, so you may not need to upload anything manually. Shopify, Webflow, and Squarespace all have one built in. For WordPress, a plugin like Yoast SEO or Rank Math adds this functionality.
How to test a robots.txt file
The best way to test your robots.txt for Googlebot is through Google Search Console.
GSC shows you whether your robots.txt file is valid and Googlebot can read it correctly. The tool also flags any rules it is ignoring.
Here’s how to test:
- Open Google Search Console and select your property.
- Go to Settings in the left sidebar.
- Under the Crawling section, find robots.txt and click Open Report.
- You will see your robots.txt file listed with its fetch status, size, and any issues flagged.
- Click on your file to open the side panel.
- Review any warnings or rules flagged as ignored by Googlebot.
- Click Open live robots.txt to view your current live file.
For other crawlers, use a robots.txt validator or checker. A simple Google search for “robots.txt tester” will bring up several free options.
Most work the same way: enter your robots.txt URL, select a user agent, and the tool tells you whether that crawler is allowed or blocked.
Alternatively, you can use WebYes to monitor your site’s technical health on an ongoing basis. It will flag robots.txt issues and website crawlability problems as part of its regular automated scans.
What are the common robots.txt mistakes to avoid?
Robots.txt mistakes can be costly. Some cause pages to vanish from search results, others waste crawl budget, and a few do nothing at all because the syntax is wrong.
Here are the 20 common ones to watch out for.
1. Using your CMS default robots.txt file
Most CMS platforms generate a default robots.txt file. These are generic and often block files or resources that should be accessible to crawlers. Every website is different. Your robots.txt should be customized to your site’s specific structure, not left on default settings.
2. Blocking canonicalized URLs
Some sites use canonical tags to manage duplicate content caused by filters, sort orders, or URL parameters. If those URLs are blocked in robots.txt, crawlers can’t visit them to read the canonical tag. The canonical is ignored, and the duplicate content issue remains. Allow crawling of any URL that carries a canonical tag.
3. Using robots.txt to noindex pages
Google no longer supports noindex directives placed inside robots.txt files. If your robots.txt includes noindex commands, they will be ignored, and those pages may still appear in search results. The correct way to keep a page out of the index is to use a meta robots noindex tag at the page level.
4. Blocking URLs that have a noindex tag
If a page has a noindex meta tag but is also blocked in robots.txt, crawlers can’t visit it to read that tag. Since the noindex is never seen, the page may remain indexed. Remove the robots.txt block and let crawlers visit the page, so the noindex tag can do its job.
5. Case sensitivity errors
Robots.txt is case-sensitive. Disallow: /Folder will not block a folder named /folder. Accidentally using the wrong case means the rule is never applied, and the content gets crawled when it shouldn’t. Always match the exact case of your file paths.
6. Blocking CSS and JavaScript files
When crawlers can’t load your scripts and stylesheets, they can’t render your pages correctly. They see broken layouts and may misclassify your content. Check your robots.txt for any Disallow rules targeting CSS or JavaScript directories and remove them.
7. Using robots.txt to hide sensitive content
Blocking a URL in robots.txt does not make it private. Your robots.txt file is publicly accessible, so anyone can read it and find the paths you are trying to hide. For truly private content, use password protection instead.
8. Blocking redirected URLs
If a URL has a 301 redirect pointing to a new location, crawlers need to visit it to follow that redirect. If it is blocked in robots.txt, they can’t crawl it, can’t follow the redirect, and your old URL may stay indexed. Make sure redirected URLs are not blocked.
9. Unnecessary trailing slashes
Disallow: /contact/ and Disallow: /contact are not always treated the same way by crawlers. One may block the URL while the other does not. Be consistent and test your rules to make sure they work as intended.
10. Using one robots.txt file for subdomains
Search engines treat subdomains as entirely separate websites. A robots.txt file at example.com does not apply to blog.example.com. Each subdomain needs its own robots.txt file with its own crawling instructions.
11. Using absolute URLs instead of relative paths
Writing Disallow: https://example.com/login is incorrect. Robots.txt only understands relative paths. Search engines may misinterpret absolute URLs or ignore the rule entirely. The correct format is Disallow: /login.
12. Moving your staging robots.txt to your live site
Staging environments often use Disallow: / to keep the site out of search results during development. If that file gets pushed to the live site by mistake, every crawler is locked out immediately. Always verify your robots.txt is environment-specific before every release.
13. Moving your live robots.txt to a staging site
The opposite problem. If your staging or development site is publicly accessible and you copy your live robots.txt to it, crawlers will have full access to it. A staging site without proper restrictions can get indexed. Keep your staging robots.txt separate and restrictive.
14. Empty user-agent
Every set of rules in your robots.txt needs a user-agent line above it. An empty or missing user-agent means crawlers don’t know who the rules apply to and may ignore them entirely. Always specify User-agent: before any Disallow or Allow directive.
15. Blocking removed pages
If you have deleted a page and it now returns a 404 or 410, crawlers need to visit it to register that status. If it is blocked in robots.txt, they never see the response code, and the page may stay indexed. Remove any robots.txt blocks on deleted pages.
16. Improper use of wildcards
The * and $ wildcards let you match patterns in your robots.txt. Used incorrectly, they can block important URLs or allow access to URLs you intended to restrict. Test any wildcard rules carefully before deploying them.
17. Blocking pages with hreflang tags
If a page has hreflang tags for alternate language or regional versions, crawlers need to visit it to read those tags. Blocking it in robots.txt means Google never sees the hreflang instructions and can’t serve the right content to the right audience. This is especially important for international SEO.
18. Using the crawl-delay directive
The crawl-delay directive is not supported by Google. Including it in your robots.txt has no effect on Googlebot. Google manages its own crawl speed based on your server load and site size.
19. Misusing comments
Comments in robots.txt start with # and are meant for human readers. Crawlers ignore them. The problem arises when a comment is placed in the wrong position, breaking the structure of the file. A misplaced comment can cause the directives around it to be misread or ignored entirely.
20. Not including your sitemap
Your robots.txt is one of the first things crawlers read. Including your sitemap URL here gives all crawlers, including smaller search engines, an easy way to discover your pages. Add a Sitemap: directive pointing to your XML sitemap.
How to manage AI crawlers in robots.txt
Managing AI crawlers in robots.txt works the same way as managing any other crawler. You specify the user-agent, set a rule, and save. The crawler will follow it on its next visit.
But first, you need the user-agent string of the crawler you want to manage.
Here is a reference table of all major AI crawlers:
| User-agent | Company | Purpose |
|---|---|---|
| GPTBot | OpenAI | AI training |
| OAI-SearchBot | OpenAI | ChatGPT search indexing |
| ChatGPT-User | OpenAI | Real-time fetch |
| ClaudeBot | Anthropic | AI training |
| Claude-SearchBot | Anthropic | Claude web search indexing |
| Claude-User | Anthropic | Real-time fetch |
| Google-Extended | Gemini AI training | |
| Applebot-Extended | Apple | AI training |
| Amazonbot | Amazon | AI training |
| Bytespider | ByteDance | AI training |
| PerplexityBot | Perplexity | Search indexing |
Now that you know which crawlers exist, the next question is which ones you should actually block.
If you want your content to appear in AI-powered search results inside tools like ChatGPT or Claude, do not block OAI-SearchBot or Claude-SearchBot.
Blocking those removes your site from those results entirely.
What is llms.txt?
Llms.txt is a proposed standard, essentially a robots.txt for AI.
It is a markdown file placed at your site’s root directory that gives AI crawlers and large language models (LLMs) a clear, structured summary of your most important content.
As of now, no AI crawlers are known to actively request or use llms.txt files. It remains a community proposal with no formal adoption from major AI companies. And there is no evidence that llms.txt improves AI visibility and citations, as of now.
So, should you create one?
Well, it won’t do any harm, but it won’t make a meaningful difference either, at least not yet. You would be better off focusing on things that actually improve your SEO.
FAQs on robots.txt
No, it is not. If you don’t have one, crawlers treat your entire site as open, and your pages will still be crawled, indexed, and ranked as normal. But without it, you have no control over what gets crawled. Admin pages, staging directories, and other low-value pages may be crawled unnecessarily, which wastes crawl budget on larger sites.
No, a robots.txt file is not bad for SEO. In fact, a well-configured one can improve your SEO by directing crawlers toward your most important pages. It only becomes a problem when it is misconfigured and accidentally blocks pages you need indexed.
Yes, but a robots.txt file is still worth having. Use it to block admin directories, login pages, and staging areas. It is also a good place to manage AI training bots. Blocking them prevents your content from being used to train AI models, regardless of your site size.
Robots.txt is not a legal document and is not legally enforceable. It is a technical convention that well-behaved crawlers choose to follow. If a bot ignores your robots.txt, you have no legal recourse based on the file alone. That said, unauthorized scraping may still be addressed through other laws depending on your jurisdiction.
Your robots.txt file is publicly accessible to anyone. This means it can also reveal the structure of your site, including directories you may want to keep private. Do not rely on robots.txt to protect sensitive content. Use proper authentication for that instead.
Yes, it can. Blocking a page in robots.txt prevents crawlers from visiting it, but it does not prevent it from getting indexed. If external sites link to that page, crawlers can infer it exists and index it without ever visiting it. If you want to prevent a page from appearing in search results, use a noindex tag instead of robots.txt.
AUTHOR
Melwyn holds a Bachelor's degree in Computer Science and has been writing professionally for over 4 years. He has a strong background in SEO and is certified in web accessibility from the World Wide Web Consortium (W3C). When he's not writing about website optimisation, you can find him enjoying a game of FIFA or exploring the open world of GTA.
Melwyn Joseph 
