Robots.txt
A file that tells search engine crawlers which URLs they can access on your site.
What is Robots.txt?
Robots.txt is a text file placed at your website's root that provides instructions to web crawlers about which pages or sections they should or shouldn't crawl. It follows the Robots Exclusion Protocol and is one of the first files crawlers check when visiting a site.
While robots.txt can prevent crawling, it doesn't prevent indexing. Pages blocked by robots.txt can still appear in search results if linked from other sites. For true de-indexing, use noindex meta tags.
Why Robots.txt matters
Robots.txt helps you control crawl budget by preventing search engines from wasting resources on unimportant pages. It's essential for large sites and helps keep admin areas, duplicate content, and development pages out of search engine crawls.
How to implement
Create robots.txt in your root directory
Specify User-agent for each rule set
Use Disallow to block paths
Use Allow to permit specific paths within blocked directories
Include sitemap location
Test with Google Search Console
Code example
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /private/public/
Sitemap: https://example.com/sitemap.xmlCommon mistakes to avoid
- Blocking CSS/JS files needed for rendering
- Using robots.txt to hide sensitive content (it's public)
- Blocking entire site during development and forgetting to remove
- Conflicting rules that confuse crawlers
Examples
Block admin area: Disallow: /admin/
Block specific file type: Disallow: /*.pdf$
Allow specific folder in blocked parent: Allow: /private/public/
Frequently asked questions
No, it only prevents crawling. Blocked pages can still be indexed if linked from other sites. Use noindex meta tag to prevent indexing.
You can block specific AI crawlers like GPTBot or ClaudeBot if you don't want your content used for training. This won't affect search rankings.