What it controls
Robots.txt controls crawling, not indexing. It can ask crawlers to skip areas like admin pages, internal search results or endless filter combinations, which saves crawl effort for pages that matter.
What it can't do
It doesn't reliably keep a page out of search results. Google can still index a blocked URL if other sites link to it. To keep a page out, use a noindex tag and let the page be crawled so the tag can be seen. Robots.txt is also public, so don't use it to hide sensitive pages.
AI crawlers
Many AI companies run their own crawlers, which can be allowed or blocked by name in robots.txt. Blocking them may keep your content out of some AI answers, so treat it as a business decision, not a default.
Why Robots.txt matters
A single wrong line in robots.txt can block search engines from an entire site. It's one of the first files to check when rankings or indexing suddenly drop.
The staging rule that went live
A team blocked all crawlers on its staging site with 'Disallow: /'. The same file was copied to the live site during launch. Pages started dropping from search until someone caught it in Search Console and removed the rule.
Common mistakes
Blocking the whole site by accident
'Disallow: /' for all user agents tells every crawler to stay out. Check robots.txt after every launch or migration.
Using it to hide pages from search
Blocked pages can still appear in results. Use noindex for pages you don't want shown.
Blocking CSS and JavaScript
Search engines need these files to render your pages. Blocking them can make pages look broken to Google.
Blocking AI crawlers without deciding
Copying a list of blocked AI bots from elsewhere may remove you from AI answers you'd want to appear in.