Every website has a small, plain-text file that most visitors never see but every search engine checks first: robots.txt. It sits quietly at the root of your domain, and it has the power to make an entire section of your site invisible to Google — sometimes by accident.

What robots.txt Actually Does

Robots.txt gives instructions to search engine crawlers about which parts of your site they are allowed to request. It uses a simple set of rules built from three directives:

• User-agent: which crawler the rule applies to (* means all crawlers)
• Disallow: a path crawlers should not request
• Allow: an exception carved out of a broader Disallow rule

A typical file looks like this:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/help/
Sitemap: https://yoursite.com/sitemap.xml

What It Is Not For

This is the part most people get wrong: robots.txt controls crawling, not indexing. If another site links to a page you've disallowed, Google can still list that URL in search results — just without a description, since it never crawled the content. To truly keep a page out of search results, use a noindex meta tag on the page itself, not a Disallow rule.

The Mistake That Deindexes Entire Sites

The single most damaging robots.txt error is this line, left over from a staging site or added by mistake:

User-agent: *
Disallow: /

That trailing slash blocks crawling of every single page on the domain. It is easy to add and easy to miss — and because search engines only re-check robots.txt periodically, a site can quietly lose its rankings over several weeks before anyone notices traffic dropping.

What Should You Actually Block?

In most cases, less is more. Reasonable candidates for Disallow include admin dashboards, internal search result pages, shopping cart and checkout flows, and duplicate content generated by filters or tracking parameters. Your actual articles, product pages, and category pages should almost always remain crawlable.

Testing Before You Publish

Google Search Console shows exactly how Googlebot interprets your live robots.txt file, including which specific rule is blocking a given URL. Always test a few key pages — your homepage, a blog post, a product page — after any change, since a syntax mistake in one line can silently affect rules below it.

Conclusion

Robots.txt is one of the smallest files on your server and one of the easiest to get wrong. Keep the rules minimal, double-check them after every edit, and remember that this file manages crawling — pairing it with proper canonical tags and noindex directives gives you full control over how search engines see your site.