What is Robots.txt?
Robots.txt is a plain text file placed in the root directory of your website (e.g., yourdomain.com/robots.txt) that instructs search engine crawlers which pages or sections of your site they are allowed or not allowed to access. It is part of the Robots Exclusion Protocol.
Why It Matters
Crawlers visit every publicly accessible URL they can find. Robots.txt helps you direct that crawl budget toward your most important pages, while blocking crawlers from admin pages, staging environments, duplicate content, or private sections you never want indexed.
How It Works
The file uses simple directives. ‘User-agent: *’ applies rules to all crawlers. ‘Disallow: /admin/’ blocks the admin directory from all crawlers. ‘Allow: /’ permits access to the root. ‘Sitemap: https://yourdomain.com/sitemap.xml’ points crawlers to your sitemap. Robots.txt controls access, but it does not prevent indexing — if another site links to a blocked page, Google may still index it.
Real-World Example
A marketing agency has a staging subdomain at staging.digitalmediarun.com that mirrors the live site. Adding ‘Disallow: /’ in the staging site’s robots.txt ensures Google doesn’t index duplicate staging pages and split crawl budget away from the live site.
