Is Your Robots.txt Blocking Search Engines?
Check your robots.txt file and see what search engines are allowed or blocked from crawling.
Free • No signup required
Why Robots.txt Matters
Robots.txt is a plain-text file that lives at the root of a domain — https://yourdomain.com/robots.txt — and it is the first thing most crawlers request. It contains groups of rules: a User-agent line names the crawler a group applies to, and the Allow and Disallow lines beneath it describe which URL paths that crawler may or may not request. A group for a named crawler such as Googlebot takes precedence over the catch-all User-agent: * group, and within a group the longest matching rule wins.
Because it governs crawling, a mistake here has an outsized effect: a stray Disallow: / left over from a staging environment can stop search engines from reading the entire site. Robots.txt is also where you can point crawlers to your XML sitemap with a Sitemap: line.
Robots.txt does not guarantee indexing, and it does not prevent it either. Blocking a URL stops crawlers from fetching it, but the URL can still be indexed from external links — with no description, because the page was never read. To keep a page out of search results, leave it crawlable and use a noindex meta tag or X-Robots-Tag header instead. That is also why important pages, and the CSS and JavaScript needed to render them, should never be blocked by accident.
Next, confirm the sitemap you declare actually works with the XML sitemap checker, review page-level directives with the SEO meta tag checker, or run the full crawlability checker.
Frequently Asked Questions
- What is robots.txt?
- Robots.txt is a plain-text file that tells search engine crawlers which parts of a website they may request. It is a set of crawling instructions, not a security control.
- Where should robots.txt be located?
- At the root of each domain or subdomain, for example https://yourdomain.com/robots.txt. Crawlers do not look for it anywhere else.
- Does robots.txt stop Google from indexing a page?
- No. Robots.txt blocks crawling, not indexing. A blocked URL can still appear in search results if other pages link to it — Google just cannot read the page. Use a noindex meta tag or header on a crawlable page to keep it out of the index.
- How do I know if Googlebot is blocked?
- Look for a User-agent: Googlebot group; if none exists the User-agent: * group applies. Then check whether any Disallow rule matches the URL you care about. This tool evaluates the homepage using Google's longest-match rules.
- Should I add my sitemap to robots.txt?
- It helps, especially when your sitemap sits at a non-standard path. It is a hint rather than a requirement, so a missing Sitemap: line is not an error.
- Can I block specific folders?
- Yes. A line such as Disallow: /admin/ inside a User-agent group stops matching crawlers from requesting anything under that folder.
- What happens if I don't have a robots.txt file?
- Crawlers treat a 404 as permission to crawl everything they can find, so the site remains crawlable. You simply lose the ability to guide crawlers or declare a sitemap.
- Can robots.txt block CSS or JavaScript?
- It can, but it usually should not. Google renders pages before evaluating them, so blocking CSS or JS files can make pages look broken to the crawler and hurt how they are assessed.