Robots.txt Parser

Analyze your website's robots.txt file to see how search engines and bots are instructed to crawl your site.

Evidence ASSESSED

Checked against: robots.txt fetched from the host root and parsed

RED TEAM 0 findings BLUE TEAM control did not hold

About Robots.txt

The robots.txt file is a standard used by websites to communicate with web crawlers and other web robots.

Directives: Use Disallow to prevent bots from crawling specific paths.
User-agent: Specify different rules for different bots (e.g., Googlebot, Bingbot).
Sitemap linked: It's best practice to include your XML sitemap URL in the robots.txt file.
Not a Security Tool: Robots.txt is a suggestion, not a firewall. Malicious bots often ignore it.

Our Robots.txt Checker retrieves and parses your robots.txt file, validating the syntax of every directive, identifying rules that block important content, and flagging the sensitive paths that robots.txt inadvertently advertises to anyone who reads it.

What is robots.txt?

robots.txt is a plain text file at the root of your domain that tells well-behaved crawlers which paths they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022 after decades as a de facto convention.

The file is a series of groups. Each begins with a User-agent line naming which crawler it applies to, followed by Disallow and Allow rules. A Sitemap directive may appear anywhere in the file and is not tied to a particular user agent.

Two properties are frequently misunderstood. It is entirely voluntary — it asks crawlers not to fetch paths, and malicious ones simply ignore it. And it controls crawling, not indexing: a disallowed URL can still appear in search results if other pages link to it, because the crawler never fetches the page to discover a noindex directive.

A plain text file at the root of the domain
Standardised as RFC 9309 in 2022
Purely advisory — hostile crawlers ignore it entirely
Controls crawling, not indexing
Publicly readable by anyone, which makes it a disclosure surface

Why robots.txt Matters

A single misplaced line can remove a site from search results. Disallow: / blocks everything, and it reaches production surprisingly often when a staging configuration is deployed unchanged. The site does not break, traffic simply stops arriving.

The security dimension runs the other way. Because robots.txt is public and lists exactly the paths you would prefer crawlers not to visit, it is a reconnaissance shortcut. Listing /admin/, /backup/ or /internal-api/ tells an attacker precisely where to look. It is one of the first files fetched in any assessment.

One wrong line can deindex an entire site
Staging rules deployed to production are a recurring cause
The file is public and reveals the paths you want hidden
Disallowed URLs can still be indexed if other pages link to them
It is among the first files an attacker requests

How This Robots.txt Checker Works

The tool retrieves the file from the domain root and parses every directive it contains.

Validates the domain input against format and SSRF safety checks
Requests /robots.txt from the domain root
Parses each user-agent group with its Allow and Disallow rules
Validates syntax and identifies directives that are ignored by major crawlers
Flags overly broad rules and sensitive paths that the file discloses

How to Use the Robots.txt Checker

Enter your domain name in the input field
Run the check to retrieve and parse the file
Confirm no rule blocks content you want indexed
Check that a Sitemap directive is present
Review the disallowed paths for anything sensitive you have inadvertently advertised

Understanding Your robots.txt Directives

Each directive has a specific meaning and some widely believed ones do nothing at all:

Field / Value	Meaning
`User-agent: *`	Opens a group of rules applying to every crawler that has no more specific group. Crawlers obey only the single most specific group that matches them, not all matching groups.
`Disallow: /`	Blocks the entire site. Correct for a staging environment, catastrophic in production. Check this first on any site that has lost its search traffic.
`Disallow:`	An empty value blocks nothing and explicitly permits everything. Easily confused with Disallow: / which does the opposite.
`Allow:`	Creates an exception within a disallowed directory. Supported by all major crawlers. The most specific matching rule wins regardless of order.
`Sitemap:`	Points to your XML sitemap using an absolute URL. It applies to the whole file rather than to a particular user-agent group.
`Crawl-delay:`	Requests a pause between requests. Google has never supported it; Bing and Yandex do. Use search console rate settings for Google instead.
`Noindex:`	Was never part of the standard and is ignored by Google since 2019. Use a meta robots tag or an X-Robots-Tag header instead.

Common Problems & How to Fix Them

The entire site is blocked from crawling and will drop out of search results.

This is almost always a staging configuration that was deployed unchanged. Remove the rule immediately and request re-crawling. Recovery typically takes days to weeks depending on how long the block was in place.

Prevent recurrence by generating robots.txt from environment configuration rather than shipping a static file.

Listing /admin/, /backup/, /.git/ or /internal-api/ tells anyone reading the file exactly where those resources are. The file is public and is among the first things an attacker fetches.

Remove them and protect those paths with authentication and server configuration instead. If a path must be excluded from crawling but not disclosed, use a broader pattern or rely on meta robots on the pages themselves.

Search engines render pages to evaluate them. Blocking the assets needed for rendering means the crawler sees a broken layout, which can be assessed as a poor mobile experience.

Allow crawlers to fetch CSS, JavaScript and images. There is rarely any benefit to blocking them and the cost is real.

Disallow prevents crawling, not indexing. A blocked URL that other sites link to can still appear in results, showing the URL with no description — because the crawler was never allowed to fetch the page and find a noindex directive.

Use a meta robots noindex tag or an X-Robots-Tag header, and allow the page to be crawled so the directive is actually seen. For genuinely private content, use authentication.

Crawlers must discover your sitemap some other way, usually through search console submission alone.

Add an absolute URL to the file:
Sitemap: https://example.com/sitemap.xml

The directive can appear anywhere in the file and applies regardless of user-agent groups.

Best Practices

Keep the file minimal — block only what genuinely needs blocking
Include an absolute Sitemap URL
Never list sensitive paths, since the file is public
Allow CSS, JavaScript and image files so pages render correctly for crawlers
Generate the file from environment config so staging rules cannot reach production
Use meta robots noindex rather than Disallow to keep pages out of the index
Re-check after every deployment and platform migration

Security Recommendations

Treat robots.txt as a public document that an attacker will read first, and never use it to point at anything sensitive
Protect administrative and internal paths with authentication, never with a Disallow rule
Remember hostile crawlers ignore the file completely, so it provides no access control whatsoever
Avoid patterns that reveal your directory structure or technology stack
Monitor the file for unauthorised changes, since an injected Disallow: / is a low-effort way to damage a site
Consider that an unusually detailed robots.txt is itself a signal of which areas you consider sensitive

Frequently Asked Questions

A plain text file at the root of your domain that tells crawlers which paths they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309, and is entirely voluntary — well-behaved crawlers honour it and hostile ones ignore it.

No. It prevents crawling, not indexing. A disallowed URL that other sites link to can still appear in results showing the URL alone, because the crawler was never able to fetch the page and discover a noindex directive.

Use a meta robots noindex tag in the page head, or an X-Robots-Tag HTTP header. Crucially, the page must be crawlable for the directive to be seen — blocking it in robots.txt prevents the crawler from ever reading the noindex.

Not at all. It is a public file that hostile crawlers ignore, and listing a path in it advertises that path to anyone who reads the file. Use authentication and server configuration for anything that genuinely needs protecting.

No. Search engines render pages to evaluate them, and blocking rendering assets means the crawler sees a broken page. This can be assessed as a poor mobile experience and harm rankings, with no compensating benefit.

Bing and Yandex honour it; Google has never supported it and ignores the directive entirely. To control Google crawl rate, use the setting in Search Console rather than robots.txt.

Check robots.txt first for Disallow: / — a staging rule deployed to production is the most common cause. Remove it and request re-crawling. Recovery usually takes days to weeks depending on how long the block was live.