The tool page is tamperbox.dev/robots-txt-checker. It checks tamperbox.dev live when opened, says which search engines and AI crawlers the rules let in, and lets you test any path. This page is the long version: how the file is fetched and read, how a crawler’s group and a path’s rule are chosen, every finding, and how to fix each one.

Where processing happens: a pasted robots.txt or sitemap is read in your browser and never uploaded. A host name is sent to the Tamperbox server, which fetches /robots.txt and up to six sitemaps without cookies and returns them; the rules are evaluated in your browser. Explicit result-action transfers use temporary browser storage; diagnostic results are not a monitoring history. Sample hosts on the reserved .example zone are answered in the browser from a fixture.

What the tool answers

1. Can search engines crawl the site? Googlebot, Bingbot, Applebot and DuckDuckBot at the home page, judged: blocked is a failure.
2. Which AI crawlers are let in? Ten AI training, search and on-request tokens, stated without a verdict, because that is the owner’s choice.
3. Is a given URL allowed, and why? The tester names the deciding line for any path and crawler.
4. Is the file and its sitemaps well formed? Every line a crawler would skip or misread, and every sitemap problem.
5. Do the rules and the sitemap agree? URLs the sitemap asks to be crawled that the rules refuse.

Inputs

A host or address

example.com or a full address. The host is what is checked; a path in a full address (https://example.com/shop/boots) becomes the tester’s path. The check runs when you stop typing, or on Enter. The usual refusals apply: spaces, other schemes, credentials, non-standard ports, IP addresses and single labels.

A pasted robots.txt

Text with a newline, a line starting #, or a line starting User-agent:, Disallow:, Allow:, Sitemap: or Crawl-delay: is read as robots.txt, in the browser. Nothing is fetched, so the Sitemaps section stays empty.

A pasted sitemap

Text starting with <?xml, <urlset or <sitemapindex is read as a sitemap, in the browser.

Files up to 10 MiB can be dropped or opened (.txt, .xml). A pasted address loses quotes, brackets and trailing punctuation and keeps the host; text that fits nothing stays as pasted so the error can explain it.

Reading the result

Verdict and chips

“Open to search engines”, “Closed to Googlebot” or “Closed to search engines”, then the AI crawlers blocked and the number of things to fix. For a missing file: “No robots.txt. Every crawler may fetch every page.” The chips count AI crawlers blocked, sitemap URLs blocked, and problems; each scrolls to its section. Two items say where the input went.

Findings

Every finding, worst first, line findings prefixed with their line; two or more passing findings are grouped under an expandable checks passed row. A single passing finding stays visible.

Test a URL

A path or full URL and a crawler. The result says Allowed or Blocked, names the group and the deciding line, and marks that line in the file. It opens on the path from your address, or on the first blocked sitemap URL, with Googlebot.

robots.txt

The file as written, numbered, each line with an issue tinted and noted. Long files show their first 60 lines, every marked line and the tested line; Show all n lines lists the rest, and Show the first 60 lines folds it again. Folding is offered only when at least ten unmarked lines after line 60 can be hidden; marked lines remain visible.

Crawlers at the home page

Two tables, search engines and AI crawlers: who runs each token, Allowed or Blocked at /, and the deciding rule or group.

Sitemaps

Each sitemap fetched, its status, URL count, last change, and up to three issues.

History

History follows the current result evidence. Break it and Heal it record input changes, newest first. Revert restores the input before the latest recorded change; Reset input restores the input before the first retained change and clears the list. Restore input before this break under Heal it also reverts the latest change.

These actions reload and check the restored input. Restoring a public address can therefore send a new request, including on the two tools whose ordinary typing waits for a run button. Editing or pasting a new input clears History. The list lives in the current page; it is not saved monitoring data and does not restore an old network response.

Take it further

Only links with an available result value appear.

Follow-up	Value carried	When offered
Redirect Checker	https://host/robots.txt	A host was checked
Website Down Checker	First included sitemap URL	A host check includes a sitemap result
Open Graph Preview	First page URL found in sitemap entries	A page URL is available, including in a pasted sitemap
security.txt Checker	Checked host	A host was checked

Related checks and input transfer

Check the rest opens related tools without carrying input. Take it further carries the specific value listed above. Pasted file contents are not transferred through these links. Transfers temporarily use browser sessionStorage with a one-use random token in the URL fragment; the input itself is not in the URL. The destination accepts a transfer for less than one minute and removes the token. Manual destination tools wait for their run button or Enter.

Each destination collects new evidence. A follow-up link does not confirm the original verdict or repair the remote site.

Result cards

Six expandable cards below the result explain its meaning, give a beginner explanation, offer an agent prompt, describe good practice, suggest fixes and state limitations. Their text follows the current result; it is not another measurement. Open a card for its full explanation and the relevant Method or Guide. The agent prompt can be copied for review; copying it does not run a repair or send it to an agent.

Method

Fetching

The server requests https://<host>/robots.txt with TamperboxBot/1.0, follows up to 5 redirects and reads the first 512 KiB. RFC 9309, section 2.3.1.2: crawlers “SHOULD follow at least five consecutive redirects”. Up to three Sitemap URLs are then fetched (or /sitemap.xml when none is listed), with sitemap index children until six sitemaps have been read; each is decompressed if gzipped and read up to 10 MiB.

The response before any rule

Response	Severity	Why
500 or above	FAIL	RFC 9309, section 2.3.1.4: on an unreachable status “the crawler MUST assume complete disallow”; Google keeps retrying and after 30 days uses its last copy or crawls everything
429	FAIL	Treated like a server error
Other 400 to 499	INFO	Section 2.3.1.3: “the crawler MAY access any resources on the server”
HTML content type or body	FAIL	A catch-all route; crawlers find no rules
Over 500 KiB	CHECK	Section 2.5: “The parsing limit MUST be at least 500 kibibytes”; Google stops there

Parsing lines

Following RFC 9309, section 2.2: comments after # are removed, blank lines skipped, field names read case-insensitively. Consecutive User-agent lines form one group, and Allow and Disallow belong to the group above them.

Line	Severity	Note
No field:	CHECK	Not a rule; skipped
Sitemap: not an absolute http(s) URL	FAIL	Sitemap is an “other record” (section 2.2.4); the Sitemaps protocol asks for the full URL
Allow/Disallow before any User-agent	CHECK	Ignored; Heal it moves it under User-agent: *
Dissallow, Disalow and similar	CHECK	Google’s parser accepts them; others ignore the line. Heal it fixes the spelling
Path not starting with / or *	CHECK	Matches nothing. Heal it adds the slash
Noindex	CHECK	Not a robots.txt rule; Google stopped reading it in 2019. Heal it removes it
Crawl-delay, Host	INFO	Google ignores both; Bing and Yandex read Crawl-delay
Unknown field	CHECK	Skipped
A group with no rules	INFO	That crawler may fetch everything
A byte order mark	INFO	Some small crawlers misread the first line

Which group a crawler obeys

RFC 9309, section 2.2.1: “If there is more than one group matching the user-agent, the matching groups’ rules MUST be combined”, and a crawler without a matching group obeys the * group. The tool matches the product token case-insensitively; if no group names it, the token is shortened at each hyphen (Google-Extended, then google) before falling back to *. A crawler follows only its own group: a User-agent: Googlebot group with one Allow line drops every * rule for Googlebot.

Which rule decides a path

RFC 9309, section 2.2.2: “the most specific match found MUST be used. The most specific match is the match that has the most octets. … If an allow rule and a disallow rule are equivalent, then the allow rule SHOULD be used.” The tester approximates those rules:

• the path includes the query; percent-escapes of unreserved characters are decoded, others upper-cased, on both sides (section 2.2.2);
• * matches any run of characters and a final $ anchors the end (section 2.2.3); every other rule is a prefix;
• the longest raw pattern wins by JavaScript string length, excluding a final $; Allow wins a tie. This is not an octet-length comparison for every Unicode or percent-encoded spelling;
• an empty Disallow matches nothing, and /robots.txt is always allowed (section 2.2.2).

With Disallow: /shop/ and Allow: /shop/sale, /shop/sale/boots is allowed: the Allow rule is longer.

The crawler table

14 tokens at /: Googlebot, Bingbot, Applebot and DuckDuckBot (search, judged), GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Bytespider (AI, stated). Google-Extended and Applebot-Extended are control tokens read by their owners’ crawlers, not crawlers themselves. When all four search crawlers are blocked, “Every search engine is blocked” leads the findings. That title describes only these four catalogued search tokens at /, not every crawler or every path.

Sitemaps

Read as the Sitemaps protocol 0.9 defines:

Sitemap issue	Severity
An HTML page; no <urlset> or <sitemapindex>; no closing tag	FAIL
An entry without <loc>; an unescaped & in <loc>; a relative <loc>	FAIL
More than 50,000 entries	FAIL
No sitemaps.org namespace	CHECK
A URL on another host	CHECK
A lastmod that is not a W3C date or is more than a day ahead	CHECK
No entries	CHECK
A listed sitemap answering 400 or more, or not at all	CHECK
Listed twice; changefreq or priority present	INFO

Then the cross-check: sitemap URLs that Googlebot may not fetch are CHECK. No Sitemap line is INFO, with a repair that adds one when /sitemap.xml answered.

Standards and conformance

RFC 9309, Robots Exclusion Protocol

Implemented: groups and combination (2.2.1), longest match with Allow on ties (2.2.2), * and $ (2.2.3), access results by status (2.3.1), the 500 KiB parsing limit (2.5), five redirects. Lenient where Google is: the misspellings of Disallow are reported but evaluated as Disallow, as Google’s open-source parser does. Not part of RFC 9309 but reported: Sitemap (widely supported, and mentioned as an “other record” in section 2.2.4), Crawl-delay, Host, Noindex. RFC 9309 has no predecessor RFC; it standardised the 1994 convention.

Sitemaps protocol 0.9

Implemented: both root elements, loc, lastmod, the 50,000 entry and 50 MiB limits, escaping, same-host URLs. Live fetching and file drops stop at 10 MiB. The parser also has a 50 MiB size rule, but those bounded inputs cannot exercise it; it is not evidence that a larger original file was fully validated. Text sitemaps and image or news extensions are not read.

Crawler documentation

Tokens and their meaning come from each operator’s own crawler documentation; their behaviour beyond RFC 9309 is not verified.

Finding reference

File and record findings

Finding	Severity	What it means for crawlers	Fix
Every search engine is blocked	FAIL	All four catalogued search tokens are denied at /	Remove Disallow: / from *
robots.txt answered 5xx or 429	FAIL	Google crawls nothing	Serve the file statically
robots.txt is an HTML page	FAIL	No rules found	Serve a text file before the catch-all
robots.txt is n KiB	CHECK	Rules past 500 KiB ignored	Use wildcards
No robots.txt (n)	INFO	Everything allowed	Optional
Line n: Sitemap must be a full URL	FAIL	Sitemap not found	Full https:// address
Line n: Rule before any User-agent	CHECK	Ignored	Move under a group
Line n: “x” is a misspelling of Disallow	CHECK	Only Google obeys	Spell Disallow
Line n: Path does not start with /	CHECK	Matches nothing	Add the slash
Line n: Noindex in robots.txt does nothing	CHECK	Pages can still be indexed	Use meta robots or X-Robots-Tag
Line n: Not a rule, Unknown field	CHECK	Skipped	Rewrite or remove
Line n: Crawl-delay is not read by Google, Host is obsolete	INFO	Partly ignored	Optional
Line n: No rules for x	INFO	That crawler may fetch everything	Add rules or remove the group
Line 1: Starts with a byte order mark	INFO	First line misread by some	Save without BOM

Sitemap findings

Finding	Severity	What it means for crawlers	Fix
n sitemap URLs are blocked by robots.txt	CHECK	Conflicting signals	Narrow the rule or drop the URL
No Sitemap line	INFO	Sitemap found more slowly	Add one (Heal it)
Sitemap answered n, did not answer	CHECK	Skipped by crawlers	Fix the address
sitemap: (each issue in the table above)	as listed

Guide

The site is closed to search engines

1. Open the file and find Disallow: / in the * group (or a search crawler’s group).
2. If the site should be public, remove it or narrow it to the paths that should stay private.
3. Publish at /robots.txt and check again. Confirm outside Tamperbox: curl -s https://example.com/robots.txt.

A page you want found is blocked

Enter its URL in the tester with Googlebot. The marked line is the one to change; add an Allow: for the page or narrow the Disallow:. Remember a crawler with its own group ignores *.

robots.txt answers 5xx, 429 or HTML

Serve it as a static file outside the application, and exempt the path from bot filters and rate limits. curl -sI https://example.com/robots.txt should show 200 and text/plain.

Blocking or allowing AI crawlers

Add a group per token you want to block (User-agent: GPTBot then Disallow: /). Training and search tokens are separate: blocking GPTBot does not block OAI-SearchBot. Google-Extended controls use of content for Google’s AI models, not Google Search.

Sitemap problems

Fix the file the generator writes; the sitemap’s own issues name the line. Announce it with Sitemap: https://example.com/sitemap.xml.

Break it cases

Case	What it loads	What to look at
Block everything	User-agent: * / Disallow: / above your file	Closed to search engines
Misspelled Disallow	Your file with Disallow misspelled	The line marked; Heal it fixes it
Rule above User-agent	Your first rule moved above every group	Heal it moves it back
Noindex line	A Noindex line after your first rule	Heal it removes it
Server error (503)	A sample host whose file answers 503	Response finding
Served as a web page	A sample single-page app	robots.txt is an HTML page
No robots.txt	A sample host without the file	No robots.txt

The first four are built from your own file when you entered a host or pasted one.

Worked example

The page opens with a live check of tamperbox.dev. It fetches the current robots.txt and a bounded selection of sitemap files. The crawler table describes permission for /; it does not summarize every path on the site.

Choose a crawler and enter a path that matters to your site. Read the highlighted deciding rule and compare it with the most specific matching group. An allowed home page can coexist with a blocked checkout or administration path. Likewise, a listed sitemap URL can be blocked by the rules even though the sitemap itself was fetched successfully.

Review any suggested repair before copying it into your repository. Heal it replaces the input with corrected text; it does not publish a policy. Recheck a published change by fetching the file independently and testing the affected paths again. Blocking a crawler is a policy choice, not an authentication barrier, and a clean syntax result does not prove that every crawler will obey it.

For agents

Use the Tamperbox MCP search tool to inspect the schema, then execute with tamperbox.request to call POST /v1/tools/robots_txt_checker. Supply the required domain field as a public domain name. The operation is named robots_txt_checker; the browser page’s hyphenated identifier is not the API operation name.

The response contains file evidence, HTTP response interpretation and a parsed report. It does not fetch the sitemap collection or reproduce the browser’s path-testing table. Sitemap declarations are parsed URLs, not proof that the files or their pages were retrieved.

Keep the input, observation time, source limitations and inconclusive results with any summary. Browser fixtures are demonstrations, not live MCP targets. Do not turn missing evidence into a passing check or infer that an inaccessible resource has been deleted.

See Connect an MCP client, Code mode and Security and privacy.

Limits

• One fetch of robots.txt (512 KiB, 5 redirects) and up to six sitemaps (10 MiB each).
• The crawler table is for / only.
• Crawler-specific extensions beyond RFC 9309 are not evaluated.
• Live checks are rate limited. Explicit result-action transfers use temporary browser storage; diagnostic results are not a monitoring history.

References

• RFC 9309, Robots Exclusion Protocol: sections 2.2, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.3.1, 2.5. Proposed standard, 2022.
• Sitemaps XML format, protocol 0.9.
• W3C Date and Time Formats.