# Robots.txt & Sitemap Checker

How the robots.txt checker fetches and parses a site's rules and sitemaps, which group each crawler obeys, which rule decides a path, every finding it reports, and how to fix each one.

Canonical: https://docs.tamperbox.dev/tools/robots-txt-checker/
Updated: 2026-10-26
Applies to: tamperbox.dev, October 2026
Prerequisites: A public host name, or a robots.txt or XML sitemap to paste
Outputs: A verdict for search and AI crawlers, findings, a path tester that marks the deciding line, the file line by line, a crawler table for the home page and the sitemaps with their URL counts
Limitations: robots.txt is a request crawlers may ignore; the crawler table judges the home page only; at most six sitemaps, 10 MiB each

The tool page is [tamperbox.dev/robots-txt-checker](https://tamperbox.dev/robots-txt-checker). It checks `tamperbox.dev` live when opened, says which search engines and AI crawlers the rules let in, and lets you test any path. This page is the long version: how the file is fetched and read, how a crawler's group and a path's rule are chosen, every finding, and how to fix each one.

<div class="tb-callout"><strong>Where processing happens:</strong> a pasted robots.txt or sitemap is read in your browser and never uploaded. A host name is sent to the Tamperbox server, which fetches <code>/robots.txt</code> and up to six sitemaps without cookies and returns them; the rules are evaluated in your browser. Explicit result-action transfers use temporary browser storage; diagnostic results are not a monitoring history. Sample hosts on the reserved <code>.example</code> zone are answered in the browser from a fixture.</div>

## What the tool answers

1. **Can search engines crawl the site?** Googlebot, Bingbot, Applebot and DuckDuckBot at the home page, judged: blocked is a failure.
2. **Which AI crawlers are let in?** Ten AI training, search and on-request tokens, stated without a verdict, because that is the owner's choice.
3. **Is a given URL allowed, and why?** The tester names the deciding line for any path and crawler.
4. **Is the file and its sitemaps well formed?** Every line a crawler would skip or misread, and every sitemap problem.
5. **Do the rules and the sitemap agree?** URLs the sitemap asks to be crawled that the rules refuse.

## Inputs

### A host or address

`example.com` or a full address. The host is what is checked; a path in a full address (`https://example.com/shop/boots`) becomes the tester's path. The check runs when you stop typing, or on Enter. The usual refusals apply: spaces, other schemes, credentials, non-standard ports, IP addresses and single labels.

### A pasted robots.txt

Text with a newline, a line starting `#`, or a line starting `User-agent:`, `Disallow:`, `Allow:`, `Sitemap:` or `Crawl-delay:` is read as robots.txt, in the browser. Nothing is fetched, so the Sitemaps section stays empty.

### A pasted sitemap

Text starting with `<?xml`, `<urlset` or `<sitemapindex` is read as a sitemap, in the browser.

Files up to 10 MiB can be dropped or opened (`.txt`, `.xml`). A pasted address loses quotes, brackets and trailing punctuation and keeps the host; text that fits nothing stays as pasted so the error can explain it.

## Reading the result

### Verdict and chips

"Open to search engines", "Closed to Googlebot" or "Closed to search engines", then the AI crawlers blocked and the number of things to fix. For a missing file: "No robots.txt. Every crawler may fetch every page." The chips count AI crawlers blocked, sitemap URLs blocked, and problems; each scrolls to its section. Two items say where the input went.

### Findings

Every finding, worst first, line findings prefixed with their line; two or more passing findings are grouped under an expandable **checks passed** row. A single passing finding stays visible.

### Test a URL

A path or full URL and a crawler. The result says Allowed or Blocked, names the group and the deciding line, and marks that line in the file. It opens on the path from your address, or on the first blocked sitemap URL, with Googlebot.

### robots.txt

The file as written, numbered, each line with an issue tinted and noted. Long files show their first 60 lines, every marked line and the tested line; **Show all n lines** lists the rest, and **Show the first 60 lines** folds it again. Folding is offered only when at least ten unmarked lines after line 60 can be hidden; marked lines remain visible.

### Crawlers at the home page

Two tables, search engines and AI crawlers: who runs each token, Allowed or Blocked at `/`, and the deciding rule or group.

### Sitemaps

Each sitemap fetched, its status, URL count, last change, and up to three issues.

### History

History follows the current result evidence. Break it and Heal it record input changes, newest first. **Revert** restores the input before the latest recorded change; **Reset input** restores the input before the first retained change and clears the list. **Restore input before this break** under Heal it also reverts the latest change.

These actions reload and check the restored input. Restoring a public address can therefore send a new request, including on the two tools whose ordinary typing waits for a run button. Editing or pasting a new input clears History. The list lives in the current page; it is not saved monitoring data and does not restore an old network response.

### Take it further

Only links with an available result value appear.

| Follow-up | Value carried | When offered |
| --- | --- | --- |
| [Redirect Checker](https://tamperbox.dev/redirect-checker) | `https://host/robots.txt` | A host was checked |
| [Website Down Checker](https://tamperbox.dev/website-down-checker) | First included sitemap URL | A host check includes a sitemap result |
| [Open Graph Preview](https://tamperbox.dev/open-graph-preview) | First page URL found in sitemap entries | A page URL is available, including in a pasted sitemap |
| [security.txt Checker](https://tamperbox.dev/security-txt-checker) | Checked host | A host was checked |

### Related checks and input transfer

Check the rest opens related tools without carrying input. Take it further carries the specific value listed above. Pasted file contents are not transferred through these links. Transfers temporarily use browser `sessionStorage` with a one-use random token in the URL fragment; the input itself is not in the URL. The destination accepts a transfer for less than one minute and removes the token. Manual destination tools wait for their run button or Enter.

Each destination collects new evidence. A follow-up link does not confirm the original verdict or repair the remote site.

### Result cards

Six expandable cards below the result explain its meaning, give a beginner explanation, offer an agent prompt, describe good practice, suggest fixes and state limitations. Their text follows the current result; it is not another measurement. Open a card for its full explanation and the relevant [Method](#method) or [Guide](#guide). The agent prompt can be copied for review; copying it does not run a repair or send it to an agent.

## Method

### Fetching

The server requests `https://<host>/robots.txt` with `TamperboxBot/1.0`, follows up to 5 redirects and reads the first 512 KiB. [RFC 9309, section 2.3.1.2](https://www.rfc-editor.org/rfc/rfc9309#section-2.3.1.2): crawlers "SHOULD follow at least five consecutive redirects". Up to three Sitemap URLs are then fetched (or `/sitemap.xml` when none is listed), with sitemap index children until six sitemaps have been read; each is decompressed if gzipped and read up to 10 MiB.

### The response before any rule

| Response | Severity | Why |
| --- | --- | --- |
| 500 or above | FAIL | [RFC 9309, section 2.3.1.4](https://www.rfc-editor.org/rfc/rfc9309#section-2.3.1.4): on an unreachable status "the crawler MUST assume complete disallow"; Google keeps retrying and after 30 days uses its last copy or crawls everything |
| 429 | FAIL | Treated like a server error |
| Other 400 to 499 | INFO | [Section 2.3.1.3](https://www.rfc-editor.org/rfc/rfc9309#section-2.3.1.3): "the crawler MAY access any resources on the server" |
| HTML content type or body | FAIL | A catch-all route; crawlers find no rules |
| Over 500 KiB | CHECK | [Section 2.5](https://www.rfc-editor.org/rfc/rfc9309#section-2.5): "The parsing limit MUST be at least 500 kibibytes"; Google stops there |

### Parsing lines

Following [RFC 9309, section 2.2](https://www.rfc-editor.org/rfc/rfc9309#section-2.2): comments after `#` are removed, blank lines skipped, field names read case-insensitively. Consecutive `User-agent` lines form one group, and `Allow` and `Disallow` belong to the group above them.

| Line | Severity | Note |
| --- | --- | --- |
| No `field:` | CHECK | Not a rule; skipped |
| `Sitemap:` not an absolute http(s) URL | FAIL | Sitemap is an "other record" ([section 2.2.4](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.4)); the [Sitemaps protocol](https://www.sitemaps.org/protocol.html#submit_robots) asks for the full URL |
| `Allow`/`Disallow` before any `User-agent` | CHECK | Ignored; Heal it moves it under `User-agent: *` |
| `Dissallow`, `Disalow` and similar | CHECK | Google's parser accepts them; others ignore the line. Heal it fixes the spelling |
| Path not starting with `/` or `*` | CHECK | Matches nothing. Heal it adds the slash |
| `Noindex` | CHECK | Not a robots.txt rule; Google stopped reading it in 2019. Heal it removes it |
| `Crawl-delay`, `Host` | INFO | Google ignores both; Bing and Yandex read Crawl-delay |
| Unknown field | CHECK | Skipped |
| A group with no rules | INFO | That crawler may fetch everything |
| A byte order mark | INFO | Some small crawlers misread the first line |

### Which group a crawler obeys

[RFC 9309, section 2.2.1](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.1): "If there is more than one group matching the user-agent, the matching groups' rules MUST be combined", and a crawler without a matching group obeys the `*` group. The tool matches the product token case-insensitively; if no group names it, the token is shortened at each hyphen (`Google-Extended`, then `google`) before falling back to `*`. A crawler follows only its own group: a `User-agent: Googlebot` group with one Allow line drops every `*` rule for Googlebot.

### Which rule decides a path

[RFC 9309, section 2.2.2](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.2): "the most specific match found MUST be used. The most specific match is the match that has the most octets. ... If an allow rule and a disallow rule are equivalent, then the allow rule SHOULD be used." The tester approximates those rules:

- the path includes the query; percent-escapes of unreserved characters are decoded, others upper-cased, on both sides ([section 2.2.2](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.2));
- `*` matches any run of characters and a final `$` anchors the end ([section 2.2.3](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.3)); every other rule is a prefix;
- the longest raw pattern wins by JavaScript string length, excluding a final `$`; Allow wins a tie. This is not an octet-length comparison for every Unicode or percent-encoded spelling;
- an empty `Disallow` matches nothing, and `/robots.txt` is always allowed ([section 2.2.2](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.2)).

With `Disallow: /shop/` and `Allow: /shop/sale`, `/shop/sale/boots` is allowed: the Allow rule is longer.

### The crawler table

14 tokens at `/`: Googlebot, Bingbot, Applebot and DuckDuckBot (search, judged), GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and Bytespider (AI, stated). Google-Extended and Applebot-Extended are control tokens read by their owners' crawlers, not crawlers themselves. When all four search crawlers are blocked, "Every search engine is blocked" leads the findings. That title describes only these four catalogued search tokens at `/`, not every crawler or every path.

### Sitemaps

Read as the [Sitemaps protocol 0.9](https://www.sitemaps.org/protocol.html) defines:

| Sitemap issue | Severity |
| --- | --- |
| An HTML page; no `<urlset>` or `<sitemapindex>`; no closing tag | FAIL |
| An entry without `<loc>`; an unescaped `&` in `<loc>`; a relative `<loc>` | FAIL |
| More than 50,000 entries | FAIL |
| No sitemaps.org namespace | CHECK |
| A URL on another host | CHECK |
| A `lastmod` that is not a [W3C date](https://www.w3.org/TR/NOTE-datetime) or is more than a day ahead | CHECK |
| No entries | CHECK |
| A listed sitemap answering 400 or more, or not at all | CHECK |
| Listed twice; `changefreq` or `priority` present | INFO |

Then the cross-check: sitemap URLs that Googlebot may not fetch are CHECK. No Sitemap line is INFO, with a repair that adds one when `/sitemap.xml` answered.

## Standards and conformance

### RFC 9309, Robots Exclusion Protocol

Implemented: groups and combination (2.2.1), longest match with Allow on ties (2.2.2), `*` and `$` (2.2.3), access results by status (2.3.1), the 500 KiB parsing limit (2.5), five redirects. Lenient where Google is: the misspellings of Disallow are reported but evaluated as Disallow, as Google's open-source parser does. Not part of RFC 9309 but reported: `Sitemap` (widely supported, and mentioned as an "other record" in [section 2.2.4](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.4)), `Crawl-delay`, `Host`, `Noindex`. RFC 9309 has no predecessor RFC; it standardised the 1994 convention.

### Sitemaps protocol 0.9

Implemented: both root elements, `loc`, `lastmod`, the 50,000 entry and 50 MiB limits, escaping, same-host URLs. Live fetching and file drops stop at 10 MiB. The parser also has a 50 MiB size rule, but those bounded inputs cannot exercise it; it is not evidence that a larger original file was fully validated. Text sitemaps and image or news extensions are not read.

### Crawler documentation

Tokens and their meaning come from each operator's own crawler documentation; their behaviour beyond RFC 9309 is not verified.

## Finding reference

### File and record findings

| Finding | Severity | What it means for crawlers | Fix |
| --- | --- | --- | --- |
| Every search engine is blocked | FAIL | All four catalogued search tokens are denied at `/` | Remove `Disallow: /` from `*` |
| robots.txt answered 5xx or 429 | FAIL | Google crawls nothing | Serve the file statically |
| robots.txt is an HTML page | FAIL | No rules found | Serve a text file before the catch-all |
| robots.txt is n KiB | CHECK | Rules past 500 KiB ignored | Use wildcards |
| No robots.txt (n) | INFO | Everything allowed | Optional |
| Line n: Sitemap must be a full URL | FAIL | Sitemap not found | Full https:// address |
| Line n: Rule before any User-agent | CHECK | Ignored | Move under a group |
| Line n: "x" is a misspelling of Disallow | CHECK | Only Google obeys | Spell Disallow |
| Line n: Path does not start with / | CHECK | Matches nothing | Add the slash |
| Line n: Noindex in robots.txt does nothing | CHECK | Pages can still be indexed | Use meta robots or X-Robots-Tag |
| Line n: Not a rule, Unknown field | CHECK | Skipped | Rewrite or remove |
| Line n: Crawl-delay is not read by Google, Host is obsolete | INFO | Partly ignored | Optional |
| Line n: No rules for x | INFO | That crawler may fetch everything | Add rules or remove the group |
| Line 1: Starts with a byte order mark | INFO | First line misread by some | Save without BOM |

### Sitemap findings

| Finding | Severity | What it means for crawlers | Fix |
| --- | --- | --- | --- |
| n sitemap URLs are blocked by robots.txt | CHECK | Conflicting signals | Narrow the rule or drop the URL |
| No Sitemap line | INFO | Sitemap found more slowly | Add one (Heal it) |
| Sitemap answered n, did not answer | CHECK | Skipped by crawlers | Fix the address |
| sitemap: (each issue in the table above) | as listed | | |

## Guide

### The site is closed to search engines

1. Open the file and find `Disallow: /` in the `*` group (or a search crawler's group).
2. If the site should be public, remove it or narrow it to the paths that should stay private.
3. Publish at `/robots.txt` and check again. Confirm outside Tamperbox: `curl -s https://example.com/robots.txt`.

### A page you want found is blocked

Enter its URL in the tester with Googlebot. The marked line is the one to change; add an `Allow:` for the page or narrow the `Disallow:`. Remember a crawler with its own group ignores `*`.

### robots.txt answers 5xx, 429 or HTML

Serve it as a static file outside the application, and exempt the path from bot filters and rate limits. `curl -sI https://example.com/robots.txt` should show `200` and `text/plain`.

### Blocking or allowing AI crawlers

Add a group per token you want to block (`User-agent: GPTBot` then `Disallow: /`). Training and search tokens are separate: blocking GPTBot does not block OAI-SearchBot. Google-Extended controls use of content for Google's AI models, not Google Search.

### Sitemap problems

Fix the file the generator writes; the sitemap's own issues name the line. Announce it with `Sitemap: https://example.com/sitemap.xml`.

### Break it cases

| Case | What it loads | What to look at |
| --- | --- | --- |
| Block everything | `User-agent: * / Disallow: /` above your file | Closed to search engines |
| Misspelled Disallow | Your file with Disallow misspelled | The line marked; Heal it fixes it |
| Rule above User-agent | Your first rule moved above every group | Heal it moves it back |
| Noindex line | A Noindex line after your first rule | Heal it removes it |
| Server error (503) | A sample host whose file answers 503 | Response finding |
| Served as a web page | A sample single-page app | robots.txt is an HTML page |
| No robots.txt | A sample host without the file | No robots.txt |

The first four are built from your own file when you entered a host or pasted one.

## Worked example

The page opens with a live check of `tamperbox.dev`. It fetches the current robots.txt and a bounded selection of sitemap files. The crawler table describes permission for `/`; it does not summarize every path on the site.

Choose a crawler and enter a path that matters to your site. Read the highlighted deciding rule and compare it with the most specific matching group. An allowed home page can coexist with a blocked checkout or administration path. Likewise, a listed sitemap URL can be blocked by the rules even though the sitemap itself was fetched successfully.

Review any suggested repair before copying it into your repository. Heal it replaces the input with corrected text; it does not publish a policy. Recheck a published change by fetching the file independently and testing the affected paths again. Blocking a crawler is a policy choice, not an authentication barrier, and a clean syntax result does not prove that every crawler will obey it.

## For agents

Use the Tamperbox MCP `search` tool to inspect the schema, then `execute` with `tamperbox.request` to call `POST /v1/tools/robots_txt_checker`. Supply the required `domain` field as a public domain name. The operation is named `robots_txt_checker`; the browser page’s hyphenated identifier is not the API operation name.

The response contains file `evidence`, HTTP `response` interpretation and a parsed `report`. It does not fetch the sitemap collection or reproduce the browser’s path-testing table. Sitemap declarations are parsed URLs, not proof that the files or their pages were retrieved.

Keep the input, observation time, source limitations and inconclusive results with any summary. Browser fixtures are demonstrations, not live MCP targets. Do not turn missing evidence into a passing check or infer that an inaccessible resource has been deleted.

See [Connect an MCP client](https://docs.tamperbox.dev/mcp/connect/), [Code mode](https://docs.tamperbox.dev/mcp/code-mode/) and [Security and privacy](https://docs.tamperbox.dev/reference/security/).

## Limits

- One fetch of robots.txt (512 KiB, 5 redirects) and up to six sitemaps (10 MiB each).
- The crawler table is for `/` only.
- Crawler-specific extensions beyond RFC 9309 are not evaluated.
- Live checks are rate limited. Explicit result-action transfers use temporary browser storage; diagnostic results are not a monitoring history.

## References

- [RFC 9309, Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309): sections [2.2](https://www.rfc-editor.org/rfc/rfc9309#section-2.2), [2.2.1](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.1), [2.2.2](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.2), [2.2.3](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.3), [2.2.4](https://www.rfc-editor.org/rfc/rfc9309#section-2.2.4), [2.3.1](https://www.rfc-editor.org/rfc/rfc9309#section-2.3.1), [2.5](https://www.rfc-editor.org/rfc/rfc9309#section-2.5). Proposed standard, 2022.
- [Sitemaps XML format, protocol 0.9](https://www.sitemaps.org/protocol.html).
- [W3C Date and Time Formats](https://www.w3.org/TR/NOTE-datetime).