Loading…
Loading…
Check whether a URL is blocked, and see the exact rule that decided it
Paste the file from https://yoursite.com/robots.txt. Lines that decide a URL below are highlighted in the rule trace.
The token is the name in a User-agent line, not the full HTTP header. Matching is case-insensitive.
Paths like /blog/post?page=2 or full URLs. The query string counts; the fragment does not.
6 allowed, 3 blocked for Googlebot.
No group names "googlebot", so it follows the "*" group (5 rules).
| URL | Verdict | Deciding rule |
|---|---|---|
| / | Allowed | No rule in the "*" group matches this path, so it is allowed. |
| /admin/ | Blocked | Disallow: /admin/line 3, group “*” |
| /admin/public/pricing | Allowed | Allow: /admin/public/line 7, group “*” |
| /search?q=robots | Blocked | Disallow: /searchline 4, group “*” |
| /search/help | Allowed | Allow: /search/helpline 6, group “*” |
| /docs/guide.pdf | Blocked | Disallow: /*.pdf$line 5, group “*” |
| /docs/guide.pdf?download=1 | Allowed | No rule in the "*" group matches this path, so it is allowed. |
| /images/private/logo.png | Allowed | No rule in the "*" group matches this path, so it is allowed. |
| /robots.txt | Allowed | /robots.txt is always allowed so crawlers can read the rules. |
Highlighted lines decided at least one URL. Lines in the selected group are shown in full colour; other groups are dimmed because this user-agent never reads them.
The verdict follows RFC 9309 and the matching rules in Google's robots.txt documentation: the most specific user-agent group, longest matching pattern, Allow on a tie, * and $ wildcards, and percent-encoding normalised before comparison. It is not Googlebot itself, and it does not fetch anything: every calculation runs in your browser. robots.txt controls crawling only: to keep a page out of search results use a noindex rule. Once a page is crawlable, what Google can show for it depends on its markup; our schema markup generator builds the JSON-LD, and the guide to JSON-LD rich results that still work in 2026 lists which types still earn a rich result.
Need SEO and content templates?
Blog post frameworks, keyword research sheets, and on-page checklists built for real traffic growth. Starting at $4.
Browse SEO TemplatesOne Disallow line in the wrong place hides a section of a site from every crawler that reads it, and the file gives no feedback when that happens. Google removed the Search Console robots.txt Tester on 15 November 2023; the robots.txt report that replaced it reports fetch status and parse issues, but it cannot answer the question most edits raise: is this URL blocked for this crawler, and which line did it? This tester answers that for a pasted file, before it goes live.
The parser reads the file line by line. It strips comments after #, trims whitespace, lower-cases field names and keeps user-agent, allow, disallow and sitemap. User-agent lines open a group; the Allow and Disallow lines that follow belong to it. Only an Allow or Disallow line ends that run of User-agent lines: a Sitemap, crawl-delay or unknown line between two User-agent lines does not split the group, which is how Google's open-source parser reads it. Malformed lines, unknown fields and rules that appear before any User-agent line are listed as warnings with their line numbers. crawl-delay gets a note, because Google ignores it. A file above 500 KiB, the most Google reads, gets a warning as well.
Group selection follows Google's documented precedence. The crawler you pick is reduced to its product token, the token is compared case-insensitively with every User-agent in the file, and the most specific match wins: googlebot-image over googlebot over *. All groups for that token are merged. If nothing matches and there is no * group, every URL is allowed.
Each URL is reduced to its path and query string. Both the URL and every pattern are percent-normalised as RFC 9309 describes, so /caf%C3%A9 and /café compare equal while /a%2Fb and /a/b stay distinct. Every rule in the selected group is tried; the longest matching pattern decides, Allow wins a tie, and /robots.txt is always allowed. The verdict table shows the directive, pattern, line number and group for each URL, and the rule trace highlights the deciding lines in the source so you can see the decision in context.
An SEO auditing a client site pastes the live robots.txt and twenty key landing pages to confirm a redesign did not block them for Googlebot.
A developer adding Disallow: /*?sessionid= checks that product pages with other query parameters stay allowed before the change ships.
A publisher deciding what GPTBot may crawl tests the GPTBot group against article, feed and image URLs and sees which line decides each one.
A technical SEO migrating a site compares Allow and Disallow rules of equal length and confirms which one wins on the shared prefix.
Scope note: The tester reads the file you paste; it does not fetch robots.txt or any URL, so it cannot see redirects, fetch errors or what Google actually downloaded. It follows Google's documented rules and RFC 9309, not a live Googlebot, and crawlers that do not follow the protocol will ignore your file regardless of what this tool reports. robots.txt controls crawling only: use noindex to keep a page out of search results. Google's own parser accepts some leniencies, such as a missing colon and common misspellings of a field name, that this tester flags as malformed and ignores, so a line it warns about may still be read by Google. Google's special-case crawlers (AdsBot-Google, AdsBot-Google-Mobile, Mediapartners-Google, APIs-Google) ignore the * group and obey only a group that names them (https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers), and the tester applies that rule.
Paste your robots.txt into the first box. The example file is preloaded, so you can edit it instead of starting blank
Pick the crawler: Googlebot, Googlebot-Image, Bingbot, GPTBot or * for any other crawler. Choose "Custom…" to type any product token, such as ClaudeBot
Enter the URLs or paths to test, one per line. Full URLs are reduced to their path and query string
Read the verdict table: Allowed or Blocked per URL, the deciding rule, its line number and the group it came from. No matching rule means allowed
Fix what the warnings list flags (malformed lines, unknown fields, rules before any user-agent, a file above 500 KiB), then copy the report or re-test
About the Robots.txt Tester
Yes. Google retired the Search Console robots.txt Tester on 15 November 2023 and replaced it with the robots.txt report under Settings. The report lists the robots.txt files Google fetched for the top 20 hosts of a property, with the fetch status and any parse issues, and it is available for domain-level properties only. It does not take a URL and tell you whether it is blocked. For one live URL, the URL Inspection tool answers "Crawl allowed?". For a draft file, or for a crawler other than Google, this tester runs the same documented rules in your browser.
No. Google documents robots.txt as a way to manage crawler traffic, not as a way to keep a page out of Google. A URL that robots.txt blocks can still be indexed if other pages link to it. To keep a page out of search results, use a noindex rule in the page (a robots meta tag or an X-Robots-Tag header). A crawler can only read that rule on a page it is allowed to fetch, so do not block a page you also mark noindex.
The one whose User-agent token is the most specific match for the crawler, compared case-insensitively on the product token alone (the name, not the full HTTP header). With groups for googlebot-image, googlebot and *, Googlebot-Image follows googlebot-image; Googlebot follows googlebot; Bingbot follows *. If a crawler has no specific group and there is no * group, nothing is disallowed for it. Several groups naming the same token are merged into one, and a group opened by several User-agent lines applies to each of them. Only one group ever applies to a crawler. The Googlebot-Image to googlebot fallback models the precedence Google documents for its own crawlers; RFC 9309 compares tokens for equality, so a crawler from another company may honour only the group that names its exact token.
The longest matching pattern, counted in characters, with * and $ included in the count. When an Allow and a Disallow of equal length both match, Allow wins. Patterns are prefixes: Disallow: /fish blocks /fish, /fish.html and /fishheads/, but not /Fish.asp, because matching is case-sensitive. * stands for any run of characters and $ anchors the end of the path, so Disallow: /*.pdf$ blocks /docs/a.pdf but not /docs/a.pdf?download=1. The query string is part of the matched path; the fragment is not. An empty Disallow: means disallow nothing. /robots.txt itself is always allowed.
The tester implements the matching rules in Google's robots.txt documentation and RFC 9309, and Google's own parser is open source at github.com/google/robotstxt if you want to compare. It is not Googlebot: it does not fetch your file or your pages, and it does not know how Google treats a robots.txt that fails to load. For the authoritative answer on one URL of a verified property, use "Crawl allowed?" in the Search Console URL Inspection tool. The tester also notes two things Google documents: crawl-delay is ignored by Google, and only the first 500 KiB of a robots.txt is read.
No. Parsing and matching run in your browser with JavaScript. Nothing you paste is uploaded, no URL is fetched, and no account is needed.
Generate JSON-LD structured data for Google rich results
Open →SEO & ContentPreview how your site looks on Google, Twitter & more
Open →SEO & ContentBuild and validate an llms.txt file for AI agents
Open →DeveloperFind what breaks when Chrome 158 turns XSLT off on 17 Nov 2026
Open →