robots.txt Validator
Lint your robots.txt for syntax errors, missing user-agent groups and bad sitemap URLs.
Quick answer: robots.txt Validator: lint your robots.txt for syntax errors, missing user-agent groups and bad sitemap URLs. Runs entirely in your browser, free, no signup.
Last updated
Good to know
A robots.txt file lives at the root of your domain and tells crawlers which paths they may or may not request. Its syntax looks simple but is unforgiving: rules are grouped under a User-agent line, and a group with no preceding user-agent, a misspelled directive, or the wrong order can silently be ignored — meaning a rule you thought was blocking a section is doing nothing at all. This linter parses the file line by line and flags exactly those structural mistakes.
The most important thing to understand is what robots.txt does not do: it is a crawl directive, not an access-control or a de-indexing tool. A well-behaved crawler will not fetch a disallowed path, but the URL can still appear in search results if it is linked elsewhere, and malicious bots ignore the file entirely. To keep a page out of the index use a noindex meta tag or header, and to protect private content use real authentication.
A few directive quirks are worth knowing: Google honors wildcards (*) and end-of-URL anchors ($) in path patterns but ignores Crawl-delay, whereas Bing and Yandex still respect the delay. Absolute Sitemap URLs belong here too and are read independently of any user-agent group. Once the syntax is clean, generate matching declarations with the robots.txt generator and register your URL list with the sitemap generator.
Frequently asked questions
- Is the validator free?
- Yes — free with no signup.
- Does it match Google's parser?
- It implements the Google robots.txt RFC subset (User-agent, Allow, Disallow, Sitemap, Crawl-delay, Host) — the rules every modern crawler honors.
- Are uploads sent anywhere?
- No — every check runs in your browser.
- Why does it warn about Crawl-delay?
- Google ignores Crawl-delay; Bing and Yandex still honor it. The warning is informational.
- What about wildcards?
- Wildcards (*) and end-of-string ($) are accepted in Allow/Disallow patterns — that's the modern Google syntax.