robots.txt-controle
Lint je robots.txt op syntaxisfouten, ontbrekende user-agent groepen en onjuiste sitemap URL's.
Snel antwoord: robots.txt-controle: controleer je robots.txt op syntaxisfouten, ontbrekende user-agent-groepen en onjuiste sitemap-URL's. Werkt volledig in je browser, gratis en zonder registratie.
Laatst bijgewerkt
Goed om te weten
A robots.txt file lives at the root of your domain and tells crawlers which paths they may or may not request. Its syntax looks simple but is unforgiving: rules are grouped under a User-agent line, and a group with no preceding user-agent, a misspelled directive, or the wrong order can silently be ignored — meaning a rule you thought was blocking a section is doing nothing at all. This linter parses the file line by line and flags exactly those structural mistakes.
The most important thing to understand is what robots.txt does not do: it is a crawl directive, not an access-control or a de-indexing tool. A well-behaved crawler will not fetch a disallowed path, but the URL can still appear in search results if it is linked elsewhere, and malicious bots ignore the file entirely. To keep a page out of the index use a noindex meta tag or header, and to protect private content use real authentication.
A few directive quirks are worth knowing: Google honors wildcards (*) and end-of-URL anchors ($) in path patterns but ignores Crawl-delay, whereas Bing and Yandex still respect the delay. Absolute Sitemap URLs belong here too and are read independently of any user-agent group. Once the syntax is clean, generate matching declarations with the robots.txt generator and register your URL list with the sitemap generator.
Veelgestelde vragen
- Is de validator gratis?
- Ja — gratis zonder aanmelding.
- Komt het overeen met Google's parser?
- Het implementeert de subset van de Google robots.txt RFC (User-agent, Allow, Disallow, Sitemap, Crawl-delay, Host) — de regels die elke moderne crawler respecteert.
- Worden uploads ergens naartoe gestuurd?
- Nee — elke controle wordt in je browser uitgevoerd.
- Waarom waarschuwt het over Crawl-delay?
- Google negeert Crawl-delay; Bing en Yandex respecteren het nog steeds. De waarschuwing is informatief.
- Wat betreft jokertekens?
- Jokertekens (*) en het einde-van-string ($) worden geaccepteerd in Allow/Disallow patronen — dat is de moderne Google syntaxis.