Validador de robots.txt
Revisa tu robots.txt en busca de errores de sintaxis, grupos de user-agent faltantes y malas URLs de sitemap.
Respuesta rápida: Validador de robots.txt: analiza tu archivo robots.txt para detectar errores de sintaxis, grupos user-agent ausentes y URL de sitemaps incorrectas. Funciona íntegramente en el navegador, gratis y sin registro.
Última actualización
Bueno saber
A robots.txt file lives at the root of your domain and tells crawlers which paths they may or may not request. Its syntax looks simple but is unforgiving: rules are grouped under a User-agent line, and a group with no preceding user-agent, a misspelled directive, or the wrong order can silently be ignored — meaning a rule you thought was blocking a section is doing nothing at all. This linter parses the file line by line and flags exactly those structural mistakes.
The most important thing to understand is what robots.txt does not do: it is a crawl directive, not an access-control or a de-indexing tool. A well-behaved crawler will not fetch a disallowed path, but the URL can still appear in search results if it is linked elsewhere, and malicious bots ignore the file entirely. To keep a page out of the index use a noindex meta tag or header, and to protect private content use real authentication.
A few directive quirks are worth knowing: Google honors wildcards (*) and end-of-URL anchors ($) in path patterns but ignores Crawl-delay, whereas Bing and Yandex still respect the delay. Absolute Sitemap URLs belong here too and are read independently of any user-agent group. Once the syntax is clean, generate matching declarations with the robots.txt generator and register your URL list with the sitemap generator.
Preguntas frecuentes
- ¿Es el validador gratuito?
- Sí, gratuito y sin registro.
- ¿Coincide con el analizador de Google?
- Implementa el subconjunto del RFC de robots.txt de Google (User-agent, Allow, Disallow, Sitemap, Crawl-delay, Host) — las reglas que todos los rastreadores modernos respetan.
- ¿Se envían las cargas a algún lugar?
- No, cada verificación se realiza en tu navegador.
- ¿Por qué advierte sobre Crawl-delay?
- Google ignora Crawl-delay; Bing y Yandex aún lo respetan. La advertencia es informativa.
- ¿Qué pasa con los comodines?
- Los comodines (*) y el final de cadena ($) son aceptados en los patrones Allow/Disallow — esa es la sintaxis moderna de Google.