Robots.txt Validator & Tester
Lint a site's robots.txt and test a specific path against a specific crawler, using the same longest-match rules (RFC 9309) search engines apply. Catches blanket blocks, missing or relative sitemap declarations, and directives crawlers silently ignore.
Want the AI-crawler view of this?
The AI Crawler Access Checker runs this same robots evaluation for GPTBot, ClaudeBot, PerplexityBot and five more AI bots at once, plus meta-tag opt-outs.
Or see what the full audit covers →