Skip to main content

Free tool · no signup

robots.txt Checker

Parse any site's robots.txt and see what each AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot) is allowed or blocked from doing. Catches common mistakes — missing file, syntax errors, accidentally blocking AI search.

See the full picture — not just robots.txt

The Nova Insight Website Audit checks AI crawler access alongside 300+ other technical factors — schema markup, structured data, page-speed, security headers — and delivers a branded PDF with prioritised fixes.

Free robots.txt Checker + Analyser — Nova Insight