Step 1
Enter a website URL
We fetch the live robots.txt from the site origin using SSRF-safe server requests.
Free SEO Tool
Robots.txt Tester
Fetch a live robots.txt, parse Allow/Disallow groups, inspect sitemaps, and test path access for Googlebot, Bingbot, and AI crawlers.
Enter a website URL to fetch robots.txt, inspect directives, and test whether a path is allowed for a crawler.
How It Works
Step 2
Choose crawler and path
Pick Googlebot, Bingbot, GPTBot, or another crawler and optionally test a specific path.
Step 3
Review allow/disallow outcome
See matched rules, sitemap declarations, warnings, and clear allow/disallow guidance.
Use Cases
Indexing troubleshooting
Confirm important pages are not accidentally blocked for Googlebot.
AI crawler policy checks
Review whether GPTBot and similar crawlers are allowed or blocked on key paths.
Sitemap discovery
Quickly inspect which sitemap URLs are declared in robots.txt.
Frequently Asked Questions
Is robots parsing deterministic?
Yes. Path allow/disallow decisions use deterministic robots.txt matching rules, not AI guesses.
Which crawlers are supported?
Presets include Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, and the wildcard * group.
What if robots.txt is missing?
If the file cannot be fetched, the tool returns an error. An empty file is generally treated as fully allowed.
Related Tools
Technical SEO
Robots.txt Generator
Generate valid robots.txt rules quickly with crawler-specific directives.
Open toolTechnical SEO
Meta Tag Checker
Audit title tags, descriptions, canonical links, robots directives, H1s, and Open Graph basics from a live URL.
Open toolTechnical SEO
XML Sitemap Validator
Validate an XML sitemap for syntax issues, structure problems, and inaccessible sample URLs.
Open toolTurn Insights Into Published Content
These free tools help you validate opportunities and technical quality. RankHit takes the next step: keyword opportunities to content plan, optimized article creation, and publishing workflow.