Is your site blocking AI crawlers?
Many websites block AI crawlers by accident through robots.txt or firewall rules, which means ChatGPT, Claude, and Perplexity cannot read or cite them. Enter a domain and see the rules for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google Extended, and Bingbot, with a corrected robots.txt to copy.
No email required · No account · Reads only your public robots.txt
What the result means
The check reads robots.txt at the root of your domain and applies its rules to your homepage the way the robots.txt standard (RFC 9309) describes.
- Allowed
- The rules let this crawler fetch your homepage. When some folders are still closed to it, the row says so.
- Blocked
- A Disallow rule matches your homepage for this crawler, either in a group that names it or in the User-agent: * group it falls back to.
- No robots.txt
- The site answers /robots.txt with a 404 or another 4xx status. Under the standard that means there are no rules, so every crawler may fetch every page.
- Firewalls are a separate check
- This tool reads robots.txt only. A firewall, CDN bot setting, or bot challenge can still turn AI crawlers away when robots.txt allows them, and that does not show in this result.
How to fix it
Put one group that names the AI crawlers at the top of robots.txt. A group that names a crawler replaces the User-agent: * group for it, and Allow: / opens the whole site. Remove any other group that blocks them.
# Let AI crawlers read the site
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
User-agent: Bingbot
Allow: /Does robots.txt stop AI crawlers from reading my site?
Only crawlers that choose to follow it. robots.txt is a set of requests that well behaved crawlers read before they fetch a page; it does not enforce anything. OpenAI, Anthropic, Google, and Perplexity publish the user agent names their crawlers use, so you can write rules for each one.
What happens if my site has no robots.txt?
Crawlers treat the whole site as open. Under the robots.txt standard (RFC 9309), a 404 or other 4xx answer means there are no rules. A server error is different: while robots.txt returns a 5xx error, crawlers that follow the standard treat the whole site as blocked.
How does a crawler decide which rules apply to it?
It uses the group whose User-agent line names it, and falls back to the User-agent: * group only when no group names it. Inside that group the longest matching rule wins, and Allow wins a tie with Disallow. That is how a site can block everything for * and still let GPTBot in with its own group.
Is Google-Extended a separate crawler?
No. Google-Extended is a robots.txt token, not a crawler that visits your site. Googlebot does the crawling, and a Disallow for Google-Extended tells Google not to use your content to train or ground its Gemini models. Google states that it does not affect your inclusion or ranking in Google Search.
AI crawler questions
Decide with evidence, not a pitch.
Open a neutral research prompt in the AI you already use. It asks where NextCTL fits, where a freelancer or an in house hire may be enough, and what to verify before you choose.
- 01ChatGPTOpen the NextCTL comparison prompt in ChatGPT (opens in a new tab)
- 02ClaudeOpen the NextCTL comparison prompt in Claude (opens in a new tab)
- 03PerplexityOpen the NextCTL comparison prompt in Perplexity (opens in a new tab)
AI answers can be incomplete. Check nextctl.com/methodology before deciding.
See where your operations
are leaking revenue.
A 14 day audit shows where missed calls and slow follow up cost you most. Credited toward any build within 90 days.
Or run a free tool first:Website AuditAI Readiness Scan