Robots.txt AI Bot Access
Scan another siteWhat This Check Covers
Checks whether major AI shopping agents, AI search indexers, and AI training crawlers are allowed in robots.txt.
Frequently Asked Questions
- What is an AI Bot Access Policy in robots.txt?
- It's the set of rules in your site's robots.txt file that specifically allow or block user-agents associated with AI services, such as ChatGPT-User, Anthropic-ai, or Google-Extended.
- Why should I allow AI shopping agents?
- AI shopping agents actively browse the web to find products for users. Blocking them means your products won't be recommended when users ask chatbots for purchasing advice.
Traditional Crawlers vs AI Agents
| Feature | Traditional Search Bots | AI Shopping Agents |
|---|---|---|
| Primary Goal | Index pages for search results | Extract specific facts and answers |
| Traffic Impact | Drives clicks to your site | Often provides zero-click answers, but highly qualified leads |
| Robots.txt Group | Googlebot, Bingbot | ChatGPT-User, ClaudeBot, PerplexityBot |
Implementation Checklist
- Audit your current robots.txt file.
- Identify which AI agents you want to allow (e.g., search indexers) vs block (e.g., pure scrapers).
- Add explicit Allow or Disallow directives for major AI user-agents.
- Regularly monitor server logs to discover new AI bots.
Field Guide
| Field | Meaning |
|---|---|
tiers.*.allowed/blocked/not_mentioned |
Per bot tier: how many bots are explicitly allowed, blocked, or not mentioned. |
overall |
Totals across all tracked bots. |
blocked_operators |
Provider names where one or more bots are blocked. |
status_code |
HTTP response status for robots.txt. |
reason |
Why the result is fail/inconclusive when robots.txt could not be evaluated. |