Security & compliance
AI Crawler Access Checker
Tested and ready
See which AI crawlers each website allows or blocks in robots.txt (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended, Meta and more) and whether it publishes llms.txt / llms-full.txt. Bulk, for GEO/AI-visibility audits.
What data it extracts
- website
- Blocked Count
- Blocked Ai Bots
- Allows Ai Training
- Allows Ai Search
- Has Llms Txt
- Llms Txt Bytes
- Has Llms Full Txt
- Robots Txt Status
- Robots Txt Bytes
- bots
- sitemaps
Output format
Each run writes an Apify dataset you can download as JSON, CSV or Excel, or read through the Apify API.
Schedule
Run it on demand in the Apify Console, or put it on an hourly, daily or weekly schedule.
Price
$1.75 per 1,000 results
Pay per result.
Get access
This scraper is available. Request access and we’ll set you up today.
Use the request form on this site, or email [email protected].