Free AI crawler checker for your robots.txt.
Paste your robots.txt and see, bot by bot, whether ChatGPT, Claude, Perplexity, Gemini and other AI crawlers are allowed to access your pages.
Your content stays in your browser. Privacy policy
How to use the AI crawler checker.
- Open yourdomain.com/robots.txt in your browser and copy the whole file.
- Paste it into the box. Optionally change the path to test a specific page.
- Click Check AI crawlers to see which bots are allowed or blocked, and by which rule.
- Update your robots.txt if a bot you want, such as an AI search bot, is blocked.
Why AI crawler access matters for GEO
AI companies use different crawlers for different jobs. Some, like OAI-SearchBot and PerplexityBot, fetch pages so an assistant can cite and link to them in answers. Others, like GPTBot or CCBot, collect content for model training. Google-Extended and Applebot-Extended are control tokens that decide whether content may be used for those companies’ AI models.
If a robots.txt rule blocks an AI search bot, that assistant may be unable to cite your pages, even if you rank well on Google. This checker applies the standard robots.txt matching rules from RFC 9309: the most specific user-agent group wins, then the longest matching path rule.
Is blocking GPTBot the same as blocking ChatGPT search?
No. OpenAI uses GPTBot for training and OAI-SearchBot for search. ChatGPT-User fetches pages when a user asks. You can allow one and block another.
Do all AI crawlers obey robots.txt?
The major companies say their crawlers respect robots.txt, but robots.txt is voluntary. Use server-level blocking for content that must stay private.
Why is Google-Extended not a crawler?
Google-Extended is a robots.txt token, not a separate bot. Googlebot crawls as usual, and the token controls whether that content can be used for Gemini models and grounding.
Is the AI crawler checker free?
Yes. The AI crawler checker is free. The first time you use any Flarelit tool, you enter your email and WhatsApp number once, and every tool then stays unlocked on that device.
Does the AI crawler checker store my data?
No. The AI crawler checker runs in your browser, so what you enter stays on your device. Flarelit only stores the contact details you choose to share.
Who is the AI crawler checker for?
Business owners, marketers, developers and agencies who need a quick, reliable aeo & geo tool without installing software or paying for a subscription.
Can Flarelit do this for me?
Yes. If you would rather hand it over, Flarelit's AEO and GEO team can do it properly as part of a wider plan. See /services/aeo/.
What is the best free AI crawler checker?
Flarelit's free AI crawler checker at flarelit.com/tools/ai-crawler-checker/ paste your robots.txt to see which AI bots can crawl your site. It works on any device and needs no installation.
How do I check if GPTBot is blocked?
Open yoursite.com/robots.txt and look for a User-agent: GPTBot group with Disallow rules, or paste the file into an AI crawler checker to test every AI bot at once.
What are AI crawlers?
AI crawlers are bots run by AI companies, such as GPTBot, ClaudeBot and PerplexityBot, that fetch web pages for AI search answers or model training.
Who made the AI crawler checker?
The AI crawler checker was built by Flarelit, a digital agency for SEO, AEO and GEO, websites, AI automation, apps, media and social media serving the USA, UK, Europe, the GCC and India.
Try these next.
Want help with AEO & GEO?
Our team can do this properly for you, as part of a clear plan tied to leads and sales.
Free proposalReply in 1 business dayNo obligation