Can AI engines read your site?
Check your robots.txt against the crawlers ChatGPT, Perplexity, Claude and Google use to find and cite websites, plus your llms.txt and homepage schema.
Free · No sign-up · Results in a few seconds
- Fetching robots.txt
- Matching rules for each AI crawler
- Checking for llms.txt
- Reading your homepage without JavaScript
Verdict
AI search & answer crawlers these decide citations
AI training crawlers optional to allow
AI readiness
Crawlable isn't the same as recommended.
See whether ChatGPT, Perplexity, Gemini and Claude actually name your business, and who they recommend instead. Rankless tracks it and fixes what's in the way.
Questions about AI crawlers
Why does crawler access matter?
AI answer engines only cite pages their crawlers are allowed to fetch. Blocking OAI-SearchBot or ChatGPT-User keeps you out of ChatGPT search; blocking PerplexityBot keeps you out of Perplexity; blocking Claude-SearchBot keeps you out of Claude's web answers; and blocking Googlebot removes you from Google, AI Overviews and Gemini grounding.
Should I block AI training crawlers?
Training crawlers (GPTBot, ClaudeBot, Google-Extended) are different: blocking them stops your content being used to train models, but does not stop AI search from citing you. That's a legitimate business choice.
How does this check work?
- We fetch
/robots.txtand apply RFC 9309 rules: the most specific matchingUser-agentgroup wins, then the longest matchingAllow/Disallowpath for/. - We fetch
/llms.txtand your homepage's raw HTML (no JavaScript, like most AI crawlers) to look for Organization schema, sameAs links, a meta description, an H1 and readable text.