Can AI crawlers actually read your site?

A free ~10-second check of your robots.txt rules, live bot access, llms.txt, structured data, and render visibility. No signup — just enter a URL.

No signup · Free instant results · ~10-second check

What this tool checks

  • Crawler access — robots.txt rules for 8 AI crawlers, plus a live fetch of your page as GPTBot vs a normal browser
  • Structure signals — llms.txt, sitemap.xml, JSON-LD structured data, meta basics, and how much text is readable without JavaScript
  • Scored findings — three scores — Crawlability, Access, Structure — and prioritized findings with specific fixes

How this check works

When you enter a URL, our server makes a small number of direct requests to your site — robots.txt, your page fetched as GPTBot and as a normal browser, llms.txt, and sitemap.xml. Every line of your report is something we actually found, or verifiably didn't find, in those responses — or something we explicitly tell you we couldn't verify. Deterministic by design: the same site always produces the same report.

Access — can AI crawlers get in?

  • robots.txt rules for 8 AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended, CCBot
  • A live fetch of the page you enter as GPTBot, compared against the same fetch as a normal browser — catches edge/WAF blocks that robots.txt can't show
  • Whether robots.txt declares a sitemap

Structure — can they understand what they find?

  • llms.txt — present or not
  • sitemap.xml — present or not
  • JSON-LD structured data — presence and detected types
  • Meta basics — title, meta description, canonical URL, Open Graph tags
  • Raw-HTML text visibility — how much readable text exists before JavaScript runs, and whether your page looks like an empty JS shell

Scoring is deterministic — the same site always produces the same scores. The exact weights are shown in the 'How this check works' panel of every report.

What this check can't see

This is a crawlability diagnostic — it checks whether AI crawlers can read your site. It does not measure AI visibility.

  • AI mentions & recommendations — whether ChatGPT, Claude, Perplexity, or Gemini actually mention or recommend your brand. This check never queries an AI engine.
  • Competitor share of voice — how often AI assistants surface competitors instead of you — that lives in the engines, not on your site.
  • The third-party corpus — best-of lists, Reddit threads, and review sites that AI answers draw from — none of it lives on your domain.
  • JavaScript-rendered content — we read your raw HTML the way AI crawlers do. Content that only appears after JavaScript runs is invisible to this check — and to most AI crawlers.
  • Other pages on your site — we check the page you enter — not your whole site, and not pages behind a login.

Measuring real AI visibility takes a live multi-engine audit — that's a paid engagement. Email hello@oneroomdigital.com if you want one.

Honest by design

  • A few requests, that's all — scanning sends a small number of direct requests to your site from our server: robots.txt, the page you enter (as an AI crawler and as a normal browser), llms.txt, and sitemap.xml.
  • What we store — if you unlock your report, we store your email, the URL you scanned, and your three scores — nothing else. The full report stays on this page.
  • Scores are our rubric — the score bands are our triage labels, not a certification or industry standard.
  • Not a visibility measurement — this check tells you whether AI crawlers can read your site — not whether AI assistants mention or recommend you. Those are different questions.

AI Crawlability Check is a free tool by One Room Digital.