Loading…
Free AI crawler tool
Compare declared robots policy, optional simulated User-Agent responses and the provider verification still required in your own logs.
Direct answer
| Purpose | Examples | Score treatment |
|---|---|---|
| Search and answers | Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot | Accidental root blocks reduce readiness. |
| Model training | GPTBot, ClaudeBot, Google-Extended | Reported only; permission never adds points. |
| User-triggered retrieval | ChatGPT-User, Perplexity-User, Claude-User | Informational because robots behavior varies by provider. |
Published methodology
The score uses observable public-web fundamentals. It does not estimate citations, traffic or rankings. Live User-Agent probes, training permission and optional llms.txt discovery contribute zero points because none proves verified provider access.
robots.txt states publisher intent. The optional matrix observes how the site responds to a simulated User-Agent from Assetzaar. Only provider IP ranges, reverse DNS or signatures combined with CDN and origin logs can prove that an official bot was allowed.
The free checker samples at most three public HTML pages and conventional discovery files. Live mode adds one neutral request and at most ten 96 KB probes with concurrency capped at three. Every request rejects localhost, private IP ranges, unsafe ports, credentials, DNS rebinding, redirect loops, timeouts and oversized payloads. No indexable result URL is created.
Evidence-led diagnosis
A matching Disallow wins for the exact requested path.
robots.txt returned no response, 429 or a server error.
The neutral response succeeds while a simulated bot string is challenged or blocked.
The bounded bot response returns HTTP 429.
The response is reachable but materially thinner or changes content type.
The server response lacks enough visible content, title or H1 without executing JavaScript.
Bot names, IP ranges and robots behavior change. The live report links each detected token to its provider; these pages remain the authority for production decisions.
July 16, 2026
Added exact-path robots evaluation, unknown states for unstable robots responses, an opt-in neutral-versus-User-Agent matrix, Cloudflare/Vercel evidence, raw HTML diagnostics, official verification links, per-bot recipes and spreadsheet-safe CSV. User-Agent differences are now explicitly separated from verified provider identity.
July 15, 2026
Added a synthetic sample, Markdown export, official per-bot sources, wildcard-aware robots parsing, lifecycle analytics and an explicit 100-point methodology. Removed llms.txt and training permission from scoring.
FAQ
No. OpenAI documents GPTBot for potential model training and OAI-SearchBot for ChatGPT search. Publishers can block GPTBot while allowing OAI-SearchBot. The checker never penalizes a training opt-out.
No. Google states that Google-Extended does not affect inclusion or ranking in Google Search. The checker reports that policy as an unscored publisher preference.
Provider behavior differs because these fetches are initiated by users. OpenAI says robots.txt may not apply to ChatGPT-User, while Perplexity says Perplexity-User generally ignores robots.txt. The tool shows detected rules for context but does not score them.
No. Live mode compares a neutral response with requests carrying documented bot strings or tokens, all from Assetzaar IP space. A difference is a debugging lead, not proof that an official bot is blocked. Reliable verification requires current provider IP ranges, reverse DNS or signatures plus CDN and origin logs.
No. It inspects bounded raw HTTP responses and reports visible word, title, heading, canonical and schema evidence without executing client JavaScript. A Chromium rendering comparison requires a separately isolated, owner-authorized browser crawl.
They add up to ten bounded requests and may appear in firewall logs. The default policy audit sends only the normal bounded crawl. Enable live probes only for a public site you manage or are authorized to test.
No. llms.txt is an optional convention with no confirmed ranking or citation benefit. It is shown as informational and contributes zero points to the readiness score.
No result URL is created. The report remains in the current session unless you explicitly save it to local browser storage or download a private Markdown, JSON or CSV file.