AI readiness

Why this is a separate category

Classic SEO optimises for a crawler that renders JavaScript, indexes whole pages and ranks them in a list. AI discovery differs on all three counts, and the differences are not cosmetic:

• Most AI crawlers do not execute JavaScript. Content that renders client-side is invisible to them even when Googlebot sees it perfectly.
• Assistants cite passages, not pages. Clear headings and self-contained answers determine whether you get quoted.
• Access is opt-out per user-agent. A blanket robots.txt block aimed at scrapers routinely takes out the retrieval bots that would have sent you traffic.

Retrieval crawlers versus training crawlers

This is the distinction that matters most, and the one most sites get wrong.

Retrieval crawlers send referral traffic with a citation. Training crawlers do not. Whether you want to be in training data is a legitimate business decision either way — but it should be a decision, not a side effect.

Opting out of training, staying in search

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Retrieval crawlers stay allowed — these send traffic
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Server-render your content

The highest-impact item in this category, by a wide margin. To check whether you have a problem, view the page source — not the inspector, which shows the DOM after JavaScript has run — and search for a sentence from your content. If it is not there, AI crawlers cannot see it.

Our engine deliberately does not execute JavaScript, so its word count is a direct measure of what a non-rendering crawler gets.

Make passages quotable

Assistants match a user's question against your headings, then lift the passage beneath. Structure follows from that:

• Phrase headings as questions. “How much does X cost?” beats “Pricing”, because it matches what someone actually typed.
• Answer in the first two sentences under each heading. An answer buried in paragraph four rarely gets extracted.
• Use real lists and tables. These are the structures extraction pipelines handle most reliably. A comparison written as prose is far less likely to be quoted accurately.
• Add a summary near the top of long pages, so there is a self-contained answer to lift rather than one to synthesise.

Say who you are

Structured data is the least ambiguous way to state a fact. When a model chooses between inferring your price from prose and reading it from a Product schema, the schema wins.

Add Organization markup with a sameAs array pointing at your authoritative profiles — Wikipedia, Wikidata, LinkedIn, Crunchbase. Those links are the edges that let a model connect your site to a known entity rather than treating it as an unknown domain.

Add machine-readable authorship and dates too. Assistants weight recency and identifiable expertise when choosing between competing sources, and unattributed content loses that comparison.

Check your bot protection

WAF and bot-management rules frequently block AI crawlers as a side effect of blocking scrapers, and the site owner rarely finds out. If your robots.txt allows retrieval crawlers but you still never appear in AI answers, this is the usual reason. Allowlist the retrieval user-agents explicitly in your bot rules, not just in robots.txt.

Common questions