AI Readiness — the new SEO surface area in 2026. The five signals that decide whether ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews cite you, summarise you accurately, or quietly ignore you.

Three things make AI readiness distinct from classical SEO. None subtle once you see them — most teams still optimise for the old surface.

• Crawler identity AI bots use distinct user-agents from search bots — Disallow: Googlebot does not affect GPTBot, and vice versa. Your robots.txt from 2019 almost certainly says nothing about the dozen new crawlers fetching your pages every day.
• Rendering tolerance Most AI training crawlers do not execute JavaScript. A perfectly optimised Next.js SPA can be invisible to them. If your homepage hydrates to a blank <div>, an LLM sees an empty page.
• Editorial signals llms.txt is a markdown table of contents you write for LLMs. No equivalent in classical SEO — there is no sitemap.xml convention that tells a search engine which pages matter more, only which pages exist.

Before you can decide what to allow and deny, you need to know who is knocking. The mid-2026 lineup:

• GPTBot (OpenAI) Fetches pages for training ChatGPT's base models. Respects robots.txt.
• OAI-SearchBot (OpenAI) Separate crawler for ChatGPT's real-time browsing. Distinct from GPTBot — you can allow one and block the other.
• ChatGPT-User (OpenAI) Fired on-demand when a user asks ChatGPT to fetch a URL. Per-request, not bulk.
• Claude-Web / ClaudeBot (Anthropic) Training and retrieval crawlers. Separate user-agents for separate purposes.
• PerplexityBot Perplexity's fetcher. Cites sources prominently, so showing up here has direct visibility upside.
• Google-Extended Google's opt-out signal for Gemini training. Does not affect Googlebot indexing.
• CCBot (Common Crawl) Open dataset that many smaller models train on. Block here and you block downstream training too.
• Long tail Bytespider, Diffbot, FacebookBot, Amazonbot — assorted training and retrieval bots from the next tier of operators.
• Implication "Allow all bots" and "allow Google, block everyone else" are both probably wrong choices. You need an opinion per crawler.

From most to least leverage, these dimensions decide whether an LLM cites you, summarises you accurately, or quietly ignores you:

• 1. JSON-LD structured data The single highest-ROI addition. Without it, LLMs guess your entity from prose — and guess wrong, sometimes.
• 2. llms.txt Editorial control over which pages enter the LLM's "important" list. Cheap to ship; almost no one has it.
• 3. Server-side rendering Training crawlers do not execute JS. Hydration is a death sentence for citation.
• 4. Bot-aware robots.txt Decide your stance per-bot, not blanket. Most sites accidentally allow training they would not opt into.
• 5. Canonical + sitemap hygiene Table stakes. AI bots get confused by duplicate-content fragmentation, same as Googlebot did in 2010.

The single highest-leverage change. LLMs are trained on web text but have a strong bias toward explicit, machine-readable claims when they exist. A JSON-LD block on your homepage that says "this organisation is called X, founded in year Y, based in country Z" is trusted over the same claim in prose — the model has seen the pattern millions of times in training data tied to accurate citations.

• Organization or LocalBusiness Block on the homepage. Names the entity. Highest single piece of weight.
• Product or SoftwareApplication Block on each product page. Pins the offering.
• Article Block on every blog post, with populated author and datePublished.
• FAQPage Block wherever you answer questions — these get pulled into AI Overviews almost verbatim.
• BreadcrumbList On category and detail pages. Helps the model situate the page in your site hierarchy.

llms.txt is the AI-era equivalent of sitemap.xml — except editorial. A sitemap says "here are all my URLs". An llms.txt says "here are the URLs that matter, in this order, with this framing". Lives at the domain root. Plain markdown.

• Cost Nothing to ship. Vast majority of the web does not have one yet — both a competitive edge and a fast win.
• Deeper read Companion post walks through the format with real-world examples from GitHub, Anthropic, Cloudflare, Stripe and Vercel — see /blog/llms-txt-explained.

Almost no AI training crawler executes JavaScript. Bulk training crawlers walk through pages reading raw HTML. If your homepage is a hydrated SPA, the version those bots see has no content, no headings, no paragraphs to cite — just <div id="root"></div> and a script tag.

• 5-second test Run curl -A "GPTBot/1.0" https://yoursite.com. If what comes back is mostly empty, your AI readiness score is capped low regardless of what else you do.
• Fixes Server-side rendering (Next.js App Router, Remix, Nuxt, SvelteKit), static site generation, or — if you can't change the framework — prerendered HTML snapshots served conditionally on user-agent.

Most robots.txt files in 2026 are still Googlebot-era — a single User-agent: * stanza that either allows everything or denies everything. Neither is right. Take a position per-crawler.

# Search bots — allow.
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI on-demand retrieval — allow (they cite you).
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

# AI training — your call. Opt out if you want.
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Everyone else.
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

• Retrieval bots = upside They fetch your page and cite it when the user asks something relevant. Pure upside — block them and you give up free visibility.
• Training bots = your call They fetch and may or may not regurgitate it later, with no attribution. That trade is debatable and you should make it deliberately.

Table stakes — but plenty of sites still get it wrong, and AI bots are unforgiving:

• One canonical per page Every page should declare a single <link rel="canonical"> pointing to itself or a canonical equivalent. Conflicting canonicals confuse LLMs and dilute citation weight.
• Sitemap matches canonicals sitemap.xml URLs match canonical URLs exactly — no trailing-slash mismatches, no ?utm noise, no mixed protocols.
• Consolidate duplicates If two URLs serve the same content, pick one as canonical and 301 the rest. LLMs that see fragmented duplicates often cite the wrong copy or none of them.

Drop this into your <head>. Five minutes of work, often 15–18 points of score lift in the AI Readiness Check:

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type":    "Organization",
  "name":     "Your Company",
  "url":      "https://example.com",
  "logo":     "https://example.com/logo.png",
  "sameAs": [
    "https://twitter.com/yourcompany",
    "https://linkedin.com/company/yourcompany"
  ]
}
</script>

• Why this works Tells every LLM that ever ingests your homepage: this is the entity, this is its name, these are the verified identities on other platforms. From there the model resolves any question about your business to a structured, attributable answer.

The audit breaks into five mechanical checks:

• 1. JSON-LD presence Does the homepage have at least an Organization block? Do inner pages have type-appropriate structured data?
• 2. llms.txt presence Does /llms.txt resolve with markdown content?
• 3. SSR check Does a no-JS HTTP fetch return meaningful content?
• 4. Robots policy Does robots.txt mention AI-specific user-agents, or is it silent on them?
• 5. Canonical hygiene Do canonicals match the sitemap, and is there one canonical per page?

Pick one signal — JSON-LD is usually the right starting point — and ship it on Monday. Re-scan on Friday. Repeat for a month. You'll out-score 95% of the web by quarter-end.

• If you only have one hour Add the Organization JSON-LD block to your homepage.
• If you have one day Add JSON-LD plus a basic llms.txt.
• If you have one week Do both plus the robots.txt rewrite plus an SSR audit of your top ten pages.
• On the trade-off Some teams worry that allowing AI crawlers means giving content away for free. That trade existed with Google in 2001 too. The teams that opted out then are not the teams running the web today.

• What is AI readiness? A set of signals on your domain that determine whether AI systems (ChatGPT, Claude, Perplexity, Gemini, Google AI Overviews) cite you accurately. Overlaps with SEO but adds structured data, llms.txt, server-side rendering and a per-crawler robots.txt policy.
• Is AI readiness the same as Generative Engine Optimization (GEO)? Roughly yes. GEO is the umbrella term the industry settled on. AI readiness is the score; GEO is the discipline of improving it. Used interchangeably.
• Should I block GPTBot? Depends on your business. If your content is your moat (publishing, research, paid courses), blocking training crawlers is defensible. If your content is your marketing (most B2B, e-commerce, SaaS), blocking them costs visibility when users ask AI questions you should be the answer to.
• How do I check if my site is AI-ready? Run the AI Readiness scan — checks all five signals in one pass and returns a score plus the specific gaps to fix. Most teams score 30–50/100 on the first scan and reach 80+ within two weeks of focused work.
• What's the difference between an AI training bot and an AI retrieval bot? Training bots (GPTBot, ClaudeBot, CCBot) fetch your pages to add them to a training corpus. Retrieval bots (OAI-SearchBot, ChatGPT-User, PerplexityBot) fetch on demand when a user asks a question, and typically cite the source. The two should usually have different rules in robots.txt.