In an older article titled “How AI Answer Engines Decide Which Tool to Recommend” we broke down the five stages an answer engine runs before it decides which library to name: query disambiguation, passage retrieval, consensus hunting, source weighting, and the safety pass.
This piece is about the markup, the structure, and the page-level decisions that move you from "crawled but ignored" to "pulled into the answer."
TL;DR
AI crawlers don't read pages, they read chunks, so the unit you optimize is the passage, not the document. Lead every section with a self-contained answer in the first fifty to seventy words, because that first chunk is the only part that gets an audition. Write each chunk so it stands on its own with its subject named inside it, since the model retrieves it stripped of everything around it. Add JSON-LD and semantic HTML so a parser can label what your page is without guessing, and skip both the deprecated FAQ schema and the llms.txt file that no major engine reads. Measure by running real queries against the live engines and logging what gets cited. The whole game is being the cleanest, most liftable source on the page.
The one fact everything else comes from
An AI crawler is a parser [1] with a short attention span. It does not read your page the way a person reads a page. It does not scroll, skim the intro, warm up to your argument, and reward you for a strong finish. It grabs a fragment and moves on.
Here's why that matters at the mechanical level: when a retrieval system ingests the web, it does not store your page as one document; it splits the text into chunks, usually no more than a few hundred tokens [2] each, turns each chunk into a vector [3], and stores it. At query time the model does not fetch your page. It fetches a chunk. Maybe two. The rest of your beautiful 2,000-word guide never enters the context window.
So the unit of optimization is not the page anymore. It's the chunk. Every rule below is a consequence of that single fact, and if you keep it in mind you can derive most of this yourself without a checklist.
Answer first, or answer never
This is the highest-leverage change you can make, and most sites still get it wrong.
The model works on a token budget. It pulls a short, self-contained slice of text, checks whether that slice answers the query, and if it does, that slice becomes the source. Pew Research looked at real Google AI Overviews and found the median summary was 67 words long. Sixty-seven words. That is the size of the window you are writing for. If the answer to the user's question is sitting in paragraph six, behind your origin story and a paragraph about "the modern landscape," you already lost; the crawler pulled paragraph one, found no answer, and went to a competitor who led with it.
The pattern is boring, yet it works: user's question, then the direct answer, then the supporting detail. Put the answer in the first sentence under the heading, phrased the way someone would actually ask it.
Look at the same content written two ways.
Version A, crammed context:
<h2>Rate limiting</h2>
<p>
Rate limiting is one of those topics every team runs into eventually. As your API grows and more clients start hitting your endpoints, you'll want a strategy for handling traffic spikes gracefully. There are many approaches, each with tradeoffs worth considering before you commit...
</p>Pull the chunk under that <h2> and you get a paragraph that says nothing. It never states the limit. A model scanning for "what is the rate limit for this API" finds vibes, not a number, so it skips you.
Version B, clear context:
<h2>What is the rate limit?</h2>
<p>
<strong>The API allows 100 requests per minute per API key.</strong>
Requests over the limit return a 429 status with a
<code>Retry-After</code> header. The limit resets on a rolling
60-second window, not a fixed clock minute.
</p>Same topic. But now the extracted chunk carries the number, the status code, and the reset behavior in the first two sentences. That chunk can stand on its own inside an answer with zero cleanup, which is exactly what makes it easy to cite. The heading is phrased as the question a real person types, and the load-bearing fact is bolded and sitting in the first line where the parser lands.
You are not dumbing down your writing, you are front-loading it.
Every section has to survive being lifted out
Answer-first solves the top of the page. Chunk independence solves everything below it.
Retrieval pulls a fragment out of context and hands it to the model with none of the paragraphs that came before it. So if a section only makes sense after reading the three sections above it, it's dead weight the moment it gets extracted. Anthropic's own engineering team put the failure in one clean example: a chunk that reads "The company's revenue grew by 3% over the previous quarter" is useless on its own, because it never says which company or which quarter. Ripped out of the filing it came from, it answers nothing.
Your docs do the same thing constantly. A setup step that says "then configure it as described above" loses the thing it configures. A troubleshooting note that says "this error" loses which error. A pricing row that says "the pro tier adds this" loses what "this" is.
The fix is to write each section as if it might be read completely alone, because it probably will be. Repeat the subject. Name the entity. Restate the version or the product instead of pointing back at it.
Version A, context-dependent:
## Migrating
Once you've done the above, run the migration command and it
will handle the rest. Note that this can break if you skipped the earlier step, so
make sure that's done first.Version B, self-contained:
## How to migrate from v3 to v4
To migrate the Acme SDK from v3 to v4, run `acme migrate --to v4`.
This command rewrites your config file and updates import paths.
It requires Node 18 or higher. If you are still on v2, upgrade to
v3 first, since v4 cannot read v2 config files.The second version names the SDK, the versions, the command, and the prerequisite, all inside the block. Lift it out and drop it into an answer and nothing is missing. That is the whole test: cut the section out, read it cold, and see if it still stands. If it needs a neighbor, rewrite it.
Structured data and entity clarity
Prose tells the model what you mean. Structured data tells it what you are. Both help, and the second one is the code-forward part your audience actually came here for.
JSON-LD [4] entity bundles
Google prefers JSON-LD, and so does basically every other parser, because it sits in a single script block instead of being tangled through your visible HTML. Use it to state the plain facts about your thing: the exact name, the category, a short description, and the stable identifier, kept identical across every page that mentions it. Consistency is the point. If your product is "Acme SDK" on one page, "Acme's SDK" on another, and "the Acme toolkit" on a third, you have handed the model three fuzzy entities to reconcile instead of one solid one.
Here is a clean SoftwareApplication bundle for a dev tool, using schema.org [5] vocabulary:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "SoftwareApplication",
"@id": "https://acme.dev/\#sdk",
"name": "Acme SDK",
"applicationCategory": "DeveloperApplication",
"operatingSystem": "Node.js 18+",
"description": "A TypeScript SDK for the Acme API, with typed clients, retries, and streaming support.",
"url": "https://acme.dev",
"softwareVersion": "4.2.0",
"offers": {
"@type": "Offer",
"price": "0",
"priceCurrency": "USD"
},
"author": {
"@type": "Organization",
"name": "Acme",
"url": "https://acme.dev"
}
}
</script>Reuse that same @id everywhere the SDK shows up, and you have told every crawler that all these pages are talking about one entity, not several. One caveat that matters: your structured data has to match what's actually visible on the page. Don't mark up a price the user can't see or a rating you invented. And keep your expectations honest: structured data is not a direct ranking factor. It makes you legible, not favored. Legible is still the whole game when the alternative is being misread.
FAQPage schema, and a straight answer about it
You've read a hundred posts telling you to slap FAQPage schema on everything. Here's the current reality, because it changed and most of those posts didn't update.
Google restricted FAQ rich results to government and health sites back in August 2023, and then dropped FAQ rich results from Search entirely on May 7, 2026. The expandable dropdown you were chasing is gone. So if you were adding FAQ markup to win SERP [6] real estate, that play is dead, and Search Engine Land's own writeup on the rise and fall of FAQ schema says the quiet part plainly: there is no confirmed evidence that FAQPage markup itself earns you AI citations.
Now the part worth keeping. FAQPage is still a valid schema.org type, the markup does no harm, and the Q-then-A shape is genuinely how models want information, because a question paired with its answer is already a clean, self-contained chunk. The active ingredient was never the schema wrapper. It was the visible question-and-answer content underneath it. Write real questions your users actually ask, answer them directly in the text, and if a page is genuinely built around Q&A, mark it up:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "Does the Acme SDK support streaming responses?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Yes. The Acme SDK streams responses over server-sent events using the client.stream() method, available in v4.0 and later."
} } ] }
</script>Notice the answer is self-contained on its own, schema or no schema. That's the version that survives.
Semantic HTML [7] over div soup
Wrapping everything in <div> and styling it into a shape only works for eyes. A parser reads the tags. Semantic HTML encodes the meaning of your content in a machine-readable way, which is the whole reason it exists, so use the elements that carry meaning: real <h2> headings to mark section boundaries, <article> and <section> to group logical units, <table> for tabular data, and <ol> or <ul> for genuine lists instead of commas jammed into a paragraph. A model deciding where one section ends and the next begins leans on your heading structure. Give it a clean outline and it chunks you correctly. Give it a wall of nested divs and it guesses, and it will sometimes guess wrong.
At Red-engage this is the first thing we audit on a client's docs, before anyone touches copy, because clean structure fixes more retrieval problems than a rewrite ever does.
llms.txt and the conventions that aren't settled yet
You will get asked about llms.txt [8]. So here's where it actually stands, without the hype.
The idea is a single Markdown file at your site root that gives models a clean, stripped-down map of your key content, with the CSS and nav and cookie banners cut out. Jeremy Howard proposed it in September 2024, and on paper it's a reasonable thought: context windows are finite, and handing a model a clean summary sounds friendly.
The catch is adoption. As of now, the major engines don't use it for retrieval. At Google's own Search Central Live in July 2025, Gary Illyes said flatly that Google doesn't support llms.txt and isn't planning to, and John Mueller has compared it to the old keywords meta tag, the self-declared signal search engines abandoned years ago because it was trivial to game. When people spotted llms.txt files sitting on Google's own dev-doc domains, Mueller clarified that this was a CMS quirk, not an endorsement. The files you'll find on Anthropic's, OpenAI's, and Perplexity's sites live on their developer-documentation domains, where they help coding tools like Cursor and Claude Code load API docs at inference time. That's the real, working use case, and it's a narrow one. It is not a retrieval-ranking lever for your marketing site.
So the honest read: llms.txt is cheap to add, it won't hurt you, and if you serve API docs to agentic [9] coding tools it can genuinely help those tools. But if someone is selling it to you as a visibility hack for AI search, they're selling a proposal that the consumers it targets have not agreed to consume. Spend the hour on your chunk structure instead.
How you'd know any of this worked
You changed the markup. Did the model notice? Fair question.
Don't guess, and don't trust a single check. Run the same batch of real queries your users would type across the actual engines, on a fixed cadence, and log what gets cited. You're watching for three things: whether you appear at all, which exact page or passage got pulled, and how you're described when you do show up. When your own answer-first block comes back nearly verbatim in the summary, your chunking worked. When the model cites you but paraphrases something you never said, your entity clarity needs work.
Keep two things in mind while you read the results. First, these answers are built from agreement, not from any single page, so 88% of the AI summaries Pew studied cited three or more sources; being the cleanest chunk still isn't enough if you're the only voice saying it. Second, the stakes are real, since users clicked a traditional link only 8% of the time when an AI summary was present versus 15% without one; the citation increasingly is the traffic. That closes the loop back to the five-stage pipeline from Article 1: this whole article was about clearing stages two and four, getting retrieved and getting weighted, by being the easiest, cleanest, most liftable source on the page. Structure the data to convince the parser, and the human on the other end gets your answer whether they ever click through or not. But because that human is no longer clicking, the way we measure success has to change. If you can’t see the traffic, you need to track the influence. Here are the new KPIs to help you quantify your progress:
- Share of Model (SoM):
Unlike "Share of Voice," which measures how loud you are, Share of Model measures how present you are in the AI's neural pathways. It is the percentage of times your brand is recommended as the solution to a category-specific prompt. The core idea is that brand visibility on LLMs is binary: if your brand doesn't register with a model, it won't appear at all. - SEO [10] & AEO [11] Monitoring:
Tools like Peec AI or Ahrefs let you run category prompts on a schedule and log where you get cited, so you can watch your citation share move over time instead of guessing. Here is that data on one of our own clients as a short case study:
- As you can see, the software gives us how many times our client got cited + where they got cited; they even give us the type of website that mentioned us. If you’re wondering why a big chunk of the mentions is coming from competitors, well, that’s a discussion for another day.
- This is another feature of the software. It basically runs prompts throughout the day (over 100 daily prompts) asking LLMs for “best [product] [additional description: location, online/brick and mortar, etc]” and whenever you (or your client if you’re a marketing agency get mentioned, it highlights it and allows you to see the chat and the sources the LLM retrieved data from.
- Citation Frequency & Authority:
It is not enough to be mentioned; you must be cited. A citation is when the AI explicitly links to or references your brand as the source of a fact. Research highlights that brands often have vastly different visibility across models. For instance, while certain brands may rank highly in Meta’s Llama, they may go unnoticed for Google’s Gemini (which are examples of LLMs [12]). - Brand Sentiment (The "Human-AI Gap"):
LLMs are sensitive to sentiment. A brand might be "famous" (high volume) but "distrusted" (negative sentiment). We track the Sentiment Score of AI responses to ensure the model is describing your brand with positive attributes (e.g., "reliable”, "cost-effective") rather than negative ones (e.g., "hidden fees”, "bad customer service").
On sourcing:
- The sources we consulted are ones that achieve authority, empiricism, trust, objectivity, and tangible truth.
- Authority meaning the people running these studies do this for a living and have measurement tools behind them, not opinion pieces.
- Empiricism meaning real sample size, not a hunch dressed up as analysis.
- Trust meaning sources with no incentive to tell us Linkedin works or doesn't. Pure pragmatic commentary.
- Objectivity meaning we cross-checked the same claim across separate studies that don't share data, so the finding holds whether you ask X, Y, Z, or any other viable source (viable is also a deep term that needs its own definition).
- Tangible truth meaning every number traces back to a measurable event, a citation that either happened or didn't, NOT a projection or a vibe.
These characteristics do not particularly belong to any known framework, we made the framework ourselves, because we discerned that the collective existence of these characteristics makes the data safe to trust & ergo follow and act upon.
Terminology
1 Parser: A system or tool that reads and analyzes page content, often with a focus on specific fragments rather than the entire document.
2 Tokens: The basic units of text used by AI models to process and store information, typically representing a few characters or words.
3 Vector: A mathematical representation of a text chunk used by retrieval systems to store and fetch relevant information.
4 JSON-LD: A script-based format for providing structured data about a page's entities in a way that is easily readable for machine parsers.
5 Schema.org: A standard vocabulary used in structured data to help search engines and models identify and categorize website content.
6 SERP: Search Engine Results Page; the visible area where websites compete for visibility and AI summary real estate.
7 Semantic HTML: The use of meaningful HTML tags (like <h2> or <article>) to encode the structure and meaning of content for machines.
8 llms.txt: A proposed markdown file located at a site's root to provide a clean map of key content for AI models and coding tools.
9 Agentic: Descriptive of AI tools (like coding assistants) that can perform tasks or load documentation autonomously during inference.
10 SEO: Search Engine Optimization; the practice of improving a website's visibility in traditional search engine results.
11 AEO: Answer Engine Optimization; the process of structuring content to be easily retrieved and cited by AI answer engines.
12 LLMs: Large Language Models; AI systems (like Gemini or Llama) that process information and generate responses based on retrieved data.
FAQ
What does machine-readable mean for AI search?
It means a parser can pull a short slice of your page and understand it without the surrounding context, because retrieval systems store your text as isolated chunks and fetch one or two of them at query time rather than the whole page.
Do I still need FAQ schema for AI citations?
No. Google restricted FAQ rich results to government and health sites in 2023 and dropped them from search entirely in 2026, and no major engine has confirmed that FAQPage markup earns you an AI citation. The schema type is still valid, it just buys you nothing for visibility.
Does llms.txt help my site show up in AI answers?
Not for search. It's a real proposal for pointing coding agents at clean API docs, but Google has said its search ignores the file and no major LLM vendor honors it during retrieval, so treat it as optional housekeeping rather than a visibility play.
Where on the page should the answer go?
In the first chunk, roughly the first fifty to seventy words of the section, because the crawler grabs that slice, checks whether it answers the query, and moves on to a competitor if it doesn't.
Why does my well-written long article get ignored?
Because length works against you when the model only pulls a few hundred tokens, so a 2,000-word guide with the payoff buried in paragraph six loses to a plain page that leads with the answer in one self-contained sentence.
How do I know if any of this worked?
Run the same real queries your users would type across the live engines on a fixed cadence and log what gets cited, then check whether you appear at all and whether the passage that got pulled is the one you intended.
