How AI Crawlers Read Your Site (And What llms.txt Actually Does)

The previous three posts in this series covered the baseline visibility measurement, the distinction between mentions and citations, and the FAQ schema structure that generates reason-led citations. This post covers the earlier layer: how AI retrieval systems find your site in the first place and what you can do to make sure they surface the right pages.

The short version: AI retrieval is not Google crawling. Understanding how it works changes which technical fixes actually matter.

How AI retrieval actually works

Google's Googlebot crawls your entire site over time, follows every link it can reach, and builds a ranked index across billions of pages. That index is what gets queried when someone runs a Google search.

AI retrieval systems work differently. When Perplexity or the retrieval layer behind ChatGPT is forming a response to a buyer query, it pulls a small set of pages on demand. It is not working from a pre-built index of your site in the same way Google is. It is making real-time requests to sources it believes are relevant.

The retrieval priority order, based on how these systems are documented and how citation patterns behave across the audits we have run:

1. Pages your sitemap lists as high-priority
2. Pages your homepage links to directly
3. Pages with structured data (Article schema, FAQPage schema, Product schema)
4. Pages listed in your llms.txt

The practical implication: a 2,000-word blog post with good FAQ schema and a link from your homepage can outperform your main product page in AI retrieval if your product page has no structured data. Structured signals matter more per page in AI retrieval than they do in traditional SEO, because the retrieval system is selecting from a much smaller pool.

What llms.txt is and what it is not

llms.txt is a plain-text markdown file placed at the root of your domain, at yourdomain.com/llms.txt. It was proposed as a convention by Answer.AI in 2024, following the model of robots.txt: a simple, standardized file that tells a specific type of automated system how to interact with your content.

What it does: it signals to AI retrieval systems which pages on your site are most important, and gives each page a brief description the system can use when deciding which page to retrieve for a given query.

What it does not do: it does not guarantee citation. It does not control how ChatGPT or Claude, which rely heavily on training data rather than live retrieval, represent your brand. It does not replace structured data on individual pages.

For Perplexity specifically, which uses live web retrieval for most responses, llms.txt is the fastest way to signal your content hierarchy. Perplexity checks for the file. If it finds a well-structured llms.txt, it can use it to prioritize which pages to retrieve when a buyer query matches your domain.

The format that actually works

Here is the structure of an llms.txt file that generates the right retrieval behavior.

# OperatorIQ

> OperatorIQ helps B2B SaaS companies diagnose and fix their AI search visibility. Our core product is the LLMRadar Audit, a 48-hour measurement of how often and how specifically LLMs cite your brand vs. list it generically across 40 buyer query variations.

## Key pages

- [LLMRadar Audit](https://operatoriq.io/tools/): The $197 AI visibility audit that measures your brand's citation rate across ChatGPT, Claude, Gemini, and Perplexity using 40 buyer-intent query variations. Delivers a full report within 48 hours.
- [About OperatorIQ](https://operatoriq.io/about/): Who we are and why we built this.

## Key content

- [How to Set an AI Visibility Baseline](https://operatoriq.io/blog/ai-visibility-baseline-how-to-set-one/): The four-metric measurement framework for tracking LLM brand visibility over time.
- [AI Search Brand Mentions vs Citations](https://operatoriq.io/blog/ai-search-brand-mentions-vs-citations/): Why appearing in a list is different from being cited with a reason, and what drives each outcome.
- [How to Structure FAQ Schema for LLM Citations](https://operatoriq.io/blog/how-to-structure-faq-schema-llm-citation/): The question types, answer length, and JSON-LD structure that generates reason-led citations.

The format is markdown. A single H1 with your site name. A blockquote with a one-paragraph site description. One or two H2 sections: key pages with a one-sentence description for each, and key content with titles, URLs, and a sentence of context.

Keep the entire file under 200 lines. If you have more content worth indexing, put it in llms-full.txt and link to it from the main file.

The gap between having the file and having one that works

Most sites that have an llms.txt have the wrong content in it. The two failure patterns we see most often:

Failure pattern 1: The file is an unfiltered sitemap.

Listing 150 URLs in your llms.txt does not help the retrieval system prioritize. It gives the AI the same problem it would have without the file: too many options and no signal about which pages represent your highest-value content. A good llms.txt has five to fifteen entries, all of them buyer-facing, with explicit descriptions.

Failure pattern 2: The file exists but does not match the structured data on the pages it lists.

If your llms.txt says your product page is the most important page, but your product page has no FAQ schema, no Article schema, and no structured description, the retrieval system will check the page and find thin content. The llms.txt signal directs the retrieval; the structured data on the page is what produces the citation. Both have to work together.

The AEO citation retest we ran across five queries in June returned zero citations, partly because of this gap: an llms.txt file with 35 entries, most of them without matching structured data on the underlying pages. Adding the schema to the priority pages is what the current AEO work is closing.

How to check whether your llms.txt is working

After publishing or updating your llms.txt, wait two weeks, then run this test in Perplexity.

Pick one page from your llms.txt that you have not specifically optimized for Google SEO. It should be a page that would not rank organically for competitive keywords, but that answers a specific buyer query clearly. Run that buyer query in Perplexity. Check the source list.

If Perplexity surfaces the specific page from your llms.txt, the file is being read and the retrieval is working. If Perplexity surfaces a different page from your domain, or does not surface your domain at all, the file may be malformed, the content on the listed pages may be thin, or the retrieval system is not yet indexing your domain.

A second check: look at the response text. If Perplexity uses language close to your llms.txt descriptions or to your FAQ schema answers, you are getting citation-quality retrieval. If your brand appears without a reason, the retrieval is working but the content is not yet citation-ready.

What llms-full.txt is and when to use it

llms-full.txt is the companion file to llms.txt. Where llms.txt is an index with brief descriptions, llms-full.txt is the full markdown content of your key pages concatenated into a single file.

The use case: some AI retrieval systems can process a single large file more efficiently than making individual requests to ten separate pages. By putting your key content into one file, you reduce the chance that the retrieval system skips a page because it timed out, returned a rendering issue, or hit a JavaScript dependency.

For most B2B SaaS sites, start with llms.txt. Get that right first. Add llms-full.txt after you have confirmed your product page and top two authority posts are structured correctly, because the full-text file amplifies what is already there. If your content is not citation-ready, making it easier to retrieve will not change the citation rate.

The sequence: what to do in what order

Getting AI retrieval right is a four-step sequence. This matters because doing step three before step one is why most llms.txt implementations produce no measurable change.

Step 1. Identify your five highest-priority buyer-facing pages. These are the pages an AI assistant should retrieve when a buyer asks about your product category.

Step 2. Add structured data to those pages: Article schema with a clear description, FAQPage schema with buyer-language questions and 50-80 word answers that follow the situation-mechanic-anchor structure.

Step 3. Create your llms.txt file listing those same five pages with one-sentence descriptions that match the value proposition on each page.

Step 4. Submit those URLs to IndexNow (for Bing and Yandex to pick up quickly) and check your sitemap to confirm they are all listed with priority 0.8 or higher.

Run a Perplexity spot check two weeks after completing step four. If your citation rate has not improved after 90 days, the bottleneck is usually step two: the structured data on the pages is not producing citation-ready answers.

For a complete measurement across 40 query variations and all four major LLMs before and after implementation, the $197 LLMRadar Audit runs that measurement automatically and delivers the report within 48 hours.