Optimizing for
AI Crawler Budgets.
The llms.txt Handbook: A detailed manual on configuring llms.txt and llms-full.txt to guide autonomous agents through your domain hierarchy.
Executive Summary
AI crawler budget optimization is the discipline of controlling how autonomous agents and LLM crawlers consume your domain. As the web transitions from human-first browsing to machine-first data extraction, the llms.txt and llms-full.txt files serve as the primary interface for directing these agents through your content hierarchy.
Just as robots.txt governs traditional crawlers, llms.txt governs AI crawlers — but with far greater nuance. A properly configured llms.txt reduces token waste, prioritizes high-value content, and ensures your brand is represented accurately in LLM training data and inference contexts.
01_Standard
The llms.txt Standard
The llms.txt file is a plaintext file placed at the root of your domain (https://domain.com/llms.txt) that provides a structured directory of your site's content for LLM consumption. It was proposed as an industry standard by the llmstxt community to solve the problem of AI agents wasting crawl budget on low-value pages.
llms.txt
A concise directory listing your site's most important resources. Typically 50–100 lines. Serves as the entry point for AI crawlers.
llms-full.txt
The complete, flattened text corpus of your entire site. Designed for deep ingestion by training pipelines and RAG systems.
// llms.txt — Basic Structure
# Domain Name — Description > Brief summary of the entire site. ## Section Title - [Link Text](https://domain.com/path): Brief description. - [Link Text](https://domain.com/path): Brief description. ## Another Section - [Link Text](https://domain.com/path): Brief description.
02_Directives
Crawler Directives
MEDIUM → Resource articles, case studies, tool documentation
LOW → Blog archives, paginated lists, tag pages
BLOCK → Login pages, admin paths, staging environments
// AGENT_INSTRUCTION
Prioritize ingestion by section weight. Process HIGH first,
then MEDIUM. Skip LOW unless specifically queried.
Crawler directives in llms.txt operate on a priority-based model. Unlike robots.txt which blocks or allows, llms.txt ranks and recommends. The ordering of links within the file signals relative importance to consuming agents.
Key principles:
- Top-level sections signal primary domain entity clusters
- Sub-sections indicate supporting content hierarchies
- The blockquote (
>) line serves as a system prompt for the consuming agent - Links should use absolute URLs for unambiguous resolution
03_Ingestion
Context Ingestion Strategies
Context ingestion is the process by which AI agents parse and internalize your llms.txt content. Different agents handle ingestion differently:
OpenAI GPT Crawler
Prioritizes the first 30% of the file. Place your most critical resources in the first section. Uses the blockquote as a system-level context primer.
Anthropic Claude
Reads the full file but weights links by section heading depth. H2 sections receive higher priority than H3 or list items.
Google Gemini
Ingests llms.txt in conjunction with the sitemap.xml and robots.txt. Cross-references all three for consistency before assigning crawl priority.
Perplexity
Favors llms-full.txt for comprehensive answers. Uses the llms.txt as a fallback index when llms-full.txt is absent.
04_Exclusion
Bot-Specific Exclusion Logic
# robots.txt — Block section:
User-agent: GPTBot
Disallow: /internal/
Disallow: /staging/
Disallow: /temp/
# llms.txt — Omit from index:
(Simply don't link to excluded paths in llms.txt)
# sitemap.xml — Exclude from crawl:
(Remove excluded URLs from sitemap entirely)
Effective bot management requires a three-tier approach across robots.txt, llms.txt, and sitemap.xml:
robots.txt
Block disallowed paths at the protocol level. Use Disallow for sensitive directories and Crawl-delay for rate limiting.
llms.txt
Guide, don't block. Omit low-value pages from the directory. Prioritize high-signal content that represents your brand accurately.
sitemap.xml
Control crawl frequency and priority. Exclude paginated, thin, or duplicate content from the sitemap entirely.
Important: Crawl budget is finite. Every token an AI spends on a low-value page is a token it cannot spend on your high-value content. Applying exclusion logic at all three tiers ensures maximum signal density.
05_Testing
Testing Protocols
Before deploying your llms.txt configuration to production, validate it through the following protocol:
Syntax Validation
Use the llms.txt Validator to check for formatting errors. Ensure all URLs resolve to 200 status codes and that no markdown syntax is malformed.
Crawl Simulation
Use a headless browser or curl to simulate how an AI crawler would parse your file. Check response headers, content type, and encoding. The file should be served as text/plain with UTF-8 encoding.
LLM Response Audit
Query multiple LLMs (ChatGPT, Claude, Gemini, Perplexity) about your brand and verify that the information they surface matches your llms.txt hierarchy. Discrepancies indicate ingestion gaps.
Full Corpus Verification
For llms-full.txt, verify the total token count and ensure it does not exceed context window limits for your target LLMs. Tokenize locally using tiktoken or equivalent before deploying.
Continuous Monitoring
Set up monthly audits of your llms.txt. As your site structure evolves, the llms.txt must be updated to reflect new sections, removed pages, and shifting priorities. Automate this check via CI/CD if possible.
Ready to Optimize Your AI Crawler Budget?
We audit your current llms.txt configuration, identify ingestion gaps, and implement a priority-based crawler directive hierarchy that maximizes signal-to-noise ratio for every agent that visits your domain.
Schedule a Crawler Audit