Search#

How agents find content: access control, retrieval mechanics, and response quality strategies

agent-friendliness#

Definition: contested property describing whether a website or service’s content and infrastructure can be discovered, retrieved, and used by agents; no single consensus definition or unified scoring standard exists yet

Purpose: names the gap between content merely existing and an agent actually being able to retrieve and act on it; a page can rank well in GEO and still fail an agent’s fetch pipeline; agent-friendliness matters for any team whose APIs, documentation, or product needs to be usable by the growing share of traffic that comes from agents

Example: Fern’s Agent Score measures content accessibility with afdocs and the Agent-Friendly Documentation Spec - Markdown availability, llms.txt structure, truncation risk; Cloudflare’s isitagentready.com measures bot access control, protocol adoption; prioritize the following -

Tier	Focus	Guidance
Content Discoverability	Can an agent find what’s visible?	Add `llms.txt`, fix stale links, ship Markdown availability
Content Visibility	Can an agent see your content at all?	Fix SPA rendering, extreme truncation, and content buried under chrome
Protocol Layer Adoption	Content already visible and discoverable?	Markdown content negotiation has the best return; don’t implement a standard just to score points
Real World Testing	Does this hold up against a real agent?	Point Claude at docs and watch what happens

Example: Gao and Chen observed that with local repository access, agents read and write primarily from their own artifacts - AGENTS.md and CLAUDE.md, while API references are rarely the first failure-recovery resource, and absent from validation sequences, suggesting that agent-friendliness may depend more on machine-curated files than traditional documentation quality

Sources:

Agent-Friendly Documentation Spec#

Definition: specification defining 23 checks across 7 categories that evaluate how well a documentation site serves coding agents; grounded in empirical observation of real agent workflows

Purpose: provides a shared standard for documentation teams, tool builders, and platform providers to measure and improve how effectively agents can discover, retrieve, and consume documentation content; categories cover content discoverability, markdown availability, page size and truncation risk, content structure, URL stability, observability, and authentication

Example: afdocs is a companion CLI tool and Node.js library that implements the spec; produces a scorecard with category breakdowns, system-level diagnostics, and per-check results with fix suggestions, and includes a vitest helper for CI integration to catch regressions; Fern’s Agent Score ships the same 22 checks as a single number

Related Terms: agent-friendliness, Agent Reading Test, evaluation, robots.txt, spec

Sources:

Definition: percentage of AI-generated answers, in a defined prompt cluster, that link to or name a specific domain; a core metric for measuring visibility within AI-generated answers

Purpose: functions as the GEO equivalent of search ranking position; tracks whether content is actually being surfaced by AI search engines and how visibility shifts over time across platforms

Example: Reddit holding 3.83% of ChatGPT Search citations before August 14, 2026, then dropping to 0.52%, an 86.4% relative decline in citation share

Related Terms: GEO, query fan-out, retrieval pool

Sources:

data-licensing#

Definition: commercial agreements where publishers grant AI companies structured API access to their content for training, search retrieval, or product integration

Purpose: shapes the pipeline through which AI models access publisher content; when licensing terms change or are revoked, AI search citation behavior can shift dramatically, as seen when Reddit cut off ChatGPT’s access despite an existing partnership

Example: the May 2024 OpenAI-Reddit data-licensing partnership that gave OpenAI structured API access for training and product use, separate from ordinary web crawling

Source: explainx.ai: “Reddit’s ChatGPT Citations Collapsed 86% on August 14 — What Happened” by Yash Thakker

Cloudflare Monetization Gateway#

Definition: reverse proxy infrastructure intending to resolve identity, permission, and payment inside a single HTTP request at the edge, before the origin server responds; enables per-request charging for automated traffic using the x402 protocol

Purpose: addresses the monetization gap created when agent traffic bypasses traditional advertising and subscription models; allows content providers to charge crawlers and agents for access to premium content

Example: Cloudflare’s Pay Per Crawl feature lets site owners set per-request prices for automated traffic, with Cloudflare handling billing as merchant of record, moving value settlement from after the request to inside it, risks of this emerging technology include -

Risk	Description
Adoption Dependency	Revenue depends on callers recognizing the `402` response; adoption is currently minimal
Agent Definition Conflation	Conflates LLM agents with crawlers and training bots, diluting the specific meaning of “agent”
Concentration Risk	Too much infrastructure running through a single provider creates systemic risk
Pricing Difficulty	Pay Per Use aligns payment with value, but is harder to measure and verify than per-crawl pricing
Protocol Adoption Gap	Content Signals, MCP Server Cards, and Web Bot Auth have near-zero agent consumption today

Related Terms: agent-friendliness, data-licensing, robots.txt

Sources:

GEO#

Definition: abbreviation for Generative Engine Optimization; practice of optimizing content to be cited in AI-generated answers rather than ranked in traditional search results

Purpose: essential for content strategy as AI search platforms increasingly replace traditional search; represents a shift from SEO-focused strategies where the goal was ranking on search engine results pages to a new paradigm where the goal is response citation

Example: optimizing a help center article so it gets cited by ChatGPT when users ask questions about a product, rather than trying to rank on Google’s first page

Related Terms: context window, LLM, planning, query fan-out

Source: explainx.ai: “Reddit’s ChatGPT Citations Collapsed 86% on August 14 — What Happened” by Yash Thakker

`llms.txt`#

Definition: Markdown file published at a site’s root path (/llms.txt) that gives LLMs and agents a curated, structured index of a site’s content, distinct from the raw HTML a browser renders; a companion /llms-full.txt inlines the complete content in one file

Purpose: reduces the cost of parsing full HTML pages for content discovery by giving agents a direct, structured entry point; functions like robots.txt and XML sitemaps, but curates content for consumption rather than gating access to it

Example: only one section is required, an H1 heading naming the project or site; documentation sites keep /llms.txt under 50,000 characters using a progressive disclosure pattern, linking out to full pages for large doc sets rather than inlining everything

Sources:

query fan-out#

Definition: retrieval technique where an AI search platform breaks a single user query into multiple more specific sub-queries, fetches results for each, then synthesizes an answer from the combined results

Purpose: enables AI search engines to retrieve more relevant and comprehensive information by exploring different aspects of a query in parallel; understanding query fan-out is critical for GEO because changes to fan-out behavior can dramatically alter which domains and content get cited

Example: ChatGPT breaks “best project management software for small teams” into sub-queries such as “best project management software 2026,” “small team project management reviews,” and “affordable project management tools”, while each retrieves different sources

Related Terms: context window, GEO, LLM, prompt

Source: explainx.ai: “Reddit’s ChatGPT Citations Collapsed 86% on August 14 — What Happened” by Yash Thakker

retrieval pool#

Definition: set of documents and pages an AI search engine draws from when synthesizing an answer to a user query; populated by the sub-queries generated through query fan-out

Purpose: determines which sources are even candidates for citation; changes to retrieval pool composition, whether from fan-out adjustments, ranking changes, or crawler access blocks — directly alter citation share

Example: when ChatGPT’s query fan-out changed on August 8, 2026, the sub-queries became longer and more qualifier-heavy, which changed which pages entered the retrieval pool and disproportionately excluded Reddit thread-style content

Related Terms: citation share, GEO, query fan-out

Source: explainx.ai: “Reddit’s ChatGPT Citations Collapsed 86% on August 14 — What Happened” by Yash Thakker

`robots.txt`#

Definition: Robots Exclusion Protocol; web standard file placed at a site’s root that instructs web crawlers which paths they may or may not access

Purpose: the lowest-level access control layer for web content; a publisher can cut off an AI search engine’s citation pipeline unilaterally with a one-line change, making robots.txt a critical dependency for any GEO strategy that relies on live crawled content

Example: Reddit adding a blanket Disallow: / to its robots.txt in August 2026, which caused ChatGPT’s citation share from Reddit to collapse from 3.83% to 0.52% in one day

Sources:

truncation budget#

Definition: platform-specific character or byte limit an agent’s fetch pipeline reads from a page before cutting the response off, independent of the LLM’s own context window

Purpose: cost efficiency mechanism behind how identical page content reaches different agents in different amounts; budgets vary widely by tool and fetch method

Example: a documentation page where actual content doesn’t begin until 87% through the HTTP response, after markup and navigation elements consume most of the truncation budget, leaves an agent with only the page’s last few paragraphs even though the full page loaded successfully

Sources:

user-generated content#

Definition: also known as UGC, content created by end users rather than brands, publishers, or organizations

Purpose: in GEO context, has historically been a major citation source for AI search engines due to its breadth and conversational tone; AI platforms are increasingly deprioritizing UGC in favor of authoritative, structured first-party sources like documentation and help centers

Example: Reddit threads, forum discussions, and community Q&A sites that previously commanded significant citation share in ChatGPT but declined sharply after August 2026

Related Terms: citation share, GEO, retrieval pool

Source: explainx.ai: “Reddit’s ChatGPT Citations Collapsed 86% on August 14 — What Happened” by Yash Thakker

Search#

agent-friendliness#

Agent-Friendly Documentation Spec#

citation share#

data-licensing#

Cloudflare Monetization Gateway#

GEO#

llms.txt#

query fan-out#

retrieval pool#

robots.txt#

truncation budget#

user-generated content#

`llms.txt`#

`robots.txt`#