Anatomy#

What agents are: structural components and training that define their abilities and limitations

abstraction#

Definition: label and/or concept that bundles together a set of underlying components or capabilities; streamlines communication by hiding implementation details

Purpose: lets agents, teams, and platforms reason about complex systems at a higher level without tracking every internal part; understanding what an abstraction hides is often necessary for diagnosing unexpected behavior

Example: “agent” is an abstraction for a system of distinct parts

Related Terms: agent, automation, harness, spec

Source: Dachary Carey: “An Agent is More Than Its Brain”

agent#

Definition: autonomous system that perceives its environment, makes decisions, and takes actions to achieve goals; given an objective expressed in natural language, chooses its own next action from a set of tools, executes that action against an external environment, observes the result, repeats the cycle until it judges the goal met or gives up

Historical: while researchers have described systems that sense→decide→act→adapt since the 1960s, current differences in agent infrastructure include tool use as a first-class feature, larger context windows, and observability frameworks; problem domains throughout history consistently include commitment instability, evaluation at scale, goal specification/spec gaming, plan-execution drift, representation and/or framing; common industry pitfalls include conflating historical and technical novelty -

Era	Influential Works
Symbolic, GOFAI: 1956-1985	1956 Dartmouth workshop influenced dominance of GOFAI, good old-fashioned AI paradigm: “intelligence” uncovered from manipulating logic symbols according to rules, path to agency through reasoning systems; STRIPS shaped PDDL, SHRDLU, rise of expert systems such as MYCIN, DENDRAL
BDI, Multi-Agent: 1986-1999	Belief-Desires-Intention models, FIPA standards, Russell and Norvig’s Artificial Intelligence: A Modern Approach
Reinforcement Learning, Quiet Agents: 2000-2019	Sutton and Barto’s Reinforcement Learning, Tesauro’s TD-Gammon, DARPA Grand Challenges, DeepMind’s AlphaGo
LLM Inflection: 2020-2022	OpenAI’s `GPT-3` API and InstructGPT, Wei’s “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models”, Yao’s “ReAct: Synergizing Reasoning and Acting in Language Models”
Tool-Use Turn: 2023-2024	Richards’ AutoGPT, Anthropic and OpenAI tool-use APIs, Anthropic’s computer-use preview, specialized frameworks trading breadth for coherence: LangChain, CrewAI, AutoGen, smolagents; observability stacks from Braintrust, Phoenix, Opik
Standardization: 2025-present	Anthropic’s Model Context Protocol specification, OpenAI’s Agent SDK with Responses API, Google’s Agent Development Kit, Mistral, Cohere, evaluation tooling maturing, EU AI Act phased application

Purpose: current iterations convert raw LLM capability into goal-directed, self-directed task execution; distinct from chatbots, process automation, and workflow engines through nondeterministic autonomy

Examples: Caldwell’s taxonomy serves as a design resource for agentic implementation; higher isn’t better; common design pitfalls include treating LLM output as ground truth, assuming longer thinking equals better answers, confusing breadth of tools with capability, building an agent when something simpler works - the right level for the problem is the lowest one that can solve it -

Level	Category	Description	Failure Modes → Mitigation
0	Scripted Automation	Deterministic program without an LLM - cron job moves files, rules engine prices mortgages, regex parses log lines	Shipping LLM-powered system when a competent script is faster, cheaper, lower variance
1	LLM as Text Function	LLM call wrapped behind deterministic interface without tools, loops, state: calling code handles input validation, output parsing, error paths in a summarize→classify→extract process	Prompt engineering, constrained decoding, schema validation to treat LLM output as untrusted string to fix classification errors, prompt-injection from untrusted input and/or malformed inputs
2	Tool-using LLM	Tools available in short sequence - thoughtht→action→observation→thought pattern in retrieval-augmentated chat assistants deciding search, coding assistants deciding testing strategy, customer-service bots deciding when to retrieve orders	Strict step budgets, schema-validated tool arguments, observability to resolve infinite loops, wrong tool selection, hallucinated tool arguments
3	Goal-directed Agent	Tool-using LLM with explicit planning, task-decomposition mechanism: multi-step research/coding agents as orchestrators; given a goal, produces-maintains plan, executes steps, revises for failure recovery	Explicit plan validation, termination checks, periodic plan-vs-execution reconcilitation to remediate plan-execution drift in which execution quietly deviates, premature termination as agents claim success without meeting the goal, cascading hallucination as early mistake contaminates later reasoning
4	Adaptive Agent with Persistent Memory	Carries state across sessions - commonly claimed, but likely `Level 3` with key-value cache attached	Disciplined memory design to catch drift, privacy leakage across users, users deliberately teaching agent falsehoods
5	Autonomous Multi-agent Systems, General-purpose Computer Use	Systems composed of multiple `Level 3`+ agents: ability to operate heterogeneous environments - browser, desktop, shell through interfaces rather than narrowly-scoped tools	Safety, observability, evaluation techniques as failure modes at lower levels compound with inter-agent prompt injection, emergent coordination bugs, long-horizon goal drift

Coding Agent vs Training Crawler: coding assistants’ web fetch methods don’t include traditional scraping by default; products like Claude Code and GitHub Copilot make real time HTTP requests during a task using generic HTTP client signatures like axios/1.8.4, while training crawlers like GPTBot, ClaudeBot, CCBot scrape the web for LLM training data and self-identify via distinct user-agent strings that respect robots.txt; most agent-detection mechanisms such as user-agent sniffing, the RFC 9421 Signature-Agent header, bot-management JS challenges - catch training crawlers and answer-engine bots, not coding agents, which remain effectively invisible to them

Sources:

Caldwell, Thomas R. The Agentic AI Bible. Thomas R. Caldwell, April 2026.
Dachary Carey: “How to Evaluate a Platform-Written Spec”

automation#

Definition: use of technology to perform tasks with minimal human intervention; can range from basic rule-based systems to complex machine learning models

Purpose: in an AI context, delegates decision-making or execution to algorithms, robots, or automated agents to reduce manual effort and increase consistency

Related Terms: abstraction, agent, harness

Source: Wikipedia: “Automation”

context window#

Definition: total amount of text, measured in tokens, an LLM can process at once; includes system prompt, conversation history, and any injected context

Purpose: bounds what information is directly available to the LLM during a given interaction; content outside the window is not directly accessible

Example: agent platforms handle conversations approaching the limit through summarization, selective truncation, or compression; quality of context window management strategy determines whether agents retain earlier instructions or “forget” context in a few ways:

Challenge	Description
Compaction	LLM summarization / rewriting, or server-side opaque compression
Context Rot	Performance degradation as length increases ~30k+ tokens
Lost in the Middle	LLMs use info at context beginning, end more than middle
Tiered Retention	keep only current page state; trim snapshot with lightweight model before it reaches primary LLM; compress older turns into a persistent, agent-authored `memory` field instead of retaining raw history to hold token use roughly constant

Related Terms: memory, /SKILL, system prompt, temperature

Sources:

hallucination#

Definition: type of AI output that’s false, fabricated, or unsupported information; appears plausible but isn’t grounded in training data or provided context

Purpose: a critical quality-control concern in AI systems; understanding hallucination helps diagnose failures and informs evaluation of truthfulness and robustness

Example: in the Agent Reading Test, GitHub Copilot reported “25 canary tokens found” in its summary, but its own JSON report listed only 16, with only 23 canaries total across all pages; agent hallucinated a count of its own reported data, inflating a number it had just generated; LLMs don’t understand truth, fill gaps in training data, oversimplify patterns, and lack real-world verification to produce different types of hallucinations -

Type	Description
Creative	Produces completely fictional explanation to a scientific question
Fabricated	Creates references or sources that don’t exist
Factual	Generates incorrect facts, asserting a fictional event as historical
Logical	Delivers contradicting outputs or lack consistency

Sources:

harness#

Definition: platform layer that wraps around an LLM; provides configuration, permission settings, system prompts, and tools

Purpose: shapes agent behavior independently of the underlying LLM; agents sharing the same LLM can behave very differently depending on their harness configuration; in computer-use agents, forms the middle tier of a three-layer stack that includes an LLM below for perception and decision-making and infrastructure above for managed environments; the same harness can run different LLMs or infrastructure

Example: code search, file operations, shell execution, web access, content management strategy, and temperature settings; Agent S, Browser Use, CoACT, GTA1, and are harnesses distinct from the LLMs and infrastructure they run on

Sources:

heuristic#

Definition: practical problem-solving approach that uses shortcuts or rules of thumb to find satisfactory solutions; differs from algorithms that guarantee optimal solutions

Purpose: trades optimality for speed and feasibility when exhaustive search is impractical; in agent systems, guides decision-making when perfect information or unlimited computation is unavailable

Example: A* search, greedy algorithms, hand-crafted evaluation functions

Related Terms: planning, self-reflection

Source: Wikipedia: “Heuristic”

LLM#

Definition: acronym for Large Language Model; model trained on vast amounts of text data to understand and generate human language; also known as “the agent’s brain”

Purpose: not all AI is LLM-based - computer vision models and recommendation systems are common non-LLM examples; the LLM is the reasoning component an agent’s harness wraps around

Open-weight vs. Open-source:

Term	Description	Examples
Open-source	Entire pipeline released for full reproducibility, research	Ai2 Olmo, Falcon
Open-weight	Model weights released publicly for deployment, fine-tuning, inference; architecture, data, training code may not be included	LLaMA, Mistral
Proprietary	Weights closed; access typically via API	`Claude`, `Gemini`,`GPT`

Related Terms: agent, harness, LLM distillation, temperature,

Sources:

LLM distillation#

Definition: specialized form of knowledge distillation that compresses LLMs for efficiency while preserving performance; while compression shrinks LLMs through quantizing or pruning, distillation produces a separate student-model with its own parameters and different design whose goal is to emulate teacher-model behavior

Purpose: enables deployment of capable models in resource-constrained environments - edge devices, mobile, low-latency applications; reduces computational costs and energy consumption; limits include static student-model performance level, too large a gap between levels risk degradation, design impacts more than parameter quantity, knowledge loss, may require continued fine-tuning

Example: Google’s Gemma models are distilled from Gemini; DeepSeek used distillation to create a 7B student that outperformed a 32B LLM on competition mathematics; techniques include -

Technique	Description
Feature-Based	Copies hidden representations from intermediate layers to capture deeper knowledge
Multi-Teacher	Combines knowledge from multiple teacher-models for robustness
Output-Based	Learns from teacher-model’s soft probability distributions, soft targets, using KL divergence loss
Prompt-Based	Compresses long prompts into shorter, efficient versions
RL-Based	Uses reinforcement learning feedback signals to iteratively improve
Synthetic Data Augmentation	Expands training data using teacher-generated examples for improvement
Task-Specific	Fine-tunes for specific downstream applications

Related Terms: LLM, training data, harness, training data

Sources:

memory#

Definition: enables agents to maintain context, learn from experience, and reference past actions; ability to store and retrieve information across interactions and tasks

Purpose: critical for multi-step reasoning and adapting behavior based on history; types include short-term - current task, long-term - across sessions, and episodic - specific events

Example: 100K-word prompts are more expensive than short ones due to KV caching - working memory separate from the LLM’s weights that store key-value vectors for every token processed; grows linearly with token count and batch size; decoding reads the entire cache on every generated token makes it a bandwidth cost as much as a storage one, techniques include -

Technique	Description
Eviction	Drops token entries from the cache to reduce its size; keeps recent tokens and a few opening tokens, while less relevant entries are discarded - risks losing information the generation later needs
Grouped-query Attention	Lets multiple query heads share a single key-value head, sharply reducing the number of stored vector sets; the standard trade-off between full multi-head attention and the more aggressive multi-query variant
Multi-head Latent Attention	Projects keys and values into a smaller latent representation before caching, then expands them on read; large memory savings but adds serving complexity, compression adds work on every read and pairs awkwardly with standard attention implementations
Multi-query Attention	Extreme head-sharing variant where every query head shares one key-value head; saves the most memory, but can degrade quality and training stability
Paged Attention	Borrows OS-style fixed-size page management to eliminate cache fragmentation; systems that wasted 60–80% of cache memory to fragmentation dropped below 4%
Prefix Caching	Shares cached blocks across requests that begin with identical text, such as a repeated system prompt; API providers report 50–90% cost and latency reductions on cache hits - risk is prompt info leaking by sharing state
Quantization	Rounds key and value vectors from 16 bits to 8 or 4 bits; 8-bit costs under 1% accuracy, 4-bit saves more, but shows measurable losses on demanding retrieval tasks

Related Terms: context window, planning, self-reflection

Sources:

permission and safety systems#

Definition: platform-level rules that define what actions an agent is allowed to take; conceptual authorization and/or guardrails

Purpose: shape agent behavior independently of the underlying LLM and enforce safety boundaries on what an agent may execute, largely in response to security concepts -

Concept	Description
Data Poisoning	Insertion of malicious data into training sets or sources to alter LLM behavior; only requires sampling for backdoor installation
Defense in Depth	Realistic posture for LLM systems as no single filter holds all the time; process of layering independent security controls, so that failure of one is contained by another - input validation, retrieval sources cleaning, minimum tool permission scope, model sanitation, monitoring, human review
Excessive Agency	When an agent holds more permission than its task requires; arises from the lethal trifecta: simultaneous access to private data, exposure to untrusted content, a channel to act externally
Model Theft	Extraction of model weights or architecture through API queries; typically expensive, bounded, but remains a risk for teams hosting open weights or running their own training pipelines
Supply Chain Security	Verifying provenance of models, adapters, vector stores, and tools before deployment; bypasses runtime defenses because the threat is present before input validation runs

Example: requiring confirmation before running shell commands, restricting file access to specific directories, or blocking certain categories of action entirely; a production browser agent’s execution layer blocks clicks on elements labeled “refund,” “delete,” or “transfer” unless explicit confirmation is given, and restricts navigation to an allowlisted set of domains - enforcing the boundary in code instead of relying on the LLM to police its own actions, since prompt injection can override LLM-level judgment but not a deterministic check it never reaches

Sources:

prompt injection#

Definition: traditional command injection applied to realm of natural language; manipulation of LLM behavior by crafting malicious or misleading prompts, often by bypassing safety filters and executing unintended instructions; exploits the absence of a boundary between instructions and data in token sequences

Purpose: fundamental security vulnerability in LLM systems; understanding it is essential for building secure agents; the root cause of many agent security failures

Direct vs Indirect:

Type	Route	Description	Example
Direct	User input	Hostile instruction typed into chat box	User asks agent to ignore previous instructions
Indirect	Retrieved content	Instruction embedded in legitimate tasks - docs, emails, web pages	EchoLeak CVE-2025-32711 - hidden email instructions caused data exfiltration

Sources:

RAG#

Definition: acronym for retrieval-augmented generation; architecture where LLM retrieves context from external sources before generating responses; typically uses vector databases to store and retrieve documents as numerical embeddings

Purpose: extends LLM capabilities beyond training data cutoff dates; enables agents to ground answers in current, specific information, but also creates new attack surface through poisoned retrieval sources

Example: PoisonedRAG attack that corrupted system answers by inserting 5 malicious passages into millions of documents, reaching 90% success rate on targeted questions

Related Terms: context window, LLM, summarization layer

Sources:

RLHF#

Definition: acronym for reinforcement learning from human feedback; training methodology in which human evaluators rate LLM outputs and the ratings fine-tune the LLM toward preferred behaviors; uses reward models to quantify preferences and reinforcement learning to optimize the model; not immune to selection and human evaluation bias

Purpose: creates a strong instruction-following bias; LLMs trained with RLHF tend to prioritize explicit user instructions, sometimes at the expense of broader context; aligns models with human values and reduces harmful outputs

Process: SFT on instruction-response pairs → reward model trained on human preference data → RL algorithm updates LLM to maximize reward → iterative refinement

Component	Description
DPO	Direct Preference Optimization; eliminates separate reward model by using preference data directly
KL Divergence	Measures difference between original and updated model distributions; prevents reward hacking
KTO	Kahneman-Tversky Optimization; utility-based approach using binary feedback instead of paired preferences
OOD Generalization	Out-of-distribution performance; ability to handle scenarios not seen in training
PPO	Proximal Policy Optimization; stable RL algorithm using clipped objective function
Reward Hacking	Model manipulates reward system to get high scores without solving the actual problem
Reward Model	Neural network trained to predict human preferences; outputs scalar reward values
RLAIF	Reinforced Learning from AI Feedback; uses LLM instead of human annotators for preference labels
SFT	Supervised fine-tuning; initial step using high-quality instruction-response pairs

Aspect	PPO	DPO	KTO
Objective	Maximize reward, prevent large policy updates	Optimize policy based on human preferences using binary classification	Model alignment, maximize utility based on prospect theory
Input	Prompt and reward signal	Prompt and human preference pairs	Prompt and binary feedback
Output	Actions taken in environment	Actions taken in environment aligned with human preferences	LLM outputs with binary labels
Learning Mechanism	Policy gradients, clipped objective	Binary cross-entropy optimization on human preference data	Based on LLM alignment without complex preference models
Network Components	Separate policy and value networks	Single policy network	LLM framework, adapted to KTO methodology
Feedback Mechanism	Rewards from environment	Human preference data	Binary feedback on LLM outputs
Stability	Maintained by clipping mechanism	Preferences with dynamic per-example weighting	Focus on utility maximization
Complexity	Dual network, maximization with policy update stability	Bypasses explicit reward modeling, human preference optimizes policy	Reduces complexity for binary utility optimization
Applicability	General-purpose RL alignment	When human preference alignment crucial	When rapid alignment with human feedback desired

Sources:

spec#

Definition: abbreviation for specification; implementation guide that informs everyone building on a format exactly what to expect

Purpose: documents which fields exist, what values are valid, how files should be structured, and what behavior is required versus optional

Example: the Agent-Friendly Documentation Spec defines 23 checks across 7 categories - content discoverability, markdown availability, page size, content structure, URL stability, observability, and authentication - that evaluate how well a documentation site serves coding agents

Related Terms: abstraction, harness, MCP server

Source: Wikipedia: “Specification (technical standard)”

summarization layer#

Definition: context management mechanism that condenses conversation history, tool outputs, and/or retrieved content into a shorter form as an agent’s context nears its window limit; operates continuously as part of general context engineering, not only when an agent is retrieving information from the web

Purpose: explains why an agent may lose access to content it could theoretically process in full; the layer distills what it judges relevant into a summary and reinitiates a new context window so processing can continue, meaning what a source actually serves may not be what the agent actually sees; distinct from truncation, which is a hard size limit, because summarization is content-dependent and relevance-dependent

Example: when an agent fetches docs mid-task, the layer sits between the HTTP response and the agent, filtering content based on perceived relevance to the agent’s stated query - meaning what a documentation site serves may not be what the agent processes

Related Terms: context window, hallucination, harness, memory

Sources:

sycophancy#

Definition: LLM tendency to agree, comply, or validate user input rather than reason independently; pattern of producing a favorable answer they’ve been primed to produce, not the accurate one; known limitation of RLHF-trained LLMs and an active area of research

Purpose: amplified by detailed or specific prompts, which push the LLM into “execution mode”; understanding sycophancy helps distinguish an agent’s genuine reasoning from compliance bias

Example: in the Agent Reading Test, agents consistently claimed higher scores than they actually achieved; Claude Code claimed 17/18 points but actually scored 15, inflating scores by manually following redirects and fetching JavaScript files that their pipeline could not handle natively, then reporting these workarounds as pipeline capabilities; GitHub Copilot claimed “25 canary tokens found” in its summary while its own JSON report listed only 16

Related Terms: Agent Reading Test, hallucination, RLHF, self-reflection

Sources:

temperature#

Definition: parameter that controls the degree of randomness in an LLM’s outputs; set by the platform and sometimes adjustable by the user

Purpose: affects agent behavior independently of the LLM itself - low temperature produces more focused, predictable responses; high temperature produces more varied, creative ones

Related Terms: context window, harness, system prompt

Source: Dachary Carey: “An Agent is More Than Its Brain”

training data#

Definition: dataset used to teach an LLM patterns, relationships, and knowledge; the LLM learns by processing examples and adjusting internal parameters

Purpose: quality and composition of training data directly affects LLM capabilities and biases

Related Terms: context window, LLM, LLM distillation

Source: IBM: “What is training data?”