Quick Reference#

Brief definitions and complete entry locations, organized by the following sections:

A#

ablation study#

• experimental method that systematically removes model components to measure their contribution
• derived from ablative brain surgery in neuroscience

Glossary Term Entry Location: Research

abstraction#

• label and/or concept that bundles together a set of underlying components or capabilities
• streamlines communication by hiding implementation details

Glossary Term Entry Location: Anatomy

A/B test#

• also known as split testing and/or randomized controlled trial
• experimental method that compares two versions of something to determine which performs better

Glossary Term Entry Location: Research

accessibility tree snapshot#

• structured, text-based page representation pulled from the browser’s accessibility API, not raw HTML or a screenshot
• gives an agent compact, semantic, stable element references; often paired with vision for canvas-based content

Glossary Term Entry Location: Interaction

accuracy#

• also known as precision
• percentage of correct predictions made by an LLM

Glossary Term Entry Location: Evaluation, Metrics & Scoring

adversarial filtering#

• technique used to generate hard, plausible negatives for benchmark datasets
• algorithmically constructs incorrect answers that look realistic enough to mislead an LLM

Glossary Term Entry Location: Evaluation

affinity mapping#

• qualitative research method for organizing and grouping related ideas or observations
• commonly used in UX research and design thinking to synthesize findings

Glossary Term Entry Location: Research, Qualitative

agent#

• autonomous system that perceives environment, makes decisions, and takes actions to achieve goals
• typically LLM-based system that can use tools, maintain memory, and execute multi-step tasks

Glossary Term Entry Location: Anatomy

Agent-as-a-Judge#

• evaluation methodology where an agent assesses the performance of other agents
• related to LLM-as-judge, but focuses on agentic system evaluation rather than just outputs

Glossary Term Entry Location: Evaluation

agent-friendliness#

• contested property describing whether a site’s content and infrastructure work for agents
• measurement includes content accessibility and protocol adoption

Glossary Term Entry Location: Search

Agent-Friendly Documentation Spec#

• defines 23 checks across 7 categories for agent-accessible documentation
• evaluates authentication, content structure, discoverability, markdown availability, observability, page size, and URL stability

Glossary Term Entry Location: Search

Agent Reading Test#

• benchmark measuring whether an agent’s web fetch pipeline delivers page content
• uses canary tokens and task-based retrieval to evaluate what the agent can access

Glossary Term Entry Location: Evaluation, Benchmarks

AGENTS.md#

• Markdown file at a repository’s root giving coding agents build/test/contribution instructions
• jointly developed by OpenAI, Google, Sourcegraph, Cursor, and Factory; now stewarded by the Linux Foundation

Glossary Term Entry Location: Interaction

ARC#

• acronym for AI2 Reasoning Challenge
• benchmark measuring reasoning through and answering natural science questions

Glossary Term Entry Location: Evaluation, Benchmarks

assistant message#

• output generated by an agent or LLM during a conversational turn
• paired with user messages to form the back-and-forth history the agent uses as context

Glossary Term Entry Location: Interaction

automation#

• use of technology to perform tasks with minimal human intervention
• ranges from rule-based systems to complex machine learning models

Glossary Term Entry Location: Anatomy

B#

Bayesian experiment#

• experimental approach using Bayesian statistics to update beliefs with new data
• contrasts with frequentist A/B testing by assigning probabilities to hypotheses

Glossary Term Entry Location: Research

benchmark#

• standardized test or dataset used to evaluate and compare LLM and/or agentic performance
• enables objective, but not neutral, comparison that tracks progress in the field
• limited by bounded scoring, broad dataset, finite assessment, and overfitting

Glossary Term Entry Location: Evaluation

BERTScore#

• automatic evaluation metric for text summary quality
• computes semantic similarity rather than exact n-gram matching

Glossary Term Entry Location: Evaluation, Metrics & Scoring

bias and fairness score#

• evaluation metric detecting disparities in decision-making across different user groups
• used to identify and mitigate agent systematic favoritism or discrimination

Glossary Term Entry Location: Evaluation, Metrics & Scoring

BLEU#

• acronym for Bilingual Evaluation Understudy
• evaluates machine translation by computing matching n-grams between an LLM’s predicted translation and a human-produced translation

Glossary Term Entry Location: Evaluation, Metrics & Scoring

bulk action#

• also known as action batching
• grouping of multiple independent tool calls into one dispatched request

Glossary Term Entry Location: Interaction

C#

canary phrase#

• unique marker string embedded in content to verify its presence in a system
• named after canaries used in coal mines as early warning detectors

Glossary Term Entry Location: Evaluation

Chatbot Arena#

• benchmark platform that pits two anonymous chatbots against each other
• users have random real-world conversations with both, then vote on which they prefer before identities are revealed

Glossary Term Entry Location: Evaluation, Benchmarks

citation share#

• percentage of an AI search platform’s cited sources that come from a specific domain
• core metric for measuring visibility within AI-generated answers

Glossary Term Entry Location: Search

Cloudflare Monetization Gateway#

• reverse proxy infrastructure intending to resolve identity, permission, payment inside one HTTP request
• enables per-request charging for automated traffic using x402 protocol

Glossary Term Entry Location: Search

codebook#

• structured guide used in qualitative research to categorize and tag data consistently
• defines categories, codes, and rules for applying them to text or observations

Glossary Term Entry Location: Research, Qualitative

Cohen’s kappa coefficient#

• statistical measure of inter-rater agreement for categorical items
• quantifies the level of agreement between two raters while accounting for chance agreement

Glossary Term Entry Location: Research, Quantitative

conceptual AI experiment#

• test type in which AI exists as a label or framing device, but no AI is actually implemented
• typically uses vignettes or scenarios to model operational principles or consequences of AI

Glossary Term Entry Location: Research

confidence interval#

• range of values likely to contain the true effect size, given the statistical model assumptions
• commonly reported as 95% confidence interval - if computed repeatedly under valid conditions, 95% will contain the true value

Glossary Term Entry Location: Research, Quantitative

context window#

• total amount of text, measured in tokens, an LLM can process at once
• includes system prompt, conversation history, and any injected context

Glossary Term Entry Location: Anatomy

controlled vs natural#

• experimental design distinction based on environment
• trade-off between control/replicability and external validity/generalizability

Glossary Term Entry Location: Research

conversational flow#

• metric evaluating an agent’s ability to maintain coherent and meaningful conversations
• interaction and user experience evaluation for chatbots and virtual assistants

Glossary Term Entry Location: Evaluation, Metrics & Scoring

cost-efficiency#

• evaluation metric measuring computational resources required relative to task performance
• factors include token usage, API calls, processing time, energy consumption

Glossary Term Entry Location: Evaluation, Metrics & Scoring

CSAT#

• abbreviation for Customer Satisfaction score
• interaction metric measuring how satisfied users are with AI responses

Glossary Term Entry Location: Evaluation, Metrics & Scoring

D#

data-licensing#

• commercial agreements where publishers grant AI companies structured API access to their content
• shapes the pipeline through which AI models access publisher content for search or training

Glossary Term Entry Location: Search

Discriminability Score#

• metric used to filter benchmarks by identifying tasks that separate a good response from a bad one
• intends to create faster regression suites by removing redundant questions

Glossary Term Entry Location: Evaluation, Metrics & Scoring

dissemination#

• systematic sharing of research findings with target audiences beyond the research team
• ensures knowledge can advance the field, change practice and policy, or inform future research

Glossary Term Entry Location: Research, Qualitative

E#

EDD#

• acronym for Evaluation-driven Development
• software development methodology where evaluation guides design and iteration

Glossary Term Entry Location: Evaluation

edge case#

• scenario or condition that occurs at extreme operating parameters or unusual circumstances
• falls outside normal operating conditions but within specified boundaries

Glossary Term Entry Location: Research

empirical testing#

• validation approach based on observation and experimentation rather than theory alone
• uses real data and measurable outcomes to evaluate hypotheses

Glossary Term Entry Location: Research

engagement rate#

• interaction metric tracking how often users interact with an AI system
• part of interaction and user experience evaluation

Glossary Term Entry Location: Evaluation, Metrics & Scoring

error rate#

• evaluation metric measuring the percentage of incorrect outputs or failed operations
• inverse counterpart of success rate or task completion

Glossary Term Entry Location: Evaluation, Metrics & Scoring

evaluation#

• dynamic, ongoing, structured assessment of an agent and/or LLM performance
• infrastructure includes benchmarks, harnesses, and metrics

Glossary Term Entry Location: Evaluation

exact match#

• proportion of an LLM’s predictions that match the expected answer exactly
• valuable criterion for translation and question-answering benchmarks

Glossary Term Entry Location: Evaluation, Metrics & Scoring

experimental design#

• systematic planning of how to conduct an experiment to answer a research question
• goal is to isolate causal effects while minimizing confounding factors

Glossary Term Entry Location: Research

F#

F1 score#

• metric blending accuracy and recall into a single measure
• treats precision and recall as equally weighted to balance false positives and false negatives

Glossary Term Entry Location: Evaluation, Metrics & Scoring

few-shot#

• approach that supplies LLMs examples of how to complete a task before prompting
• demonstrates an LLM’s ability to learn from scarce data

Glossary Term Entry Location: Evaluation

Flesch–Kincaid readability tests#

• designed to indicate how difficult a passage in English is to understand
• score reflects the U.S. grade level needed to comprehend the text

Glossary Term Entry Location: Evaluation, Metrics & Scoring

function calling evaluation#

• assessment of an agent’s tool invocation behavior
• rule-based metrics cover structural correctness while semantic metrics cover grounding

Glossary Term Entry Location: Evaluation

functional correctness#

• evaluation criterion for code generation benchmarks
• judges solutions by whether they pass the corresponding unit tests

Glossary Term Entry Location: Evaluation

G#

GAIA2#

• successor to GAIA - General AI Assistants
• evaluates assistant agency inside agents research environments using asynchronous dynamic events and capability splits

Glossary Term Entry Location: Evaluation, Benchmarks

gate#

• prompt condition that must be satisfied before work sequence continues
• provides objectively evaluable agentic checkpoints: thing happens → condition → then proceed

Glossary Term Entry Location: Interaction

GEO#

• abbreviation for Generative Engine Optimization
• practice of optimizing content to be cited in AI-generated answers rather than ranked in search results

Glossary Term Entry Location: Search

golden dataset#

• curated set of prompts and verified answers for a specific business domain
• used as the basis for custom benchmarks and regression testing

Glossary Term Entry Location: Evaluation

Goodhart’s law#

• originally an economics principle, now widely applied to AI and/or agent systems
• “when a measure becomes a target, it ceases to be a good measure”

Glossary Term Entry Location: Evaluation

ground truth#

• reference data assumed to be objectively correct
• annotated data that agents and/or LLMs are tested against

Glossary Term Entry Location: Evaluation

GSM8K#

• abbreviation for Grade School Math 8K
• benchmark testing an LLM’s mathematical reasoning with math word problems

Glossary Term Entry Location: Evaluation, Benchmarks

Gunning fog index#

• readability test that estimates the years of formal education needed to understand text on first reading
• score of 12 indicates high school senior level

Glossary Term Entry Location: Evaluation, Metrics & Scoring

H#

hallucination#

• type of LLM and/or agent output that’s false, fabricated, or unsupported information
• appears plausible, but isn’t grounded in training data or provided context

Glossary Term Entry Location: Anatomy

harness#

• platform layer that wraps around an LLM
• provides configuration, permission settings, system prompts, tools

Glossary Term Entry Location: Anatomy

harmonic mean#

• mathematical average calculated by dividing the number of items by the sum of their reciprocals
• used in metrics like F1 score to combine precision and recall

Glossary Term Entry Location: Evaluation, Metrics & Scoring

Hawthorne effect#

• type of behavioral reactivity in which individuals modify an aspect of their behavior in response to their awareness of being observed
• observed in agent evaluation in which the agent becomes a more motivated, persistent reader than during normal use

Glossary Term Entry Location: Evaluation

HellaSwag#

• acronym for Harder Endings, Longer contexts and Low-Shot Activities for Situations With Adversarial Generations
• benchmark centered on common sense reasoning and natural language inference

Glossary Term Entry Location: Evaluation, Benchmarks

heuristic#

• practical problem-solving approach that uses shortcuts or rules of thumb to find satisfactory solutions
• differs from algorithms that guarantee optimal solutions

Glossary Term Entry Location: Anatomy

hook#

• script or callback that runs automatically in response to a specific event in the agent’s environment
• fires deterministically based on triggers, such as a file being edited, without going through the LLM’s interpretation loop

Glossary Term Entry Location: Interaction

HumanEval#

• benchmark assessing an LLM’s code generation performance through functional correctness
• LLMs given programming problems and evaluated on whether generated solutions pass corresponding unit tests

Glossary Term Entry Location: Evaluation, Benchmarks

human-in-the-loop#

• system design where humans actively participate in AI decision-making or evaluation process
• human provides feedback, validation, or intervention at critical points

Glossary Term Entry Location: Evaluation

K#

Krippendorff’s alpha#

• statistical measure of inter-rater agreement for categorical items
• generalizes Cohen’s kappa to handle multiple raters, missing data, and different measurement levels

Glossary Term Entry Location: Research, Quantitative

L#

latency#

• evaluation metric measuring the time taken for an agent or system to process and return results
• important resource-efficiency concern alongside cost

Glossary Term Entry Location: Evaluation, Metrics & Scoring

LLM#

• abbreviation for Large Language Model
• also known as “the agent’s brain”
• LLM trained on vast amounts of text data to understand and generate human language

Glossary Term Entry Location: Anatomy

LLM-as-judge#

• evaluation methodology where a large language model assesses quality of text outputs
• LLM scores or ranks responses based on criteria like accuracy, helpfulness, or safety

Glossary Term Entry Location: Evaluation

LLM distillation#

• specialized form of knowledge distillation that compresses LLMs into smaller, faster models
• used to transfer capabilities between teacher-student LLMs while maintaining efficiency, performance

Glossary Term Entry Location: Anatomy

LLM leaderboard#

• published ranking of LLMs based on a variety of benchmarks
• provides a way to track and compare the performance of many LLMs

Glossary Term Entry Location: Evaluation

llms.txt#

• Markdown file at a site’s root, giving agents a structured content index
• functions like robots.txt and XML sitemaps, but curates content rather than gating access

Glossary Term Entry Location: Search

M#

MBPP#

• abbreviation for Mostly Basic Programming Problems, also known as Mostly Basic Python Problems
• code generation benchmark, assesses functional correctness

Glossary Term Entry Location: Evaluation, Benchmarks

MCP server#

• acronym for Model Context Protocol server
• external server that exposes capabilities to an agent - tools, resources, and/or prompts

Glossary Term Entry Location: Interaction

memory#

• enables agents to maintain context, learn from experience, and reference past actions
• stores and retrieves information across interactions and tasks

Glossary Term Entry Location: Anatomy

METEOR#

• acronym for Metric for Evaluation of Translation with Explicit ORdering
• automatic metric for evaluating machine translation output

Glossary Term Entry Location: Evaluation, Metrics & Scoring

MMLU#

• abbreviation for Massive Multitask Language Understanding
• benchmark assessing breadth of knowledge, depth of natural language understanding and ability to solve problems from knowledge

Glossary Term Entry Location: Evaluation, Benchmarks

model checking#

• process of evaluating whether statistical model assumptions are satisfied by the data
• includes diagnostic tests for fit, examining residuals, and testing additional model terms

Glossary Term Entry Location: Research, Quantitative

MT-Bench#

• benchmark designed to test how well an LLM engages in dialogue and follows instructions
• includes open-ended multi-turn questions in coding, humanities, reasoning, roleplay, STEM, writing

Glossary Term Entry Location: Evaluation, Benchmarks

N#

natural AI experiment#

• test type that features AI in environments where it is actually used - platforms, workplaces, real services
• often A/B tests run by organizations to improve products or operations

Glossary Term Entry Location: Research

n-gram#

• fundamental building block for metrics like BLEU and ROUGE
• contiguous sequence of n items from a given sample of text

Glossary Term Entry Location: Evaluation, Metrics & Scoring

non-parametric methods#

• statistical techniques that make fewer assumptions about data distribution than parametric methods
• somewhat misleading - these methods are not assumption-free

Glossary Term Entry Location: Research, Quantitative

null hypothesis#

• serves as a baseline for testing - premise proposing zero effect or no relationship between variables
• tested to determine if observed data are unusual enough to reject the hypothesis

Glossary Term Entry Location: Research, Quantitative

O#

observability#

• tracking, logging and monitoring an agent’s behavior, decisions and intermediate actions
• enables inspection of function calls, API responses and retrieval steps to diagnose failures

Glossary Term Entry Location: Interaction

OLS regression#

• acronym for Ordinary Least Squares regression
• statistical method that estimates relationships between variables by minimizing squared differences

Glossary Term Entry Location: Research, Quantitative

one-sided hypothesis#

• also known as dividing hypothesis
• test premise about whether an effect is greater than or less than a specific value

Glossary Term Entry Location: Research, Quantitative

Online-Mind2Web#

• benchmark for evaluating web agents on live, dynamic websites
• tasks cover finance, government, shopping, travel, and other consumer workflows

Glossary Term Entry Location: Evaluation, Benchmarks

OSWorld#

• benchmark evaluating whether an AI can operate a desktop environment
• includes OSWorld-Verified and OSWorld 2.0 which includes long-horizon professional workflows

Glossary Term Entry Location: Evaluation, Benchmarks

overfitting#

• mathematical modeling concept used in machine learning, agent evaluation
• model learns training data so close, that it fails to generalize to new inputs
• for LLM and/or agent benchmarking, performing well on test data, but poorly on real-world data

Glossary Term Entry Location: Evaluation

P#

parameter value grounding#

• semantic function-calling metric based on LLM-as-a-judge
• verifies that every parameter value is directly derived from the user’s text, the context history, or API specification defaults

Glossary Term Entry Location: Evaluation, Metrics & Scoring

pass@k#

• code generation evaluation metric measuring at least one of k generated solutions passes unit tests
• captures functional correctness across multiple generated candidates

Glossary Term Entry Location: Evaluation, Metrics & Scoring

permission and safety systems#

• platform-level rules that define what actions an agent is allowed to take
• conceptual authorization and/or guardrails

Glossary Term Entry Location: Anatomy

perplexity#

• measures how good an LLM is at prediction
• the lower an LLM’s perplexity score, the better it is at comprehending a task

Glossary Term Entry Location: Evaluation, Metrics & Scoring

planning#

• fundamental building block for autonomous task execution
• agent capability to decompose complex goals into sequences of executable actions

Glossary Term Entry Location: Interaction

policy adherence rate#

• evaluation metric measuring the percentage of responses that comply with predefined organizational or ethical policies
• used to verify agents respect enterprise guardrails and compliance requirements

Glossary Term Entry Location: Evaluation, Metrics & Scoring

power#

• probability that a statistical test will reject the test hypothesis when a specific alternative is correct
• calculated before study to determine adequate sample size

Glossary Term Entry Location: Research, Quantitative

prediction model#

• algorithm or statistical model that forecasts outcomes based on input data
• learns patterns from training data to make predictions about new cases

Glossary Term Entry Location: Research, Quantitative

probability#

• in frequentist statistics, refers to hypothetical frequencies of data patterns under assumed model
• measured over many repetitions of same procedure under identical conditions

Glossary Term Entry Location: Research, Quantitative

prompt#

• input text or instructions given to an LLM and/or agent to guide its response
• quality and specificity of prompts significantly affect output quality

Glossary Term Entry Location: Interaction

prompt injection#

• manipulation of LLM behavior through malicious or misleading prompts
• exploits the absence of a boundary between instructions and data in token sequences

Glossary Term Entry Location: Anatomy

prompt injection vulnerability#

• ethical and security evaluation metric measuring the success rate of adversarial prompts that alter an agent’s intended behavior
• identifies susceptibility to manipulation or misuse

Glossary Term Entry Location: Evaluation, Metrics & Scoring

Promptions#

• portmanteau of “prompt” and “options”
• Microsoft middleware framework that generates ephemeral UI elements to steer generative AI responses
• replaces manual prompt engineering with structured, parameterized choices

Glossary Term Entry Location: Interaction

proxy test#

• indirect measure used to evaluate something difficult to assess directly
• substitutes an observable indicator for an unmeasurable or impractical characteristic

Glossary Term Entry Location: Research

p-value#

• probability that observed data, or more extreme, would occur if all model assumptions including test hypothesis were correct
• measures fit between data and entire statistical model, not just the hypothesis being tested

Glossary Term Entry Location: Research, Quantitative

Q#

qualitative research#

• produces insights about “why” and “how” rather than “how many”
• method focused on understanding meaning, experiences, and context through non-numerical data

Glossary Term Entry Location: Research, Qualitative

quasinatural AI experiment#

• test type that combines naturalness of real agentic systems with feasibility of lab experiments
• testing commercial chatbots in controlled studies, pilot experiments before product launch

Glossary Term Entry Location: Research

query fan-out#

• retrieval technique where an AI search platform breaks one query into multiple sub-queries
• fetches results for each sub-query then synthesizes an answer from combined results

Glossary Term Entry Location: Search

R#

RAG#

• acronym for retrieval-augmented generation
• architecture where LLM retrieves context from external sources before generating responses

Glossary Term Entry Location: Anatomy

recall#

• also called the sensitivity rate
• evaluation metric quantifying the number of true positives

Glossary Term Entry Location: Evaluation, Metrics & Scoring

regression testing#

• release gate discpline that checks whether an agent got worse after a change
• running a curated benchmark dataset as a test suite every time a base LLM is swapped or a system prompt is updated

Glossary Term Entry Location: Evaluation

RLHF#

• acronym for reinforcement learning from human feedback
• training methodology in which human evaluators rate LLM outputs and ratings to fine-tune LLM behavior

Glossary Term Entry Location: Anatomy

robustness#

• system’s ability to maintain performance under varying or adverse conditions
• evaluated through stress testing, edge cases, and challenging scenarios

Glossary Term Entry Location: Evaluation

ROUGE#

• acronym for Recall-Oriented Understudy for Gisting Evaluation
• metric for evaluating text summarization

Glossary Term Entry Location: Evaluation, Metrics & Scoring

rule#

• prompt instruction an LLM interprets and applies at its own discretion
• has implicit opt-out path; LLM can rationalize skipping

Glossary Term Entry Location: Interaction

retrieval pool#

• set of documents and pages an AI search engine draws from when synthesizing an answer
• populated by sub-queries generated through query fan-out

Glossary Term Entry Location: Search

robots.txt#

• Robots Exclusion Protocol - web standard file placed at a site’s root
• instructs web crawlers which paths they may or may not access

Glossary Term Entry Location: Search

S#

SAP Labs agent eval taxonomy#

• two-axis framework for organizing LLM agent evaluation, proposed by SAP Labs
• measures behavior, capabilities, reliability, safety in various modes

Glossary Term Entry Location: Evaluation

scalar#

• mathematical concept, specifically from linear algebra
• element of a field which is used to define a vector space through the operation of scalar multiplication

Glossary Term Entry Location: Research, Quantitative

self-reflection#

• agent capability to evaluate its own reasoning, actions, and outputs
• involves identifying errors, assessing performance, and adjusting strategy

Glossary Term Entry Location: Interaction

semantic evaluation#

• assessment approach based on meaning rather than structure
• uses LLM-as-a-judge to score outputs for relevance, factuality and correctness where no ground truth exists

Glossary Term Entry Location: Evaluation

separation of concerns#

• evaluation design principle in which a complex problem is divided into distinct aspects or issues
• aspects addressed and/or managed individually, even when they belong to the same system

Glossary Term Entry Location: Evaluation

SimpleQA#

• OpenAI fact-seeking benchmark for measuring an LLM’s ability to provide verifiable answers
• used to track hallucination rates for attempted answers

Glossary Term Entry Location: Evaluation, Benchmarks

/SKILL#

• bundle of instructions and reference material that gives an agent just-in-time context for a specific domain or task
• subject to the same context window attention dynamics as other injected content

Glossary Term Entry Location: Interaction

slash command#

• direct command typed into a chat interface, /compact, /init
• triggers specific agent behavior without going through the LLM’s interpretation loop

Glossary Term Entry Location: Interaction

spec#

• abbreviation for specification
• implementation guide

Glossary Term Entry Location: Anatomy

statistical inference#

• foundational methodology for evaluating whether observed results are meaningful or due to chance
• process of drawing conclusions about populations or processes from sample data

Glossary Term Entry Location: Research, Quantitative

statistical model#

• mathematical representation of data variability and all assumptions used to compute statistics
• includes assumptions about - data collection, randomization, treatment allocation, analysis choices

Glossary Term Entry Location: Research, Quantitative

stochastic#

• random probability distribution or pattern that may be analyzed statistically, but may not be predicted precisely
• commonly used in mathematics, science, and information theory

Glossary Term Entry Location: Research, Quantitative

stylized AI experiment#

• test type conducted in a controlled environment, since the AI typically doesn’t exist outside the study
• AI tailored to a research question withs rule-based algorithms, historical data replication, or reinforcement learning

Glossary Term Entry Location: Research

summarization layer#

• context management mechanism that condenses conversation history, tool outputs, and/or retrieved content as an agent’s context nears its window limit
• operates continuously as part of general context engineering

Glossary Term Entry Location: Anatomy

SWE-bench#

• code generation evaluation framework focused on issue resolution
• LLMs tasked with fixing a bug or addressing a feature request in a specific code base

Glossary Term Entry Location: Evaluation, Benchmarks

sycophancy#

• tendency in LLMs to agree with, validate, or comply with user input rather than reasoning independently
• amplified by detailed or specific prompts, which push the LLM into “execution mode”

Glossary Term Entry Location: Anatomy

synthesis#

• process of combining multiple research findings or data points into coherent insights
• transforms raw observations into patterns, themes, and actionable conclusions

Glossary Term Entry Location: Research, Qualitative

system prompt#

• set of instructions provided to the LLM by the platform before any user interaction begins
• sits at the beginning of the context window, giving it strong positional attention weight

Glossary Term Entry Location: Interaction

T#

task completion rate#

• evaluation metric measuring how effectively an agent or system helps users complete a task
• often split into short-horizon vs. long-horizon rates with cost per completed task

Glossary Term Entry Location: Evaluation, Metrics & Scoring

tau-bench#

• also known as τ-bench and/or Tool-Agent-User Interaction Benchmark
• benchmark measuring tool-use reliability in conversational customer-service settings

Glossary Term Entry Location: Evaluation, Benchmarks

taxonomy#

• classification system that organizes concepts, objects, or phenomena into hierarchical categories
• defines relationships between categories and provides structure to a domain

Glossary Term Entry Location: Research

temperature#

• parameter that controls the degree of randomness in an LLM’s outputs
• set by the platform and sometimes adjustable by the user

Glossary Term Entry Location: Anatomy

Terminal-Bench#

• benchmark measuring agentic performance in the command line
• evaluates agent output in which the environment is text-native, deterministic, and verifiable

Glossary Term Entry Location: Evaluation, Benchmarks

training data#

• dataset used to teach an LLM patterns, relationships, and knowledge
• LLM learns by processing examples and adjusting internal parameters

Glossary Term Entry Location: Anatomy

truncation budget#

• platform-specific character/byte limit an agent’s fetch pipeline reads before cutting off
• independent of the LLM’s own context window; varies widely by tool and fetch method

Glossary Term Entry Location: Search

TruthfulQA#

• benchmark measuring an LLM’s ability to generate truthful answers to questions
• addresses the tendency of LLMs to hallucinate and produce inaccurate outputs

Glossary Term Entry Location: Evaluation, Benchmarks

turn#

• single exchange in a conversation: one user message and one assistant message
• agent considers the full turn history when generating a response

Glossary Term Entry Location: Interaction

U#

uncertainty quantification#

• process of measuring and characterizing uncertainty in predictions, decisions, or model outputs
• distinguishes between aleatoric uncertainty - inherent randomness, and epistemic uncertainty - lack of knowledge

Glossary Term Entry Location: Research, Quantitative

unit transformation#

• semantic function-calling metric based on LLM-as-a-judge
• verifies unit or format conversions between values in the context and parameter values in the tool call

Glossary Term Entry Location: Evaluation, Metrics & Scoring

user-generated content#

• also known as UGC
• content created by end users rather than brands, publishers, or organizations

Glossary Term Entry Location: Search

user message#

• input sent by a human or automated system to an agent during a conversational turn
• interpreted by the LLM rather than executed as a direct command

Glossary Term Entry Location: Interaction

V#

vignette study#

• research method presenting hypothetical scenarios to elicit preferences or judgments
• participants read descriptions of situations and state what they would do

Glossary Term Entry Location: Research

VOC#

• acronym for voice of the client
• data where people often share problems they’re encountering, provide feedback, and seek further help

Glossary Term Entry Location: Evaluation, Metrics & Scoring

W#

WebArena#

• benchmark for evaluating web agents against a simulated web environment of realistic sites
• measures reproducible web task completion in an offline setting

Glossary Term Entry Location: Evaluation, Benchmarks

WebGames#

• benchmark for evaluating web agent perception, reasoning, planning, and tool use
• calibrated against a 95.7% human baseline
• failures cluster into vision, real-time-reaction, and precision-control categories

Glossary Term Entry Location: Evaluation, Benchmarks

WebVoyager#

• benchmark evaluating a web agent’s ability to complete tasks on live websites
• sanity check rather than differentiator due to high scores and saturation

Glossary Term Entry Location: Evaluation, Benchmarks

Wilson interval#

• method for estimating a confidence interval for a population proportion
• adjusts the Clopper-Pearson interval to have better coverage properties for small sample sizes

Glossary Term Entry Location: Research, Quantitative

Winogrande#

• benchmark evaluating an LLM’s commonsense reasoning capabilities
• builds upon WSC, Winograd Schema Challenge with crowdsourced problems

Glossary Term Entry Location: Evaluation, Benchmarks