Quick Reference#

Brief definitions for key terms in the Agent Ecosystem domain

A#

accuracy#

• also known as precision
• percentage of correct predictions made by an LLM

Glossary Term Entry Location: Evaluation, Metrics & Scoring

adversarial filtering#

• technique used to generate hard, plausible negatives for benchmark datasets
• algorithmically constructs incorrect answers that look realistic enough to mislead an LLM

Glossary Term Entry Location: Evaluation

ARC#

• acronym for AI2 Reasoning Challenge
• benchmark measuring reasoning through and answering natural science questions

Glossary Term Entry Location: Evaluation, Benchmarks

abstraction#

• label and/or concept that bundles together a set of underlying components or capabilities
• streamlines communication by hiding implementation details

Glossary Term Entry Location: Core Concepts

A/B test#

• also known as split testing and/or randomized controlled trial
• experimental method that compares two versions of something to determine which performs better

Glossary Term Entry Location: Research

affinity mapping#

• qualitative research method for organizing and grouping related ideas or observations
• commonly used in UX research and design thinking to synthesize findings

Glossary Term Entry Location: Research, Qualitative

agent#

• autonomous system that perceives environment, makes decisions, and takes actions to achieve goals
• typically LLM-based system that can use tools, maintain memory, and execute multi-step tasks

Glossary Term Entry Location: Core Concepts

Agent-as-a-Judge#

• evaluation methodology where an agent assesses the performance of other agents
• related to LLM-as-a-Judge, but focuses on agentic system evaluation rather than just outputs

Glossary Term Entry Location: Evaluation

assistant message#

• output generated by an agent or LLM during a conversational turn
• paired with user messages to form the back-and-forth history the agent uses as context

Glossary Term Entry Location: Core Concepts

automation#

• use of technology to perform tasks with minimal human intervention
• ranges from rule-based systems to complex machine learning models

Glossary Term Entry Location: Core Concepts

B#

benchmark#

• standardized test or dataset used to evaluate and compare LLM and/or agentic performance
• enables objective comparison and tracks progress in the field

Glossary Term Entry Location: Evaluation

bias and fairness score#

• evaluation metric detecting disparities in decision-making across different user groups
• used to identify and mitigate agent systematic favoritism or discrimination

Glossary Term Entry Location: Evaluation, Metrics & Scoring

BLEU#

• abbreviation for Bilingual Evaluation Understudy
• evaluates machine translation by computing matching n-grams between an LLM’s predicted translation and a human-produced translation

Glossary Term Entry Location: Evaluation, Metrics & Scoring

bounded scoring#

• benchmark limitation - when an agent and/or LLM reaches the highest possible score
• influences continuous evaluation and benchmark maintenance

Glossary Term Entry Location: Evaluation

C#

canary phrase#

• unique marker string embedded in content to verify its presence in a system
• named after canaries used in coal mines as early warning detectors

Glossary Term Entry Location: Evaluation

Chatbot Arena#

• benchmark platform that pits two anonymous chatbots against each other
• users have random real-world conversations with both, then vote on which they prefer before identities are revealed

Glossary Term Entry Location: Evaluation, Benchmarks

codebook#

• structured guide used in qualitative research to categorize and tag data consistently
• defines categories, codes, and rules for applying them to text or observations

Glossary Term Entry Location: Research, Qualitative

Cohen’s kappa coefficient#

• statistical measure of inter-rater agreement for categorical items
• measures the level of agreement between two raters while accounting for chance agreement

Glossary Term Entry Location: Research, Quantitative

conceptual AI experiment#

• test type in which AI exists as a label or framing device, but no AI is actually implemented
• typically uses vignettes or scenarios to model operational principles or consequences of AI

Glossary Term Entry Location: Research

confidence interval#

• range of values likely to contain the true effect size, given the statistical model assumptions
• commonly reported as 95% confidence interval - if computed repeatedly under valid conditions, 95% will contain the true value

Glossary Term Entry Location: Research, Quantitative

context window#

• total amount of text, measured in tokens, an LLM can process at once
• includes system prompt, conversation history, and any injected context

Glossary Term Entry Location: Core Concepts

context window management#

• agent platform strategies to handle conversations that approach or exceed the context window limit
• determine what gets retained, compressed, or dropped as conversations grow long

Glossary Term Entry Location: Core Concepts

controlled vs natural#

• experimental design distinction based on environment
• trade-off between control/replicability and external validity/generalizability

Glossary Term Entry Location: Research

conversational flow#

• metric evaluating an agent’s ability to maintain coherent and meaningful conversations
• interaction and user experience evaluation for chatbots and virtual assistants

Glossary Term Entry Location: Evaluation, Metrics & Scoring

cost-efficiency#

• evaluation metric measuring computational resources required relative to task performance
• factors include token usage, API calls, processing time, energy consumption

Glossary Term Entry Location: Evaluation, Metrics & Scoring

CSAT#

• abbreviation for Customer Satisfaction score
• interaction metric measuring how satisfied users are with AI responses

Glossary Term Entry Location: Evaluation, Metrics & Scoring

D#

Discriminability Score#

• metric used to filter benchmarks by identifying tasks that separate a good response from a bad one
• intends to create faster regression suites by removing redundant questions

Glossary Term Entry Location: Evaluation, Metrics & Scoring

dissemination#

• systematic sharing of research findings with target audiences beyond the research team
• ensures knowledge can advance the field, change practice and policy, or inform future research

Glossary Term Entry Location: Research, Qualitative

E#

EDD#

• acronym for Evaluation-driven Development
• software development methodology where evaluation guides design and iteration

Glossary Term Entry Location: Evaluation

edge case#

• scenario or condition that occurs at extreme operating parameters or unusual circumstances
• falls outside normal operating conditions but within specified boundaries

Glossary Term Entry Location: Research

empirical testing#

• validation approach based on observation and experimentation rather than theory alone
• uses real data and measurable outcomes to evaluate hypotheses

Glossary Term Entry Location: Research

engagement rate#

• interaction metric tracking how often users interact with an AI system
• part of interaction and user experience evaluation

Glossary Term Entry Location: Evaluation, Metrics & Scoring

error rate#

• evaluation metric measuring the percentage of incorrect outputs or failed operations
• inverse counterpart of success rate or task completion

Glossary Term Entry Location: Evaluation, Metrics & Scoring

evaluation#

• dynamic, ongoing, structured assessment of an agent and/or LLM performance
• infrastructure includes benchmarks, harnesses, and metrics

Glossary Term Entry Location: Evaluation

exact match#

• proportion of an LLM’s predictions that match the expected answer exactly
• valuable criterion for translation and question-answering benchmarks

Glossary Term Entry Location: Evaluation, Metrics & Scoring

experimental design#

• systematic planning of how to conduct an experiment to answer a research question
• goal is to isolate causal effects while minimizing confounding factors

Glossary Term Entry Location: Research

F#

F1 score#

• metric blending accuracy and recall into a single measure
• treats precision and recall as equally weighted to balance false positives and false negatives

Glossary Term Entry Location: Evaluation, Metrics & Scoring

few-shot#

• approach that supplies LLMs examples of how to complete a task before prompting
• demonstrates an LLM’s ability to learn from scarce data

Glossary Term Entry Location: Evaluation

final response evaluation#

• evaluation methodology that assesses only the end result or output of an agent’s execution
• judges success based on whether final answer or outcome is correct

Glossary Term Entry Location: Evaluation

Flesch–Kincaid readability tests#

• designed to indicate how difficult a passage in English is to understand
• score reflects the U.S. grade level needed to comprehend the text

Glossary Term Entry Location: Evaluation, Metrics & Scoring

function calling evaluation#

• assessment of an agent’s tool invocation behavior
• rule-based metrics cover structural correctness while semantic metrics cover grounding

Glossary Term Entry Location: Evaluation

functional correctness#

• evaluation criterion for code generation benchmarks
• judges solutions by whether they pass the corresponding unit tests

Glossary Term Entry Location: Evaluation

G#

gate#

• prompt condition that must be satisfied before work sequence continues
• provides objectively evaluable agentic checkpoints: thing happens → condition → then proceed

Glossary Term Entry Location: Core Concepts

golden dataset#

• curated set of prompts and verified answers for a specific business domain
• used as the basis for custom benchmarks and regression testing

Glossary Term Entry Location: Evaluation

Goodhart’s law#

• originally an economics principle, now widely applied to AI and/or agent systems
• “when a measure becomes a target, it ceases to be a good measure”

Glossary Term Entry Location: Evaluation

ground truth#

• reference data assumed to be objectively correct
• annotated data that agents and/or LLMs are tested against

Glossary Term Entry Location: Evaluation

GSM8K#

• abbreviation for Grade School Math 8K
• benchmark testing an LLM’s mathematical reasoning with math word problems

Glossary Term Entry Location: Evaluation, Benchmarks

Gunning fog index#

• readability test that estimates the years of formal education needed to understand text on first reading
• score of 12 indicates high school senior level

Glossary Term Entry Location: Evaluation, Metrics & Scoring

H#

hallucination#

• type of LLM and/or agent output that’s false, fabricated, or unsupported information
• appears plausible, but isn’t grounded in training data or provided context

Glossary Term Entry Location: Core Concepts

harness#

• platform layer that wraps around an LLM
• provides configuration, permission settings, system prompts, tools

Glossary Term Entry Location: Core Concepts

HellaSwag#

• acronym for Harder Endings, Longer contexts and Low-Shot Activities for Situations With Adversarial Generations
• benchmark centered on common sense reasoning and natural language inference

Glossary Term Entry Location: Evaluation, Benchmarks

heuristic#

• practical problem-solving approach that uses shortcuts or rules of thumb to find satisfactory solutions
• differs from algorithms that guarantee optimal solutions

Glossary Term Entry Location: Core Concepts

hook#

• script or callback that runs automatically in response to a specific event in the agent’s environment
• fires deterministically based on triggers, such as a file being edited, without going through the LLM’s interpretation loop

Glossary Term Entry Location: Core Concepts

HumanEval#

• benchmark assessing an LLM’s code generation performance through functional correctness
• LLMs given programming problems and evaluated on whether generated solutions pass corresponding unit tests

Glossary Term Entry Location: Evaluation, Benchmarks

human-in-the-loop#

• system design where humans actively participate in AI decision-making or evaluation process
• human provides feedback, validation, or intervention at critical points

Glossary Term Entry Location: Evaluation

L#

latency#

• evaluation metric measuring the time taken for an agent or system to process and return results
• important resource-efficiency concern alongside cost

Glossary Term Entry Location: Evaluation, Metrics & Scoring

LLM#

• abbreviation for Large Language Model
• also known as “the agent’s brain”
• LLM trained on vast amounts of text data to understand and generate human language

Glossary Term Entry Location: Core Concepts

LLM-as-a-Judge#

• evaluation methodology where a large language model assesses quality of text outputs
• LLM scores or ranks responses based on criteria like accuracy, helpfulness, or safety

Glossary Term Entry Location: Evaluation

LLM leaderboard#

• published ranking of LLMs based on a variety of benchmarks
• provides a way to track and compare the performance of many LLMs

Glossary Term Entry Location: Evaluation

M#

MBPP#

• abbreviation for Mostly Basic Programming Problems, also known as Mostly Basic Python Problems
• code generation benchmark, assesses functional correctness

Glossary Term Entry Location: Evaluation, Benchmarks

MCP server#

• acronym for Model Context Protocol server
• external server that exposes capabilities to an agent - tools, resources, and/or prompts

Glossary Term Entry Location: Core Concepts

memory#

• stores and retrieve information across interactions and tasks
• enables agents to maintain context, learn from experience, and reference past actions

Glossary Term Entry Location: Core Concepts

MMLU#

• abbreviation for Massive Multitask Language Understanding
• benchmark assessing breadth of knowledge, depth of natural language understanding and ability to solve problems from knowledge

Glossary Term Entry Location: Evaluation, Benchmarks

model checking#

• process of evaluating whether statistical model assumptions are satisfied by the data
• includes diagnostic tests for fit, examining residuals, and testing additional model terms

Glossary Term Entry Location: Research, Quantitative

MT-Bench#

• benchmark designed to test how well an LLM engages in dialogue and follows instructions
• includes open-ended multi-turn questions in coding, humanities, reasoning, roleplay, STEM, writing

Glossary Term Entry Location: Evaluation, Benchmarks

N#

natural AI experiment#

• test type that features AI in environments where it is actually used - platforms, workplaces, real services
• often A/B tests run by organizations to improve products or operations

Glossary Term Entry Location: Research

non-parametric methods#

• statistical techniques that make fewer assumptions about data distribution than parametric methods
• somewhat misleading - these methods are not assumption-free

Glossary Term Entry Location: Research, Quantitative

null hypothesis#

• serves as a baseline for testing - premise proposing zero effect or no relationship between variables
• tested to determine if observed data are unusual enough to reject the hypothesis

Glossary Term Entry Location: Research, Quantitative

O#

observability#

• tracking, logging and monitoring an agent’s behavior, decisions and intermediate actions
• enables inspection of function calls, API responses and retrieval steps to diagnose failures

Glossary Term Entry Location: Core Concepts

OLS regression#

• abbreviation for Ordinary Least Squares regression
• statistical method that estimates relationships between variables by minimizing squared differences

Glossary Term Entry Location: Research, Quantitative

one-sided hypothesis#

• also known as dividing hypothesis
• test premise about whether an effect is greater than or less than a specific value

Glossary Term Entry Location: Research, Quantitative

overfitting#

• limitation of LLM benchmarks - an LLM performs well on training or test data, but poorly on real-world data
• risk when an LLM is trained on the same dataset the benchmark uses

Glossary Term Entry Location: Evaluation

P#

parameter value grounding#

• semantic function-calling metric based on LLM-as-a-judge
• verifies that every parameter value is directly derived from the user’s text, the context history, or API specification defaults

Glossary Term Entry Location: Evaluation, Metrics & Scoring

pass@k#

• code generation evaluation metric measuring at least one of k generated solutions passes unit tests
• captures functional correctness across multiple generated candidates

Glossary Term Entry Location: Evaluation, Metrics & Scoring

permission and safety systems#

• platform-level rules that define what actions an agent is allowed to take
• conceptual authorization and/or guardrails

Glossary Term Entry Location: Core Concepts

perplexity#

• measures how good an LLM is at prediction
• the lower an LLM’s perplexity score, the better it is at comprehending a task

Glossary Term Entry Location: Evaluation, Metrics & Scoring

planning#

• fundamental building block for autonomous task execution
• agent capability to decompose complex goals into sequences of executable actions

Glossary Term Entry Location: Core Concepts

policy adherence rate#

• evaluation metric measuring the percentage of responses that comply with predefined organizational or ethical policies
• used to verify agents respect enterprise guardrails and compliance requirements

Glossary Term Entry Location: Evaluation, Metrics & Scoring

power#

• probability that a statistical test will reject the test hypothesis when a specific alternative is correct
• calculated before study to determine adequate sample size

Glossary Term Entry Location: Research, Quantitative

prediction model#

• algorithm or statistical model that forecasts outcomes based on input data
• learns patterns from training data to make predictions about new cases

Glossary Term Entry Location: Research, Quantitative

probability#

• in frequentist statistics, refers to hypothetical frequencies of data patterns under assumed model
• measured over many repetitions of same procedure under identical conditions

Glossary Term Entry Location: Research, Quantitative

prompt#

• input text or instructions given to an LLM and/or agent to guide its response
• quality and specificity of prompts significantly affect output quality

Glossary Term Entry Location: Core Concepts

prompt injection vulnerability#

• ethical and security evaluation metric measuring the success rate of adversarial prompts that alter an agent’s intended behavior
• identifies susceptibility to manipulation or misuse

Glossary Term Entry Location: Evaluation, Metrics & Scoring

proxy test#

• indirect measure used to evaluate something difficult to assess directly
• substitutes an observable indicator for an unmeasurable or impractical characteristic

Glossary Term Entry Location: Research

P value#

• probability that observed data, or more extreme, would occur if all model assumptions including test hypothesis were correct
• measures fit between data and entire statistical model, not just the hypothesis being tested

Glossary Term Entry Location: Research, Quantitative

Q#

qualitative research#

• produces insights about “why” and “how” rather than “how many”
• method focused on understanding meaning, experiences, and context through non-numerical data

Glossary Term Entry Location: Research, Qualitative

quasinatural AI experiment#

• test type that combines naturalness of real agentic systems with feasibility of lab experiments
• testing commercial chatbots in controlled studies, pilot experiments before product launch

Glossary Term Entry Location: Research

R#

recall#

• also called the sensitivity rate
• evaluation metric quantifying the number of true positives

Glossary Term Entry Location: Evaluation, Metrics & Scoring

regression testing#

• release gate discpline that checks whether an agent got worse after a change
• running a curated benchmark dataset as a test suite every time a base LLM is swapped or a system prompt is updated

Glossary Term Entry Location: Evaluation

RLHF#

• acronym for reinforcement learning from human feedback
• training methodology in which human evaluators rate LLM outputs and ratings fine-tune the LLM toward preferred behaviors

Glossary Term Entry Location: Core Concepts

robustness#

• system’s ability to maintain performance under varying or adverse conditions
• evaluated through stress testing, edge cases, and challenging scenarios

Glossary Term Entry Location: Evaluation

ROUGE#

• abbreviation for Recall-Oriented Understudy for Gisting Evaluation
• metric for evaluating text summarization

Glossary Term Entry Location: Evaluation, Metrics & Scoring

rule#

• prompt instruction an LLM interprets and applies at its own discretion
• has implicit opt-out path; LLM can rationalize skipping

Glossary Term Entry Location: Core Concepts

S#

SAP Labs agent eval taxonomy#

• two-axis framework for organizing LLM agent evaluation, proposed by SAP Labs
• measures behavior, capabilities, reliability, safety in various modes

Glossary Term Entry Location: Evaluation

scalar#

• mathematical concept, specifically from linear algebra
• element of a field which is used to define a vector space through the operation of scalar multiplication

Glossary Term Entry Location: Research, Quantitative

self-reflection#

• agent capability to evaluate its own reasoning, actions, and outputs
• involves identifying errors, assessing performance, and adjusting strategy

Glossary Term Entry Location: Core Concepts

semantic evaluation#

• assessment approach based on meaning rather than structure
• uses LLM-as-a-judge to score outputs for relevance, factuality and correctness where no ground truth exists

Glossary Term Entry Location: Evaluation

SimpleQA#

• OpenAI fact-seeking benchmark for measuring an LLM’s ability to provide verifiable answers
• used to track hallucination rates for attempted answers

Glossary Term Entry Location: Evaluation, Benchmarks

/SKILL#

• bundle of instructions and reference material that gives an agent just-in-time context for a specific domain or task
• subject to the same context window attention dynamics as other injected content

Glossary Term Entry Location: Core Concepts

slash command#

• direct command typed into a chat interface, /compact, /init
• triggers specific agent behavior without going through the LLM’s interpretation loop

Glossary Term Entry Location: Core Concepts

spec#

• abbreviation for specification
• implementation guide

Glossary Term Entry Location: Core Concepts

statistical inference#

• foundational methodology for evaluating whether observed results are meaningful or due to chance
• process of drawing conclusions about populations or processes from sample data

Glossary Term Entry Location: Research, Quantitative

statistical model#

• mathematical representation of data variability and all assumptions used to compute statistics
• includes assumptions about - data collection, randomization, treatment allocation, analysis choices

Glossary Term Entry Location: Research, Quantitative

stepwise evaluation#

• evaluation methodology that assesses agent performance at each individual step of task execution
• examines correctness of intermediate actions, decisions, and reasoning at granular level

Glossary Term Entry Location: Evaluation

stochastic#

• random probability distribution or pattern that may be analyzed statistically, but may not be predicted precisely
• commonly used in mathematics, science, and information theory

Glossary Term Entry Location: Research, Quantitative

stylized AI experiment#

• test type conducted in a controlled environment, since the AI typically doesn’t exist outside the study
• AI tailored to a research question: rule-based algorithms, historical data replication, or reinforcement learning

Glossary Term Entry Location: Research

SWE-bench#

• code generation evaluation framework focused on issue resolution
• LLMs tasked with fixing a bug or addressing a feature request in a specific code base

Glossary Term Entry Location: Evaluation, Benchmarks

sycophancy#

• tendency in LLMs to agree with, validate, or comply with user input rather than reasoning independently
• amplified by detailed or specific prompts, which push the LLM into “execution mode”

Glossary Term Entry Location: Core Concepts

synthesis#

• process of combining multiple research findings or data points into coherent insights
• transforms raw observations into patterns, themes, and actionable conclusions

Glossary Term Entry Location: Research, Qualitative

system prompt#

• set of instructions provided to the LLM by the platform before any user interaction begins
• sits at the beginning of the context window, giving it strong positional attention weight

Glossary Term Entry Location: Core Concepts

T#

task completion rate#

• evaluation metric measuring how effectively an agent or system helps users complete a task
• used for task-specific and interaction/user-experience evaluation

Glossary Term Entry Location: Evaluation, Metrics & Scoring

taxonomy#

• classification system that organizes concepts, objects, or phenomena into hierarchical categories
• defines relationships between categories and provides structure to a domain

Glossary Term Entry Location: Research

temperature#

• parameter that controls the degree of randomness in an LLM’s outputs
• set by the platform and sometimes adjustable by the user

Glossary Term Entry Location: Core Concepts

training data#

• dataset used to teach an LLM patterns, relationships, and knowledge
• LLM learns by processing examples and adjusting internal parameters

Glossary Term Entry Location: Core Concepts

trajectory-based evaluation#

• evaluation methodology that analyzes the complete path or sequence of actions an agent takes
• examines entire decision-making process from initial state to final outcome

Glossary Term Entry Location: Evaluation

TruthfulQA#

• benchmark measuring an LLM’s ability to generate truthful answers to questions
• addresses the tendency of LLMs to hallucinate and produce inaccurate outputs

Glossary Term Entry Location: Evaluation, Benchmarks

turn#

• single exchange in a conversation: one user message and one assistant message
• agent considers the full turn history when generating a response

Glossary Term Entry Location: Core Concepts

U#

uncertainty quantification#

• process of measuring and characterizing uncertainty in predictions, decisions, or model outputs
• distinguishes between aleatoric uncertainty - inherent randomness, and epistemic uncertainty - lack of knowledge

Glossary Term Entry Location: Research, Quantitative

unit transformation#

• semantic function-calling metric based on LLM-as-a-judge
• verifies unit or format conversions between values in the context and parameter values in the tool call

Glossary Term Entry Location: Evaluation, Metrics & Scoring

user message#

• input sent by a human or automated system to an agent during a conversational turn
• interpreted by the LLM rather than executed as a direct command

Glossary Term Entry Location: Core Concepts

V#

vignette study#

• research method presenting hypothetical scenarios to elicit preferences or judgments
• participants read descriptions of situations and state what they would do

Glossary Term Entry Location: Research

VOC#

• abbreviation for voice of the client
• data where people often share problems they’re encountering, provide feedback, and seek further help

Glossary Term Entry Location: Evaluation, Metrics & Scoring

W#

Winogrande#

• benchmark evaluating an LLM’s commonsense reasoning capabilities
• builds upon WSC, Winograd Schema Challenge with crowdsourced problems

Glossary Term Entry Location: Evaluation, Benchmarks