Benchmarks#
Standardized datasets used to measure and compare LLM and/or agent performance across capabilities
Agent Reading Test#
Definition: companion to Agent-Friendly Documentation Spec; measures whether an agent’s web fetch pipeline delivers page content, using canary tokens embedded at specific positions within large pages and task-based retrieval questions to evaluate what the agent can access vs. what gets truncated, summarized, or filtered away
Purpose: tests pipeline behavior - truncation, redirect following, JavaScript rendering, relevance filtering - rather than agent reasoning; uses a task-first design to avoid priming the agent’s relevance layer, with canary reporting separated into a post-task phase and human-side scoring to avoid self-assessment bias
Scoring: 20 points across 10 tasks; 16 from canary tokens at specific page positions, 10K, 40K, 75K, 100K, 130K characters - and 4 from qualitative assessment of task responses; an agent that scores 12 with all truncation canaries, but no tabbed content tells a different story than one that scores 12 with tabs, but no content past 40K
| Finding | Description |
|---|---|
| Self-Reporting Unreliability | agents are unreliable self-reporters because LLMs are agreeable pattern-matchers |
| Pipeline vs. Agent Behavior | infrastructure layer with HTTP, redirects, JavaScript behaves differently from the reasoning layer on top of it; conflating the two produces misleading conclusions |
| Priming Corrupts Measurement | when an agent knows what’s being looked for, summarization treats those items as relevant, skewing whether content actually survives the pipeline |
| Cascading Accuracy Problems | each attempted fix revealed a new failure mode, score inflation led to qualitative reporting issues, which led to framing problems, which exposed summarization layers |
| Evaluation Context Changes Behavior | agents perform differently once they recognize they’re being tested, attempting more complexity, more persistence than normal |
| Human Usability Research Translates | framing evaluation as “testing the documentation, not you” reduces performance anxiety, produces natural behavior in human and agent users |
| Safety Filters Block Agent-Directed Markup | patterns like <blockquote><strong>Agent:</strong> trigger intermediary safety models, preventing content delivery entirely |
| Structural Separation Works Best | most reliable separation includes agent dividing tasks, comparison with static analysis, human judgment |
Related Terms: benchmark, canary phrase, evaluation, Hawthorne effect, summarization layer
Sources:
ARC#
Definition: acronym for AI2 Reasoning Challenge; benchmark measuring question answering and reasoning through more than 7,000 grade-school natural science questions
Purpose: evaluates an LLM’s ability to reason over knowledge, includes an easy set and a challenge set of harder questions requiring multi-step reasoning
Related Terms: benchmark, evaluation, few-shot, MMLU, zero-shot
Source: arXiv: “Think you have Solved Question Answering? Try ARC” by Clark et al.
Chatbot Arena#
Definition: open benchmark platform that pits two anonymous chatbots against each other; users hold real-world conversations and vote on which they prefer before identities are revealed
Purpose: produces crowdsourced pairwise comparison data fed into statistical methods that estimate scores and create approximate LLM rankings
Example: user preference votes across matchups generate an Elo-style ranking of anonymous LLMs
Related Terms: benchmark, evaluation, LLM leaderboard, MT-Bench
Source: arXiv: “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference” by Chiang et al.
GAIA2#
Definition: successor to GAIA, General AI Assistants, addressing general assistant-style agency; suite of 800 scenarios run inside Meta’s Agents Research Environments with asynchronous dynamic events, continuous time, and capability-differentiated scoring
Purpose: measures how a system handles assistant workloads by default, including changing conditions -
scenarios evolve while the agent works - rather than a frozen question set; introduced capability splits
that report separate scores for execution, search, adaptability, temporal reasoning, and ambiguity handling
instead of a single success rate
Example: dataset evaluates seven core capabilites - execution, search, adaptability, time, ambiguity, Agent2Agent, and noise
Related Terms: benchmark, evaluation, OSWorld
Sources:
- Hugging Face: “Datasets: meta-agents-research-environments/gaia2”
- o-mega: “AI Computer Use Benchmarks 2026: Top Agents Ranked”
GSM8K#
Definition: acronym for Grade School Math 8K; benchmark testing an LLM’s mathematical reasoning with a corpus of 8,500 grade-school math word problems; one of the most-cited reasoning benchmarks used on public LLM leaderboards
Purpose: measures math reasoning where solutions are collected in natural language rather than mathematical expressions
Related Terms: benchmark, evaluation, few-shot, LLM leaderboard, MMLU
Source: arXiv: “Training Verifiers to Solve Math Word Problems” by Cobbe et al.
HellaSwag#
Definition: acronym for Harder Endings, Longer contexts and Low-Shot Activities for Situations With Adversarial Generations; benchmark centered on common sense reasoning and natural language inference
Purpose: tasks LLMs with completing sentences by choosing among endings, including wrong answers created through adversarial filtering; evaluates accuracy for few-shot and zero-shot categories
Related Terms: adversarial filtering, benchmark, evaluation, Winogrande
Source: arXiv: “HellaSwag: Can a Machine Really Finish Your Sentence?” by Zellers et al.
HumanEval#
Definition: benchmark assessing an LLM’s code generation performance through functional correctness; LLMs are given programming problems and evaluated on whether generated solutions pass corresponding unit tests
Purpose: establishes a reproducible standard for coding ability using the pass@k metric
Related Terms: benchmark, evaluation, functional correctness, pass@k, MBPP, SWE-bench
Source: arXiv: “Evaluating Large Language Models Trained on Code” by Chen et al.
MBPP#
Definition: acronym for Mostly Basic Programming Problems, also known as Mostly Basic Python Problems; code generation benchmark with a corpus of more than 900 coding tasks
Purpose: assesses functional correctness based on passing a set of test cases, evaluated in few-shot and fine-tuned settings
Related Terms: benchmark, evaluation, functional correctness, HumanEval, pass@k
Source: arXiv: “Program Synthesis with Large Language Models” by Austin et al.
MMLU#
Definition: acronym for Massive Multitask Language Understanding; benchmark assessing breadth of knowledge, depth of natural language understanding and problem-solving from knowledge; largely saturated by frontier LLMs, reducing its usefullness as a differentiator
Purpose: dataset of more than 15,000 multiple-choice general-knowledge questions across 57 subjects, scored by average per-subject accuracy in few-shot and zero-shot settings
Related Terms: benchmark, evaluation, few-shot, LLM leaderboard
Sources:
- AI Benchmarking Hub, llm-stats: “MMLU”
- arXiv: “Measuring Massive Multitask Language Understanding” by Hendrycks et al.
MT-Bench#
Definition: benchmark designed to test how well an LLM engages in dialogue and follows instructions; dataset of open-ended multi-turn questions, 10 each across coding, extraction, knowledge, math, reasoning, roleplay and writing
Purpose: uses GPT-4 as a judge to evaluate the responses of other LLMs; created by the
same researchers behind Chatbot Arena
Related Terms: benchmark, Chatbot Arena, evaluation, functional correctness, LLM-as-judge
Source: arXiv: “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” by Zheng et al.
Online-Mind2Web#
Definition: live-web benchmark for evaluating web agents on real, dynamic websites rather than cached snapshots; 300 tasks across 136 real sites, difficulty-tiered from Easy - 1-5 steps - through Hard - 11+ steps
Purpose: tests agents against real popups, layout changes, and anti-bot friction that static web datasets omit; replaced the older Mind2Web standard for live-web evaluation because agents face genuine environment variability
Related Terms: benchmark, evaluation, WebArena, WebVoyager
Source: o-mega: “AI Computer Use Benchmarks 2026: Top Agents Ranked”
OSWorld#
Definition: benchmark evaluating whether an AI can operate a desktop environment - files, browsers, office
applications, system settings - through screen observation and virtual mouse and keyboard control;
OSWorld-Verified fixed flawed tasks to enable reproducible scoring while OSWorld 2.0 includes
108 long-horizon professional workflows
Purpose: measures agentic computer use on real operating systems rather than simulated web pages;
Verified is the standard citation for desktop control while 2.0 resets a saturated benchmark
with task counts averaging hundreds of tool calls and median human completion times near 1.6 hours
Example: public task types include “How do I change my Mac desktop background?” and “Update the bookkeeping sheet with my recent transactions over the past few days in the provided folder.”
Related Terms: benchmark, evaluation, GAIA2, WebArena
Sources:
- arXiv: “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments” by Tianbao Xie et al.
- o-mega: “AI Computer Use Benchmarks 2026: Top Agents Ranked”
- “OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments”
SimpleQA#
Definition: fact-seeking benchmark for measuring an LLM’s ability to provide verifiable answers
Purpose: used to track hallucination rates for attempted answers; verifiable recall metrics from such tests provide a baseline that general reasoning benchmarks often overlook
Related Terms: benchmark, evaluation, TruthfulQA
Source: OpenAI: “Introducing SimpleQA” by Jason Wei et al.
SWE-bench#
Definition: code generation evaluation framework focused on issue resolution; LLMs are tasked with fixing a bug or addressing a feature request in a specific code base
Purpose: assessment metric is the percentage of resolved task instances, testing real-world software maintenance ability
Related Terms: benchmark, evaluation, functional correctness, HumanEval
Source: arXiv: “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” by Jimenez et al.
tau-bench#
Definition: also known as τ-bench and/or Tool-Agent-User- interaction Benchmark;
benchmark measuring tool-use reliability in conversational customer-service settings; agents must correctly
invoke tools while holding a natural back-and-forth dialogue with a user
Purpose: evaluates the combination of tool calling and dialogue handling rather than isolated function calls, identifying how agents handle cases that mix conversation with API actions
Example: public task types include conversation with interruptions, accents, and background noise, retrieving-reasoning over a knowledge base
Related Terms: benchmark, evaluation, function calling evaluation, task completion rate
Sources:
- arXiv: “
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains” by Shunyu Yao et al. - o-mega: “AI Computer Use Benchmarks 2026: Top Agents Ranked”
- Sierra AI:
τ-bench
Terminal-Bench#
Definition: collection of harbor-native benchmarks that measure agentic performance in the command line - builds, scripts, and system administration; evaluates whether agents correctly operate a terminal environment where output is text-native, deterministic, and verifiable
Purpose: targets the one computer-use surface where deterministic verification is straightforward, yielding agent scores that track ahead of GUI long-horizon results; became a shared citation across model cards for terminal and infrastructure work
Example: public task types range from “Write and execute small programs and CLI commands” to “Build an entire codebase from scratch”
Related Terms: benchmark, evaluation, OSWorld
Sources:
- o-mega: “AI Computer Use Benchmarks 2026: Top Agents Ranked”
- terminal-bench: benchmarks for ai agents in terminal environments
TruthfulQA#
Definition: benchmark measuring an LLM’s ability to generate truthful answers to questions; addresses the tendency of LLMs to hallucinate and produce inaccurate outputs
Purpose: dataset contains 817 questions spanning 38 subjects including health, law, finance, and politics;
combines human evaluation with GPTs, fine-tuned on BLEU and ROUGE to predict human assessments of
informativeness and truthfulness; largest models were generally least truthful
Related Terms: benchmark, evaluation, hallucination, SimpleQA
Source: arXiv: “TruthfulQA: Measuring How Models Mimic Human Falsehoods” by Lin et al.
WebArena#
Definition: benchmark for evaluating browser agents on reproducible, self-hosted simulated websites rather than live web traffic; 812 tasks across e-commerce, forum discussion, collaborative software development, content management, maps, and reference lookup
Purpose: provides controlled, repeatable web-agent experiments with functional success criteria rather than production website variability; scores saturate faster than live-web benchmarks because fixed environments reward memorization over adaptability
Example: public tasks include “What is the top-1 best-selling product in 2022?” and “Tell me the full address of all international airports that are within a driving distance of 50 km to Carnegie Mellon University.”
Related Terms: benchmark, evaluation, Online-Mind2Web, OSWorld, WebGames, WebVoyager
Sources:
- arXiv: “A Real-World Web Environment for Building Autonomous Agents” by Zhou et al.
- o-mega: “AI Computer Use Benchmarks 2026: Top Agents Ranked”
- Steel.dev: WebArena Leaderboard
WebGames#
Definition: benchmark of 53 diverse challenges evaluating web agent capabilities across perception, reasoning, planning, and tool use; calibrated against a human baseline
Purpose: separates reasoning and tool-use capability from perception and latency limits by clustering unsolved challenges into distinct failure categories rather than a single pass/fail score; human subjects achieve a 95.7% success rate, setting the ceiling agent performance is measured against
Example: a production browser agent achieved approximately 85% success, 45/53 challenges,
compared to approximately 50% reported for the benchmark authors’ own best-performing tested
agent - Gemini 2.5 Pro with the Browser Use framework; the 8 unsolved challenges clustered into
advanced vision requirements, “Slider Symphony,” “Pixel Copy”, real-time interaction
requiring sub-second reactions, “Brick Buster,” “Frog Crossing”, and continuous
precision control, “Block Stack”
Related Terms: benchmark, evaluation, WebArena, WebVoyager
Sources:
- arXiv: “Building Browser Agents: Architecture, Security, and Practical Solutions” by Aram Vardanyan
- Convergence: WebGames
WebVoyager#
Definition: benchmark evaluating a web agent’s ability to complete tasks on live websites through natural-language instructions and browser interaction
Purpose: measures end-to-end web task completion in a realistic environment; interpreted as solved given saturation near the top of scores, making it a sanity check rather than a differentiator
Example: public tasks include “Provide a recipe for vegetarian lasagna with more than 100 reviews and a rating of at least 4.5 stars suitable for 6 people” and “Search an Xbox Wireless controller with green color and rated above 4 stars.”
Related Terms: benchmark, evaluation, Online-Mind2Web, WebArena, WebGames
Sources:
- arXiv: “WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models” by Hongliang He et al.
- o-mega: “AI Computer Use Benchmarks 2026: Top Agents Ranked”
- Steel.dev: WebVoyager Leaderboard
Winogrande#
Definition: benchmark evaluating an LLM’s common sense reasoning capabilities; builds on the original WSC, Winograd Schema Challenge with 44,000 crowdsourced problems using adversarial filtering
Purpose: scored based on accuracy, measuring coreference resolution and common sense reasoning
Related Terms: adversarial filtering, HellaSwag
Source: arXiv: “WinoGrande: An Adversarial Winograd Schema Challenge at Scale” by Sakaguchi et al.