Benchmarks#

Standardized datasets and tasks used to measure and compare LLM and/or agent performance across capabilities.

ARC#

Definition: acronym for AI2 Reasoning Challenge; benchmark measuring question answering and reasoning through more than 7,000 grade-school natural science questions

Purpose: evaluates an LLM’s ability to reason over knowledge, includes an easy set and a challenge set of harder questions requiring multi-step reasoning

Related Terms: benchmark, evaluation, few-shot, MMLU, zero-shot

Source: arXiv: “Think you have Solved Question Answering? Try ARC” by Clark et al.

Chatbot Arena#

Definition: open benchmark platform that pits two anonymous chatbots against each other; users hold real-world conversations and vote on which they prefer before identities are revealed

Purpose: produces crowdsourced pairwise comparison data fed into statistical methods that estimate scores and create approximate LLM rankings

Example: user preference votes across matchups generate an Elo-style ranking of anonymous LLMs

Related Terms: benchmark, evaluation, LLM leaderboard, MT-Bench

Source: arXiv: “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference” by Chiang et al.

GSM8K#

Definition: acronym for Grade School Math 8K; benchmark testing an LLM’s mathematical reasoning with a corpus of 8,500 grade-school math word problems; one of the most-cited reasoning benchmarks used on public LLM leaderboards

Purpose: measures math reasoning where solutions are collected in natural language rather than mathematical expressions

Related Terms: benchmark, evaluation, few-shot, LLM leaderboard, MMLU

Source: arXiv: “Training Verifiers to Solve Math Word Problems” by Cobbe et al.

HellaSwag#

Definition: acronym for Harder Endings, Longer contexts and Low-Shot Activities for Situations With Adversarial Generations; benchmark centered on common sense reasoning and natural language inference

Purpose: tasks LLMs with completing sentences by choosing among endings, including wrong answers created through adversarial filtering; evaluates accuracy for few-shot and zero-shot categories

Related Terms: adversarial filtering, benchmark, evaluation, Winogrande

Source: arXiv: “HellaSwag: Can a Machine Really Finish Your Sentence?” by Zellers et al.

HumanEval#

Definition: benchmark assessing an LLM’s code generation performance through functional correctness; LLMs are given programming problems and evaluated on whether generated solutions pass corresponding unit tests

Purpose: establishes a reproducible standard for coding ability using the pass@k metric

Related Terms: benchmark, evaluation, functional correctness, pass@k, MBPP, SWE-bench

Source: arXiv: “Evaluating Large Language Models Trained on Code” by Chen et al.

MBPP#

Definition: acronym for Mostly Basic Programming Problems, also known as Mostly Basic Python Problems; code generation benchmark with a corpus of more than 900 coding tasks

Purpose: assesses functional correctness based on passing a set of test cases, evaluated in few-shot and fine-tuned settings

Related Terms: benchmark, evaluation, functional correctness, HumanEval, pass@k

Source: arXiv: “Program Synthesis with Large Language Models” by Austin et al.

MMLU#

Definition: acronym for Massive Multitask Language Understanding; benchmark assessing breadth of knowledge, depth of natural language understanding and problem-solving from knowledge; largely saturated by frontier LLMs, reducing its usefullness as a differentiator

Purpose: dataset of more than 15,000 multiple-choice general-knowledge questions across 57 subjects, scored by average per-subject accuracy in few-shot and zero-shot settings

Related Terms: benchmark, bounded scoring, evaluation, few-shot, LLM leaderboard

Sources:

MT-Bench#

Definition: benchmark designed to test how well an LLM engages in dialogue and follows instructions; dataset of open-ended multi-turn questions, 10 each across coding, extraction, knowledge, math, reasoning, roleplay and writing

Purpose: uses GPT-4 as a judge to evaluate the responses of other LLMs; created by the same researchers behind Chatbot Arena

Source: arXiv: “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” by Zheng et al.

SimpleQA#

Definition: fact-seeking benchmark for measuring an LLM’s ability to provide verifiable answers

Purpose: used to track hallucination rates for attempted answers; verifiable recall metrics from such tests provide a baseline that general reasoning benchmarks often overlook

Related Terms: benchmark, evaluation, TruthfulQA

Source: OpenAI: “Introducing SimpleQA” by Jason Wei et al.

SWE-bench#

Definition: code generation evaluation framework focused on issue resolution; LLMs are tasked with fixing a bug or addressing a feature request in a specific code base

Purpose: assessment metric is the percentage of resolved task instances, testing real-world software maintenance ability

Related Terms: benchmark, evaluation, functional correctness, HumanEval

Source: arXiv: “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?” by Jimenez et al.

TruthfulQA#

Definition: benchmark measuring an LLM’s ability to generate truthful answers to questions; addresses the tendency of LLMs to hallucinate and produce inaccurate outputs

Purpose: dataset contains 817 questions spanning 38 subjects including health, law, finance, and politics; combines human evaluation with GPTs, fine-tuned on BLEU and ROUGE to predict human assessments of informativeness and truthfulness; largest models were generally least truthful

Related Terms: benchmark, evaluation, hallucination, SimpleQA

Source: arXiv: “TruthfulQA: Measuring How Models Mimic Human Falsehoods” by Lin et al.

Winogrande#

Definition: benchmark evaluating an LLM’s common sense reasoning capabilities; builds on the original WSC, Winograd Schema Challenge with 44,000 crowdsourced problems using adversarial filtering

Purpose: scored based on accuracy, measuring coreference resolution and common sense reasoning

Related Terms: adversarial filtering, HellaSwag

Source: arXiv: “WinoGrande: An Adversarial Winograd Schema Challenge at Scale” by Sakaguchi et al.