Benchmarks

Mon, 01 Jan 0001 00:00:00 +0000

Benchmarks#

Standardized datasets and tasks used to measure and compare LLM and/or agent performance across capabilities.

ARC#

Definition: acronym for AI2 Reasoning Challenge; benchmark measuring question answering and reasoning through more than 7,000 grade-school natural science questions

Purpose: evaluates an LLM’s ability to reason over knowledge, includes an easy set and a challenge set of harder questions requiring multi-step reasoning

Related Terms: benchmark, evaluation, few-shot, MMLU, zero-shot

Source: arXiv: “Think you have Solved Question Answering? Try ARC” by Clark et al.

Metrics & Scoring

Mon, 01 Jan 0001 00:00:00 +0000

Metrics & Scoring#

Approaches used to quantify LLM and/or agent performance across the benchmarks and customized evaluations.

accuracy#

Definition: also known as precision; percentage of correct predictions made by an LLM; foundational to evaluation, the most widely reported scoring metric in benchmarks and/or leaderboards; paired with recall and combined into the F1 score