Quantitative Methods#
Statistical tools used to model data, test hypotheses, and quantify uncertainty
Cohen’s kappa coefficient#
Definition: statistical measure of inter-rater agreement for categorical items; quantifies the level of agreement between two raters while accounting for chance agreement
Purpose: verifies that a codebook is applied consistently across raters, supporting the reliability of qualitative coding
Example: Gao and Chen cite Cohen’s kappa in their “From Agent Behaviour to Agent-Friendly Documentation” as part of an established reliability apparatus for classifying documentation interactions, facilitate human validation of their coding scheme, but don’t implement it
Related Terms: codebook, Krippendorff’s alpha, Wilson interval
Sources:
- arXiv: “From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation” by Zhijun Gao, Jing Chen
- Wikipedia: “Cohen’s kappa”
confidence interval#
Definition: range of values likely to contain the true effect size, given the statistical model’s assumptions; commonly reported as a 95% confidence interval - if computed repeatedly under valid conditions, 95% of such intervals will contain the true value
Purpose: width indicates the precision of an estimate, with narrower intervals signaling more precise estimates; not to be confused with a “95% probability the true value is in this range” for any single interval
Example: interval selection depends on the data structure and underlying assumptions; Gao and Chen chose cluster bootstraps over Wilson intervals for their primary analysis in “From Agent Behaviour to Agent-Friendly Documentation” for their primary analysis because standard intervals like Wilson assume independence, which is violated in clustered agent interaction data
Related Terms: null hypothesis, p-value, statistical inference, Wilson interval
Sources:
- arXiv: “From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation” by Zhijun Gao, Jing Chen
- Wikipedia: “Confidence interval”
Krippendorff’s alpha#
Definition: statistical measure of inter-rater agreement for categorical items; generalizes Cohen’s kappa to handle multiple raters, missing data, and different measurement levels
Purpose: verifies that a codebook is applied consistently across raters, supporting the reliability of qualitative coding
Example: Gao and Chen cite Krippendorff’s alpha in their “From Agent Behaviour to Agent-Friendly Documentation” as part of an established reliability apparatus for classifying documentation interactions, facilitate human validation of their coding scheme, but don’t implement it
Related Terms: codebook, Cohen’s kappa coefficient, Wilson interval
Sources:
- arXiv: “From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation” by Zhijun Gao, Jing Chen
- Wikipedia: “Krippendorff’s alpha”
model checking#
Definition: process of evaluating whether statistical model assumptions are satisfied by the data; includes diagnostic tests for fit, examining residuals, and testing additional model terms
Purpose: identifies violations that could invalidate statistical inferences; itself relies on further assumptions that become part of the full model
Related Terms: statistical inference, statistical model
non-parametric methods#
Definition: statistical techniques that make fewer assumptions about data distribution than parametric methods; don’t assume data follows a specific distribution, such as a normal distribution
Purpose: somewhat misleadingly named, since these methods are not assumption-free; still require assumptions such as random sampling or randomization
Related Terms: statistical inference, statistical model
null hypothesis#
Definition: premise proposing zero effect or no relationship between variables; serves as a baseline for testing
Purpose: tested to determine if observed data are unusual enough to reject the hypothesis; failure to reject doesn’t prove the null is true, only that the data are compatible with it
Example: a treatment makes no difference in average outcome compared to control
Related Terms: confidence interval, one-sided hypothesis, p-value, power
OLS regression#
Definition: acronym for Ordinary Least Squares regression; statistical method that estimates relationships between variables by minimizing squared differences to find the best-fitting line through data points
Purpose: used in AI testing to build simple prediction models based on historical data
Related Terms: prediction model, statistical model
one-sided hypothesis#
Definition: also known as a dividing hypothesis; test premise about whether an effect is greater than or less than a specific value
Purpose: differs from two-sided tests that check whether an effect differs in either direction
Example: testing whether a new treatment is at least as good as the standard treatment
Related Terms: null hypothesis, p-value
power#
Definition: probability that a statistical test will reject the test hypothesis when a specific alternative is correct; calculated before a study to determine adequate sample size
Purpose: studies are typically designed for 80% power, meaning the test will detect a true effect 80% of the time; doesn’t measure compatibility of the alternative hypothesis with observed data and shouldn’t be used to interpret results after data collection
Related Terms: null hypothesis, p-value
prediction model#
Definition: algorithm or statistical model that forecasts outcomes based on input data; learns patterns from training data to make predictions about new cases
Purpose: ranges from basic regression models to complex neural networks; accuracy depends on data quality, feature selection, and algorithm sophistication
Related Terms: OLS regression, statistical model
probability#
Definition: in frequentist statistics, refers to hypothetical frequencies of data patterns under an assumed model, measured over many repetitions of the same procedure under identical conditions; in Bayesian statistics, represents the degree of belief in a hypothesis given observed data
Purpose: often confused with the probability of a hypothesis being true or false, leading to common statistical misinterpretations; essential for distinguishing between deterministic and probabilistic approaches in testing and evaluation
| Concept | Description |
|---|---|
| Deterministic | Same input always produces same output; no randomness |
| Probabilistic | Output may vary across runs due to randomness or sampling |
| Prior Probability | Initial belief about a hypothesis before observing any data |
| Posterior Probability | Updated belief after incorporating observed data using Bayes’ theorem |
Example: LLM inference is probabilistic due to temperature, sampling introduce variability - while executing a compiled program is deterministic - same binary + same input = same output; deterministic test execution ensures reproducibility, while probabilistic evaluation, such as LLM-as-a-judge, introduces variability across runs
Related Terms: Bayesian experiment, LLM-as-judge, p-value, statistical inference
p-value#
Definition: probability that observed data, or more extreme, would occur if all model assumptions - including the test hypothesis - were correct; ranges from 0, complete incompatibility, to 1, perfect compatibility
Purpose: used to determine whether an observed effect is likely due to chance; measures the evidence the outcome provides against a given claim while accounting for variability; a low p-value indicates the test procedure had little probability of producing an outcome as large as the observed, were the claim true; commonly misinterpreted as the probability that the hypothesis is true or false
| Concept | Description |
|---|---|
| evidence against claim | p-value quantifies how surprising the outcome is assuming the claim holds |
| null hypothesis | the default assumption that there is no effect or difference; the claim being tested |
| point estimate | the observed effect, which is the best guess of the true effect given the sample |
| variability | random variation in outcomes due to user differences and sampling |
Example: in an A/B test observing 10% lift with p-value 0.01, outcomes ≥10% would occur in only 1 out of 100 tests if the true lift were zero; this rarity provides evidence against the claim of no effect
Related Terms: confidence interval, null hypothesis, probability, statistical model
Source: Analytics Toolkit: “P-values and Confidence Intervals Explained” by Georgi Georgiev
scalar#
Definition: mathematical concept, specifically from linear algebra; element of a field which is used to define a vector space through the operation of scalar multiplication
Purpose: a “scalar value” refers to a single numerical quantity that has magnitude but no direction, distinguishing it from vector-valued quantities
Related Terms: statistical model
statistical inference#
Definition: process of drawing conclusions about populations or processes from sample data; includes hypothesis testing, confidence interval estimation, and parameter estimation
Purpose: foundational methodology for evaluating whether observed results are meaningful or due to chance; accounts for uncertainty and random variation when making generalizations
Related Terms: confidence interval, null hypothesis, statistical model, uncertainty quantification
statistical model#
Definition: mathematical representation of data variability and all assumptions used to compute statistics; embodies a full web of assumptions beyond just equations with parameters, including data collection, randomization, treatment allocation, and analysis choices
Purpose: violation of any assumption, not just the test hypothesis, can produce misleading P values; often presented in compressed form, with many assumptions left unstated or unrecognized
Related Terms: model checking, non-parametric methods, p-value, statistical inference
stochastic#
Definition: random probability distribution or pattern that may be analyzed statistically, but may not be predicted precisely
Purpose: commonly used in mathematics, science, and information theory to describe processes governed by randomness
Related Terms: probability, uncertainty quantification
Wilson interval#
Definition: method for estimating a confidence interval for a population proportion; adjusts the Clopper-Pearson interval to have better coverage properties for small sample sizes and extreme proportions
Purpose: provides more accurate coverage than the Wald interval, especially when the sample size is small or the proportion is near 0 or 1; commonly used in A/B testing and comparative experiments
Example: Gao and Chen cite Wilson intervals in their “From Agent Behaviour to Agent-Friendly Documentation” as a reference for small and zero counts in their primary intervals, which are cluster bootstraps
Related Terms: A/B test, Cohen’s kappa coefficient, confidence interval, Krippendorff’s alpha
Sources:
- arXiv: “From Agent Behaviour to Agent-Friendly Documentation: An Empirical Study of How Coding Agents Discover, Read, and Write Technical Documentation” by Zhijun Gao, Jing Chen
- Wikipedia: “Binomial proportion confidence interval”
uncertainty quantification#
Definition: process of measuring and characterizing uncertainty in predictions, decisions, or model outputs; distinguishes between aleatoric uncertainty - inherent randomness - and epistemic uncertainty - lack of knowledge
Purpose: enables AI systems to express confidence levels and identify when additional data or validation is needed; critical for safe deployment in high-stakes domains like healthcare, autonomous systems, and decision support
Example: common methods include Bayesian inference, ensemble approaches, and Monte Carlo techniques
Related Terms: prediction model, statistical inference, stochastic