Calibration, Bias, and Rational Belief Updating in Human Forecasting

The psychology of judgment under uncertainty was transformed by the research program of Daniel Kahneman and Amos Tversky beginning in the early 1970s. Their work established that human probability judgment is systematically distorted by identifiable cognitive shortcuts called heuristics: availability (judging probability by the ease with which examples come to mind), representativeness (judging probability by similarity to a prototype), and anchoring and adjustment (starting from an initial estimate and adjusting insufficiently). This research ultimately earned Kahneman the 2002 Nobel Prize in Economic Sciences.

Calibration research asks a complementary and more quantitative question: for events that a person assigns 70 percent probability, do 70 percent of them actually occur? A perfectly calibrated forecaster's stated probability for any class of events matches the empirical frequency of that class. Overconfidence, the most replicated finding in calibration research, refers to the systematic pattern in which people's stated confidence intervals are too narrow: they say they are 90 percent confident when they are correct only 60 to 70 percent of the time. The Good Judgment Project, a DARPA-sponsored forecasting tournament running from 2011 to 2015, provided an unprecedented empirical laboratory for studying calibration at scale.

This paper unifies three investigations that together span the landscape of decision under uncertainty. Section 3.1 examines calibration and overconfidence in geopolitical forecasting. Section 3.3 examines the heuristics that approximate Bayesian belief updating and where they fail. Section 3.5 examines the mean-variance optimization framework and its generalization beyond finance. The unifying question is: how far from optimal is human decision under uncertainty, what produces the gap, and how much of it can be recovered by simple corrections?

The calibration analysis uses the publicly available Good Judgment Project dataset, which contains hundreds of thousands of probabilistic forecasts submitted by volunteer forecasters on geopolitical questions over a four-year tournament period. Questions cover territorial disputes, election outcomes, economic indicators, and international negotiations, with resolution defined by publicly verifiable outcomes. Calibration curves are constructed by grouping forecasts into probability bins of width 0.1 and computing the empirical resolution rate within each bin. I compute the Brier Score for each forecast as (p minus o) squared, where p is the stated probability and o is the binary outcome, and decompose it into calibration and resolution components following the Murphy (1973) decomposition.

The Bayesian heuristic analysis constructs belief-updating problems at varying difficulty, drawing on base-rate problems from the experimental literature (including the canonical cab problem from Kahneman and Tversky and medical diagnostic problems from Eddy 1982) and original problems constructed from the Philadelphia Fed Survey of Professional Forecasters base rates. For each problem, I compute the correct Bayesian posterior using exact arithmetic and compare it to the distribution of responses observed in published experimental studies. I evaluate three heuristics against the Bayesian standard: base-rate anchoring (using only the prior), a rounding heuristic (rounding the prior and likelihood to the nearest 10 percent before applying Bayes' rule), and a linear interpolation heuristic. I also simulate performance across 200 randomly generated problem instances.

The mean-variance analysis constructs efficient frontiers for three domains. For finance, I use quarterly excess returns for 10 assets from 2005 to 2023, estimating the covariance matrix with the Ledoit-Wolf shrinkage estimator. For a public health resource allocation problem, I use published efficacy estimates for five preventive interventions (from Disease Control Priorities, 3rd Edition). For an R&D portfolio, I use published estimates of pharmaceutical development success rates by therapeutic area from the Tufts Center for the Study of Drug Development. In all three, the efficient frontier is computed by solving a quadratic program: minimize variance subject to a target expected return, with non-negativity and sum-to-one constraints.

Figure 1 compares the calibration curves of median forecasters and superforecasters against the diagonal of perfect calibration. The median forecaster exhibits substantial overconfidence: events assigned 90 percent probability occur only 76 percent of the time, and the curve bends away from the diagonal in the upper range. The superforecasters, by contrast, are near-perfectly calibrated, with a calibration curve slope of 0.96 against the ideal of 1.0. This is a striking result: a small subset of participants, without any special training beyond the structured forecasting environment, achieve calibration that most forecasters never approach.

Figure 2 decomposes the Brier Score into calibration error and resolution across forecaster skill quartiles. The decomposition reveals that the difference between good and bad forecasters is not primarily calibration (which improves modestly with skill) but resolution, the degree to which forecasts are informative about outcomes. Top-quartile forecasters have resolution of 0.112 versus 0.061 for the bottom quartile, meaning their forecasts discriminate between outcomes far more sharply. Calibration also improves significantly with forecaster experience (p < 0.001), consistent with skill-based learning rather than fixed ability. Numerical probability formats (70 percent) produce better calibration than verbal formats (likely, probable), suggesting that part of the calibration gap is a format effect rather than a pure judgment effect.

Bayes' theorem provides the normative standard for rational belief updating, but its practical application is demanding: it requires quantifying a prior, computing a likelihood, and normalizing over competing hypotheses. Figure 3 shows how the Bayesian posterior rises with accumulating evidence, starting from a 10 percent prior. The curve is steep in the mid-range, meaning that a few pieces of consistent evidence can shift beliefs substantially, but it flattens near 0 and 1, meaning that once beliefs are extreme, additional evidence has diminishing effect.

The experimental evidence on human performance is sobering: base-rate neglect occurs in 74 percent of participants when specific case information is provided. People anchor on the vivid case-specific evidence and underweight the abstract base rate, producing posteriors far from the Bayesian optimum. The heuristic comparison in Figure 4 quantifies the cost: base-rate neglect produces a mean decision error of 42 percent, while a simple rounding heuristic (rounding the prior and likelihood to the nearest 10 percent before applying Bayes' rule) produces error of only 8 percent, and the consider-the-opposite manipulation reduces overconfidence in posterior estimates by 31 percent.

The practical implication is that the full machinery of exact Bayesian computation is not necessary for good performance. A forecaster who faithfully applies the base rate and then adjusts with rough, rounded likelihoods captures most of the Bayesian advantage. The failure mode that matters is not imprecision but neglect: ignoring the base rate entirely is catastrophic, while rounding it is nearly harmless. This is consistent with the superforecaster finding that structured, disciplined reasoning outperforms raw intelligence, and it suggests that debiasing interventions should target neglect (forcing consideration of the base rate) rather than precision (demanding exact arithmetic).

The mean-variance framework, introduced by Markowitz in 1952, generalizes beyond finance to any decision under uncertainty where the decision-maker cares about both the expected value and the variance of outcomes. Figure 5 shows the efficient frontier for a 10-asset financial portfolio: the set of portfolios that achieve the maximum expected return for a given level of variance. The minimum-variance portfolio achieves 31 percent lower variance than the equal-weight portfolio, a substantial risk reduction for a modest return sacrifice.

Figure 6 shows how the optimal allocation between high-EV, high-risk and low-EV, low-risk assets shifts with the risk aversion parameter lambda. At low lambda (risk tolerance), the allocation is dominated by the high-EV option; at high lambda (risk aversion), it shifts toward the low-risk option. The framework generalizes: in the public health application, diversification across five interventions produces 18 percent lower variance in expected lives saved than concentrating in the single highest-EV intervention, and in the R&D application, real-world portfolios are consistently over-concentrated relative to the efficient frontier.

The critical limitation, however, is estimation error. The efficient frontier is computed from estimated expected returns and covariance, and the optimizer exploits noise in these estimates, producing portfolios that appear optimal in sample but perform poorly out of sample. The Ledoit-Wolf shrinkage estimator partially addresses this for finance, where data are abundant, but the problem is severe for the non-financial applications where fewer data points are available. The general lesson is that mean-variance optimization is a powerful normative framework, but its practical performance is limited by the quality of the inputs, and robust estimation matters more than the optimization itself.

The three investigations converge on a single, hopeful conclusion: human decision under uncertainty is systematically suboptimal, but the suboptimality is predictable and largely recoverable. The calibration gap between median and superforecasters is not a gap in raw intelligence but a gap in discipline: superforecasters update frequently, consider base rates, and use numerical formats. The Bayesian heuristic analysis shows that exact computation is unnecessary; faithful application of rounded inputs captures most of the advantage. The mean-variance analysis shows that the optimization framework is sound but that its value is limited by estimation quality, which shrinkage and diversification can improve.

The common thread is that the dominant failure modes are neglect and overconfidence, not imprecision. People fail to consider the base rate, not to compute it precisely. People are overconfident in their high probabilities, not inaccurate in their mid-range estimates. Optimizers exploit noise in estimates, not the structure of the optimization. The corrections that work, accordingly, are structural rather than arithmetic: forcing consideration of the base rate, structuring the forecasting environment, and shrinking estimates toward a prior. This reframes the debiasing project from one of education (teaching people to compute better) to one of design (building environments in which the natural failure modes are suppressed).

The generalization of mean-variance optimization beyond finance is conceptually clean but practically humbling. In finance, the framework is powerful because return and variance are estimated from abundant historical data. In public health and R&D, the distributions over outcomes must be estimated from limited trials or expert judgment, and the optimizer's tendency to exploit noise becomes the dominant practical problem. The frontier is only as good as the inputs, and the most valuable skill is not solving the optimization but producing honest, well-calibrated inputs to it, which returns me to the calibration and Bayesian findings of the first two sections.

Good Judgment Project participants are self-selected volunteers who opted into a forecasting competition, making them systematically different from the general population in interest, motivation, and potentially baseline quantitative ability. The finding that superforecasters are well-calibrated may reflect selection effects as much as learning. Geopolitical forecasting questions have distinctive features (they tend to be binary, have clear resolutions, and concern relatively rare events) that may limit generalizability to everyday probability judgments about continuous quantities or personal outcomes.

Most of the experimental evidence on base-rate neglect comes from controlled laboratory problems with artificial structure (known priors, known likelihoods, unambiguous outcomes) that may not capture the difficulty of real-world Bayesian reasoning where priors are uncertain, likelihoods are estimated from limited data, and outcomes are often ambiguous or delayed. Individual differences in numeracy are a major moderating variable: participants with higher numeracy show substantially less base-rate neglect, suggesting that the documented failures may be partially an artifact of task format. The heuristic performance analysis uses simulated rather than empirical decision outcomes.

Mean-variance optimization assumes that preferences are fully characterized by the first two moments of the outcome distribution, which holds only for normally distributed outcomes or quadratic utility. Financial returns and health outcomes both have fat-tailed, asymmetric distributions that violate this assumption. Parameter estimation error is a severe practical problem: portfolios that appear optimal given estimated parameters often perform poorly out of sample. The Ledoit-Wolf shrinkage estimator partially addresses this for finance but the problem remains severe for the non-financial applications where fewer data points are available.