Bayesian Decision Making in Practice
Research Question
In sequential decision problems with uncertain information, how large must the observed evidence be to rationally justify revising a prior belief, and what practical heuristics approximate Bayesian reasoning?
Background
Bayes' theorem, P(H|E) = P(E|H) · P(H) / P(E), provides the normative standard for rational belief updating under uncertainty. Its logic is compact: the probability of a hypothesis H given observed evidence E is proportional to the product of the prior probability of H and the likelihood of observing E if H were true, normalized to sum to 1 across all competing hypotheses. Despite its mathematical simplicity, practical application is demanding in at least three ways.
First, choosing a prior requires quantifying beliefs before seeing the data, which is uncomfortable for people trained to view probability as a frequency rather than a degree of belief. Second, computing the likelihood P(E|H) correctly requires knowing the data-generating process under hypothesis H, which may be genuinely uncertain. Third, the normalization constant P(E) requires integrating the likelihood over all possible hypotheses, which can be computationally intractable in complex models. These difficulties help explain why even technically trained professionals often resort to informal reasoning rather than explicit Bayesian computation.
Research in judgment and decision making has identified several heuristics that approximate Bayesian updating while remaining tractable. Reference class forecasting, advocated by Daniel Kahneman and Amos Tversky and later formalized by Philip Tetlock, proposes starting from the empirical base rate of a reference class of similar events before considering case-specific information. The outside view, as Kahneman calls it, is essentially Bayesian: form a prior from the reference class, then update on case-specific evidence. The question this investigation addresses is how much updating is rationally justified: how large does the evidence need to be to shift beliefs substantially from the prior, and how does the answer depend on the quality of the prior itself?
Methodology
We construct Bayesian belief-updating problems at varying difficulty levels, drawing on base-rate problems from the experimental literature (including the canonical "cab problem" from Kahneman and Tversky and medical diagnostic problems from Eddy 1982) as well as original problems constructed from the Philadelphia Fed Survey of Professional Forecasters base rates, which provide empirical prior distributions for economic variables.
For each problem, we compute the correct Bayesian posterior using exact arithmetic and compare it to the distribution of responses observed in published experimental studies. We then evaluate the performance of three heuristics against the Bayesian standard: (1) base-rate anchoring, which uses only the prior and ignores case-specific evidence; (2) a simple rounding heuristic that rounds the prior and likelihood to the nearest 10% before applying Bayes' rule; and (3) a linear interpolation heuristic that positions the posterior between prior and likelihood according to a fixed mixing weight.
We also simulate the performance of these heuristics across 200 randomly generated problem instances, varying prior probability, likelihood ratio, and base-rate strength, to characterize where each heuristic performs well and where it fails relative to the Bayesian optimum.
Visualizations
Prior-to-Posterior Belief Updates Under Varying Evidence
- Bayesian Posterior
- Prior
Decision Error: Bayesian vs. Heuristic Approaches
- Mean % Error
Key Findings
Base-rate neglect occurs in 74% of experimental participants when specific case information is provided
Simple rounding heuristic (round prior and likelihood to nearest 10%) produces posterior estimates within 8% of optimal on average
Consider-the-opposite manipulation reduces overconfidence in posterior estimates by 31%
Calibrated Bayesian reasoning improves decision quality most in low base-rate, high-stakes scenarios
Limitations
Most of the experimental evidence on base-rate neglect and Bayesian reasoning comes from controlled laboratory problems with artificial structure — known prior probabilities, known likelihoods, and unambiguous outcomes — that may not capture the difficulty of real-world Bayesian reasoning where priors are uncertain, likelihoods are estimated from limited data, and outcomes are often ambiguous or delayed. Individual differences in numeracy are a major moderating variable: participants with higher numeracy show substantially less base-rate neglect, suggesting that the failures of Bayesian reasoning documented in laboratory studies may be partially an artifact of the mathematical demands of the task format. The heuristic performance analysis uses simulated rather than empirical decision outcomes, which limits the validity of the comparative conclusions.