Risk vs. Reward: Optimization Under Uncertainty

Harry Markowitz's 1952 paper "Portfolio Selection," published in the Journal of Finance when he was a graduate student at the University of Chicago, introduced a mathematical framework for optimal decision making under uncertainty that transformed finance and has since been applied to domains ranging from medical decision making to environmental policy to drug development. The core insight is deceptively simple: rational agents should care not only about the expected value of their choices but also about the variance, because variance represents real risk in the sense that high-variance outcomes can produce catastrophically bad realizations even if the average is good.

The efficient frontier is the set of all portfolios (or, more generally, decision strategies) that achieve the maximum expected return for a given level of variance, or equivalently the minimum variance for a given expected return. Points below the efficient frontier represent suboptimal choices: more variance than necessary for their expected return, or less return than achievable for their variance. Points above the efficient frontier are infeasible given the available options. The optimal point on the frontier for any particular agent is determined by their risk aversion coefficient λ, which characterizes how much expected return they are willing to sacrifice per unit of variance reduction.

The generalization of this framework beyond finance is conceptually straightforward but raises important practical questions. In financial applications, expected return and variance are estimated from historical price data, which is abundant. In medical, public health, or R&D applications, the distributions over outcomes must be estimated from clinical trials, simulation models, or expert judgment, introducing estimation uncertainty that can substantially distort the optimal allocation. This investigation examines both the theoretical generalizability of mean-variance optimization and its empirical performance across three problem domains.

We construct mean-variance efficient frontiers for three distinct problem domains using different data sources. For the financial domain, we use quarterly excess returns (over T-bills) for a set of 10 assets (sector ETFs and broad asset classes) from Quandl Sharadar data spanning 2005 to 2023, estimating the covariance matrix using the Ledoit-Wolf shrinkage estimator to reduce estimation error.

For a stylized public health resource allocation problem, we construct a decision matrix based on published efficacy estimates for five preventive health interventions (from Disease Control Priorities, 3rd Edition), with expected lives saved per million dollars spent as the return measure and bootstrap uncertainty in efficacy estimates as the variance measure.

For an R&D project portfolio, we use published estimates of pharmaceutical development success rates and expected returns by therapeutic area from the Tufts Center for the Study of Drug Development, constructing a variance-return tradeoff across 8 hypothetical project types.

In all three domains, the efficient frontier is computed by solving a quadratic program: minimize portfolio variance subject to achieving a target expected return, with non-negativity constraints on allocations and the constraint that allocations sum to 1. We use the OSQP solver for the financial application and scipy.optimize.minimize with method='SLSQP' for the other domains. The efficient frontier is traced by solving this problem across a grid of target return values.