Is This Predictor More Informative than Another? A Decision-Theoretical Comparison

In many real-world applications, a model provider provides probabilistic forecasts to downstream decision-makers who use them to make decisions under diverse payoff objectives. The provider may have access to multiple predictive models, each potentially miscalibrated, and must choose which model to deploy in order to maximize the usefulness of predictions for downstream decisions. A central challenge arises: how can the provider meaningfully compare two predictors when neither is guaranteed to be well-calibrated, and when the relevant decision tasks may differ across users and contexts?&nbsp; <div> <br> </div> <div> To answer this, our first contribution introduces the notion of the informativeness gap between any two predictors, defined as the maximum normalized payoff advantage one predictor offers over the other across all decision-making tasks. Our framework strictly generalizes several existing notions: it subsumes U-Calibration [Kleinberg et al., 2023] and Calibration Decision Loss [Hu and Wu, 2024], which compare a miscalibrated predictor to its calibrated counterpart, and it recovers Blackwell informativeness [Blackwell, 1951, 1953] as a special case when both predictors are perfectly calibrated. Our second contribution is a dual characterization of the informativeness gap, which gives rise to a natural informativeness measure that can be viewed as a relaxed variant of the earth mover's distance (EMD) between two prediction distributions. We show that this measure satisfies natural desiderata: it is complete and sound, and it can be estimated sample-efficiently in the prediction-only access setting. Along the way, we also obtain novel combinatorial structural results when applying this measure to perfectly calibrated predictors. We complement our theory with experiments on LLM-based forecasters in real-world weather and Bitcoin prediction tasks, showing that the informativeness gap offers a more decision-relevant alternative to traditional metrics like the Brier score/ECE, and provides a principled lens for evaluating how ad hoc calibration post-processing affects downstream decision usefulness. </div>