Nonparametric Predictive Inference for Ranking and Selection using Loss Functions

Selection and ranking of multiple groups are fundamental problems in many scientific and practical applications, where decisions must be made under uncertainty about future outcomes. This paper develops a Nonparametric Predictive Inference (NPI) framework for ranking and selection using loss functions to quantify the consequences of suboptimal decisions. Uncertainty is quantified through NPI lower and upper expected losses, capturing the imprecision inherent in predictive inference under minimal modelling assumptions. For selection problems, zero–one, linear, and quadratic loss functions are applied in both pairwise and multiple comparison settings, covering events such as selecting the best group, selecting a subset of best groups, and selecting a subset that contains the best group. For ranking problems, zero–one and general multi-level loss functions are introduced for ranked subsets of the best groups. While the zero–one loss provides a binary assessment of correctness, the multi-level loss assigns graded penalties according to the severity of ranking errors. The framework is further extended to general selection-and-ranking events in which groups are allocated to ordered buckets. Examples indicate that zero–one, linear, and quadratic loss functions often agree in overall conclusions, while linear, quadratic, and multi-level losses offer a more informative assessment of suboptimal selections and ranking errors. The findings highlight the importance of careful loss specification and demonstrate the usefulness of NPI lower and upper expected losses for ranking and selection under uncertainty.