Crowd-Sourcing for Data Science and Quantifiable Challenges: Optimal Contest Design

We study the design of crowd-sourcing contests in settings where the output (from the contestants) is quantifiable -- for example, a data science challenge. This setting is in contrast to settings where the output is only qualitative and cannot be quantified in an objective manner -- for example, when the goal of the contest is to design a logo. The rapidly growing literature on the design of crowd-sourcing contests focuses largely on ordinal contests -- these are contests where contestants' outputs are ranked by the organizer and awards are based on the relative ranks. Such contests are ideally suited for the latter setting, where output is qualitative. For our setting (quantitative output), it is possible to design \emph{cardinal} contests -- contests where awards could be based on the actual outputs and not on their ranking alone -- thus, the family of cardinal contests includes the family of ordinal contests. We derive an optimal cardinal contest using Myerson's mechanism design framework and design an easy-to-implement contest that achieves the same outcome as the former. We also perform a set of numerical experiments to examine the benefit provided by the optimal cardinal contest over the most popular ordinal contest -- namely, the Winner-Takes-All (WTA) contest. On our testbed of problem instances, the average improvement provided by the optimal cardinal contest to the contest designer's objective is 23.82%; moreover, this improvement is higher when the number of participants in the contest is smaller.