Multi-armed Bandit Algorithms for Cournot Games

Abstract We investigate using a multi-armed bandit (MAB) setting for modeling repeated Cournot oligopoly games. Agents interact with separate bandit problems. An agent can choose from a set of arms/actions representing discrete production quantities; here, the action space is ordered. Agents are independent and autonomous and cannot observe anything from the environment; they can only see their own rewards after taking action and only work towards maximizing these rewards. We first study Cournot models with stationary market demand where random entry or exit from the market is not allowed. We propose two novel approaches that take advantage of the fact that the action space is ordered: ϵ-greedy+HL and ϵ-greedy+EL. These are based on the ϵ-greedy approach as an underlying mechanism because the ϵ-greedy method does not require any knowledge of even the priors of the reward distributions, unlike other popular methods like UCB or Thompson sampling. Our proposed approaches help firms focus on more profitable actions by eliminating less profitable choices and are designed to optimize the exploration. However, in real-world scenarios, market demands evolve over a product’s lifetime for a myriad of reasons. Therefore, we also investigate repeated Cournot games with non-stationary demand such that firms/agents face independent instances of the non-stationary multi-armed bandit problem. We propose a novel algorithm Adaptive with Weighted Exploration (AWE) ϵ-greedy that is loosely based on the ϵ-greedy approach. We use computer simulations to study the emergence of various equilibria in the outcomes and empirically analyze joint cumulative regrets. Using our proposed method, agents are able to swiftly change their course of action according to the changes in demand. In most of the simulations, firms overall produce collusive outcomes, i.e., outcomes better than the Nash equilibrium.