Counterfactual Shapley Values for Explaining Reinforcement Learning

Abstract This paper introduces an approach based on Counterfactual Shapley Values, which enhances explainability in reinforcement learning by integrating counterfactual analysis with Shapley Values. The approach aims to quantify and compare the contributions of different state dimensions to various action choices. To more accurately analyze the impacts of these contributions, we introduce new characteristic value functions, the Counterfactual Difference based Characteristic Value functions and the Average Counterfactual Difference based Characteristic Value functions. These functions help to evaluate the differences in contributions between optimal and non-optimal actions. Experiments across several RL domains, such as GridWorld, FrozenLake, and Taxi, demonstrate the effectiveness of the Counterfactual Shapley Values method. The results show that this method not only improves transparency in complex RL systems but also quantifies the differences across various decisions.