Reward centering in multi-agent reinforcement learning

Reward centering (RC) has recently emerged as a simple yet effective paradigm to stabilize training in single-agent reinforcement learning. However, its potential in multi-agent reinforcement learning (MARL) remains unexplored. In this paper, we first construct a reward centering value decomposition framework for MARL, in which global reward signals can be dynamically adjusted to effectively mitigate instability in value function estimation. Then a reward centering policy optimization framework for MARL is also generated, which aims to reduce the instability of policy updates. Based on these two reward centering frameworks, we propose five novel RC-based MARL methods. Extensive experimental results show that these methods can substantially outperform their baselines. Particularly, RC-QMIX and RC-MAPPO methods are best choices for off-policy and on-policy MARL scenarios respectively. This implies that our reward centering mechanisms can efficiently mitigate reward bias and improve the robustness of MARL methods. Our work provides a simple yet general mechanism for enhancing MARL methods and opens a perspective for reward engineering in complex multi-agent scenarios.