Koopman Spectrum RL for Bifurcation Control: Data-Driven Policy Optimization in Spectral Subspaces

This paper presents a reinforcement learning (RL) framework based on the Koopman operator for high-dimensional nonlinear control. By leveraging nonlinear eigenvalue dynamics, the approach enables scalable and efficient policy optimization. We examined the challenge of controlling complex systems by embedding high-dimensional states xt∈Rn into a Koopman-invariant subspace ϕx∈Rm, where evolution becomes linear under the Koopman operator K. By spectrally decomposing K=UΛU−1, the eigenvalue dynamics are obtained, and K is reconstructed iteratively via dominant eigenpairs vi,wi. A policy network πa|s selects actions ut, while a value function Vs, expressed in Koopman eigenfunction coordinates, guides gradient-based policy updates. The framework integrates spectral stability constraints (ρX<1) and Lyapunov-based analysis to ensure convergence. We derive perturbation bounds for Koopman eigenvalues under policy updates and establish conditions for nonlinear mode interactions in the lifted space. The spectral policy gradient theorem for Koopman RL links eigenvalue dynamics to policy optimization, includes a constrained Bellman formulation in Koopman coordinates, and analyzes bifurcation of learning-induced eigenvalue shifts.