Adaptive Recursive Learning in Mixture of Experts: A Systematic Review

Abstract Mixture-of-Experts (MoE) architectures have become an important mechanism for increasing neural-network capacity while keeping per-input computation substantially below the cost of evaluating the complete parameter set. The basic principle is to maintain multiple expert functions and use a learned gating or routing mechanism to activate a subset of experts for each input. Since the adaptive mixtures of local experts introduced in the early 1990s, MoE research has progressed through hierarchical expert systems, sparsely gated neural layers, distributed sparse Transformers, top-1 and top-k routing, expert-choice routing, vision MoE, instruction-tuned MoE, fine-grained expert specialization, and very large open MoE language models [1–13]. [1–13] Despite this rapid development, the literature uses overlapping terminology for adaptation, recurrence, dynamic computation, routing refinement, expert specialization, and continual updating. Consequently, adaptive routing is sometimes discussed as if it were recursive learning even though a one-pass router does not necessarily contain a feedback loop. This review distinguishes these concepts and proposes Recursive Adaptive Recursive Learning (RARL) as an integrative analytical framework. RARL is not presented as an already standardized algorithm; instead, it denotes a family of MoE mechanisms in which information generated during one computational cycle influences routing, expert participation, expert parameters, or computational depth in a subsequent cycle [3, 6, 7, 14–18]. [3, 6, 7, 14–18] The review follows a PRISMA 2020-oriented reporting structure and synthesizes foundational and contemporary literature across algorithmic, architectural, optimization, and systems dimensions. Particular attention is given to routing functions, sparse top-k selection, expert capacity, load balancing, expert-choice allocation, expert specialization, adaptive computation, distributed communication, and recursive stopping policies. Mathematical formulations are provided for routing probability, sparse aggregation, auxiliary balancing objectives, recursive state updates, adaptive halting, routing churn, and quality-per-compute efficiency [3–12, 19–23]. [3–12, 19–23] The synthesis indicates that the principal unresolved problem is not simply how to increase the number of experts, but how to allocate computation intelligently under quality, latency, memory, communication, and energy constraints. A recursive MoE controller could potentially allocate additional computation only to difficult or uncertain inputs, but it also introduces risks of routing oscillation, error amplification, communication overhead, over-recursion, and expert instability. The review therefore develops a research agenda centered on uncertainty-aware recursive routing, stable expert specialization, dynamic expert populations, communication-aware objectives, continual adaptation, multimodal routing, energy-aware inference, and standardized RARL benchmarks [6–13, 18, 20, 22–29]. [6–13, 18, 20, 22–29]