A fractional-order adaptive numerical scheme for nonlinear optimization problems

Fractional gradient descent (FGD) methods, due to their flexible order selection, offer greater potential applications compared to integer order methods. The improved FGD based on the Caputo definition has demonstrated the advantages of fractional order methods in deep neural networks (DNNs). However, these advantages often rely on selecting an appropriate order for specific scenarios. To mitigate the dependence of DNNs on order selection, recent studies have explored variable-order schemes. Nevertheless, existing designs of variable order FGD are primarily based on predetermined changes with respect to iteration numbers, lacking flexibility to adapt to the dynamic training states of the network, which limits their practical effectiveness. To address this issue, this paper proposes a novel adaptive order fractional gradient descent method (AOFGD), where the fractional order can self-adapt during training. We integrate AOFGD into Caputo-based fractional optimizers, constructing a new family of fractional optimizers. Specifically, we conduct numerical simulations to analyze the relationship between the order, convergence speed, and accuracy. We propose an AOFGD method based on a convergence evaluation factor, which not only retains the fast convergence property of FGD but also improves convergence precision. Moreover, the proposed method eliminates the need for manual order tuning, significantly enhancing the generalization capability of fractional optimizers. Experimental results show that the proposed method achieves consistent performance gains across multiple datasets and network architectures, and demonstrates robust behavior under varying learning rates, batch sizes, and training strategies. Overall, it outperforms integer-order and fixed-order fractional optimization methods, highlighting its practical potential for deep learning applications.