RLion: A Refined Lion Optimizer for Deep Learning

Abstract Optimization algorithms play a fundamental role in training neural networks. The optimizer focuses on the updating weights ofmomentum and velocity on learning rates and losses, furthermore the complexity of the optimizer and the quantity of updatedparameters are considered. In this paper, a RLion(Refined Lion Optimizer) based on the Lion optimizer is denoted with ascaleable factor α and arctan operation to denote the update rule θt = θt−1 − ηt ( 2π arctan(α ∗ ˆmt ) + λ θt−1). arctan is continousmonotonic function and its expectation, variation are less than those of sign, also the θ ’s fluctuation of RLion is less than that ofLion. The higher α, the convey faster. The RLion is able to smooth out the fluctuations, converge faster and more reliable.The FasterNet, EfficientNetV2 and the YOLO_V8 with ImageNet1k dataset are trained without warm up for classificationleveraging the RLion optimizer. Object detection with Vision Transformers on Caltech 101 dataset and the DeepLabV3+ forsemantic segmentation on camera data are trained with AdamW, Lion and RLion optimizer too. Compared to the AdamWand Lion optimizer, the loss and accuracy present the RLion can promote the validation accuracy about 0 ∼ +20% higher thanAdamW on many models even the learning rate is as high as AdamW. The RLion has better convergence performance andversatility.