Carrying ReLU’s Simplicity Beyond Its Successors

Activation functions are fundamental components governing the learning behavior of deep neural networks. ReLU retains its importance through its simplicity and efficiency, yet its zero gradients in the negative region give rise to the dying neuron problem. Modern self-gated activations (GELU, Swish, Mish) mitigate this but abandon the exact positive-region identity that ReLU possesses and incur higher cost due to transcendental operations. We propose XReLU (eXtended ReLU), a rational activation that carries ReLU’s simplicity up to the performance level of these smooth successors: it preserves the exact ReLU identity for positive inputs while producing nonzero gradients and lower-bounded outputs (asymptote: -0.5) for negative inputs without any transcendental operation: f(x) = x/(1-2 min(x, 0)). XReLU attains C1 continuity and strict monotonicity at only 6 FLOPs per element. Its defining advantage is not higher accuracy; rather, it reaches the accuracy level of smooth activations while simultaneously preserving exact identity, strict monotonicity, C1 continuity, and a transcendental-free form — a combination offered by no existing activation. Experiments span four domains: CNN/CIFAR- 10/100 (8 architectures), YOLO26n/COCO, CSWin-UNet/ISIC 2018, and Transformer/IWSLT14. XReLU achieves accuracy competitive with smooth activations while exhibiting 1.7×, 3.9×, and 3.5× lower latency than Swish, GELU, and Mish, respectively. In the Transformer experiments it attains one of the lowest gradient variances (0.0013) and the smallest generalization gap (0.0043); a depth ablation in residual-free 50-layer networks reveals only a 0.51% accuracy drop. These findings show that the accuracy of smooth activations is attainable with ReLU-level simplicity, without sacrificing identity, monotonicity, or efficiency.