An enhanced version of vulture optimization algorithm for robust signal modulation detection in modern communication systems

Signal modulation detection plays a critical role in contemporary communication systems, serving as a foundation for accurate information decoding, spectral analysis, and system security. Conventional techniques—such as statistical analysis, time-frequency transformations, and cyclostationary feature extraction—are effective under controlled conditions but often fail to deliver robust performance in noisy, dynamic environments, particularly when dealing with sophisticated modulation formats like OFDM and higher-order QAM (e.g., 16-QAM, 64-QAM). While deep learning models have automated the feature extraction process and improved classification accuracy, their dependence on large-scale labeled datasets, high computational costs, and limited adaptability in non-stationary scenarios hinders their practical deployment in real-time applications. To bridge these limitations, this study introduces the Enhanced Vulture Optimization Algorithm (EVOA), a bio-inspired metaheuristic that mimics the adaptive foraging strategies of vultures. EVOA is proposed for modulation classification due to its accurate parameter measurement and good feature selection in complex conditions. Experimental results show that EVOA outperforms traditional methods and deep learning methods, improving the accuracy by up to 20% in complex modulations such as 16-QAM and 64-QAM, and reducing the processing time by ∼30%. In addition, EVOA has strong detection capabilities in unknown modulations and multi-user simultaneous operation, making it a strong candidate for use in mobile telecommunications and data transmission and eavesdropping devices, as well as in cognitive radio networks, secure communication systems, and future 6G infrastructures. This research evaluates modulation classification with methods such as deep learning architectures like fully connected networks and optimization techniques like particle swarm optimization, the original vulture optimization algorithm, and the proposed EVOA. The modulation types tested include AM, PM, PSK, and higher-order QAM variants. In this study, we also used several traditional classification methods, including support vector machines, decision trees, k-nearest neighbors (k-NN), naïve Bayes, neural networks, and random forests. Standard evaluation criteria were used to validate the effectiveness of each method.