The Role of Token Pruning in Efficient Transformer Architectures

The rapid advancements in deep learning have led to the widespread adoption of Transformer-based models, which now power a variety of natural language processing (NLP) applications, from search engines to conversational AI. While these models deliver state-of-the-art performance, their high computational cost presents challenges for real-time inference, mobile deployment, and large-scale applications. As a result, numerous model compression techniques have been explored to enhance efficiency without compromising accuracy. Among these, token pruning has gained attention as a promising strategy that selectively removes less informative tokens during inference, reducing computational complexity while preserving model effectiveness. This survey provides a comprehensive review of token pruning methods, categorizing them into static and dynamic approaches and analyzing their underlying principles. We examine key evaluation metrics used to measure pruning effectiveness, explore its impact across various NLP tasks, and compare different pruning strategies in terms of efficiency, accuracy trade-offs, and generalization. Additionally, we highlight critical challenges, including maintaining long-range dependencies, ensuring robustness to distribution shifts, and scaling pruning techniques to large language models. Finally, we outline open research directions and discuss potential integrations with other efficiency-driven techniques, such as quantization and knowledge distillation. By consolidating recent progress in token pruning, this survey aims to serve as a valuable resource for researchers and practitioners striving to develop more efficient NLP models.