Hardware Accelerators for Softmax in Large Language Models: A Survey

Transformers underpin today's large language models (LLMs): stacked self-attention blocks contextualize input token embeddings and a final projection produces next-token probabilities. Softmax is central to both stages, turning attention scores into weights and converting output logits into a normalized distribution. As sequence lengths and vocabularies grow, Softmax becomes a bottleneck: it couples expensive exponentials and division with reductions over long vectors, demands multiple passes, drives memory traffic and synchronization, and is sensitive to numeric range in low-precision hardware. These characteristics make Softmax increasingly memory-and latency-bound in modern deployments. This survey reviews hardware strategies to mitigate the Softmax bottleneck, organizing prior work into six categories: (1) mathematical reformulations and approximations, (2) data representation and precision, (3) microarchitectural optimization, (4) efficiency-oriented algorithms, (5) attention-and LLM-level adaptations, and (6) alternative normalization functions that replace exponentials or division. A comparison of reported accuracy, latency/throughput, power, and area is performed. This survey equips readers with a structured understanding of the Softmax acceleration landscape, highlighting practical trade-offs across accuracy, latency, power, and area to guide future hardware design for Softmax in LLMs.