Token-Centric Representations in Large Language Models: Analyzing Llama and Mistral Through the Lens of Rate-Distortion Theory

Token-centric representations play a crucial role in how language models understand and generate human language, influencing the accuracy and efficiency of various downstream tasks. The novel application of rate-distortion theory to analyze token representations offers a significant contribution to the understanding of how compression impacts the retention of linguistic information within language models. Through a systematic evaluation of two prominent models, Llama and Mistral, the study provides a detailed examination of the trade-offs between token compression and representational fidelity, revealing distinct patterns in their respective tokenization strategies. Experimental results demonstrated that Llama maintains a higher level of accuracy under increasing compression rates compared to Mistral, indicating a more robust tokenization approach. The analysis further demonstrates the importance of optimizing token embeddings to achieve scalable and adaptable models capable of performing a wide range of language processing tasks. The findings contribute to the broader discourse on model efficiency, offering a framework for the development of future models that balance complexity and performance effectively.