Summarizing Smarter: RAG vs Vanilla in Modern Transformers An Empirical Study on Retrieval-Augmented Generation Methods

Recent advances in abstractive summarization have predominantly relied on pre-trained transformer models, which excel at generating fluent summaries but often fail to handle long or fact-heavy contexts effectively. To address this limitation, we explore the use of Retrieval-Augmented Generation (RAG), which incorporates external retrieval of relevant information to enhance model performance. In this study, we evaluate the impact of RAG on three popular transformer architectures: BART, T5, and FLAN-T5, in both their vanilla-augmented and RAG-augmented variants. We evaluated the models on a custom dataset of longform question-answer pairs using ROUGE-1, ROUGE-2, and ROUGE-L metrics. Our results indicate that BART benefits from RAG, showing consistent improvements in ROUGE scores, while T5 and FLAN-T5 show no significant gains from RAG. These findings underscore the importance of the model architecture and task-specific context in determining the effectiveness of retrieval augmentation. We conclude that RAG can enhance summarization performance in certain models but is not universally beneficial, highlighting the necessity for careful consideration of model design when applying retrieval techniques.