Making Foundation Models Adaptable: A Review of Parameter-Efficient Fine-Tuning Approaches

Foundation models have demonstrated remarkable capabilities across a wide range of tasks, but their immense size poses significant challenges for fine-tuning. Traditional full fine-tuning approaches require substantial computational resources and storage, making them impractical for many real-world applications. Parameter-efficient fine-tuning (PEFT) methods have emerged as a promising alternative, allowing for efficient adaptation of large models with a minimal number of trainable parameters. This survey provides a comprehensive overview of PEFT techniques, categorizing them into four primary approaches: adapter-based tuning, low-rank adaptation (LoRA), prefix tuning, and prompt tuning. We analyze the theoretical foundations, practical implementations, and empirical performance of these methods, highlighting their advantages and limitations. Additionally, we discuss the deployment considerations for PEFT models, including computational efficiency, scalability, and integration with existing model architectures. Furthermore, we identify open challenges in PEFT, such as robustness across diverse domains, mitigating catastrophic forgetting, privacy-preserving adaptation, and the need for standardized benchmarking. We explore future research directions that could enhance the efficiency, interpretability, and generalization of PEFT methods. By enabling efficient fine-tuning with reduced computational overhead, PEFT is poised to play a crucial role in democratizing access to powerful AI models. This survey aims to serve as a valuable resource for researchers and practitioners interested in leveraging PEFT for scalable and cost-effective model adaptation.