Fine-tuning Whisper model for end-to-end Kannada to English speech transcription

Abstract Building reliable automatic speech recognition (ASR) systems for low-resource languages such as Kannada is challenging due to the scarcity of training data and the cross-lingual translation requirements. This study investigates fine-tuning Whisper models – Tiny, Small, and Medium – to achieve end-to-end Kannada-to-English speech transcription. Using a dataset of Kannada speech, the multilingual Whisper models are fine-tuned to improve transcription and translation performance. The accuracy of the fine-tuned models was evaluated using word error rate (WER) and character error rate (CER), revealing clear improvements across the different model sizes. Based on the analysis of the experimental results, the Whisper-Medium model was found to perform the best, with significant reductions in WER and CER compared to the zero-shot baselines. The results highlight the feasibility of lightweight and midsize Whisper architectures for cross-lingual ASR in low-resource Indian languages and indicate that these models are more suitable for real-world applications in multilingual speech systems.