Empowering Dysarthric Speech: Leveraging Advanced LLMs for Accurate Speech Correction and Multimodal Emotion Analysis

Dysarthria or Dysarthric speech as called is kind of a motor speech disorder which is caused by neurological damage that affects the muscles for speech production, which results in slurred, slow, or difficult-to-understand speech which has been affecting millions of people worldwide including those with conditions such as stroke, traumatic brain injury, cerebral palsy, Parkinson's disease, and multiple sclerosis, dysarthric presents a significant communication barrier, which impacts the quality of life and social interaction of individuals. The entire aim of this paper is to come up with a mechanism that can recognize and translate the speech of dysarthric users and empower their ability to communicate effectively. In this paper, we are proposing a novel approach leveraging Advanced Large Language models for accurate speech correction and multimodal emotion analysis. Our methodology involves converting dysarthric speech to text using OpenAI's whisper model, followed by fine-tuned open source models such as LlaMa 3.1 70B and Mistral 8x7B models on Groq AI accelerators to predict the intended sentences from distorted input speech accurately. The entire dataset used was made by combining TORGO dataset with Google speech data and then manually labeling emotional contexts. Our framework identifies emotions such as happiness, sadness, neutral, surprise, anger and fear highlighting the potential understanding of dysarthric speech. Our approach effectively reconsturcts the intended sentences and detects emotions with high accuracy.