Lip2Text: Sentence-Level Lipreading on English Speakers Using the Deep Learning Approach

Abstract Lipreading is the skill of visually analyzing lip movements and facial cues to understand spoken language. This valuable skill finds application in assisting individuals with hearing impairments and providing auditory information in noisy environments. Advancements in deep learning have revolutionized lipreading with the use of networks such as Temporal Convolutional Networks(TCNs), Gated Recurrent Units(GRUs), Recurrent Neural Networks(RNNs) and Long Short Term Memory Networks(LSTMs) which perform exceptionally well on sequential data such as lip sequences. They excel at recognizing subtle lip movements of the speaker to predict what is being spoken. Most of the current approaches perform extremely well on speakers on which the model has been trained as they learn the speaker-specific visual cues thoroughly, but the obstacle lies in dealing with unseen speakers. To overcome this challenge, we have proposed a deep learning framework based on Long Short Term Memory Networks(LSTMs) and Spatiotemporal Convolutional Neural Networks (STCNNs) which focuses on accurately mapping lip sequences to their corresponding text trained on a multi-speaker dataset with unconstrained, open vocabulary. The model uses a Wav2Lip-GFPGAN(Generative Facial Prior- Generative Adversarial Network) network which converts the input video into a video of the dataset’s speaker speaking the same utterances using lip-synchronisation and then applied LipNet model for Lip-Reading after that applied spell corrector by using python Text-Blob library. This is a novel approach in the domain of lipreading which makes our model speaker-independent, increasing the applications of this work immensely in multiple fields, including assisting hearing-impaired individuals and enhancing communication accessibility. The other application of this proposed work may be lipreading of suspicious people from CCTV footage can potentially offer some benefits in certain security and investigative contexts. However, it's important to note that lipreading is a challenging task and often relies on various factors, including video quality, lighting conditions, and the same has been tackled in this work.