Conversational AI Video Assistant

This research paper introduces a Conversational AI Video Assistant developed to enhance user interaction with video content through the processing of inputs, transcription of audio, analysis of scenes, and delivery of context-aware responses in near real-time. The system is equipped with Whisper for accurate audio transcription, custom object detection models built using OpenCV and TensorFlow for visual analysis, and Coqui TTS for natural-sounding audio feedback, all integrated seamlessly via a user-friendly Gradio-based interface. Extensive evaluation across multiple test videos demonstrates efficient performance, with processing times scaling linearly with video length and an average real-time factor of 0.173, confirming suitability for real-time applications. The system also exhibits robust effectiveness, achieving an overall accuracy of 0.86, precision of 0.83, recall of 0.88, and F1-score of 0.85, which reflects its reliability in delivering relevant responses. Designed for practical applications, the assistant supports diverse domains such as education—enabling interactive learning from instructional videos—accessibility, by providing audio descriptions for visually impaired users, and smart home systems, through contextual assistance. By combining multimodal processing with an intuitive interface, this Conversational AI Video Assistant provides a transformative solution for engaging with video content interactively and meaningfully.