Enhanced visual multi-modal fusion framework for dense video captioning

Abstract Dense video captioning is a machine translation task that aims to localize events from full video and describe them separately. Human observers who parse the video are also impressed by frames containing a high amount of information and duplication, but video often appears unrelated frames in the subject. However, existing works have largely ignored these details. To fully incorporate human visual perception into the process of understanding video, we propose an enhanced visual multi-modal fusion framework (Evmff), which utilizes the captions of video keyframes to improve dense video captioning performance. We first extract video keyframes through time stamps and then apply the recently proposed image captioning method DLCT to obtain a temporally aligned caption of the keyframe. Evmff fuses the textual information of speech, the caption of image keyframes, video features, and audio features, applying transformer architecture to convert the data into text descriptions. The performance of our model is verified using the ActivityNet Captions dataset with four different indicators, Bleu@N, METEOR, Rouge_L, and CIDEr-D. Ablation experiments show that employing the video keyframe description as the input of the multi-modal model compensates for the deficiency in visual information understanding. Our code will be released.