Detecting deepfakes using emotional irregularities

Recent advances in deep learning have spawned a new class of media forgeries known as deepfakes, which typically consist of artificially generated human faces or voices. The creation and distribution of deepfakes raises many legal and ethical concerns. As a result, the ability to distinguish between deepfakes and authentic media is vital. Since deepfake generation methods are numerous and rapidly evolving, it is unrealistic to hope that a single method will be able to detect all deepfakes. Rather, a multiplicity of detection methods is needed, so that the burden of creating an undetectable deepfake becomes extremely high. While deepfakes can create plausible video and audio, it may be challenging for them to accurately synthesize high-level concepts such as emotion. Unnatural displays of emotion, as measured by valence and arousal, can provide important evidence that a video has been synthesized. In this work, we propose a novel method for detecting deepfakes of a human speaker using the emotion predicted from the face and voice. First, we train long short-term memory (LSTM) networks to predict emotion from low-level descriptors (LLDs) of audio and video of human speakers from the SEMAINE database. Then, we use the trained networks as feature extractors to extract sequences of valence and arousal from videos in a subset of the Deepfake Detection Challenge (DFDC) dataset. Finally, we classify the videos as authentic videos or deepfakes with high accuracy using features derived from the predicted emotion.