Direction-Guided Two-Stream Convolutional Neural Networks for Skeleton-Based Action Recognition

Abstract In skeleton-based action recognition, the approach of treating skeleton data as pseudo-images using convolutional neural networks (CNN) has proven to be effective. However, among the existing CNN-based approaches, most of them focus on modeling information at the joint-level ignoring the size and direction information of the skeleton edges, which play an important role in action recognition, and these approaches may not be optimal. In addition, combining the directionality of human motion to portray the motion variations information of the action, which is more natural and reasonable for action sequence modeling, is rarely considered in existing approaches. In this work, we propose a novel direction-guided two-stream convolutional neural networks (DG-2sCNN) for skeleton-based action recognition. On the first stream, our model focuses on our defined edge-level information (including edge and edge\_motion information) with directionality in the skeleton data to explore the spatio-temporal features of the action. On the second stream, since the motion is directional, we define different skeleton edge directions and extract different motion information (including translation and rotation information) in different directions in order to better exploit the motion features of the action. Besides, we propose the description of human motion inscribed by a combination of translation and rotation, and explore the way they are integrated. We conducted extensive experiments on two challenging datasets, NTU-RGB+D 60 and NTU-RGB+D 120, to verify the superiority of our proposed method over state-of-the-art methods. The experimental results demonstrate that the proposed direction-guided edge-level information and motion information complement each other for better action recognition.