A Learnable Motion Preserving Pooling for Fine-Grained Video Classification

Using deep neural network (DNN) for video understanding tasks is expensive in terms of computation cost. Pooling layers in DNN which are widely used in most of vision tasks to resize the spatial dimensions play crucial roles in reducing the computation and memory cost . In video-related task, pooling layers are also applied, mostly in the spatial dimension only as the standard average pooling in the temporal domain can significantly reduce its performance. This is because conventional temporal pooling degrades the underlying important motion features in consecutive frames. Such phenomenon is rarely investigated and most of state-of-art methods simply do not adopt temporal pooling, leading to enormous computation cost. In this work, we propose a learnable motion-preserving pooling (MPPool) layer which is able to preserve the general motion progression after the pooling. This pooling layer firstly locates the frames with strongest motion features and then keeps these crucial features during pooling. Our experiments demonstrate that MPPool not only reduces the computation cost for video data modelling, but also increases the final prediction accuracy on various motion-centric datasets.