CNN-Based Bi-Directional Motion Compensation for High Efficiency Video Coding

The state-of-the-art High Efficiency Video Coding (HEVC) standard adopts the bi-prediction to improve the coding efficiency for B frame. However, the underlying assumption of this technique is that the motion field is characterized by the block-wise translational motion model, which may not be efficient in the challenging scenarios such as rotation and deformation. Inspired by the excellent signal level prediction capability of deep learning, we propose a bi-directional motion compensation algorithm with convolutional neural network, which is further incorporated into the video coding pipeline to improve the performance of video compression. Our network consists of six convolutional layers and a skip connection, which integrates the prediction error detection and non-linear signal prediction into an end-to-end framework. Experimental results show that by incorporating the proposed scheme into HEVC, up to 10.5% BD-rate savings and 3.1% BD-rate savings on average for random access (RA) configuration have been observed.