Learning Spatio-Temporal Resolutions for Deep Video Compression

We propose a spatio-temporal adaptive deep video compression scheme, which is capable of intelligently adjusting the spatial resolution and temporal frame rate for content adaptive compression, with the aim of pursuing enhanced rate-distortion performance. In particular, a neural network-based spatio-temporal adaptation network is integrated into the deep video coding paradigm, enabling the adaptive determination of the optimal rescaling ratios for compression, leading to the further reduction of spatial and temporal redundancies. Moreover, learning-based modules for rescaling parameter determination are incorporated into the spatio-temporal adaptation network. The proposed scheme can be easily plugged into, and seamlessly collaborate with the existing deep video coding frameworks. Experimental results demonstrate that, compared to the original neural video codecs, the proposed method achieves significant bitrate savings in terms of both PSNR and MS-SSIM.