iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks
- PDF / 2,429,932 Bytes
- 12 Pages / 612 x 808 pts Page_size
- 59 Downloads / 222 Views
iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks Aman Chadha1 (
), John Britto2 , and M. Mani Roja3
c The Author(s) 2020.
demonstrate that iSeeBetter offers superior VSR fidelity and surpasses state-of-the-art performance.
Abstract Recently, learning-based models have enhanced the performance of single-image superresolution (SISR). However, applying SISR successively to each video frame leads to a lack of temporal coherency. Convolutional neural networks (CNNs) outperform traditional approaches in terms of image quality metrics such as peak signal to noise ratio (PSNR) and structural similarity (SSIM). On the other hand, generative adversarial networks (GANs) offer a competitive advantage by being able to mitigate the issue of a lack of finer texture details, usually seen with CNNs when super-resolving at large upscaling factors. We present iSeeBetter, a novel GAN-based spatio-temporal approach to video super-resolution (VSR) that renders temporally consistent super-resolution videos. iSeeBetter extracts spatial and temporal information from the current and neighboring frames using the concept of recurrent back-projection networks as its generator. Furthermore, to improve the “naturality” of the superresolved output while eliminating artifacts seen with traditional algorithms, we utilize the discriminator from super-resolution generative adversarial network. Although mean squared error (MSE) as a primary loss-minimization objective improves PSNR/SSIM, these metrics may not capture fine details in the image resulting in misrepresentation of perceptual quality. To address this, we use a four-fold (MSE, perceptual, adversarial, and total-variation loss function. Our results
Keywords super resolution; video upscaling; frame recurrence; optical flow; generative adversarial networks; convolutional neural networks
1
Introduction
The goal of super-resolution (SR) is to enhance a low resolution (LR) image to a higher resolution (HR) image by filling in missing fine-grained details in the LR image. The domain of SR research can be divided into three main areas: single image SR (SISR) [1–4], multi image SR (MISR) [5, 6], and video SR (VSR) [7–11]. Consider an LR video source which consists of a sequence of LR video frames LRt-n , ..., LRt , ..., LRt+n , where we super-resolve a target frame LRt . The idea behind SISR is to super-resolve LRt by utilizing spatial information inherent in the frame, independently of other frames in the video sequence. However, this technique fails to exploit the temporal details inherent in a video sequence resulting in temporal incoherence. MISR seeks to address just that—it utilizes the missing details available from the neighboring frames LRt-n , ..., LRt , ..., LRt+n and fuses them for super-resolving LRt . After spatially aligning frames, missing details are extracted by separating differences between the aligned frames from missing details observed only in one or some of the frames. However, in MISR, the alignment of the frames is do
Data Loading...
