| PhD Viva

Name of the Speaker: Maitreya Suin (EE17D201)
Guide: Dr. Rajagopalan A. N
Online meeting link: meet.google.com/uwb-scka-gnn
Date/Time: 19-09-2022, 3:00 PM
Title: Pixel Adaptive Neural Network Design for Image and Video Restoration

Abstract

Images and videos captured in our day-to-day life are often degraded due to various inevitable elements such as blur, rain, haze, etc. The majority of such visual degradations are spatially varying. Existing restoration algorithms aim to recover the original clean information using input-agnostic and spatially-invariant processing in purely convolutional architectures. They fail to adequately address the inherent spatial and temporal variations observed in degraded inputs. Our work presents pixel-adaptive neural networks for various restoration tasks, where the network can adjust its behavior depending on the pixel content. Blur is a common phenomenon when capturing a dynamic scene with handheld devices. We design a pixel adaptive and feature attentive image deblurring network that can handle large blur variations across different spatial locations and different images. Our content-aware global-local filtering module considers not only the global dependencies but also the neighboring information dynamically for each pixel. We extend this framework to the spatio-temporal domain to handle blurry videos. We adaptively extract the additional information from the temporal axis to refine each frame using non-local attention. Our work is more efficient than existing designs while outperforming them by a large margin.

Subsequently, we address removing bad-weather-caused degradations such as rain streaks, haze, and raindrops from images. We delve deeper into the limitations of the pixel-adaptive modules and present a two-stage neural framework. We decompose the restoration task into two stages of degradation localization and degraded region-guided restoration. The additional knowledge of the degraded regions helps the network focus on restoring the most difficult regions. We demonstrate that this knowledge transfer technique can be equally beneficial for the highly ill-posed image inpainting task. We design a distillation-based approach, where we provide direct feature-level supervision while training. This additional guidance helps the network to reach a much better optima and produce superior results. We conduct extensive evaluations on multiple datasets to demonstrate the superiority of our method for each task separately.