Projects | Computational Imaging Lab

A Deep Learning Framework for Compressive Light Field Reconstruction

In proceedings of ACPR 2017 [paper] [code soon]

Anil Kumar Vadathya, Saikiran Cholleti, Gautham Ramajayam, Vijaylakshmi Kanchana, Kaushik Mitra

learning-light-field-reconstruction-computational-imaging-lab-iit-madras

Figure 1: We propose a learning based solution for full sensor resolution light field reconstruction from a single coded image. Our approach consists of three convolutional neural networks (CNNs); ViewNet reconstructs center view from the coded image, DispNet estimates disparity map, WarpNet generates light field by warping center view with the disparity. As can be seen from EPI images, our approach recovers parallax very well. Generated EPI is shown with dashed line border and ground truth with thick line.

Abstract

Light field imaging is a rich way of representing the 3D world around us. However, due to limited sensor resolution capturing light field data inherently poses spatio-angular resolution trade-off. In this paper, we propose a deep learning based solution to tackle the resolution trade-off. Specifically, we reconstruct full sensor resolution light field from a single coded image. We propose to do this in three stages 1) reconstruction of center view from the coded image 2) estimating disparity map from the coded image and center view 3) warping center view using the disparity to generate light field. We propose three neural networks for these stages. Our disparity estimation network is trained in an unsupervised manner alleviating the need for ground truth disparity. Our results demonstrate better recovery of parallax from the coded image. Also, we get better results than dictionary learning approaches on simulated data.

Our proposed architecture

Figure 2: ViewNet and DispNet; figure shows the convolutional architecture details of both the networks. ViewNet is a convolutional deconvolutional architecture with three symmteric skip connections. It is shown compactly and can be easily inferred from Mao et al. 2016. DispNet is an encoder decoder (bottom) with a skip connection (top). The `sk' specified below the block implies stride k convolution. Each block shows the filter size (c) and number of output channels (h) as c x c, h.

Results

Full sensor resolution light field reconstruction from a single coded image

With the reconstructed scene from ViewNet and disparity from DispNet, we warp the scene using disparity to reconstruct full resolution light field.

Figure below shows the reconstructed light field (7x7 angular resolution) for different scenes. The left image shows the estimated disparity, middle image shows the reconstructed centerview along with EPI images. The right image shows the animation of full sensor resolution light field after warping and refining.

DispNet: geometry estimation from the coded image and the reconstructed center view

We learn to estimate scene geometry in an unsupervised manner. Ideally training dispnet would need ground-truth(GT) disparity maps. Instead we follow the recent approaches of unsupervised geometry estimation by Garg et al. 2016, Godard et al. 2017, where the photo-consistency loss between stereo pairs is used to learn the disparity maps. Thus, alleviating the need for GT disparity.

Disparity map comparison: Below figure compares our estimated disparities from the coded image and reconstructed center view with that of Jeon et al. CVPR 2015, which estimates depth from ground truth light field. Note that our estimation is without any explicit regularization unlike Jeon et al. 2015.

Our reconstructed center view	Our estimated disparity from a single coded image	Jeon et al. disparity from GT light field	Our reconstructed center view	Our estimated disparity from a single coded image	Jeon et al. disparity from GT light field

Light field reconstruction comparison with dictionary learning

Below figure shows our reconstructed center and EPIs of light field (5x5 angular resolution). For the patches indicated we show the corresponding reconstruction from dictionary learning method of Marwah et al. 2012. Comparisons show that our reconstructions are sharp and parallax is also reconstructed better as indicated by PSNR and SSIM values.

Light field reconstruction comparison with view synthesis of Kalantari et al.

Kalantari et al. 2016 reconstruct 8x8 light field from corner 2x2 views. Here we compare our reconstructions with them. Note that our task is much difficult than theirs given we reconstruct from one single coded image whereas Kalantari et al. have four corner views. Also, information around occlusion edges is explicitly available to them unlike ours. We compare our 7x7 LF with corresponding reconstruction from Kalantari et al. after removing the views that go as input. Our method produces results close to Kalantari's except at the occlusion edges.

Our light field reconstruction from a single coded image. PSNR: 31.88, SSIM: 0.953.

Kalantari et al. LF reconstruction from four corner views. PSNR: 32.65, SSIM: 0.971. Note that to calculate the PSNR and SSIM we perform the same preprocessing here to that of Kalantari et al. to account for contrast differences.

Refocusing with the reconstructed light field

An application of light field is post-capture refocusing. Here we demonstrate the refocusing application with the light field reconstructed from our approach. Hover over the image to see the estimated disparity maps. Note that refocusing is done by shifting and adding of light field images and not the depth based blurring.

ViewNet: scene reconstruction from the input coded image

We need to reconstruct scene for warping based light field synthesis. We use a CNN to reconstruct center view of the light field from the coded image. Scene reconstruction from the coded image can be seen as image restoration problem. We use skip connection based architecture proposed by Mao et al. 2016 which yielded state-of-the-art results in image denoising and super-resolution. In figure beside you can notice the recovery of sharp details (text over the glass and pattern edges below) on using the symmetric skip connections.

Acknowledgments

This project is supported by Qualcomm Innovation Fellowship, India 2016.

Publications

Anil Kumar Vadathya, Saikiran Cholleti, Gautham Ramajayam, Vijaylakshmi Kanchana, Kaushik Mitra, "Learning Light Field Reconstruction from a Single Coded Image", accepted at ACPR, 2017.