A unified learning-based framework for light field reconstruction from coded projections

Abstract

Light fields present a rich way to represent the 3D world by capturing the spatio-angular dimensions of the visual signal. However, the popular way of capturing light fields (LF) via a plenoptic camera presents a spatio-angular resolution trade-off. To address this issue, computational imaging techniques such as compressive light field and programmable coded aperture have been proposed, which reconstruct full sensor resolution LF from coded projections of the LF. Here, we present a unified learning framework that can reconstruct LF from a variety of multiplexing schemes with minimal number of coded images as input. We consider three light field capture schemes; heterodyne capture scheme with code placed near the sensor, coded aperture scheme with code at the camera aperture and finally the dual exposure scheme of capturing a focus-defocus pair where there is no explicit coding. Our algorithm consists of three stages; Firstly, we recover the all-in-focus image from the coded image. Secondly, we estimate the disparity maps for all the LF views from the coded image and the all-in-focus image. And finally, we render the LF by warping the all-in-focus image using the estimated disparity maps. We show that our proposed learning algorithm performs either on par with or better than the state-of-the-art methods for all the three multiplexing schemes. LF from focus-defocus pair is especially attractive as it requires no hardware modification and produces LF reconstructions that are comparable to the current state of the art learning-based view synthesis approaches from multiple images. Thus, our work paves the way for capturing full-resolution LF using conventional cameras such as DSLRs and smartphones.