Image and Depth from a Conventional Camera with a Coded Aperture

Anat Levin,  Rob Fergus,  Fredo Durand,  Bill Freeman

Abstract
A conventional camera captures blurred versions of scene information away from the plane of focus. Camera systems have been proposed that allow for recording all-focus images, or for extracting depth, but to record both simultaneously has required more extensive hardware and reduced spatial resolution. We propose a simple modi_cation to a conventional camera that allows for the simultaneous recovery of both (a) high resolution image information and (b) depth information adequate for semi-automatic extraction of a layered depth representation of the image.
Our modification is to insert a patterned occluder within the aperture of the camera lens, creating a coded aperture. We introduce a criterion for depth discriminability which we use to design the preferred aperture pattern. Using a statistical model of images, we can recover both depth information and an all-focus image from single photographs taken with the modified camera. A layered depth map is then extracted, requiring user-drawn strokes to clarify layer assignments in some cases. The resulting sharp image and layered depth map can be combined for various photographic applications, including automatic scene segmentation, post-exposure refocussing, or re-rendering of the scene from an alternate viewpoint.

  • The Paper (PDF) , appeared in SIGGRAPH 2007.
  • PowerPoint slides from our SIGGRAPH talk.
  • Supplementary file with hi-resolution results (PDF)
  • Video demonstrating novel view synthesis produced using the estimated depth map.
  • Deconvolution using natural image prior- this package (last updated: Sep 27, 2007) contains an implementation of the (non blind) debluring algorithm used in the paper, relying on a natural image prior. This deblurring algorithm significantly outperforms the traditional Richardson-Lucy deconvolution algorithm. A document describing the algorithm in detail can be download here.
  • Data - this package contains 16bit images captured by our coded aperture camera and our calibrated kernel (at different scales). It also includes our all focused images at full resolution.


Conventional Aperture

Coded Aperture
Aperture pattern
Image of a point light source



Input (Single Image)

Estimated Depth



Input Image (zoomed)

Focused Image (zoomed)