Light-Field Appearance Manifolds
 
Principal Investigators:
Abstract:
Statistical shape and texture appearance models are powerful image representations, but previously had been restricted to 2D or 3D shapes with smooth surfaces and Lambertian reflectance. In this work we present a novel 3D appearance model using image-based rendering techniques, which can represent complex lighting conditions, structures, and surfaces. We construct a light field manifold capturing the multi-view appearance of an object class and extend the direct search algorithm of Cootes and Taylor to match new light fields or 2D images of an object to a point on this manifold. When matching to a 2D image the reconstructed light field can be used to render unseen views of the object. Our technique differs from previous view-based active appearance models in that model coefficients between views are explicitly linked, and that we do not model any pose variation within the shape model at a single view. It overcomes the limitations of polygonal based appearance models and uses light fields that are acquired in real-time..
Light Field Appearance Manifolds:
A light field [3,5] is a structured set of images that capture a static object or scene from a range of viewing directions. A new view of the object is synthesized from the light field using light field rendering, an efficient image-based rendering algorithm that does not require knowledge of scene geometry. Light fields can model the appearance of objects with non-Lambertian surface reflectance and complex geometry. A light field appearance manifold is defined by the set of light fields of an object class and is a surface in the space of light fields. In this work we present a method for representing light field appearance manifolds using Principle Component Analysis (PCA). More generally, we develop a shape-and-texture appearance model [1,4] parameterized over light fields of objects. A light field appearance manifold of the human face, modelled using our algorithm, is displayed below. Light field appearance models define a powerful technique for representing the appearance of objects with complex surface reflectance and geometry.
 
Light-Field Appearance Manifold
Figure 1.   Light field appearance manifold of the human face. Provided an image of a previously unseen object with unknown pose, our algorithm matches the object to a point on the manifold that best approximates the object's appearance in the input view. A full light field of the object is recovered.
 
Model Matching and Evaluation:
Provided an image of a previously unseen object with unknown pose, we would like to match the object to a point on the appearance manifold that best approximates the object's appearance in the input view. With our algorithm, the object's pose is estimated and example light fields are combined to recover a full light field of the input object. Synthetic light fields, generated from the light field appearance manifold of Figure 1 are displayed by Figure 2. In the figure, select views of the 6 x 8 light fields are shown. Note our method is able to infer 48 views of the human face provided a single view.
 
Matching to 2D images with unknown pose.
Figure 2.   Matching a light field appearance model to 2D images of novel subjects with unknown pose. The reconstructed light field, is rendered over the input view and is displayed aside each match. The light field appearance model generates convincing light field reconstructions from 2D images. In particular, the overall shape and texture of each subject are well approximated across each view.
 
We compare our algorithm to a competing approach in Figure 3. The View-based Active Appearance Model [2] defines a piecewise linear representation of the multi-view appearance of an object. It is defined by fitting local Active Appearance Models (AAMs) over regions of pose space. Since the View-based AAM blends several poses at a single model, it is difficult to represent objects with non-Lambertian surface reflectance and complex geometry with this approach. This point is illustrated by Figure 3. The View-based AAM is unable to represent the specular reflection in the subject's glasses and is adversely affected by the interpose occlusion of the glasses. The light field appearance model can faithfully represent the appearance of the subject's glasses.
 
Comparison to View-Based AAM.
Figure 3.   Comparison of a light field appearance model to a View-based AAM. The left column shows the input, the middle column the best fit with a 2D AAM, and the right column the light field fit. The View-based AAM and light field appearance models both exhibit qualitatively good fits when the surface is approximately smooth and lambertian.When glasses are present, however, the 2D method fails and the light field appearance model succeeds.
 
Publication:
 
C. Mario Christoudias, Louis-Philippe Morency, Trevor Darrell, Light Field Appearance Manifolds, Proceedings of the European Conference on Computer Vision, 2004.
 
References:

  1. T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearance models,” Lecture Notes in Computer Science, vol. 1407, pp. 484–98, 1998.

  2. T. F. Cootes, G. V. Wheeler, K. N. Walker, and C. J. Taylor, “View-based active appearance models,” Image and Vision Computing, vol. 20, pp. 657–664, 2002.

  3. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The lumigraph,” Computer Graphics, vol. 30, no. Annual Conference Series, pp. 43–54, 1996.

  4. M. J. Jones and T. Poggio, “Multidimensional morphable models,” in ICCV, 1998, pp. 683–688.

  5. M. Levoy and P. Hanrahan, “Light field rendering,” Computer Graphics, vol. 30, pp. 31–42, 1996.