Statistical shape-and-texture appearance models (also known as deformable models) [1,4] use image morphing to define a rich, compact representation of object appearance. Deformable models are useful in a variety of applications including object recognition, tracking and segmentation [4, 5, 6]. These techniques, however, have been limited to object classes that exhibit relatively simple appearance variation and as such they cannot be applied to many interesting object classes such as hands or mouths. The appearance manifold of such object classes is highly nonlinear and their appearance cannot be linearized into shape and texture by applying a global morphing operator. To see this consider morphing an open and closed mouth---the intermediate images of the morph would contain holes and folds caused by folding the open mouth and stretching the open mouth. Instead these objects have an apperarance manifold with a varying topology, with different regions of the space correlated with the same parts of the mouth or hand being visible and holes in the manifold corresponding to inadmissible appearance, i.e., that caused by interpolating the appearance of an open and closed mouth.

In addition to having a complex appearance manifold, these object classes also have shape vectors with varying dimensionality determined by what parts of the object are visible in the image. For example, the image of an open mouth contains shape features for the teeth that are absent from the image of a closed mouth. In this work we propose two complementary, nonlinear deformable models that can model object classes whose appearance manifold has a varying topology and whose shape has a varying dimensionality. To the author's knowledge, this is the first work to describe a nonlinear model of shape and appearance. We demonstrate our models using mouth images from speaking person video sequences and compare them to a baseline linear model. An extensive discussion of this work can be found in the publication below. Due to space constraints, additional results not in the publication are available below.

Additional Results