WATSON: Real-time Head Tracking and Gesture Recognition   Principal Investigators: Louis-Philippe Morency (lmorency@csail.mit.edu) Goal: Watson can track rigid objects in real-time with 6 degrees of freedom using a tracking framework called Adaptive View-Based Appearance Model.  The tracking library can estimate the pose of the object for a long period of time with bounded drift.  Our main application is head pose estimation and gesture recognition using a USB camera or a stereo camera. Our Approach: Our approach combines an Adaptive View-based Appearance Model (AVAM) with a robust 3D view registration algorithm. AVAM is a compact and flexible representation of the object that can be used during the tracking to reduce the drift in the pose estimates. The model is acquired online during the tracking and can be adjusted according to the new pose estimates.  Relative poses between frames are computed using a hybrid registration technique which combine the robustness of ICP (Iterative Closest Point) for large movement and the precision of the normal flow constraint.  The complete system runs at 25Hz on a Pentium 4 3.2GHz. Videos: Video 1 (7.19 MB): Head pose estimation for a long period of time using the tracking system (CVPR 2003).  Video 2 (3.78 MB): Rigid object (puppet) pose tracking using the adaptive tracking system (CVPR 2003). Video 3 (4.66 MB): Mouse cursor moved using the head pose tracker (FG 2002). Video 4 (108 MB): Head tracker used with an interactive kiosk (ICMI 2004) Video 5 (11.83 MB): Head pose tracking (IUI 2006). Video 6 (45.04 MB): Evolution of the keyframes during tracking (CVIU 2006). Download: Watson tracking library is available for download (for research purpose only). This library also include a module for head gesture recognition.  To download it, please send an email to Louis-Philippe Morency (lmorency@csail.mit.edu).  You can look at the documentation of Watson:  Watson23.pdf . Related Publications: Louis-Philippe Morency, Candace Sidner, Christopher Lee, Trevor Darrell, Contextual Recognition of Head Gestures, Proceedings of the International Conference on Multimodal Interactions, 2005. Louis-Philippe Morency, Ali Rahimi, Trevor Darrell,Adaptive View-based Appearance Model, Proceedings IEEE Conf. on Computer Vision and Pattern Recognition, 2003. Louis-Philippe Morency, Trevor Darrell, Stereo Tracking using ICP and Normal Flow, Proceedings Int. Conf. on Pattern Recognition, 2002. Louis-Philippe Morency, Ali Rahimi, Neal Checka, Trevor Darrell, Fast Stereo-Based Head Tracking for Interactive Environment, Proceedings of the Int. Conference on Automatic Face and Gesture Recognition, 2002. Ali Rahimi, Louis-Philippe Morency, Trevor Darrell, Reducing Drift in Parametric Motion Tracking, Proceedings of the International Conference on Computer Vision, 2001.