Current Projects

Micro-Air Vehicle Navigation and Control

In environments where GPS is noisy and maps are unavailable, such as indoors or in dense urban environments, a UAV (unmanned air vehicle) runs the risk of becoming lost, operating in high threat regions, or colliding with obstacles. The size, weight and budget limitations of micro air vehicles (MAVs) typically preclude high-precision inertial navigation units that can mitigate the loss of GPS. We are developing estimation and planning algorithms that allow MAVs to use environmental sensors such as laser range finders or cameras to estimate their position, build maps of the environment and fly safely and robustly.

For more details, see the Micro-Air Vehicle Navigation and Control page.

  • A. Bry, C. Richter, A. Bachrach and N. Roy. "Aggressive Flight of Fixed-Wing and Quadrotor Aircraft in Dense Indoor Environments". International Journal of Robotics Research, 37(7):969-1002, June 2015.
    [PDF] [Bibtex Entry]
  • Charles Richter, Adam Bry, Nicholas Roy. (2013). "Polynomial Trajectory Planning for Aggressive Quadrotor Flight in Dense Indoor Environments." International Symposium of Robotics Research (ISRR), Singapore, 2013.
    [PDF] [BiBTeX Entry]
  • Adam Bry, Abraham Bachrach and Nicholas Roy. "State Estimation for Aggressive Flight in GPS-Denied Environments Using Onboard Sensing". Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). St Paul, MN 2012. (Nominated, best conference paper.).
    [PDF] [BiBTeX Entry]
  • A. Bachrach, S. Prentice, R. He, and N. Roy. "RANGE - Robust Autonomous Navigation in GPS-denied Environments". Journal of Field Robotics. 28(5):646-666, September 2011.
    [Compressed postscript] [PDF] [Bibtex Entry]
  • A. Bachrach, R. He, N. Roy. "Autonomous Flight in Unknown Indoor Environments". International Journal of Micro Air Vehicles, 1(4): 217-228, December 2009.
  • Ruijie He, Sam Prentice and Nicholas Roy. "Planning in Information Space for a Quadrotor Helicopter in a GPS-denied Environments''. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA 2008). Los Angeles, 2008.
  • Abraham Bachrach, Alborz Garamifard, Daniel Gurdan, Ruijie He, Sam Prentice, Jan Stumpf and Nicholas Roy. "Co-ordinated Tracking and Planning using Air and Ground Vehicles''. In Proceedings of the International Symposium on Experimental Robotics (ISER), Athens, 2008.

Natural Language Interaction for Manipulation and Exploration Tasks

Advances in robot autonomy have moved humans to a different level of interaction, where the ultimate success hinges on how effectively and intuitively humans and robots can work together to correctly accomplish the task. A key requirement for robots operating alongside humans, particularly in manufacturing, warehouse, and search and rescue tasks, is the ability to infer situational awareness and goal information embedded in natural language instructions.

Inferring the grounding for a language description in the context of the agentís perceived world representation is computationally challenging due to the rich diversity of human language, tasks and environments. We have developed probabilistic models like Generalized Grounding Graphs and Distributed Correspondence Graphs to infer a grounding for language descriptions in the context of the agentís perceived representation.

Our immediate research direction is focused on expanding the scope of state of the art models for understanding abstract spatial concepts necessary to carry out manipulation, navigation and planning tasks. These include references to aspects of the workspace at multiple spatial resolution: collections, parts or sequences satisfying a particular semantic property. Further, we investigate language understanding for specifying high-level tasks like search and rescue operations. Here, language understanding can guide planning and state-action space exploration for task execution.

Wind Field Estimation and Planning

A wind field estimate across the MIT campus.

With unmanned aerial vehicles (UAVs) becoming more prolific and capable, and regulations evolving, their eventual operation in urban environments seems all but certain. As UAVs begin to fly in these environments, they will be presented with a host of unique challenges. One of these challenges will be the complex wind fields generated by urban structures and terrain. Although much effort has been directed towards developing planning and estimation strategies for wind fields at high altitudes or in large open spaces, these approaches contain an implicit assumption that the wind field evolves over relatively large temporal and spatial scales. Given this simplification, a history of local measurements can be used to estimate the global wind field with sufficient accuracy. However, urban wind fields are highly variant in both space and time and are therefore resistant to this estimation method and require an approach that models the complex interaction between the flow and surrounding environment.

Our approach is to use prevailing wind estimates from local weather stations and a 3D model of the surrounding environment as inputs to a computational fluid dynamics solver to obtain both steady and unsteady wind field estimates. Unlike many approaches, these wind field estimates account for the strong coupling between the wind flow and nearby structures. Once obtained, these wind field estimates can be used to find minimum-energy trajectories between points of interest. Further work hopes to leverage a library of precomputed wind fields to find a wind field covariance estimate within a region. This uncertainty estimate could be used to infer a global wind field from local measurements, or predict future wind conditions.

Past Projects

Understanding Natural Language Commands

Our system understands commands
such as "Pick up the tire pallet off
the truck and set it down."

Natural language is an intuitive and flexible modality for human-robot interaction. A robot designed to interact naturally with humans must be able to understand instructions without requiring the person to speak in any special way. We are building systems that robustly understand natural language commands produced by untrained users. We have applied our work to understanding spatial language commands for a robotic wheelchair, a robotic forklift, as well as a micro-air vehicle. More information is at

  • Thomas Kollar, Stefanie Tellex, Deb Roy and Nicholas Roy. "Grounding Verbs of Motion in Natural Language Commands to Robots", International Symposium on Experimental Robotics (ISER), New Delhi, India, Dec. 2010. [PDF]
  • Stefanie Tellex, Thomas Kollar, George Shaw, Nicholas Roy, and Deb Roy. "Grounding Spatial Language for Video Search," Proceedings of the Twelfth International Conference on Multimodal Interfaces (ICMI), 2010. (Winner, Best Student Paper award.) [PDF]
  • Thomas Kollar, Stefanie Tellex, Deb Roy and Nick Roy, "Toward understanding natural language directions," Human-Robot Interaction 2010. [PDF]

Human-Robot Interaction for Assistive Robots

We are developing planning and learning algorithms that can be used to optimize awheelchair dialogue manager for a human-robot interaction system. The long-term goal of this research is to develop intelligent assistive technology, such as a robotic wheelchair, that can be used easily by an untrained population. We are working with the residents and staff of The Boston Home, a specialized care residence for adults with advanced multiplesclerosis and other progressive neurological diseases, to develop an intelligent interface to the residents' wheelchairs. An adaptive, intelligent dialogue manager will be essential for allowing a diverse population with a variety of physical and communication impairments to interact with the system.

  • F. Doshi and N. Roy. "Spoken Language Interaction with Model Uncertainty: An Adaptive Human-Robot Interaction System''. Connection Science, To appear.
  • Finale Doshi and Nicholas Roy. "The Permutable POMDP: Fast Solutions to POMDPs for Preference Elicitation''. Proceedings of the Seventh International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2008). Estoril, Portugal, 2008.

Planning Under Uncertainty

Continuous-state POMDPs provide a natural representation for a variety of tasks, including many in robotics. However, existing continuous-state POMDP approaches are limited by their reliance on a single linear model to represent the world dynamics. We have developed new switching-state (hybrid) dynamics models that can represent multi-modal state-dependent dynamics, and a new point-based POMDP planning algorithm for solving continuous-state POMDPs using this dynamics model. Additionally, POMDPs have succeeded in many planning domains because they can optimally trade between actions that increase an agent's knowledge and actions that increase an agent's reward. Unfortunately, most real-world POMDPs are defined with a large number of parameters which are difficult to specify from domain knowledge alone.

We have shown that the POMDP model parameters can be incorporated as additional hidden states in a larger 'model-uncertainty' POMDP, and we have developed an approximate algorithm for planning in the induced `model-uncertainty' POMDP. This approximation, coupled with model-directed queries, allows the planner to actively learn the true underlying POMDP and the accompanying policy.

  • E. Brunskill, L. Kaelbling, T. Lozano-Perez and Nicholas Roy. "Continuous-State POMDPs with Hybrid Dynamics''. Proceedings of the Tenth International Symposium on Artificial Intelligence and Mathematics, Fort Lauderdale, FL, 2008.
  • F. Doshi, J. Pineau and N. Roy. "Bayes Risk for Active Learning in POMDPs''. Proceedings of the International Conference on Machine Learning (ICML), Helsinki, Finland, 2008, pp. 256-263.


Mapping as a research problem has received considerable attention in robotics recently. Mature mapping techniques now allow practitioners to reliably and consistently generate 2-D and 3-D maps of objects, office buildings, city blocks and metropolitan areas with a comparatively small number of errors. Nevertheless, the ease of construction and quality of map are strongly dependent on the exploration strategy used to acquire sensor data. We have shown that reinforcement learning can be used to optimize the trajectory of a vehicle exploring an unknown environment. One of the primary technical challenges of exploration is being able to predict the value of different sensing strategies efficiently. We have shown that a robot can learn the effect of sensing strategies from past experience using kernel-based regression techniques. The local regression model can then be used inside a global planner to optimize a trajectory. We have demonstrated this technique both for a mobile robot building a map of an unknown environment, and an airborne mobile sensor collecting data for weather prediction.

  • T. Kollar and N. Roy. "Trajectory Optimization using Reinforcement Learning for Map Exploration''. International Journal of Robotics Research, 27(2): 175-197, 2008.
  • N. Roy, H. Choi, D. Gombos, J. Hansen, J. How and S. Park. "Adaptive Observation Strategies for Forecast Error Minimization''. Proceedings of the International Conference on Computational Science, Beijing, 2007.

Mobile Manipulation

Robot manipulators largely rely on complete knowledge of object geometry in order to plan their motion and compute successful grasps. If an object is fully in view, the object geometry can be inferred from sensor data and a grasp computed directly. If the object is occluded by other entities in the environment, manipulation based on the visible part of the object may fail; therefore, to compensate, object recognition is often used to identify the location of the object and compute the grasp from a prior model. We are developing algorithms for geometric inference and manipulation planning that allow grasp plans to be computed with only partial information about the objects in the environment and their geometry. We are developing these ideas both for small-object manipulation in the home, and large-object supply-chain manipulation.

  • J. Glover, D. Rus and N. Roy. "Manipulation using Probabilistic Models of Object Geometry''. Proceedings of Robotics: Science and Systems (R:SS), 2008.