Grounding Spatial Relations for Human-Robot Interaction

TitleGrounding Spatial Relations for Human-Robot Interaction
Publication TypeConference Paper
Year of Publication2013
AuthorsGuadarrama, S., Riano L., Golland D., Göhring D., Jia Y., Klein D., Abbeel P., & Darrell T.
Other Numbers3614

We propose a system for human-robot interactionthat learns both models for spatial prepositions and for objectrecognition. Our system grounds the meaning of an inputsentence in terms of visual percepts coming from the robot’ssensors in order to send an appropriate command to the PR2or respond to spatial queries. To perform this grounding, thesystem recognizes the objects in the scene, determines whichspatial relations hold between those objects, and semanticallyparses the input sentence. The proposed system uses the visualand spatial information in conjunction with the semantic parseto interpret statements that refer to objects (nouns), their spatialrelationships (prepositions), and to execute commands (actions).The semantic parse is inherently compositional, allowing therobot to understand complex commands that refer to multipleobjects and relations such as: “Move the cup close to the robotto the area in front of the plate and behind the tea box”. Our


This work was partially supported by funding provided to ICSI by the U.S. Defense Advanced Research Projects Agency (DARPA) as part of Activity E within the Broad Operational Language Translation (BOLT) program. Additional support was provided by the Spanish Deparment of Science and Education (ref. EX2009-1073), including EDRF/FEDER funding. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors or originators and do not necessarily reflect the views of DARPA or of the U.S. Government.

Bibliographic Notes

Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, (IROS 2013), Tokyo Big Sight, Japan

Abbreviated Authors

S. Guadarrama, L. Riano, D. Golland, D. Goehring, Y. Jia, D. Klein, P. Abbeel, and T. Darrell

ICSI Research Group


ICSI Publication Type

Article in conference proceedings