A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects

Anthony Simeonov   Yilun Du   Beomjoon Kim   Francois R. Hogan  

Joshua Tenenbaum   Pulkit Agrawal   Alberto Rodriguez


CoRL 2020


We present a framework for solving long-horizon planning problems involving manipulation of rigid objects that operates directly from a point-cloud observation, i.e. without prior object models. Our method plans in the space of object subgoals and frees the planner from reasoning about robot-object interaction dynamics by relying on a set of generalizable manipulation primitives. We show that for rigid bodies, this abstraction can be realized using low-level manipulation skills that maintain sticking contact with the object and represent subgoals as 3D transformations. To enable generalization to unseen objects and improve planning performance, we propose a novel way of representing subgoals for rigid-body manipulation and a graph-attention based neural network architecture for processing point-cloud inputs. We experimentally validate these choices using simulated and real-world experiments on the YuMi robot. Results demonstrate that our method can successfully manipulate new objects into target configurations requiring long-term planning. Overall, our framework realizes the best of the worlds of task-and-motion planning (TAMP) and learning-based approaches.


Latest version (Nov 16, 2020): arXiv:2011.08177


Anthony Simeonov, Yilun Du, Beomjoon Kim, Francois R. Hogan, Joshua Tenenbaum, Pulkit Agrawal, and Alberto Rodriguez. A Long Horizon Planning Framework for Manipulating Rigid Pointcloud Objects. Conference on Robot Learning (CoRL) 2020


Project Summary Video

CoRL Presentation

CoRL Supplementary Video

Single-step Execution and Multi-step Planning (Real World)

Multi-step Planning with Cuboids (Simulation)

Multiple Placement Surfaces and Geometry Generalization (Simulation)