How to Select and Use Tools? Active Perception of Target Objects Using Multimodal Deep Learning

Namiko Saito, Tetsuya Ogata, Satoshi Funabashi, Hiroki Mori, Shigeki Sugano

研究成果: Article査読

18 被引用数 (Scopus)


Selection of appropriate tools and use of them when performing daily tasks is a critical function for introducing robots for domestic applications. In previous studies, however, adaptability to target objects was limited, making it difficult to accordingly change tools and adjust actions. To manipulate various objects with tools, robots must both understand tool functions and recognize object characteristics to discern a tool-object-action relation. We focus on active perception using multimodal sensorimotor data while a robot interacts with objects, and allow the robot to recognize their extrinsic and intrinsic characteristics. We construct a deep neural networks (DNN) model that learns to recognize object characteristics, acquires tool-object-action relations, and generates motions for tool selection and handling. As an example tool-use situation, the robot performs an ingredients transfer task, using a turner or ladle to transfer an ingredient from a pot to a bowl. The results confirm that the robot recognizes object characteristics and servings even when the target ingredients are unknown. We also examine the contributions of images, force, and tactile data and show that learning a variety of multimodal information results in rich perception for tool use.

ジャーナルIEEE Robotics and Automation Letters
出版ステータスPublished - 2021 4月

ASJC Scopus subject areas

  • 制御およびシステム工学
  • 生体医工学
  • 人間とコンピュータの相互作用
  • 機械工学
  • コンピュータ ビジョンおよびパターン認識
  • コンピュータ サイエンスの応用
  • 制御と最適化
  • 人工知能


「How to Select and Use Tools? Active Perception of Target Objects Using Multimodal Deep Learning」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。