Vision-Touch Fusion for Predicting Grasping Stability Using Self Attention and Past Visual Images

Gang Yan, Zhida Qin, Satoshi Funabashi, Alexander Schmitz, Tito Pradhono Tomo, Sophon Somlor, Lorenzo Jamone, Shigeki Sugano

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Predicting the grasp stability before lifting an object, to be detailed, whether a gripped object will move with respect to the gripper, gives more time to modify unstable grasps compared to after-lift slip detection. Recently, deep learning relying on visual and tactile information becomes increasingly popular. However, how to combine visual and tactile data effectively is still under research. In this paper, we propose to fuse visual and tactile data by introducing self attention (SA) mechanisms for predicting grasp stability. In our experiments, we use tactile sensors (uSkin) and camera sensor (Spresense). An image of the object, not collected immediately before or during grasping, is used, as it might be more readily available. Dataset collection is done by grasping and lifting 1050 times on 35 daily objects in total with various forces and grasping positions. As a result, the predicted accuracy improves over 9% compared to previous attention-based visual-tactile fusion research. Furthermore, our analysis reveals that the introduction of self-attention mechanisms enables more effective and widespread feature extraction for both visual and tactile data.

Original languageEnglish
Title of host publication2023 IEEE International Conference on Development and Learning, ICDL 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages339-345
Number of pages7
ISBN (Electronic)9781665470759
DOIs
Publication statusPublished - 2023
Event2023 IEEE International Conference on Development and Learning, ICDL 2023 - Macau, China
Duration: 2023 Nov 92023 Nov 11

Publication series

Name2023 IEEE International Conference on Development and Learning, ICDL 2023

Conference

Conference2023 IEEE International Conference on Development and Learning, ICDL 2023
Country/TerritoryChina
CityMacau
Period23/11/923/11/11

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Human-Computer Interaction
  • Control and Optimization

Fingerprint

Dive into the research topics of 'Vision-Touch Fusion for Predicting Grasping Stability Using Self Attention and Past Visual Images'. Together they form a unique fingerprint.

Cite this