Vision-Touch Fusion for Predicting Grasping Stability Using Self Attention and Past Visual Images

Predicting the grasp stability before lifting an object, to be detailed, whether a gripped object will move with respect to the gripper, gives more time to modify unstable grasps compared to after-lift slip detection. Recently, deep learning relying on visual and tactile information becomes increasingly popular. However, how to combine visual and tactile data effectively is still under research. In this paper, we propose to fuse visual and tactile data by introducing self attention (SA) mechanisms for predicting grasp stability. In our experiments, we use tactile sensors (uSkin) and camera sensor (Spresense). An image of the object, not collected immediately before or during grasping, is used, as it might be more readily available. Dataset collection is done by grasping and lifting 1050 times on 35 daily objects in total with various forces and grasping positions. As a result, the predicted accuracy improves over 9% compared to previous attention-based visual-tactile fusion research. Furthermore, our analysis reveals that the introduction of self-attention mechanisms enables more effective and widespread feature extraction for both visual and tactile data.