Tactile Object Property Recognition Using Geometrical Graph Edge Features and Multi-Thread Graph Convolutional Network

Performing dexterous tasks with a multi fingered robotic hand remains challenging. Tactile sensors provide touch states and object features for multifingered tasks, yet the variety in shapes, sizes, textures, deformabilities and masses of everyday objects makes the task conditions diverse. Despite these challenges, humans accomplish these difficult tasks by producing a sensory motor representation of their body. This combined tactile and proprioceptive representation enables humans to accommodate the diversity in daily objects. Referring to this concept, this paper presents a method for object property recognition using Graph Convolutional Networks (GCNs), leveraging robot hand proprioception and morphology with spatial embeddings derived from geometrical graph edge features acquired from real tactile sensor alignments on an Allegro Hand. Additionally, a Multi Thread GCN (MT GCN) architecture is introduced to process these edge features and basically multi modal data in a graph. Training data was acquired using a data glove, from tri axial tactile sensors distributed across the fingertips, finger phalanges, and palm of an Allegro Hand, producing a total of 1152 tactile measurements. MT GCN with proposed edge features, tactile features and joint angles achieved a high recognition rate, 86.08% for six classes of object property combinations from eight objects. The effect of variation in graph adjacency on MT GCN was examined. The proposed network showed clusters following the robot hand configuration with t SNE analysis. Furthermore, analysis of learned parameters in the edge feature encoder demonstrated its ability to discern joint positions on the hand, acquiring proprioceptive features effectively. Consequently, we could confirm that the proposed method was effective for multi fingered dexterous tasks.