Multi-modal feature fusion for better understanding of human personality traits in social human–robot interaction

Zhihao Shen*, Armagan Elibol, Nak Young Chong

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

12 Citations (Scopus)


Since the dynamic nature of human–robot interaction becomes increasingly prevalent in our daily life, there is a great demand for enabling the robot to better understand human personality traits and inspiring humans to be more engaged in the interaction with the robot. Therefore, in this work, as we design the paradigm of human–robot interaction as close to the real situation as possible, the following three main problems are addressed: (1) fusion of visual and audio features of human interaction modalities, (2) integration of variable length feature vectors, and (3) compensation of shaky camera motion caused by movements of the robot's communicative gesture. Specifically, the three most important visual features of humans including head motion, gaze, and body motion were extracted from a camera mounted on the robot performing verbal and body gestures during the interaction. Then, our system was geared to fuse the aforementioned visual features and different types of vocal features, such as voice pitch, voice energy, and Mel-Frequency Cepstral Coefficient, dealing with variable length multiple feature vectors. Lastly, considering unknown patterns and sequential characteristics of human communicative behavior, we proposed a multi-layer Hidden Markov Model that improved the classification accuracy of personality traits and offered notable advantages of fusing the multiple features. The results were thoroughly analyzed and supported by psychological studies. The proposed multi-modal fusion approach is expected to deepen the communicative competence of social robots interacting with humans from different cultures and backgrounds.

Original languageEnglish
Article number103874
JournalRobotics and Autonomous Systems
Publication statusPublished - 2021 Dec


  • Human personality traits
  • Human–robot interaction
  • Machine learning
  • Multi-modal feature fusion

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Mathematics(all)
  • Computer Science Applications


Dive into the research topics of 'Multi-modal feature fusion for better understanding of human personality traits in social human–robot interaction'. Together they form a unique fingerprint.

Cite this