Rhythmic body movement analysis for robot-based music therapy

Y. H. Ma, J. Y. Lin, S. Cosentino*, A. Takanishi

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)


The ability to correctly perceive time and extract accurate timing information is crucial during social interaction. In fact, several activities during social interaction, such as appropriate feedback, turn-taking, coordination with peers, and even empathy and engagement exhibition directly depend on it. One of the aspects of cognitive malfunctioning in children with Autistic Spectrum Disorders is time perception deficit. Learning to pay attention to and correctly assess timing is thus a critical first step to improve social skills for children with Autism. In this paper, we present a novel sensing system and algorithm for estimating a subject's rhythmic motion timing from visual information using Recurrent Neural Network (RNN) coupled with FFT. This system will enable a robot saxophonist to estimate the rhythmic period from a child's motion during a robot-based music therapy session. Fast-Fourier- Transform (FFT) is an algorithm widely applied in rhythmic body movement detection, due to advantages such low computation and easy integration. However, long transient time delay is a critical limitation, reducing the correct motion timing estimation during period transitions. The novel system presented in this article is shown to significantly reduce transient time delay. The results of both a simulation and an evaluation experiment show that, compared with FFT processing alone, this algorithm gives a better performance due to its smaller average offset error and shorter transient time delay, allowing a more precise assessment of the child's synchronization response.

Original languageEnglish
Title of host publication2021 IEEE International Conference on Advanced Robotics and Its Social Impacts, ARSO 2021
PublisherIEEE Computer Society
Number of pages6
ISBN (Electronic)9781665449533
Publication statusPublished - 2021 Jul 8
Event2021 IEEE International Conference on Advanced Robotics and Its Social Impacts, ARSO 2021 - Tokoname, Japan
Duration: 2021 Jul 82021 Jul 10

Publication series

NameProceedings of IEEE Workshop on Advanced Robotics and its Social Impacts, ARSO
ISSN (Print)2162-7568
ISSN (Electronic)2162-7576


Conference2021 IEEE International Conference on Advanced Robotics and Its Social Impacts, ARSO 2021


  • Autism Spectrum Disorder
  • Human-robot-interaction
  • Long Short-Term Memory
  • Music therapy
  • Recurrent Neural Network

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Science Applications
  • Computer Vision and Pattern Recognition
  • Electrical and Electronic Engineering


Dive into the research topics of 'Rhythmic body movement analysis for robot-based music therapy'. Together they form a unique fingerprint.

Cite this