A multimodal tempo and beat-tracking system based on audiovisual information from live guitar performances

Tatsuhiko Itohara*, Takuma Otsuka, Takeshi Mizumoto, Angelica Lim, Tetsuya Ogata, Hiroshi G. Okuno


研究成果: Article査読

3 被引用数 (Scopus)


The aim of this paper is to improve beat-tracking for live guitar performances. Beat-tracking is a function to estimate musical measurements, for example musical tempo and phase. This method is critical to achieve a synchronized ensemble performance such as musical robot accompaniment. Beat-tracking of a live guitar performance has to deal with three challenges: tempo fluctuation, beat pattern complexity and environmenta noise. To cope with these problems, we devise an audiovisual integration method for beat-tracking. The auditory beat features are estimated in terms of tactus (phase) and tempo (period) by Spectro-Temporal Pattern Matching (STPM), robust against stationary noise. The visual beat features are estimated by tracking the position of the hand relative to the guitar using optical flow, mean shift and the Hough transform. Both estimated features are integrated using a particle filter to aggregate the multimodal information based on a beat location model and a hand's trajectory model. Experimental results confirm that our beat-tracking improves the F-measure by 8.9 points on average over the Murata beat-tracking method, which uses STPM and rule-based beat detection. The results also show that the system is capable of real-time processing with a suppressed number of particles while preserving the estimation accuracy. We demonstrate an ensemble with the humanoid HRP-2 that plays the theremin with a human guitarist.

ジャーナルEurasip Journal on Audio, Speech, and Music Processing
出版ステータスPublished - 2012

ASJC Scopus subject areas

  • 音響学および超音波学
  • 電子工学および電気工学


「A multimodal tempo and beat-tracking system based on audiovisual information from live guitar performances」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。