Musical-based interaction system for the Waseda Flutist Robot: Implementation of the visual tracking interaction module

Since 1990, at Waseda University the development on the Anthropomorphic Flutist Robot has been focused on mechanically reproducing the physiology of the organs involved during the flute playing (i.e. lungs, lips, etc.) and implementing basic cognitive capabilities to interact with flutist beginners. As a results of the research efforts done until now, the Waseda Flutist Robot is considered to play the flute nearly similar to the performance of a intermediate human player. However, we consider that in order to extend the interaction capabilities of the flutist robot with musical partners, further research efforts should be done. In this paper, we propose as a long-term goal to enable the flutist robot to interact more naturally with musical partners on the context of a Jazz band. For this purpose a Musical-Based Interaction System (MbIS) is proposed to enable the robot the process both visual and aural cues coming throughout the interaction with musicians. In particular, in this paper, the details of the implementation of the visual tracking module on the Waseda Flutist Robot No. 4 Refined IV (WF-4RIV) is presented. The visual tracking module is composed by two levels of interaction: basic (visual interface for the musician based on controlling virtual buttons and faders) and advanced (instrument tracking system so that the robot can process motion gestures performed by the musical partner in real-time which are then directly mapped into musical parameters of the performance of the robot). The experiments carried out were focused in verifying the effectiveness and usability of the proposed levels of interaction. In particular, we focused on determining how well our the WF-4RIV dynamically changes musical parameters while interacting with a human player. From the experimental results we observed that the physical constraints of the robot play an important role during the interaction. Although further improvements should be done to overcome such constrains, we expect that the interaction experience may become more natural.