A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars

Haruki Sato, Tatsunori Hirai, Tomoyasu Nakano, Masataka Goto, Shigeo Morishima

Research output: Chapter in Book/Report/Conference proceedingConference contribution

4 Citations (Scopus)


This paper presents a system that can automatically add a sound-track to a video clip by replacing and concatenating an existing song's musical bars considering a user's preference. Since a soundtrack makes a video clip attractive, adding a soundtrack to a clip is one of the most important processes in video editing. To make a video clip more attractive, an editor of the clip tends to add a soundtrack considering its timing and climax. For example, editors often add chorus sections to the climax of the clip by re-placing and concatenating musical bars in an existing song. How-ever, in the process, editors should take naturalness of rearranged soundtrack into account. Therefore, editors have to decide how to replace musical bars in a song considering its timing, climax, and naturalness of rearranged soundtrack simultaneously. In this case, editors are required to optimize the soundtrack by listening to the rearranged result as well as checking the naturalness and synchro-nization between the result and the video clip. However, this repe-titious work is time-consuming. [Feng et al. 2010] proposed an automatic soundtrack addition method. However, since this meth-od automatically adds soundtrack with data-driven approach, this method cannot consider timing and climax which a user prefers. Our system takes all the patterns of rearranged musical bars into account and finds the most natural soundtrack considering a user's preference of intention for an audio-visual alignment and a climax of the resulting soundtrack. Specifically, musical sections between user's specified points and the beginning and the ending of the song are automatically interpolated by replacing and concatenat-ing musical bars based on dynamic programming. To consider user's intention for a climax of the soundtrack, the system allows the user to specify the intended climax by an editing interface. The system immediately reflects the intention and the soundtrack will be interactively re-rearranged. These semi-automated pro-cesses of rearranging soundtrack for a video clip help the users adding songs without considering naturalness of the rearranged song by their own.

Original languageEnglish
Title of host publicationACM SIGGRAPH 2015 Posters, SIGGRAPH 2015
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450336321
Publication statusPublished - 2015 Jul 31
EventInternational Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015 - Los Angeles, United States
Duration: 2015 Aug 92015 Aug 13

Publication series

NameACM SIGGRAPH 2015 Posters, SIGGRAPH 2015


OtherInternational Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2015
Country/TerritoryUnited States
CityLos Angeles

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Computer Vision and Pattern Recognition
  • Human-Computer Interaction


Dive into the research topics of 'A music video authoring system synchronizing climax of video clips and music via rearrangement of musical bars'. Together they form a unique fingerprint.

Cite this