Joint speaker diarization and speech recognition based on region proposal networks

Zili Huang*, Marc Delcroix, Leibny Paola Garcia, Shinji Watanabe, Desh Raj, Sanjeev Khudanpur


研究成果: Article査読

4 被引用数 (Scopus)


Speaker diarization, the process of partitioning an input audio stream into homogeneous segments according to the speaker identity, is an important task for speech processing. The standard clustering-based diarization pipeline (1) segments the whole utterance into small chunks, (2) extracts speaker embedding for each chunk, and (3) groups the chunks into clusters, where each cluster represents one speaker. It has two major disadvantages: first, it contains several individually optimized modules in the pipeline, and second, it cannot handle overlapping speech. To address these issues, we proposed region proposal network-based speaker diarization (RPNSD) (Huang et al., 2020). In this paper, we perform a detailed study of the RPNSD system, and make two important contributions. First, we report its diarization performance on additional datasets and empirically investigate the impact of different system settings. Second, we integrate an automatic speech recognition (ASR) component into the RPNSD system and propose a new framework called RPN-JOINT that simultaneously performs diarization and ASR. Our experiments reveal that (1) the RPNSD system can consistently achieve diarization results that are competitive with state-of-the-art methods, and (2) the RPN-JOINT system offers several advantages over the conventional cascade of diarization and ASR systems.

ジャーナルComputer Speech and Language
出版ステータスPublished - 2022 3月

ASJC Scopus subject areas

  • ソフトウェア
  • 理論的コンピュータサイエンス
  • 人間とコンピュータの相互作用


「Joint speaker diarization and speech recognition based on region proposal networks」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。