Multi-stream end-To-end speech recognition

Ruizhi Li*, Xiaofei Wang, Sri Harish Mallidi, Shinji Watanabe, Takaaki Hori, Hynek Hermansky

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

14 Citations (Scopus)


Attention-based methods and Connectionist Temporal Classification (CTC) network have been promising research directions for end-To-end (E2E) Automatic Speech Recognition (ASR). The joint CTC/Attention model has achieved great success by utilizing both architectures during multi-Task training and joint decoding. In this article, we present a multi-stream framework based on joint CTC/Attention E2E ASR with parallel streams represented by separate encoders aiming to capture diverse information. On top of the regular attention networks, the Hierarchical Attention Network (HAN) is introduced to steer the decoder toward the most informative encoders. A separate CTC network is assigned to each stream to force monotonic alignments. Two representative framework have been proposed and discussed, which are Multi-Encoder Multi-Resolution (MEM-Res) framework and Multi-Encoder Multi-Array (MEM-Array) framework, respectively. In MEM-Res framework, two heterogeneous encoders with different architectures, temporal resolutions and separate CTC networks work in parallel to extract complementary information from same acoustics. Experiments are conducted on Wall Street Journal (WSJ) and CHiME-4, resulting in relative Word Error Rate (WER) reduction of \text{18.0}\!-\!\text{32.1}\% and the best WER of \text{3.6}\% in the WSJ eval92 test set. The MEM-Array framework aims at improving the far-field ASR robustness using multiple microphone arrays which are activated by separate encoders. Compared with the best single-Array results, the proposed framework has achieved relative WER reduction of \text{3.7}\% and \text{9.7}\% in AMI and DIRHA multi-Array corpora, respectively, which also outperforms conventional fusion strategies.

Original languageEnglish
Article number8932598
Pages (from-to)646-655
Number of pages10
JournalIEEE/ACM Transactions on Audio Speech and Language Processing
Publication statusPublished - 2020
Externally publishedYes


  • End-To-end speech recognition
  • connectionist temporal classification
  • encoder-decoder
  • hierarchical attention network (HAN)
  • joint CTC/attention
  • multi-encoder multi-Array (MEM-Array)
  • multi-encoder multi-resolution (MEM-Res)

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Acoustics and Ultrasonics
  • Computational Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Multi-stream end-To-end speech recognition'. Together they form a unique fingerprint.

Cite this