Hybrid CTC/Attention Architecture for End-to-End Speech Recognition

Shinji Watanabe*, Takaaki Hori, Suyoun Kim, John R. Hershey, Tomoki Hayashi

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

537 Citations (Scopus)


Conventional automatic speech recognition (ASR) based on a hidden Markov model (HMM)/deep neural network (DNN) is a very complicated system consisting of various modules such as acoustic, lexicon, and language models. It also requires linguistic resources, such as a pronunciation dictionary, tokenization, and phonetic context-dependency trees. On the other hand, end-to-end ASR has become a popular alternative to greatly simplify the model-building process of conventional ASR systems by representing complicated modules with a single deep network architecture, and by replacing the use of linguistic resources with a data-driven learning method. There are two major types of end-to-end architectures for ASR; attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC) uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes hybrid CTC/attention end-to-end ASR, which effectively utilizes the advantages of both architectures in training and decoding. During training, we employ the multiobjective learning framework to improve robustness and achieve fast convergence. During decoding, we perform joint decoding by combining both attention-based and CTC scores in a one-pass beam search algorithm to further eliminate irregular alignments. Experiments with English (WSJ and CHiME-4) tasks demonstrate the effectiveness of the proposed multiobjective learning over both the CTC and attention-based encoder-decoder baselines. Moreover, the proposed method is applied to two large-scale ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and exhibits performance that is comparable to conventional DNN/HMM ASR systems based on the advantages of both multiobjective learning and joint decoding without linguistic resources.

Original languageEnglish
Article number8068205
Pages (from-to)1240-1253
Number of pages14
JournalIEEE Journal on Selected Topics in Signal Processing
Issue number8
Publication statusPublished - 2017 Dec
Externally publishedYes


  • Automatic speech recognition
  • attention mechanism
  • connectionist temporal classification
  • end-to-end
  • hybrid CTC/attention

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering


Dive into the research topics of 'Hybrid CTC/Attention Architecture for End-to-End Speech Recognition'. Together they form a unique fingerprint.

Cite this