Fast Algorithm and VLSI Architecture of Rate Distortion Optimization in H.265/HEVC

Heming Sun, Dajiang Zhou*, Landan Hu, Shinji Kimura, Satoshi Goto

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

16 Citations (Scopus)


In H.265/high efficiency video coding (HEVC) encoding, rate distortion optimization (RDO) is an important cost function for mode decision and coding structure decision. Despite being near-optimum in terms of coding efficiency, RDO suffers from a high complexity. To address this problem, this paper presents a fast RDO algorithm and its very large scale implementation (VLSI) for both intra-and inter-frame coding. The proposed algorithm employs a quantization-free framework that significantly reduces the complexity for rate and distortion optimization. Meanwhile, it maintains a low degradation of coding efficiency by taking the syntax element organization and probability model of HEVC into consideration. The algorithm is also designed with hardware architecture in mind to support an efficient VLSI implementation. When implemented in the HEVC test model, the proposed algorithm achieves 62% RDO time reduction with 1.85% coding efficiency loss for the 'all-intra' configuration. The hardware implementation achieves 1.6 × higher normalized throughput relative to previous works, and it can support a throughput of 8k@30fps (for four fine-processed modes per prediction unit) with 256 k logic gates when working at 200 MHz.

Original languageEnglish
Article number7918540
Pages (from-to)2375-2390
Number of pages16
JournalIEEE Transactions on Multimedia
Issue number11
Publication statusPublished - 2017 Nov


  • Encoding
  • high efficiency video coding (HEVC)
  • rate distortion optimization (RDO)
  • video coding

ASJC Scopus subject areas

  • Signal Processing
  • Media Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Fast Algorithm and VLSI Architecture of Rate Distortion Optimization in H.265/HEVC'. Together they form a unique fingerprint.

Cite this