Multi-frame super-resolution for versatile video coding

Toshiya Hori, Remina Yano, Hiroshi Watanabe, Takeshi Chujoh, Tomohiro Ikai, Eiichi Sasaki, Takuya Suzuki, Norio Ito

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The latest video coding standard, Versatile Video Coding (VVC), has been finalized in July 2020. In traditional codecs, it is not possible to refer to different resolution pictures in inter-picture prediction except for inter layer cases. However, VVC does support Reference Picture Resampling (RPR), which allows pictures to refer to different resolution pictures so that video resolution can be adaptively switched in sequence. This work's objective is to improve compression efficiency in the case of RPR. In the preliminary experiments, the compression efficiencies of reduced size encoded case where frames are encoded in half size and up-scaled to their original size and original size encoded case are compared. The result shows that most sequences get worse BD-rate when the sequences are encoded/decoded in reduced size. Here, it is noted that up-scaling is done by VVC Test Model, VTM-10.0, which uses a linear interpolation filter. Thus, the frames cannot recover high-frequency components sufficiently and cannot use temporal correlation. Therefore, this paper proposes a multi-frame super-resolution between frames of different resolutions where some frames are encoded in high quality frames in their original size and other frames in half size by using RPR. The proposed method uses the original size frames, which contain a lot of information, to perform super-resolution to the reduced frame. By doing so, it is observed it saves the bitrate by reducing the frame resolution and increases the quality of reduced frames by using multiframe super-resolution, which leads to improving the compression efficiency.

Original languageEnglish
Title of host publicationInternational Workshop on Advanced Imaging Technology, IWAIT 2021
EditorsMasayuki Nakajima, Jae-Gon Kim, Wen-Nung Lie, Qian Kemao
ISBN (Electronic)9781510643642
Publication statusPublished - 2021
Event2021 International Workshop on Advanced Imaging Technology, IWAIT 2021 - Kagoshima, Virtual, Japan
Duration: 2021 Jan 52021 Jan 6

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X


Conference2021 International Workshop on Advanced Imaging Technology, IWAIT 2021
CityKagoshima, Virtual

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Multi-frame super-resolution for versatile video coding'. Together they form a unique fingerprint.

Cite this