Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective

Quan Cui, Bingchen Zhao, Zhao Min Chen, Borui Zhao, Renjie Song*, Boyan Zhou, Jiajun Liang, Osamu Yoshie

*この研究の対応する著者

研究成果: Conference contribution

3 被引用数 (Scopus)

抄録

This work simultaneously considers the discriminability and transferability properties of deep representations in the typical supervised learning task, i.e., image classification. By a comprehensive temporal analysis, we observe a trade-off between these two properties. The discriminability keeps increasing with the training progressing while the transferability intensely diminishes in the later training period. From the perspective of information-bottleneck theory, we reveal that the incompatibility between discriminability and transferability is attributed to the over-compression of input information. More importantly, we investigate why and how the InfoNCE loss can alleviate the over-compression, and further present a learning framework, named contrastive temporal coding (CTC), to counteract the over-compression and alleviate the incompatibility. Extensive experiments validate that CTC successfully mitigates the incompatibility, yielding discriminative and transferable representations. Noticeable improvements are achieved on the image classification task and challenging transfer learning tasks. We hope that this work will raise the significance of the transferability property in the conventional supervised learning setting.

本文言語English
ホスト出版物のタイトルComputer Vision – ECCV 2022 - 17th European Conference, 2022, Proceedings
編集者Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, Tal Hassner
出版社Springer Science and Business Media Deutschland GmbH
ページ20-37
ページ数18
ISBN(印刷版)9783031198083
DOI
出版ステータスPublished - 2022
イベント17th European Conference on Computer Vision, ECCV 2022 - Tel Aviv, Israel
継続期間: 2022 10月 232022 10月 27

出版物シリーズ

名前Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
13686 LNCS
ISSN(印刷版)0302-9743
ISSN(電子版)1611-3349

Conference

Conference17th European Conference on Computer Vision, ECCV 2022
国/地域Israel
CityTel Aviv
Period22/10/2322/10/27

ASJC Scopus subject areas

  • 理論的コンピュータサイエンス
  • コンピュータサイエンス一般

フィンガープリント

「Discriminability-Transferability Trade-Off: An Information-Theoretic Perspective」の研究トピックを掘り下げます。これらがまとまってユニークなフィンガープリントを構成します。

引用スタイル