Collaborative Teacher-Student Learning via Multiple Knowledge Transfer

By Liyuan Sun et al

Published on Jan. 27, 2021

Read the original document by opening this link in a new tab.

Table of Contents

1. Introduction
2. Related Work
3. The Proposed CTSL-MKT Method
4. Algorithm 1: The proposed CSL-MKT

Summary

Knowledge distillation (KD), as an efficient and effective model compression technique, has been receiving considerable attention in deep learning. The key to its success is to transfer knowledge from a large teacher network to a small student one. This paper introduces a collaborative teacher-student learning via multiple knowledge transfer (CTSL-MKT) that promotes self-learning and collaborative learning. It allows multiple students to learn knowledge from individual instances and instance relations in a collaborative way. The proposed CTSL-MKT significantly outperforms the state-of-the-art KD methods in experiments on four image datasets. The framework integrates self-distillation and online distillation to bridge the capacity gap and improve performance. The main contributions include a new mutual learning framework, self-learning enhanced collaborative learning, extensive experiments on peer teacher-student networks, and ablation studies providing insights into knowledge distillation.