Improving Knowledge Distillation by Revising Soft Labels and Selecting Appropriate Training Data

The core message of this paper is to improve the reliability of the teacher's supervision in knowledge distillation by revising the soft labels of the teacher using ground truth, and selecting appropriate training samples to be supervised by the teacher, in order to mitigate the negative impact of incorrect predictions from the teacher model.