Leveraging Teacher Embedding Structure for Efficient Knowledge Distillation in Few-Class Classification Tasks

A novel method called Learning Embedding Linear Projections (LELP) that extracts informative linear subspaces from the teacher's embedding space and uses them to create pseudo-subclasses, which are then used to guide the training of the student model. LELP achieves superior performance compared to existing distillation methods, especially in binary and few-class classification tasks.