The content discusses the challenge of enabling large language models (LLMs) to collaborate and learn from each other while preserving privacy. LLMs are powerful but come with high inference costs and need to run in data centers far from local contexts where private data is available. Conversely, local models that can run on user devices have more limited capabilities.
The authors introduce the first privacy-preserving approach to cascade systems, where a local model (the student) can query a more capable remote model (the teacher) for help, without revealing any private information. They propose three methods for the student to generate queries to the teacher:
To evaluate privacy, the authors introduce two metrics: the entity leak metric that counts entities leaked from the original examples, and the mapping leak metric that measures how well a curious teacher with auxiliary information could map the student's queries back to the original examples.
Experiments on diverse datasets show that the authors' methods can significantly improve the student's performance compared to baselines, while minimizing privacy leakage. Method 3 (replacing entities) generally achieves the best quality results while leaking few entities, while Method 2 (generating new examples) with grouping offers the strongest privacy metrics.
To Another Language
from source content
arxiv.org
Deeper Inquiries