แนวคิดหลัก
This study introduces a novel Conformal Prediction method tailored for API-only Large Language Models without logit-access, ensuring efficient prediction sets with a statistical coverage guarantee.
บทคัดย่อ
The study addresses the challenge of quantifying uncertainty in Large Language Models (LLMs) without logit-access. It proposes a novel Conformal Prediction method that leverages coarse-grained and fine-grained uncertainty notions to improve efficiency and accuracy in predicting responses. Experimental results demonstrate the effectiveness of the approach in outperforming logit-based baselines across various tasks.
Key points:
- Addressing uncertainty quantification challenges in Large Language Models (LLMs) without logit-access.
- Introducing a novel Conformal Prediction method tailored for API-only LLMs.
- Leveraging frequency as a coarse-grained measure and introducing fine-grained notions like NE and SS to enhance prediction efficiency.
- Demonstrating superior performance over logit-based methods through experiments on open-ended and close-ended Question Answering tasks.
สถิติ
A minimum of 9,604 samples is required to achieve a 95% confidence level with a 1% margin of error.
The proposed nonconformity score function combines frequency, NE, and SS measures to enhance uncertainty estimation.
The method ensures rigorous statistical coverage guarantees without relying on model logits.
คำพูด
"Our proposed approach does not rely on model logits and can alleviate the known miscalibration issue when using logits."
"Experiments demonstrate the superior performance of our approach compared to logit-based and logit-free baselines."