The paper focuses on improving the generalization and consistency of factual knowledge extraction from pre-trained language models. It identifies two key biases in the factual probing objective: the object likelihood bias and the template prior bias.
The object likelihood bias refers to the likelihood of a predicted object given only the prompt template, without the subject, being biased. This positively correlates with the predictions from subject-given prompts and negatively influences the performance of factual extraction.
The template prior bias refers to the inconsistency among outputs from prompt paraphrases due to the domination of specific verbalizations during pre-training.
The paper proposes UniArk, a parameter-free framework that uses adapter-tuning to debias these two objectives. For the object likelihood bias, UniArk introduces a max entropy loss to equalize the likelihood distribution over the top retrieved candidates. For the template prior bias, UniArk uses a self-data augmentation method to average the output distribution over different prompt templates.
Extensive experiments on the LAMA dataset and two paraphrased datasets, ParaTrex and ParaRel, show that UniArk can significantly improve the model's out-of-domain generalization as well as consistency under various prompts, without harming in-domain performance.
The paper also introduces ParaTrex, a large-scale and diverse dataset for measuring the inconsistency and out-of-domain generation of models, which offers a reference method for constructing paraphrased datasets using large language models.
To Another Language
from source content
arxiv.org
Deeper Inquiries