The authors develop a framework to quantitatively measure the opinions represented in large language models (LLMs) compared to diverse global perspectives. They first compile a dataset, GlobalOpinionQA, from cross-national surveys designed to capture opinions on global issues across different countries.
The authors then define a metric to quantify the similarity between LLM-generated survey responses and human responses, conditioned on country. They run three experiments on an LLM trained to be helpful, honest, and harmless:
Default Prompting (DP): The model's responses tend to be more similar to the opinions of certain populations, such as the USA, Canada, Australia, and some European and South American countries, highlighting potential biases.
Cross-national Prompting (CP): When prompted to consider a particular country's perspective, the model's responses shift to be more similar to the opinions of the prompted populations, but can reflect harmful cultural stereotypes.
Linguistic Prompting (LP): Translating the questions to different languages does not necessarily make the model's responses more similar to the opinions of speakers of those languages.
The authors release the GlobalOpinionQA dataset and provide an interactive visualization to further explore these findings. They discuss the limitations of their approach and the need for developing models that better represent diverse global perspectives.
toiselle kielelle
lähdeaineistosta
arxiv.org
Tärkeimmät oivallukset
by Esin Durmus,... klo arxiv.org 04-15-2024
https://arxiv.org/pdf/2306.16388.pdfSyvällisempiä Kysymyksiä