Leveraging Large Language Models to Measure Ideology from Text: Semantic Scaling for Policy Preferences and Affective Polarization

Core Concepts

Semantic Scaling is a novel method that leverages large language models to classify documents based on their expressed stances, and then uses item response theory to scale subjects from these data. This approach allows researchers to explicitly define the ideological dimensions they measure, and produces valid estimates of both mass and elite ideology, including differentiating between policy preferences and in-group/out-group affect.

Abstract

The paper introduces "Semantic Scaling", a new method for ideal point estimation from text. The key aspects are:

It uses large language models to classify documents based on their expressed stances, rather than relying on word counts or word vectors. This allows for more detailed semantic inferences.

It then extracts survey-like data from these document classifications and uses item response theory to scale subjects.

This approach can be used to measure the ideologies of citizens, elites, and groups, and allows researchers to explicitly define the type of ideology to be estimated (e.g., policy-based or affective).

The method is validated using two examples from American politics:

Estimating the policy preferences of Twitter users, where Semantic Scaling outperforms the leading Tweetscores approach according to human judgement.

Estimating the policy and affective ideological positions of members of the 117th U.S. Congress. Semantic Scaling produces policy-based ideology scores that match DW-NOMINATE, while also allowing for the credible measurement of legislators' in-group/out-group affect - something not possible with existing methods.

Overall, Semantic Scaling represents a significant advance in text-based ideal point estimation, providing flexibility, robustness, and the ability to differentiate between policy preferences and affective polarization.

Stats

"Semantic Scaling significantly improves on existing text-based scaling methods, and allows researchers to explicitly define the ideological dimensions they measure."
"Semantic Scaling out-preforms Tweetscores according to human judgement; in Congress, it recaptures the first dimension DW-NOMINATE while allowing for greater flexibility in resolving construct validity challenges."
"Semantic Scaling produces policy-based ideology scores that match DW-NOMINATE, while also allowing for the credible measurement of legislators' in-group/out-group affect - something not possible with existing methods."

Quotes

"Semantic Scaling significantly improves on existing text-based scaling methods, and allows researchers to explicitly define the ideological dimensions they measure."
"Semantic Scaling out-preforms Tweetscores according to human judgement; in Congress, it recaptures the first dimension DW-NOMINATE while allowing for greater flexibility in resolving construct validity challenges."
"Semantic Scaling produces policy-based ideology scores that match DW-NOMINATE, while also allowing for the credible measurement of legislators' in-group/out-group affect - something not possible with existing methods."

Key Insights Distilled From

Semantic Scaling: Bayesian Ideal Point Estimates with Large Language Models

by Michael Burn... at arxiv.org 05-07-2024

https://arxiv.org/pdf/2405.02472.pdf

Deeper Inquiries

How could Semantic Scaling be extended to measure ideology in non-political domains, such as social or cultural attitudes?

Semantic Scaling can be extended to measure ideology in non-political domains by adapting the classification process to identify stances or attitudes relevant to the specific domain of interest. For example, in social or cultural contexts, researchers can define items related to social issues, cultural beliefs, or values that they want to measure. By using large language models to classify documents based on the semantics or meaning of the text, researchers can extract survey-like data and apply item response theory to estimate ideological positions in these non-political domains.
To apply Semantic Scaling in non-political domains, researchers would need to:

Define the specific ideological dimensions or attitudes they want to measure in the social or cultural context.
Create a set of hypotheses for entailment classification that represent the stances or beliefs relevant to the domain.
Match documents to relevant hypotheses using keywords or topic labels and classify the documents based on the identified stances.
Use Bayesian Markov Chain Monte Carlo techniques to estimate the ideological positions based on the classified documents.

By customizing the items and hypotheses to align with the unique characteristics of social or cultural attitudes, Semantic Scaling can provide valuable insights into ideological dimensions beyond the political realm.

What are the potential limitations of Semantic Scaling in contexts where individuals may strategically obfuscate their true ideological positions in their text?

One potential limitation of Semantic Scaling in contexts where individuals may strategically obfuscate their true ideological positions in their text is the risk of misclassification or biased estimation. If individuals intentionally mask or distort their ideological beliefs in their text, it can lead to inaccurate classification of stances and ultimately affect the reliability of the ideal point estimates derived from Semantic Scaling.
Additionally, the effectiveness of Semantic Scaling relies on the assumption that the text accurately reflects the author's true beliefs. When individuals purposefully conceal or misrepresent their ideologies, the classification process may not capture the genuine attitudes, leading to skewed results. This challenge highlights the importance of considering the potential for strategic obfuscation when applying Semantic Scaling in contexts where authenticity in textual expression is questionable.
Researchers using Semantic Scaling in such contexts may need to implement additional validation measures, such as cross-referencing with other sources of data or incorporating sentiment analysis techniques to detect inconsistencies or anomalies in the text that could indicate strategic obfuscation of ideological positions.

How might Semantic Scaling be combined with other data sources, such as voting records or campaign contributions, to provide a more comprehensive measure of ideology?

Combining Semantic Scaling with other data sources, such as voting records or campaign contributions, can enhance the comprehensiveness and accuracy of ideological measurement by triangulating information from multiple channels. By integrating text-based ideological estimation with quantitative data from voting behavior or financial contributions, researchers can create a more robust and nuanced understanding of individuals' or groups' ideologies.
One approach to combining Semantic Scaling with traditional data sources is to use a mixed-methods approach. Researchers can leverage Semantic Scaling to extract ideological positions from text data and then compare these estimates with ideological scores derived from voting records or campaign contributions. By aligning the results from different methods, researchers can validate and cross-validate the ideological measurements, enhancing the overall reliability of the findings.
Furthermore, integrating Semantic Scaling with other data sources allows for a more comprehensive analysis of ideology by capturing both explicit and implicit expressions of beliefs. Text data can provide insights into nuanced or subtle ideological nuances that may not be evident in voting records or financial transactions alone, offering a more holistic view of individuals' or groups' ideological orientations.
Overall, the combination of Semantic Scaling with traditional data sources offers a multi-faceted approach to ideological measurement, enabling researchers to gain deeper insights and a more comprehensive understanding of ideology across various contexts.