Centrala begrepp
The core message of this article is to introduce a "Calibrate-Extrapolate" framework for efficiently processing and analyzing content to estimate the prevalence of toxic comments on social media platforms, using a pre-trained black box classifier.
Sammanfattning
The article introduces a "Calibrate-Extrapolate" framework for prevalence estimation using a pre-trained black box classifier.
Calibration Phase:
- A limited sample of data is selected and ground truth labels are obtained through manual annotation.
- A calibration curve is estimated, mapping the classifier outputs to calibrated probabilities.
- The base dataset's joint distribution between classifier outputs and ground truth labels is inferred.
Extrapolation Phase:
- The target dataset's classifier outputs are obtained.
- Stability assumptions are made to link the base and target joint distributions.
- Two techniques are discussed - assuming stable calibration curve or stable class-conditional densities.
- The linked joint distribution is used to estimate the prevalence in the target dataset.
The framework is applied to estimate the weekly prevalence of toxic comments on news topics across Reddit, Twitter/X, and YouTube in 2022, using Jigsaw's Perspective API as the black box classifier. The results show consistently higher prevalence of toxic comments on YouTube compared to Twitter/X and Reddit.
The article also conducts simulation experiments to analyze the impacts of classifier predictive power and violations of stability assumptions on the accuracy of prevalence estimates.
Statistik
The dataset contains over 15,000 hard news URLs with at least 10 comments each on Reddit, Twitter/X, and YouTube in 2022.
On average, the dataset includes 5,631 distinct comments per day on Reddit, 5,355 on Twitter/X, and 4,271 on YouTube.
A calibration sample of 1,144 Reddit comments, 1,154 Twitter/X replies, and 1,162 YouTube comments from August 2021 were manually annotated for toxicity.
Citat
"Measuring the frequency of certain labels within a data sample is a common task in many disciplines. This problem, generally called "prevalence estimation" or "quantification", has a wide range of real world applications, from quantifying the number of infected COVID-19 patients in a country (Sempos and Tian 2021), to automated accounts in a social platform (Yang et al. 2020), and to anti-social posts in an online community (Park, Seering, and Bernstein 2022)."
"The Calibrate-Extrapolate framework is broadly applicable to many real world settings. It is also flexible because researchers can still customize design elements in some steps."