Belangrijkste concepten
NUMTEMP is a large, diverse dataset of 15,514 real-world numerical claims from various fact-checking domains, designed to evaluate the challenges of verifying claims involving numerical quantities and temporal expressions.
Samenvatting
The NUMTEMP dataset is a comprehensive collection of 15,514 real-world numerical claims from various fact-checking domains. The dataset addresses the challenge of verifying claims involving numerical quantities and temporal expressions, which are prevalent in political discourse but often overlooked by existing fact-checking datasets.
The dataset construction process involves:
Collecting real-world claims from 45 fact-checking websites worldwide.
Identifying quantitative segments in the claims to extract numerical claims.
Collecting evidence for the claims from web sources, excluding fact-checking websites to avoid leakage.
Enhancing the evidence diversity by using claim decomposition approaches like CLAIMDECOMP and PROGRAM-FC.
The dataset is divided into training, validation, and test sets, with a distribution of 'True', 'False', and 'Conflicting' claims. The authors also categorize the numerical claims into four types: temporal, statistical, interval, and comparison.
The authors evaluate various fact-checking approaches on the NUMTEMP dataset, including claim decomposition, pre-trained models for numerical understanding, and different NLI models. The results show that NUMTEMP poses a significant challenge for fact-checking, with the best approach achieving a weighted-F1 of 64.89 for unified evidence and 69.79 for gold evidence. The authors also find that claim decomposition and models pre-trained on numerical understanding tasks can improve performance on numerical claims.
Statistieken
Only 8 million households with incomes up to $86k face tax increases under the GOP plan, not all families making $86k.
6.5% of the 122 million households in the bottom three quintiles will face a tax increase under the GOP plan.
Citaten
"Numerical claims are a significant component of political discourse. For instance, our analysis of the CLAIMBUSTER DATASET (Hassan et al., 2017) reveals that a substantial 36% of all check-worthy claims in U.S. presidential debates involve numerical quantities or temporal expressions."
"Numerical claims verification poses a unique challenge, where a fact-checking system must critically analyze and reason about the numerical data presented in both the claim and its evidence."