Kernkonzepte
Combining provenance tracking and assistive labeling techniques, INSPECTOR empowers users to efficiently identify and retain high-quality synthetic text data for improving the robustness of text classification models.
Zusammenfassung
The paper presents INSPECTOR, a human-in-the-loop approach for inspecting and curating synthetically generated text data for text classification tasks. INSPECTOR combines two key techniques to reduce human effort:
-
Provenance Tracking:
- INSPECTOR allows users to group the generated texts by their common transformation provenance (i.e., the transformations applied to the original text) or their feature provenance (i.e., the linguistic features of the original text).
- This enables users to efficiently inspect groups of related texts and identify patterns in data quality.
-
Assistive Labeling:
- INSPECTOR computes quality metrics such as label alignment, grammaticality, and fluency for each generated text and its corresponding label.
- It also provides the predictions of a large language model, which users can compare against the text labels to identify discrepancies.
The authors conducted a within-subject user study with 15 participants to evaluate INSPECTOR. The results show that using INSPECTOR, participants were able to identify 3-4 times more texts with correct labels compared to a baseline without the provenance tracking and assistive labeling features. Participants found the transformation provenance to be the most useful technique, as it allowed them to systematically inspect groups of texts and make informed decisions about their quality. The human-inspected data also improved the robustness of the text classification models by up to 32% compared to randomly sampled data.
The paper highlights that no single technique of INSPECTOR was found to be universally useful, suggesting that effective inspection of generated texts requires combining complementary techniques.
Statistiken
"Using INSPECTOR, participants identified an average of 277 and 259 high-quality instances compared to 82 and 63 instances using the baseline on the SST2 and TweetEval's Hate Speech dataset, respectively."
"On the SST2 dataset, 4 out of 8 participants marked data that led to more robust models than randomly sampled data. On the TweetEval dataset, all 7 participants identified data that led to more robust models than randomly sampled data."
"The attack success rate of DeepWord on models trained with randomly selected data was 0.61 on the SST2 dataset and 0.5 on the TweetEval dataset. Using the inspected data, the attack success rate decreases to an average of 0.59 and 0.34 on SST2 and TweetEval, respectively."
Zitate
"Grouping data by their shared common transformations to be the most useful technique."
"Assistive labeling allowed them to build trust in the tool."
"No single technique of INSPECTOR was found to be universally useful, suggesting that effective inspection of generated texts requires combining complementary techniques."