Evaluating Instruction-Tuned Language Models as Crowd Annotators: Exploring Model Specialization and Label Aggregation

Core Concepts

Aggregating the predictions of multiple instruction-tuned language models can outperform individual models, but still falls short of supervised learning approaches on subjective text classification tasks.

Abstract

The paper explores the feasibility of using four open-source instruction-tuned language models (Flan-T5, Flan-UL2, T0, and Tk-Instruct) as "annotators" for five subjective text classification tasks across four languages (English, French, German, and Spanish).
The key findings are:

The language models exhibit specialization, with different models performing better on different tasks and languages. This suggests that aggregating their predictions could be beneficial.

Aggregating the model predictions using majority voting or a Bayesian annotation model (MACE) indeed outperforms the individual models on average. The MACE aggregation, in particular, correlates well with the actual competence of each model.

However, even the best aggregated performance is still well below that of simple supervised learning models, let alone more advanced Transformer-based supervised models. The performance gap is over 10 F1 points on average.

Surprisingly, few-shot learning with carefully selected seed examples does not consistently improve over zero-shot learning, likely due to the difficulty in selecting high-quality exemplars.

The authors discuss the trade-offs between using language models versus human annotators, considering aspects like performance, cost, bias, and ethical implications. They conclude that while language model annotation can be a quick and cost-effective solution, human annotation is still vital for achieving high performance, especially on subjective tasks.

Stats

"Aggregated labels are 4.2 F1-points better than the average LLM."
"Even the best-aggregated performance is still well below that of even simple supervised models trained on the same data, and substantially lower than Transformer-based supervised models (by over 10 F1 points on average)."

Quotes

"Different models indeed excel on some tasks or languages, but not on others. Some models even specialize on certain labels in a given task, but perform poorly on the others."
"Aggregating several ZSL-prompted LLMs is better than using a single LLM. Surprisingly, FSL-prompting is too varied to consistently improve performance."
"However, treating LLMs as annotators cannot rival using human annotators for fine-tuning or supervised learning."

Key Insights Distilled From

Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation

by Flor Miriam ... at arxiv.org 04-16-2024

https://arxiv.org/pdf/2307.12973.pdf

Deeper Inquiries

How can we effectively select high-quality seed examples to improve the performance of few-shot learning with language models?

In few-shot learning, selecting high-quality seed examples is crucial for improving performance. One effective approach is to leverage entropy-based strategies to choose the seed examples. By calculating the entropy of each example, we can identify instances where the models are less confident or disagree more, indicating higher difficulty. These high-entropy examples can be valuable for challenging the model and improving its robustness. On the other hand, low-entropy examples where models are more confident or agree more can serve as easier instances for the model to learn from. By incorporating a mix of high-entropy and low-entropy examples, we can provide a diverse set of instances for the model to learn from, enhancing its ability to generalize and perform well on unseen data.

What are the potential biases and ethical concerns in using language models as annotation tools, especially for sensitive or subjective tasks, and how can these be mitigated?

Using language models as annotation tools can introduce biases and ethical concerns, particularly in sensitive or subjective tasks. One major concern is the potential for bias in the training data of the language models, which can lead to biased annotations. Additionally, language models may not fully understand the nuances and context of sensitive topics, resulting in inappropriate or harmful annotations. To mitigate these issues, it is essential to carefully evaluate the performance of language models on sensitive tasks and continuously monitor and address biases in the training data. Implementing diverse training data, bias detection algorithms, and regular audits can help reduce biases and ensure ethical annotation practices. Transparency in the annotation process, clear guidelines, and human oversight can also help mitigate ethical concerns and ensure the responsible use of language models in annotation tasks.

How might the role of human annotation evolve as language models continue to advance, and what new models of human-AI collaboration could emerge in the future?

As language models continue to advance, the role of human annotation may evolve to focus more on tasks that require human judgment, creativity, and domain expertise. Human annotators could be involved in curating training data, providing context, and verifying the accuracy of annotations generated by language models. This human-AI collaboration could lead to a hybrid approach where human annotators work alongside language models to ensure high-quality annotations. New models of collaboration may involve human annotators in the loop, where they guide and supervise the annotation process, correct errors, and provide feedback to improve the performance of language models. This collaborative approach can leverage the strengths of both humans and AI, leading to more accurate and reliable annotations in complex and nuanced tasks.