Kernekoncepter
A zero-shot, open-vocabulary pipeline system that integrates domain classification and dialogue state tracking, enabling efficient and adaptable task-oriented dialogue understanding without relying on predefined ontologies.
Resumé
The authors propose a zero-shot, open-vocabulary pipeline system for task-oriented dialogue understanding. The system consists of two main components:
-
Domain Classification:
- The pipeline starts by identifying the active domain for each turn of the dialogue using a self-refined prompt tailored to the language model.
- This crucial step is often overlooked in existing approaches, which either rely on predefined domains or attempt to track slots across all domains.
-
Dialogue State Tracking (DST):
- The authors introduce two complementary approaches for DST:
a. DST-as-QA: Reformulates DST as a multiple-choice question-answering task, providing a strong adaptation for smaller or less capable language models.
b. DST-as-SRP: Employs self-refining prompts, treating the language model as a black-box dialogue state tracker and guiding it through structured instructions for efficient zero-shot DST.
- Both approaches are designed to be open-vocabulary, dynamically adapting to new slot values without additional fine-tuning, unlike ontology-based methods.
The authors conduct extensive experiments on the MultiWOZ and Schema-Guided Dialogue (SGD) datasets, comparing their approaches with state-of-the-art fully-trained and zero-shot models. They demonstrate that their DST-as-SRP approach achieves new state-of-the-art results, outperforming previous methods by up to 20% in Joint Goal Accuracy (JGA) while using up to 90% fewer requests to the language model API.
The key innovations of this work are:
- Integrating domain classification and DST in a single pipeline to enable practical and adaptable dialogue understanding.
- Reformulating DST as a question-answering task and employing self-refining prompts to leverage the capabilities of large language models in a zero-shot and open-vocabulary setting.
- Achieving state-of-the-art performance while significantly reducing the computational cost compared to existing approaches.
Statistik
The MultiWOZ 2.4 dataset contains over 10,000 conversations across 8 domains.
The Schema-Guided Dialogue (SGD) dataset consists of over 16,000 conversations across 26 services and 16 domains.
Citater
"Our approach includes reformulating DST as a question-answering task for less capable models and employing self-refining prompts for more adaptable ones."
"Unlike ontology-based approaches that need to process all possible slot value pairs within the ontology, open-vocabulary approaches only use the generic slot definition and generate/extract the values directly from the dialogue."
"We show that DST-as-SRP achieves new state-of-the-art results with up to 90% fewer requests to the LLM API, improving the strict Joint Goal Accuracy (JGA) score by 20%, 3%, and 2% on the MultiWOZ 2.1, MultiWOZ 2.4, and SGD datasets, respectively."