Core Concepts
The author proposes the DeCCaF framework to address cost-sensitive scenarios and human work capacity constraints in learning to defer systems.
Abstract
The content discusses the DeCCaF framework, a novel approach for cost-sensitive learning to defer decisions to multiple experts under workload constraints. It addresses key limitations of existing methods and demonstrates superior performance in fraud detection scenarios.
The research highlights the importance of human-AI collaboration, emphasizing the complementary strengths of humans and AI models. It introduces the concept of learning to defer (L2D) as a state-of-the-art framework for managing assignments in collaborative systems.
Key aspects such as multi-expert settings, human capacity constraints, and cost-sensitive scenarios are explored. The proposed DeCCaF method is tested in realistic fraud detection experiments with synthetic expert teams and an ML classifier.
Results show that DeCCaF outperforms baselines in various scenarios, achieving an average reduction in misclassification costs. The study also evaluates the predictive performance and calibration of classifiers and expert models used in the framework.
Overall, the content provides valuable insights into improving human-AI collaboration through innovative approaches like DeCCaF.
Stats
Achieving an average 8.4% reduction in misclassification cost.
Synthetic fraud analysts team comprised of 9 experts.
Testing conducted on cost-sensitive fraud detection scenarios.
Proposed DeCCaF framework employs supervised learning.
Average expected misclassification costs per 100 instances reported.
Quotes
"We propose L2D architectures that allow for assigners to be trained assuming that each instance is accompanied by the prediction of only one expert out of the team."
"DeCCaF: a novel L2D method that models human behavior under limited data availability."