Estimating the Proportion of Positives in Unlabeled Data When the Selected Completely At Random Assumption Does Not Hold

Core Concepts

The authors propose two positive unlabeled (PU) learning algorithms, PULSCAR and PULSNAR, to estimate the proportion of positives in the unlabeled data when the selected completely at random (SCAR) assumption does not hold.

Abstract

The authors address the problem of estimating the fraction (α) of positives among unlabeled instances in positive and unlabeled (PU) learning, where only positive instances are labeled and the unlabeled set contains a mix of positive and negative instances. The key points are: Most PU learning algorithms make the SCAR assumption, where positives are randomly selected from the universe of positives. However, in many real-world applications, this assumption does not hold (selected not at random, SNAR), leading to poor estimates of α and poor model calibration. The authors propose two algorithms: PULSCAR: for SCAR data, it uses kernel density estimates of positive and unlabeled distributions to estimate α. PULSNAR: for SNAR data, it uses a divide-and-conquer approach, clustering the positive set into homogeneous subsets that better approximate SCAR, and then applying PULSCAR to each subset. The authors also propose methods to calibrate the probabilities of PU examples and improve classification performance using PULSCAR/PULSNAR. Experiments on synthetic and real-world benchmark datasets show that PULSCAR and PULSNAR outperform state-of-the-art PU learning methods, especially when the SCAR assumption does not hold.

Stats

The fraction of positives among the unlabeled examples (α) ranges from 1% to 50%. The synthetic datasets have 2,000 positives and 6,000 unlabeled examples with 50 continuous features. The real-world datasets include Bank, KDD Cup 2004 Particle Physics, Statlog (Shuttle), and Firewall.

Quotes

"In many real-world applications, such as healthcare, positives are not SCAR (e.g., severe cases are more likely to be diagnosed), leading to a poor estimate of the proportion, α, of positives among unlabeled examples and poor model calibration, resulting in an uncertain decision threshold for selecting positives." "PU learning algorithms can estimate α or the probability of an individual unlabeled instance being positive or both."

Key Insights Distilled From

Positive Unlabeled Learning Selected Not At Random (PULSNAR): class proportion estimation when the SCAR assumption does not hold

by Praveen Kuma... at arxiv.org 05-07-2024

https://arxiv.org/pdf/2303.08269.pdf

Deeper Inquiries

How can the PULSNAR algorithm be extended to handle datasets with more than two classes (i.e., multi-class PU learning)

To extend the PULSNAR algorithm to handle datasets with more than two classes, we can modify the clustering approach to accommodate multiple classes. Instead of dividing the positive examples into clusters based on their similarities, we can cluster the positive examples into distinct groups representing each class. This way, we can estimate the proportion of positives for each class separately among the unlabeled examples. By applying the PULSCAR algorithm to each class cluster and the unlabeled data, we can calculate the alpha values for each class and combine them to determine the overall proportion of positives among the unlabeled instances in a multi-class PU learning scenario.

What are the potential limitations of the clustering approach used in PULSNAR, and how could it be improved to handle more complex data distributions

The clustering approach used in PULSNAR may have limitations when dealing with complex data distributions. One potential limitation is the assumption of homogeneity within each cluster of positive examples. In real-world scenarios, the positive examples may not form distinct clusters, leading to inaccurate estimates of the proportion of positives among the unlabeled instances. To improve the clustering approach, we can consider more advanced clustering algorithms that can handle non-linear and complex data distributions, such as spectral clustering or density-based clustering. Additionally, incorporating feature selection techniques to identify the most relevant features for clustering can enhance the accuracy of the clustering process and improve the estimation of alpha values in SNAR data.

Can the proposed methods be adapted to work in an online or streaming setting, where the unlabeled data arrives incrementally over time

Adapting the proposed methods to work in an online or streaming setting where unlabeled data arrives incrementally over time is feasible but requires some modifications. In an online setting, the algorithms need to be updated dynamically as new data points become available. One approach is to periodically re-cluster the positive examples and update the alpha estimates based on the new data. Additionally, the calibration of probabilities and the classification performance metrics should be continuously monitored and adjusted as new data is incorporated. Implementing a mechanism to handle concept drift and model adaptation in response to changing data distributions is essential for the successful application of the PU learning algorithms in an online setting.