Core Concepts
The Rashomon ratio measures the proportion of classifiers in a family that yield a loss less than a given threshold. This work explores methods to estimate the Rashomon ratio for infinite hypothesis sets and demonstrates how a large Rashomon ratio can enable efficient learning by allowing good classifiers to be found through random sampling.
Abstract
This paper investigates the Rashomon ratio, which measures the proportion of classifiers in a family that yield a loss less than a given threshold, in the context of infinite hypothesis sets. The key contributions are:
-
Methodology for estimating the Rashomon ratio numerically when the true or reducible error of classifiers is unknown. This involves generating random samples from the classifier family and using the empirical loss to approximate the Rashomon ratio, with guarantees on the accuracy of the approximation.
-
Analysis of the Rashomon ratio for two specific examples:
- Affine classifiers applied to a mixture of Gaussian distributions. The authors show analytically that the Rashomon ratio approaches 1 as the distance between the Gaussian means increases, and has a strictly positive minimum value that depends on the dimensionality.
- Two-layer ReLU neural networks, where a lower bound on the Rashomon ratio is derived based on properties of the Gram matrix and label vector.
-
Demonstration of how a large Rashomon ratio can enable efficient learning. If the Rashomon ratio is large, then with high probability a good classifier can be found by randomly sampling a small subset of the hypothesis set. This provides guarantees on the performance of the best classifier in the random subset compared to the best in the full hypothesis set.
The results show that the Rashomon ratio can be substantial, even for infinite hypothesis sets, providing a theoretical foundation for methods that leverage random sampling to find accurate yet simple models.
Stats
The following sentences contain key metrics or figures:
The Rashomon ratio Rratio(F, γ) is a number between 0 and 1 that quantifies the proportion of the functions that belong to the Rashomon set Rset(F, γ) within F.
The empirical Rashomon ratio ˆRratio(F, γ) is an approximation of the true Rashomon ratio based on a finite dataset.
The reducible error of an affine classifier sign(p · x + t) ∈Faf is Eµ1,µ2,σ(p, t) = Φ(∥µ2 −µ1∥/2σ) - ζΦ((max(p · µ1, p · µ2) - t)/(σ∥p∥)) - (1-ζ)Φ((t - min(p · µ1, p · µ2))/(σ∥p∥)).
The lower bound on the empirical Rashomon ratio of a two-layer ReLU neural network depends on the dimension of the data, the number of nodes in the hidden layer, the smallest eigenvalue of H∞ and yT(H∞)−1y.
Quotes
"A large Rashomon ratio guarantees that choosing the classifier with the best empirical accuracy among a random subset of the family, which is likely to improve generalizability, will not increase the empirical loss too much."
"The Rashomon ratio can be estimated using a training dataset along with random samples from the classifier family and we provide guarantees that such an estimation is close to the true value of the Rashomon ratio."