Core Concepts
Majority-of-Three is proven to be the simplest optimal algorithm achieving optimal error bounds in PAC learning.
Abstract
The article discusses the development of an optimal PAC learning algorithm, focusing on the Majority-of-Three approach. It addresses the complexity of existing algorithms and aims to determine the simplest yet optimal solution. Theoretical analysis and proofs demonstrate that Majority-of-Three achieves optimal error bounds both in expectation and high probability regimes. Comparisons with other algorithms like Bagging and one-inclusion graph are made, highlighting the simplicity and optimality of Majority-of-Three. The study provides insights into the challenges and solutions in developing efficient learning algorithms.
Stats
Classic work by Blumer et al. [BEHW89] shows that for any δ > 0, it holds with probability 1 − δ over S that any bfS ∈ F consistent with f ⋆ on S has errP bfS = O(dn log(n/d) + 1/n log(1/δ)).
Recent work by Aden-Ali et al. [ACSZ23a] shows that for any d ∈ N, sample size n ≥ d and confidence parameter δ ≥ cd/n, there exists a function class F ⊆ {0, 1}X with VC dimension d and a hard distribution P such that a certain implementation of the one-inclusion graph algorithm has, with probability at least δ, errP bfOIG = Ω(d/δn).
Quotes
"Developing an optimal PAC learning algorithm in the realizable setting was a major open problem."
"Hanneke’s algorithm returns the majority vote of many ERM classifiers trained on subsets of data."
"The study conjectures that Majority-of-Three is optimal for all confidence levels."