Severe Pitfalls in Empirical Evaluations of Machine Learning Privacy Defenses

Core Concepts

Existing empirical evaluations of machine learning privacy defenses are severely misleading, as they fail to capture the privacy leakage of the most vulnerable samples, use weak attacks, and compare to incomparable differential privacy baselines.

Abstract

The paper identifies three key pitfalls in existing empirical evaluations of machine learning privacy defenses:

Aggregating attack success over a dataset population, rather than focusing on the most vulnerable samples. This can lead to defenses that fully violate the privacy of some samples passing existing evaluations.

Using weak or non-adaptive attacks that do not reflect the current state-of-the-art, and fail to account for peculiarities of the defense mechanism.

Comparing empirical defenses to weak differential privacy (DP) baselines that achieve much lower utility than the defenses.

To address these issues, the paper proposes a reliable evaluation protocol with three key components:

Measuring membership inference attack success (true positive rate at low false positive rate) specifically for the most vulnerable sample(s) in the dataset, approximated efficiently using a carefully designed set of "canary" samples.

Adapting the membership inference attack to the specifics of each defense, to ensure the attack is as strong as possible.

Comparing empirical defenses to a properly tuned DP-SGD baseline that achieves similar utility to the defenses, rather than a weak DP baseline.

The paper then applies this evaluation protocol to five representative empirical privacy defenses, revealing that none of them provide effective protection against properly adapted attacks targeted at the most vulnerable samples. In contrast, the authors' tuned DP-SGD baseline outperforms all the empirical defenses in terms of both privacy and utility.

Stats

"Empirical defenses for machine learning privacy forgo the provable guarantees of differential privacy in the hope of achieving higher utility while resisting realistic adversaries."
"We find that prior evaluations underestimate privacy leakage by an order of magnitude."
"Under our stronger evaluation, none of the empirical defenses we study are competitive with a properly tuned, high-utility DP-SGD baseline."

Quotes

"Empirical defenses are typically compared to weak DP-SGD baselines [9, 12, 28, 52] with utility below the state-of-the-art."
"We reveal much stronger privacy leakage and a completely different ranking than the original evaluations suggest."
"None of the five defenses provide effective protection against properly adapted attacks targeted at the most vulnerable samples."

Key Insights Distilled From

Evaluations of Machine Learning Privacy Defenses are Misleading

by Mich... at arxiv.org 04-29-2024

https://arxiv.org/pdf/2404.17399.pdf

Deeper Inquiries

How can the proposed evaluation methodology be extended to other types of privacy attacks beyond membership inference, such as attribute inference or data extraction

The proposed evaluation methodology can be extended to other types of privacy attacks beyond membership inference by adapting the evaluation criteria to suit the specific characteristics of each attack. For attribute inference attacks, where the goal is to infer specific attributes of individuals from a model, the evaluation could focus on the model's ability to protect sensitive attributes. This could involve creating canaries that represent the most sensitive attributes and evaluating the attack's success in inferring these attributes. Similarly, for data extraction attacks, which aim to extract entire training samples from a model, the evaluation could involve testing the model's resilience to such attacks by measuring the attacker's ability to extract complete training samples. Canaries in this case could represent the most vulnerable training samples that are at risk of being extracted. By adapting the evaluation methodology to suit the specific goals and characteristics of each privacy attack, researchers can provide a more comprehensive assessment of the effectiveness of privacy defenses against a range of threats.

What are the implications of the findings in this paper for the broader field of machine learning security and privacy

The findings in this paper have significant implications for the broader field of machine learning security and privacy. Firstly, they highlight the limitations of existing empirical evaluations of privacy defenses, particularly in the context of machine learning models memorizing sensitive information from their training data. By demonstrating the shortcomings of current evaluation methodologies, the paper underscores the importance of developing more robust and reliable evaluation frameworks for assessing the privacy guarantees of machine learning models. This could lead to more accurate assessments of privacy defenses and better-informed decisions regarding their deployment in real-world applications.
Furthermore, the findings emphasize the need for a more nuanced approach to designing and evaluating privacy defenses, taking into account the specific vulnerabilities and attack vectors that may be present in different types of machine learning models. This could lead to the development of more tailored and effective privacy defenses that are better equipped to protect against a wide range of privacy attacks. Additionally, the paper's focus on the trade-off between differential privacy and utility highlights the ongoing challenge of balancing privacy guarantees with model performance. This could spur further research into alternative privacy frameworks that offer stronger guarantees while still allowing for practical deployments in real-world settings.
Overall, the findings in this paper are likely to influence the design and evaluation of future privacy defenses by encouraging researchers to adopt more rigorous evaluation methodologies, consider a broader range of privacy attacks, and explore alternative privacy frameworks that strike a better balance between privacy and utility.

How might they influence the design and evaluation of future privacy defenses

Given the limitations of differential privacy in achieving high utility, researchers may explore alternative formal privacy frameworks that could provide stronger guarantees while still allowing for practical deployments. One such framework is Secure Multi-Party Computation (MPC), which enables multiple parties to jointly compute a function over their private inputs without revealing individual inputs to each other. MPC ensures privacy by design, as each party's input remains encrypted throughout the computation, and the final result is revealed without exposing any individual inputs. This framework offers strong privacy guarantees while allowing for collaborative machine learning tasks to be performed securely.
Another alternative is Homomorphic Encryption, which allows computations to be performed on encrypted data without decrypting it. This enables privacy-preserving machine learning models to be trained and evaluated on encrypted data, ensuring that sensitive information remains protected throughout the process. While homomorphic encryption can be computationally intensive, advancements in this area are making it more practical for real-world applications.
By exploring these alternative formal privacy frameworks, researchers can potentially achieve stronger privacy guarantees while still maintaining high utility in machine learning models. This could open up new avenues for developing privacy-preserving machine learning techniques that are both effective and practical for deployment in sensitive applications.