Practical Attacks on Web-Scale Training Datasets Through Domain Poisoning and Frontrunning

Core Concepts

Adversaries can reliably poison web-scale training datasets by exploiting weaknesses in how these datasets are constructed and distributed, enabling targeted mistakes in downstream machine learning models.

Abstract

The paper introduces two new dataset poisoning attacks that can reliably introduce malicious examples into web-scale training datasets:
Split-view poisoning:

Exploits the mutable nature of internet content and lack of integrity checks to ensure clients observe the same data as when the dataset was initially collected.
An adversary can purchase expired domains that previously hosted images in the dataset, and return malicious content when clients later attempt to download those images.
This attack is feasible due to the prevalence of expired domains in large datasets, and the lack of cryptographic integrity checks in most dataset downloaders.
Frontrunning poisoning:

Targets datasets that aggregate content from crowdsourced web pages, such as Wikipedia.
An adversary can precisely time malicious edits to Wikipedia articles just prior to when they are scraped for inclusion in the next dataset snapshot.
This attack is feasible due to the predictable nature of Wikipedia's snapshot process, latency in content moderation, and the immutability of snapshots once created.
The authors demonstrate the feasibility of these attacks in practice, showing that for just $60 USD, an adversary could have poisoned at least 0.01% of 10 popular web-scale datasets. They also propose defenses involving integrity checks and randomized crawling to mitigate these attacks.

Stats

For just $60 USD, an adversary could have poisoned at least 0.01% of 10 popular web-scale datasets.
The authors found that at least 0.02%-0.79% of the images in these 10 datasets were hosted on expired domains that could be purchased for under $10,000 total.
The authors observed that these datasets are still frequently downloaded, with hundreds of downloads per month for even the oldest datasets.

Quotes

"Our attacks are immediately practical and could, today, poison 10 popular datasets."
"For just $60 USD, we could have poisoned 0.01% of the LAION-400M or COYO-700M datasets in 2022."

Key Insights Distilled From

Poisoning Web-Scale Training Datasets is Practical

by Nich... at arxiv.org 05-07-2024

https://arxiv.org/pdf/2302.10149.pdf

Deeper Inquiries

How might the proposed defenses of integrity checks and randomized crawling be extended or improved to provide stronger protection against these attacks?

The proposed defenses of integrity checks and randomized crawling are effective measures to mitigate the risks of dataset poisoning attacks, but they can be further extended and improved for stronger protection. One way to enhance integrity checks is to implement more advanced cryptographic techniques, such as digital signatures or blockchain technology, to ensure the authenticity and integrity of the dataset. By using cryptographic hashes not only for the index but also for the actual content of the URLs, the dataset maintainers can verify the integrity of the entire dataset, including images and text. Additionally, incorporating multi-factor authentication for dataset access can add an extra layer of security to prevent unauthorized modifications.
Randomized crawling can be improved by introducing dynamic crawling schedules that vary the timing and order of data collection. By randomizing the intervals between snapshots and the sequence in which articles are scraped, the predictability of the data collection process is reduced, making it harder for attackers to time their malicious edits. Furthermore, implementing machine learning algorithms to detect anomalous patterns in data collection can help identify and prevent potential poisoning attempts in real-time. Continuous monitoring and analysis of data collection activities can also help detect any suspicious behavior and trigger immediate responses to mitigate the risks of poisoning attacks.

What other types of vulnerabilities might exist in the data collection and curation processes used for web-scale training datasets?

Apart from the vulnerabilities discussed in the context, there are several other potential weaknesses in the data collection and curation processes for web-scale training datasets. One vulnerability is the lack of robust access controls and authentication mechanisms, which can lead to unauthorized access and tampering of the dataset. Weaknesses in data validation and sanitization processes can also introduce vulnerabilities, allowing for the injection of malicious content or code into the dataset.
Another vulnerability is the reliance on third-party tools and services for dataset downloading and processing, which may have their security flaws or vulnerabilities that could be exploited by attackers. Inadequate data governance practices, such as poor data quality management, lack of data lineage tracking, and insufficient data privacy measures, can also create vulnerabilities in the dataset. Moreover, the use of outdated or insecure protocols for data transfer and storage can expose the dataset to potential security breaches and data leaks.

How could the security and robustness of web-scale training datasets be improved from the ground up, rather than relying on post-hoc defenses?

To enhance the security and robustness of web-scale training datasets from the ground up, a proactive and comprehensive approach is essential. One key aspect is to implement secure-by-design principles throughout the entire data collection, processing, and storage lifecycle. This includes incorporating security controls and mechanisms at every stage, such as encryption, access controls, and data integrity checks, to prevent and detect any unauthorized access or modifications.
Furthermore, adopting a zero-trust security model, where no entity or user is inherently trusted, can help mitigate the risks of insider threats and unauthorized access. Implementing continuous monitoring and auditing of dataset activities can provide real-time visibility into data interactions and help identify any suspicious behavior or anomalies. Regular security assessments and penetration testing can also help identify and address vulnerabilities proactively.
In addition, fostering a culture of security awareness and training among dataset maintainers, curators, and users is crucial to ensure that everyone understands their roles and responsibilities in maintaining the security of the dataset. By promoting a security-first mindset and incorporating security best practices into every aspect of dataset management, organizations can build a strong foundation for secure and resilient web-scale training datasets.