Ghost Sentence: A Tool for Copyrighting Data from Large Language Models
Core Concepts
Users can protect their data from large language models by embedding ghost sentences in their public documents.
Abstract
Web user data is crucial for pre-training and fine-tuning large language models (LLMs).
Users can insert personal passphrases, known as ghost sentences, to confirm if LLMs are using their data.
Ghost sentences help users safeguard their data from unauthorized use by LLMs.
The effectiveness of ghost sentences is evaluated through user training data identification tasks.
Different datasets and evaluation metrics are used to assess the memorization performance of LLMs with ghost sentences.
Longer ghost sentences and strategic insertion positions enhance memorization accuracy.
The impact of model sizes, learning rates, and training epochs on memorization performance is explored.
Continuing pre-training with ghost sentences requires larger repetition times for effective memorization.
Ghost Sentence
Stats
11 out of 16 users with ghost sentences identify their data within the generation content.
61 out of 64 users with ghost sentences identify their data within the LLM output.
Users who have more than 10 public documents should consider using a diceware passphrase.
The success rate of memorization is determined by the number of users and average repetition times of ghost sentences.
The bigger the model, the smaller the repetition times required for memorization.
Quotes
"Users can intentionally insert personal passphrases into their public documents to protect their data from misuse by large language models."
"Ghost sentences act as hidden guards within user documents, remaining concealed from LLM trainers but known to users."
"Longer ghost sentences are generally easier to memorize for LLMs."