[2310.16789] Detecting Pretraining Data from Large Language Models 

 Skip to main content 

 Search 

 Submit 
 Donate 

 Log in 

 Search arXiv 

 Press Enter to search · Advanced search 

--> 

 Computer Science > Computation and Language

 arXiv:2310.16789 (cs)

 [Submitted on 25 Oct 2023 ( v1 ), last revised 9 Mar 2024 (this version, v3)]

 Title: Detecting Pretraining Data from Large Language Models

 Authors: Weijia Shi , Anirudh Ajith , Mengzhou Xia , Yangsibo Huang , Daogao Liu , Terra Blevins , Danqi Chen , Luke Zettlemoyer 
 View a PDF of the paper titled Detecting Pretraining Data from Large Language Models, by Weijia Shi and 7 other authors

 View PDF 
 HTML (experimental) 

 Abstract: Although large language models (LLMs) are widely deployed, the data used to train them is rarely disclosed. Given the incredible scale of this data, up to trillions of tokens, it is all but certain that it includes potentially problematic text such as copyrighted materials, personally identifiable information, and test data for widely reported reference benchmarks. However, we currently have no way to know which data of these types is included or in what proportions. In this paper, we study the pretraining data detection problem: given a piece of text and black-box access to an LLM without knowing the pretraining data, can we determine if the model was trained on the provided text? To facilitate this study, we introduce a dynamic benchmark WIKIMIA that uses data created before and after model training to support gold truth detection. We also introduce a new detection method Min-K% Prob based on a simple hypothesis: an unseen example is likely to contain a few outlier words with low probabilities under the LLM, while a seen example is less likely to have words with such low probabilities. Min-K% Prob can be applied without any knowledge about the pretraining corpus or any additional training, departing from previous detection methods that require training a reference model on data that is similar to the pretraining data. Moreover, our experiments demonstrate that Min-K% Prob achieves a 7.4% improvement on WIKIMIA over these previous methods. We apply Min-K% Prob to three real-world scenarios, copyrighted book detection, contaminated downstream example detection and privacy auditing of machine unlearning, and find it a consistently effective solution.

 Subjects:

 Computation and Language (cs.CL) ; Cryptography and Security (cs.CR); Machine Learning (cs.LG)

 Cite as:

 arXiv:2310.16789 [cs.CL] 

 (or 
 arXiv:2310.16789v3 [cs.CL] for this version)

 https://doi.org/10.48550/arXiv.2310.16789 

 Focus to learn more 

 arXiv-issued DOI via DataCite

 Submission history
 From: Weijia Shi [ view email ] 
 [v1] 
 Wed, 25 Oct 2023 17:21:23 UTC (2,405 KB)

 [v2] 
 Fri, 3 Nov 2023 05:27:37 UTC (2,404 KB)

 [v3] 
 Sat, 9 Mar 2024 22:26:06 UTC (2,404 KB)

 Full-text links: 
 Access Paper:

View a PDF of the paper titled Detecting Pretraining Data from Large Language Models, by Weijia Shi and 7 other authors
 View PDF 
 HTML (experimental) 
 TeX Source

 view license 

 Additional Features

 Audio Summary 

 Current browse context:

 cs.CL

 < prev 

   |   
 next > 

 new 
 | 
 recent 
 | 2023-10 

 Change to browse by:

 cs 
 cs.CR 
 cs.LG 

 References & Citations

 SciX 
 Google Scholar 

 Semantic Scholar 

 export BibTeX citation 
 Loading... 

 BibTeX formatted citation

 × 

 loading... 

 Data provided by: 

 Bookmark

 Bibliographic Tools 

 Bibliographic and Citation Tools

 Bibliographic Explorer Toggle 

 Bibliographic Explorer ( What is the Explorer? ) 

 Connected Papers Toggle 

 Connected Papers ( What is Connected Papers? ) 

 Litmaps Toggle 

 Litmaps ( What is Litmaps? ) 

 scite.ai Toggle 

 scite Smart Citations ( What are Smart Citations? ) 

 Code, Data, Media 

 Code, Data and Media Associated with this Article

 alphaXiv Toggle 

 alphaXiv ( What is alphaXiv? ) 

 Links to Code Toggle 

 CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) 

 DagsHub Toggle 

 DagsHub ( What is DagsHub? ) 

 GotitPub Toggle 

 Gotit.pub ( What is GotitPub? ) 

 Huggingface Toggle 

 Hugging Face ( What is Huggingface? ) 

 ScienceCast Toggle 

 ScienceCast ( What is ScienceCast? ) 

 Demos 

 Demos

 Replicate Toggle 

 Replicate ( What is Replicate? ) 

 Spaces Toggle 

 Hugging Face Spaces ( What is Spaces? ) 

 Spaces Toggle 

 TXYZ.AI ( What is TXYZ.AI? ) 

 Related Papers 

 Recommenders and Search Tools

 Link to Influence Flower 

 Influence Flower ( What are Influence Flowers? ) 

 Core recommender toggle 

 CORE Recommender ( What is CORE? ) 

 Author 

 Venue 

 Institution 

 Topic 

 About arXivLabs

 arXivLabs: experimental projects with community collaborators

 arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

 Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

 Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

 Which authors of this paper are endorsers? |
 Disable MathJax ( What is MathJax? )

 We gratefully acknowledge support from our major funders ,
 member institutions , ,
 and all contributors.

 About · 
 Help · 
 Contact · 
 Subscribe · 
 Copyright · 
 Privacy · 
 Accessibility · 
 Operational Status (opens in new tab) 

 Major funding support from

 (opens in new tab) 

 (opens in new tab) 

 (opens in new tab)